Security Compliance

SEM goes down daily and requires constant restarts

This article provides a resolution to the issue when SEM functions normally for a period of time, but goes offline regularly requiring a reboot or a service restart to bring it back up.

First published date

10/10/2018 4:38 PM

Last published date

12/8/2022 2:22 PM

Overview

This article describes the issue when This article applies to Security Event Manager (formerly Log & Event Manager) functions normally for a period of time but becomes unstable and goes offline regularly requiring a reboot or a service restart to bring it back up. Symptoms may include:

  • Unable to log into the web console
  • All nodes show disconnected
  • No connectors are started
  • No events are coming into the Monitor tab
  • Very slow console performance

Product section

Security Event Manager

Cause

This issue may be caused by the following: 

  • Inadequate resource reservations
  • Too many events per day (EPD)
  • Too many rules firing
  • File Integrity Monitoring (FIM) is returning too many events
  • Duplicate or unnecessary connectors

Resolution

  1. Determine how many events your SEM receives every day using an nDepth search and verify you have the proper amount of resource reservations to handle that load. Learn about allocating resources with reservations in the SEM Administrator Guide.
  2. Verify how many rules are firing in SEM on a daily basis:
    1. On the SEM menu bar, navigate to Explore > nDepth.
    2. Drag the InternalRuleFired event to the search bar.
    3. Change the time frame to last day, and then click Search.
    4. SEM should be firing no more than a few hundred events per day. If it's much more than that, you should determine which rules are firing really often and look at the rule to see how it can be fine tuned.
  3. If you've recently added a FIM connector to your agents, and the FIM filters are set too broadly, you might be getting inundated with FIM events. Try disabling your FIM connectors to see if performance improves.
  4. On the SEM menu bar, navigate to Manage > Appliances > left gear icon > Connectors. Select the Configured check box and look through the list of configured connectors for duplicate connectors, connectors for devices that you aren't actually logging, etc. See Configuring connectors.

If the issue is still unresolved, contact SolarWinds Support for assistance.