Submit a ticketCall us

Training ClassThe Orion® Platform Instructor-led Classes

Provided by SolarWinds® Academy, these trainings will introduce users to the Orion Platform and its features, management, and navigation. These courses are suitable for users looking to discover new tips, tricks, and ways to adapt their Orion products to better suit their monitoring needs:
Deploying the Orion Platform
Configuring Orion views, maps, and accounts
Configuring Orion alerts and reports

Reserve your seat.

Home > Success Center > Log & Event Manager (LEM) > LEM - Knowledgebase Articles > LEM is queueing and dropping event data

LEM is queueing and dropping event data

Overview

The manager.log file shows that LEM begins to queue data and drops event data.

The following is an example of when the LEM shows dropping alerts in manager.log:

10:40:49 PST 2016) II:INFO [SnakQ] {EventPump:Rules:75} :postAlert:Total alerts dropped: 792000

Environment

LEM 6.2 and earlier

Cause 

LEM queues and eventually drops data when there are too many alerts received for the resources reserved for LEM. The following are examples:

  • Something in the environment is causing a spike of alert data, such as an attack, a device that is misconfigured or broken.
  • New nodes were added and LEM does not have the resources necessary to handle the new load.
  • Rules are firing often, either because of an influx of alert data or because the rule has been misconfigured.
  • This can also happen due to known HSQL DB max file size limitation of 16 GB, which causes /tmp to become full and "cleantemp" does not help

Resolution

Increase the resources reserved for LEM or reduce the alert data:

  1. Verify which queue is filling and causing the issue:
    1. Log in to CMC.
      • Virtual Console: Click Advanced Configuration and then press Enter.
      • SSH Client: Log in using your CMC credentials.
    2. Type appliance, and then press Enter.
    3. Type diskusage, and then press Enter.
  2. Check the areas below to identify the cause and solution:

     
    1. LEM Partition is 100%
    2. Logs/Data partition is 100%
    3. Temp is 10% or more, and Database Queues has a high number of alerts waiting in memory.
    4. Rules Queue and/or EPIC Rules Queue has a high number of alerts waiting in memory.
    5. Console Queue has a high number of alerts waiting in memory.

 

Alternate solutions

Apply 6.2.1 HF2 or upgrade to 6.3.1. 

Backup your settings and rules and properly backup the LEM appliance.

If you are still having an issue, contact SolarWinds Support.

 

Last modified

Tags

Classifications

Public