Security Compliance

SEM fails to send rule notifications and emails

Over time, L=SEM may experience delays or failures in sending notifications for rules. Restarting resolves this issue, but over several hours or days, emails from rules will continue to fail. The information is clearly visible in smac-hist.log in the InetAddressModule(5000) columns (the queue size; the number of successful resolves).

First published date

4/23/2019 1:29 PM

Last published date

4/23/2019 1:29 PM

Overview

Over time, SEM may experience delays or failures in sending notifications for rules. Restarting resolves this issue, but over several hours or days, emails from rules will continue to fail.

Product section

Security Event Manager

Cause

SEM sending queries to DNS servers that do not use query forwarding. This adds a resolve delay.

Resolution

There are two methods to resolve this issue. One is configuring the DNS server, the second requires assistance with root access from SolarWinds Customer Support.

Contact SolarWinds Customer Support and ask them to verify DNS resolve times with the following command:

~ # dig @DNS-IP Test-Node-IP | grep "Query time"

;; Query time: 1161 msec

~ # dig @DNS-IP Test-Node-FQDN | grep "Query time"

;; Query time: 5 msec

Any results over 100ms will eventually lead to delays in rule processing. Ideally, the DNS should resolve in under 10ms.

Resolution 1

Configure query forwarding on the DNS servers.

Note: Requires the DNS team's assistance.

Resolution 2

Note: Please contact SolarWinds Customer Support for assistance.

  1. Increase queue size from 5k to 20k, increase parallel resolutions to 50, and cache results for two hours instead of 15 minutes.
  2. Create a backup of the manager.conf file. 
  3. Edit the manager.conf file, and then add the following:
    • IpLookupQueueSize=20000
    • CorrelationThreads=50
    • IpCacheTimeout=7200000
  4. Save manager.conf.
  5. Verify the user/group and permissions are correct for manager.conf.
  6. Restart SEM.