Submit a ticketCall us

WebinarFREE IT Monitoring Webcast

Don’t miss out on our webcast, Essential IT Monitoring with SolarWinds ipMonitor, where we will show you how to keep an eye on your IT environment from one centralized, affordable, and lightweight monitoring tool: SolarWinds® ipMonitor®.

Register now.

Home > Success Center > Log & Event Manager (LEM) > Log & Event Manager (LEM) Documentation > LEM fails to send rule notifications and emails

LEM fails to send rule notifications and emails

Updated: July 3, 2018

Overview

Over time, LEM may experience delays or failures in sending notifications for rules. Restarting resolves this issue, but over several hours or days, emails from rules will continue to fail.

Environment

  • All LEM versions

Cause

 

LEM sending queries to DNS servers that do not use query forwarding. This adds a resolve delay.

Resolution

There are two methods to resolve this issue. One is configuring the DNS server, the second requires assistance with root access from SolarWinds Customer Support.

Contact SolarWinds Customer Support and ask them to verify DNS resolve times with the following command:

 

~ # dig @DNS-IP Test-Node-IP | grep "Query time"

;; Query time: 1161 msec

~ # dig @DNS-IP Test-Node-FQDN | grep "Query time"

;; Query time: 5 msec

 

Any results over 100ms will eventually lead to delays in rule processing. Ideally, the DNS should resolve in under 10ms.

Resolution 1

  • Configure query  forwarding on the DNS servers.

Requires the DNS team's assistance.

Resolution 2

Please contact SolarWinds Customer Support for assistance.

Increase queue size from 5k to 20k; increase parallel resolutions to 50; cache results for 2 hours instead of 15 minutes.

  • Create backup for manager.conf 
  • Edit manager.conf

Add 

  • IpLookupQueueSize=20000
  • CorrelationThreads=50
  • IpCacheTimeout=7200000
  • Save manager.conf
  • Verify user/group and permissions are correct for manager.conf
  • Restart LEM

 

 
Last modified

Tags

Classifications

Public