SolarWinds Platform Troubleshooting - Alerts

7/17/2019 7:09 PM

10/24/2025 9:50 PM

Description

One of the more common SolarWinds customer support calls involves Alerting issues.  The most common issues are alerts not triggering/not sending emails. These can be one and the same, but not always, so it is crucial to understand and properly identify the issue. There are many factors that come into play, so it is important that you understand how to address this issue in a logical manner. 

Issues can be described as:

• Alert/s are not triggering
• False alert triggering
• Emails are not being sent out or received
• Alerts not working

Scoping the Issue

What to Ask 

Here are a few questions to ask before jumping into the problem. You don't have to stop at these questions. Different scenarios might need you to probe differently, but this is a good starting point.

• Reproduce the issue
  • If you can't reproduce the issue, you can't confirm whether it's been fixed
  • If it's a once-off occurrence, the most you can do is check the log files and try to determine what occurred at that time.
  • Can it be easily reproduced, or is it intermittent? Is there any pattern?
  • What are the expected results?

 

• Narrow the focus
  • Is this affecting all alerts?
    • If NO, are there specific alerts that are not working?
      • Reboot alert
      • Down alert
      • Node alert
      • Interface alert
      • Volume alert
      • Application alert
      • Component alert
  • Are the alerts triggering but not sending emails?
    • If yes, Were there any changes made to the SMTP server?
  • Is this a new alert or one that was working and stopped?
    • If new, verify logic and configuration - this is the most likely cause
    • If it worked previously, ask if there were changes to the configuration. If not, investigate the Alerting services and logs for issues that may have caused it to stop working

 

• When was this issue first observed?
  • Try to establish what happened before the onset of the problem
  • Were there any changes made before the problem started?
    • Upgrades
    • Server migration
    • Hostname or IP Address changed on SolarWinds server/s

 

• Understand the Environment
  • What are the SolarWinds Platform server specs? SQL server specs? (are they under spec'd?)
  • Check CPU and Memory usage on both servers ( SolarWinds and SQL)
  • Is there an Anti-Virus agent on the server?
    • Has it been configured for on-access file scans?
    • Have SolarWinds files been added to the exclusion list?

Orion Platform

How Alerting Works

There are two types of processes how alerts are triggered:
 

• Event-based alerts: Based on specific actions in various parts of the SolarWinds Platform (BL plugins work, user actions on the website, SWIS plugins like the one for Containers, from Collector about new values of properties). Those indications are sent from their sources to SWIS and then they’re resent into the Alerting Service which triggers an alert.

• Basic alerts: Based on periodic execution of SQL/SWQL queries. If those queries return rows, alerts are triggered based on them. Workflow is that Alerting Service is the initiator which contacts SWIS to decide which alert to trigger. In case they’re containing some condition of type ‘trigger alert when the condition is valid at least 5 minutes then the state is saved into the database and verified again next round until time condition is fulfilled or alert condition invalidated.

Services Involved:

Service Name	Description	Software Package	Role
Alerting Service V2	Responsible for evaluating alerts	SolarWinds Orion Core / Platform	Saves alert conditions into the SQL Database Evaluates alert conditions and contacts "Core.MessageAction.dll" if the alert is triggered and actions need to be performed
Information Service V3	Communication and Notification Service	SolarWinds Orion Core / Platform	Checks query-based alert conditions against the DB Notifications received from other sources passed to Alerting Engine v2 for evaluation  (For example, Collector Services can raise notification that a "property changed" to SWIS)
Core Business Layer	Service used to launch and execute module engine plug-ins for all SolarWinds Orion products	SolarWinds Orion Core / Platform	Processes Alert Actions Responsible for sending emails

 

Examine the Logs:

Orion produces numerous log files. These are the log files that would likely yield clues for this particular issue, in order of importance:

Not finding the errors you're looking for?

Increase the logging level to DEBUG and reproduce the issue once again.  This applies to all services/processes. See Adjust logging levels with the Orion Log Adjuster.

Alerting V2 Service:
 

If alerts are not triggering or there is an issue with an alert this is the first place to check

Orion Server: C:\ProgramData\Solarwinds\Logs\Orion\Alerting.Service.V2.log

Example:

When the service is unable to connect to the database to evaluate the alert condition:

2019-11-20 19:50:56,899 [55] ERROR SolarWinds.Orion.Core.Alerting.Service.ConditionsStateEvaluator - 
Condition 'AlertId: 64, AlertLastEdit: 11/20/2019 7:50:56 PM, ConditionIndex: 0, Type: Trigger' Evaluator failed - 
Condition evaluation failed for query = 
(SELECT E0.[Uri], E0.[DisplayName]FROM Orion.Nodes AS E0WHERE ( ( E0.[Status] = @p0 ) )), condition = (AlertConditionDynamic: scope=EMPTY: (AND ([Orion.Nodes|Status] = '2'))) - System.ServiceModel.CommunicationException:  
The socket connection was aborted. This could be caused by an error processing your message or a receive timeout being exceeded by the remote host, or an underlying network resource issue. 

Information Service / SWIS. See Orion Platform Troubleshooting Guide - Services - SWIS.

Core Business Layer. See Orion Platform Troubleshooting Guide - Service - Module Engine.

 

Issues

Alerts are not triggering:

• Verify the alert did not actually trigger or if the action/s were not performed
• Verify the alert is enabled
• Verify the alert trigger condition is correct
• Verify alert suppression is not configured
• Verify an alert delay is not configured or has been met
• Verify that Alerting Service and Alerting Service V2 is installed (If not, run the Core Installer, Reboot Server, and run the configuration wizard to complete)

Alert not triggering. See A configured alert is not triggered.
 

Alert actions are not being performed or generated/received late:

• Verify if this involves all actions or only some
  • If only email action is configured create a new action " NetPerfMon Event  Log"
  • Reproduce the action and see if it writes to the NPM Events
• Verify alert action/s are not disabled. This is located on the "All Active Alerts" page.  Click on "Resume" to re-enable actions

• If Email action is failing, verify SMTP server is configured
• Under the Alert, Trigger Action section Simulate the action
• If logs revealed that an email action was performed, have the customer investigate Exchange server is not blocking the email
• Re-create the Actions to see if they would work

False-positive Alerts:

Investigating why a false positive alert occurred can be tricky:

• Verify the type of alert that triggered
  • Down alert - verify polling is working properly. See A node is showing as down but you can ping the same node successfully from the Orion Server or Orion Platform Troubleshooting - Polling if you believe polling is the root cause
  • Reboot alerts - This is tracked by the SNMP service if the device is monitored via SNMP.  Verify that the SNMP service did not restart.  If it did this will cause the alert to trigger, as the OID being tracked for 'Reboots' can re-initialize when the SNMP service restarts.
• If the customer is basing this on the email or Event message verify the variables are correct or that there is no static message typed in the body of the message.
  • Check reset action messages as well - customers can copy Reset messages from the Trigger messages, and if they are not edited to confirm they are Reset conditions, it can cause confusion.
• Check the logic conditions in the alert - both trigger and reset logic, and ensure it's doing exactly as the customer is expecting
  • Complex logic conditions that have not been fully thought through could behave in ways the customer has not expected, leading to the customer believing the alert was triggered or reset incorrectly