Alerts not firing, not receiving emails, or Receiving False Alerts in Orion

The article is the most advanced and comprehensive troubleshooting guide in the SolarWinds KB (Knowledge-Based) database. Can solve most alerting issues or concerns. If this does not solve an alerting problem, contact SolarWinds Support with a new support ticket.

It helps to resolve possibly 

• Alerts are not triggering/firing.
• Trigger Actions or Reset Actions are not firing.
• The alert email is not received or sent.
• Suspected false alerts.

• Improperly defined alerting conditions.
• Invalid trigger actions.
• Email to the field is wrong, blank, or custom property not being filled in.
• SMTP server settings reject the email reply field.

Validate Alert Trigger Conditions:

The first step in all alerting situations is to determine the validity of the trigger conditions. Trigger conditions can be validated from the Summary tab of the edit alert wizard. Look for the notification in the lower right near the Submit button to verify your alert trigger conditions. If this states This alert will be immediately triggered on 0 object(s) in alert scope, then you need to go back to the drawing board - navigate to the Trigger Conditions page. Recheck your conditions and verify that they are correct.
 

NOTE: This popup is NOT displayed when using the "Enable Complex conditions" setting - the option is set on the trigger conditions page.

Once you have validated that the alert will trigger on at least one object with the popup near the submit button, go ahead and submit the alert.
 

More advanced troubleshooting can be performed through the use of SWIS queries. If you're proficient enough with queries, you can utilize the generated SWQL query available on the Trigger Conditions page. Click the carrot button > "Show SWQL". Copy the SWQL in the popup dialog box. 
 

Now to execute this query, navigate to this page in a new browser tab and fill in your specific Orion site name: 
http:// [HostName/IP_Address] /Orion/admin/swis.aspx#
Execute the query to see the resulting objects that will actively fire the alert.

TIPS:

• These queries work just like regular SQL. Percent is a wildcard, functions like LIKE or JOIN work, etc.
• You can try modifying the query to see what needs to be changed to trigger expected objects. Try changing your trigger conditions and regenerate a new SWQL query.
• If you need to add a column to your query to understand the objects' names listed, you can generate a query to view all the columns in a specific table. Click in the drop-down to navigate to Orion.Nodes. Next, click on the Generate Select Query (clear out your query first). It will display all the columns available for you to select (note: Caption is a good column for determining node name).

If the alert does not show up in Active Alerts and/or the database view [AlertHistoryView], consider the following information.

• Ensure the alert is not toggled off in the Manage Alerts menu. You can verify on the summary tab of the alert or the main Manage Alerts page.
• If using a Node alert: ensure that node alerts are not muted. Settings > Manage Nodes > Select the node > Maintenance Mode > Resume Alerts (Unmute).
• If using an Application alert: applications have their muting separate from nodes. Resume alerting in the SAM application view: Settings > All Settings > SAM Settings > Manage Application Monitors. Select the Application > Maintenance Mode > Mute or Resume.
• Muted objects will not generate entries in the [AlertHistoryView] database view but create entries in the [AlertSupression2] database table.

Validate the Alert has Triggered:

Go to the Alerts & Activity > Alerts in the menu bar. Sort by 'Active Time'. If you have just submitted the alert it should be the most recent or at the very least, near the top.

Alternatively, you can check the database directly. 

-- Scripts are not supported under any SolarWinds support program or service.
-- Scripts are provided AS IS without warranty of any kind. SolarWinds further
-- disclaims all warranties including, without limitation, any implied warranties
-- of merchantability or of fitness for a particular purpose. The risk arising
-- out of the use or performance of the scripts and documentation stays with you.
-- In no event shall SolarWinds or anyone else involved in the creation,
-- production, or delivery of the scripts be liable for any damages whatsoever
-- (including, without limitation, damages for loss of business profits, business
-- interruption, loss of business information, or other pecuniary loss) arising
-- out of the use of or inability to use the scripts or documentation.

SELECT TOP 100 * FROM [AlertHistoryView]
--WHERE Name LIKE '%Insert the alert name here and delete the hyphens(optional)%'
ORDER BY TimeStamp DESC

This database view can also help determine if the alert is resetting quickly. The column [EventTypeWord] will indicate if the alert has Reset.

This view is also helpful in tracking down all alerts that a specific node has triggered. Here is an example that can be modified.
 

-- Scripts are not supported under any SolarWinds support program or service. 
-- Scripts are provided AS IS without warranty of any kind. SolarWinds further 
-- disclaims all warranties including, without limitation, any implied warranties 
-- of merchantability or of fitness for a particular purpose. The risk arising 
-- out of the use or performance of the scripts and documentation stays with you. 
-- In no event shall SolarWinds or anyone else involved in the creation, 
-- production, or delivery of the scripts be liable for any damages whatsoever 
-- (including, without limitation, damages for loss of business profits, business 
-- interruption, loss of business information, or other pecuniary loss) arising 
-- out of the use of or inability to use the scripts or documentation.

SELECT TOP 100  n.caption, a.* FROM [AlertHistoryView] a
JOIN [NodesData] n ON a.RelatedNodeCaption = n.Caption
--WHERE n.Caption LIKE '%Your node name here (optional)%'
ORDER BY TimeStamp DESC

Validate the status of the Alert Action:

The [AlertHistoryView] database view is very useful for tracking down the sequence of events upon the alert trigger. With a standard alert, you will see an entry for the trigger - [EventTypeWord] column with a value of Triggered indicates the alert triggered. If an action is configured for the alert, you will see an ActionSucceeded or ActionFailed. If an action failed exists for the alert, visit the log file on the main polling engine: 
C:\ProgramData\SolarWinds\Logs\Orion\ActionsExecutionAlert.log

Search from the bottom up for the most recent entries.

Most frequent causes of failure for alert actions: 

• The use of custom properties in email recipients.
  • Custom Property was not filled in for the node that triggered the alert. A blank value will be sent to a blank recipient.
  • Improper custom property value is set for the recipient. It is recommended to fill in the correct variable using the "insert variable" button which is only available to use in the Subject and Email Body text fields. Then copy and paste the variable to the recipient text box. Alternatively, consult the guide: Commonly Used Alert Email Variables.
  • If the custom property was very recently created, try bouncing services to force a refresh.
• The reply address of the email action is an invalid recipient and rejected by your SMTP server. There should be an error in the log file indicating this. 
• SMTP server is invalid - this would be indicated in the log file.

NOTES:

• If the action does not show in the database table view, ensure the action is toggled ON in the Action Manager (Settings > All Settings > Manage Alerts > click the Action Manager tab. Alternatively, you can click on the hyperlink of the actions by finding the alert in the 'Alert Manager' tab, this will take you to the actions associated with the alert).
• In some cases, Try to duplicate and edit the Alert then check the Summary where it shows "triggered on X objects". If there are objects to trigger, click submit and enable the alert to check if the alert would trigger.
• In rare cases, it is advised to recreate the alert actions especially when the alert was duplicated from another alert. Please see: Alert is not working and does not trigger.
• If the trigger action is "execute an external program" it is recommended to contact support for troubleshooting.