Network Management
Third party alerting service not receiving all alerts
When using 3rd party applications such as ServiceNow, the service or firewall may be blocking all or some of the alert messages passed on. Especially when bursts of several alerts are sent all at once.
First published date
Last published date
Overview
Product section
Cause
Firewall or other security service is mistaking the repeated alert message traffic as a potential threat.
Check the Alerting.ServiceV2.log for an error like the following
[{"Key":"ActionTitle","Value":"BWC-->Node Notification"},{"Key":"ErrorMessage","Value":"Failed to execute HTTP request"}]
- Check the ActionsExecutionAlert.log and found it flooded with the below.
ERROR AlertingLogger - (null) Action [Action: ID: 154, ActionType: SendHttpRequest, Title: BWC-->Node Notification, Description: Post to http://xxxxxxxx/EAWebService/rest/events?apiKey=xxxxxx, Enabled: True, Order: 2 , Context: SolarWinds.Orion.Core.Models.Actions.Contexts.AlertingActionContext, EnviromentType: Alerting, ExecutionMode: Trigger, EntityType: Orion.Nodes, EntityUri: swis://xxxxxxx .Green.net/Orion/Orion.Nodes/NodeID=78, AlertContext: AlertName: BWC - Node is down, CreatedBy: xxxxxxx_x, AlertActiveId: 20634, AlertObjectId: 1299] execution has failed.
System.Exception: Failed to execute HTTP request ---> System.Net.WebException: The underlying connection was closed: A connection that was expected to be kept alive was closed by the server. ---> System.IO.IOException: Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host. ---> System.Net.Sockets.SocketException: An existing connection was forcibly closed by the remote host
Resolution
1. Test to see if regular email alerts going through.
2. If the email alerts re going through, but the third party alerts are not. Check the logs to confirm ERROR message.
3. Check to see if network traffic is being filtered; add exception in firewall