All alerts stopped sending and come back after restarting services in Orion Platform

7/1/2021 9:17 AM

10/20/2022 7:04 AM

In big environments with a large number of triggered alerts Solarwinds system may stops triggering alerts suddenly.

Alerts start to work on their own after restarting All Solarwinds Orion Services on Main poller
 

Orion Platform

There could be two main reasons why alert action execution does not work:

1. Too many actions
2. Some particular actions take a long time to execute
3. A combination of both

Scenario I
In the Alerting service V2 logs, you will see warnings log messages like the below ones repeated:

WARN SolarWinds.Orion.Core.Alerting.Service.AlertConfigurationLock - Acquire(#157) Elapsed: 0:01:45.0147797 at SolarWinds.Orion.Core.Alerting.Service.AlertConfigurationLock.Acquire(Nullable`1 alertId)
at SolarWinds.Orion.Core.Alerting.Service.AlertConfigurationLock.AcquireShared()

AlertConfigurationLock was acquired after a long period of time. This lock is a sort of consistency check for alert conditions and is required for both condition execution and action triggering.

It also could be some performance issue in the database which is under pressure and cannot perform more triggering at that time. 

Scenario II
Failed Invalid Alert Actions - forwarding message to traps/syslogs - error processing your message or a receive timeout being exceeded by the remote host
 

2022-10-19 02:29:06,236 [32] ERROR SolarWinds.Orion.Core.Alerting.Service.ActionsResolverInternal.PendingExecutionActions - Action ID: 191, ActionType: SnmpTrap, Title: Copy of Sev-1 Trap, Description: Template: ForwardSyslog<br/>Message: ${N=Alerting;M=AlertName} Severity: Sev-1
NodeIP: ${N=SwisEntity;M=IP_Address;F=OriginalValue}
NodeName: ${N=SwisEntity;M=SysName}
Company: ${N=SwisEntity;M=CustomProperties.CompanyCode}
Comments: ${N=SwisEntity;M=CustomProperties.Comments}
DeviceType: ${N=SwisEntity;M=MachineType;F=OriginalValue}<br/>, Enabled: True, Order: 1 failed. alertActiveId: 1034215 alertObjectId: 21430. Error: The socket connection was aborted. This could be caused by an error processing your message or a receive timeout being exceeded by the remote host, or an underlying network resource issue. Local socket timeout was '00:05:00'.

2022-10-19 02:29:09,351 [61] ERROR SolarWinds.Orion.Core.Alerting.Service.ActionsResolver - Exception while ExecuteActions for [Alert:147 Escalation level:00:00:00 minutes]. Attempt #1 of 5, retry delay is 120 second(s). System.ServiceModel.EndpointNotFoundException: There was no endpoint listening at net.tcp://localhost:17777/SolarWinds/InformationService/v3/Orion/Streamed/certificate that could accept the message. This is often caused by an incorrect address or SOAP action. See InnerException, if present, for more details.

There are three possible solutions:

1. Lower the number of actions (Check the Trigger action tab when you edit the alert and check the number of actions) 
2. Investigate why actions do take too long to execute (see Alert action and alert service log)
3. Increase action execution parallelism (see below)
4. A combination of above

Scenario I
To increase the action execution parallelism you can follow the steps below:

1. Go to http://your-orion-page/Orion/Admin/advancedconfiguration/global.aspx
2. Search SolarWinds.Orion.Core.Alerting.Settings.AlertSettings.
3. Increase MaxConcurrentActions from 4 to 12:
   • 
4. Save the change and then click Restart All to restart the alerting service:
   • 

Scenario II
Non existing Action for forwarding traps is no longer in use
Log in to the Orion Web console
Disable Invalid Alert Actions from Action Manager
Restart the Alert Engine Service