Polling randomly stops after upgrading SolarWinds Platform to version 2023.2 and 2023.2.1 causing loss of data

5/31/2023 1:54 PM

10/24/2025 9:20 PM

The Job Engine service is crucial in polling information from nodes. However, a bug in SolarWinds versions 2023.2 and 2023.2.1 affects the JobEngine service, disrupting the polling process. 

In the SolarWinds.JobEngineService_v2023_2.log, the following error message will appear:

WARN  SolarWinds.JobEngine.Routing.Router - (null)   Error distributing throttling data to router 'sw-agent://faedfa3a-26c5-4ace-8b77-0d1ccfb158a7/'. System.NullReferenceException: Object reference not set to an instance of an object.    
at SolarWinds.AgentManagement.Messaging.Contract.MessagingServiceProxy.IsAgentConnected(String agentGuid)    
at SolarWinds.JobEngine.AgentSupport.MessagingService.SendMessageAsync[TData](String endpointAddress, String messageType, TData messageData, Boolean ignoreFailuresAndTimeouts)    
at SolarWinds.JobEngine.AgentSupport.Routing.JobRouterAgentProxy.UpdateThrottlingDataAsync(ThrottleData data)    
at SolarWinds.JobEngine.AgentSupport.Routing.JobRouterAgentProxy.UpdateThrottlingData(ThrottleData data)  
at SolarWinds.JobEngine.Routing.Router.UpdateRouterThrottlingData(Uri routerUri, ThrottleData data)
...
 ERROR SolarWinds.JobEngine.Engine.WorkerMonitor - (null)     Attempt to proces jobs on worker dff46970-615a-48b3-837d-770df1272641 failed.
System.IndexOutOfRangeException: Index was outside the bounds of the array.
   at SolarWinds.JobEngine.Scheduling.Scheduler.HandleJobResult(Guid cleanUpGuid, ScheduledJobInfo jobInfo, JobResult jobResult)
   at SolarWinds.JobEngine.Scheduling.Scheduler.HandleFinishedJobResult(JobCompleteEventArgs e, ScheduledJobInfo jobInfo)
   at SolarWinds.JobEngine.Scheduling.Scheduler.OnJobComplete(Object state, JobCompleteEventArgs e)
   at SolarWinds.JobEngine.Engine.JobEngine.OnJobCompleted(Object sender, JobCompleteEventArgs e)
   at System.EventHandler`1.Invoke(Object sender, TEventArgs e)
   at SolarWinds.JobEngine.Engine.WorkerMonitor.OnJobComplete(JobCompleteEventArgs e)
   at SolarWinds.JobEngine.Engine.WorkerMonitor.ProcessTerminalState(Guid workerId, Guid jobExecutionId, ActiveJobSnapshot activeJob)
   at SolarWinds.JobEngine.Engine.WorkerMonitor.ProcessJobResult(Guid workerId, Guid jobId, JobResult jobResult)
   at SolarWinds.JobEngine.Engine.WorkerMonitor.ProcessWorkerProcess(Guid workerId)


When the problem occurs the following symptoms may occur:

1. Nodes Status (including agent status) will suddenly report Unknown status
2. Polling engines may experience high CPU / Memory problems (due to jobs are piling up and not completed)

Network Performance Monitor

The following issue impacts the following versions:

• SolarWinds Platform 2023.2 and above

  <jobEngineSettings_Full jobExecutionCount="10000000">
    <nodeSources></nodeSources>
  </jobSchedulerSettings_Full>

The root cause of the JobEngine bug (in SolarWinds versions 2023.2 and 2023.2.1) was due removal of the polling completion counter. When the polling completion counter reaches close to 100,000, the increase of the bug trigger is possible.

To resolve the issue, please upgrade to SolarWinds Platform 2023.3 and above.  If an upgrade is not possible, perform the following steps to apply the workaround:

Clarification of the workaround:  The workaround is increasing the threshold to reduce the frequency of the bug. This does not resolve the bug and would require upgrading to 2023.3 for a permanent fix.  For a more permanent short-term solution without upgrade is increase the threshold (as below) and schedule a periodic restart of the Job Engine (since the counter will reset to 0 after a reboot).
 

1. Create a backup of the SWJobEngineSvc2.exe.config file located in the default directory: C:\Program Files\SolarWinds\Orion or C:\Program Files (x86)\SolarWinds\Orion.
2. Open the SWJobEngineSvc2.exe.config file and locate the <jobSchedulerSettings_Full tag.
3. Add the parameter jobExecutionCount="10000000" within the tag. The modified line should resemble: "<jobSchedulerSettings_Full jobExecutionCount="10000000" ...additional parameters ".

4. Save the changes to the file.
5. Restart the JobEngine service.

If issues arise after applying the workaround, revert the SWJobEngineSvc2.exe.config file to its previous state and restart the JobEngine service again. This will restore the original configuration and allow for further troubleshooting if needed.

Details:

look for the following in the config file:

 <jobSchedulerSettings_Full certificateBasedProtection="true" resultDispatcherThreadCount="2" deleteOldJobStatesInterval="00:15:00" agentJobsTTLExtension="0" jobExecutionMode="Poller">
    <nodeSources></nodeSources>
  </jobSchedulerSettings_Full>

change it to :

 

<jobSchedulerSettings_Full jobExecutionCount="10000000" certificateBasedProtection="true" resultDispatcherThreadCount="2" deleteOldJobStatesInterval="00:15:00" agentJobsTTLExtension="0" jobExecutionMode="Poller"> <nodeSources></nodeSources> </jobSchedulerSettings_Full>

 

 

 

Work Around 2:
restart job engine once a day via task scheduler:

As this work around will only work for a few days, you can also restart the jobengine service on schedule:

1. RDP to the Polling engine
2. Open Notepad and paste the following:

@echo off

 

echo Restarting SWJobEngineSvc2 and collector service...

 

taskkill /f /fi "SERVICES eq SWCollectorService"
taskkill /f /fi "SERVICES eq SWJobEngineSvc2"

 

net start SWJobEngineSvc2
net start SWCollectorService

 

echo Service restarted successfully

3. When saving it, choose all files and name it restart_SWJobEngineSvc2.bat

4. Open Task Scheduler and create a Task under Microsoft:

5. Set a name for the task and select "Run whether user is logged on or not and run with highest privileges:


6. On the trigger tab set a daily schedule:



7. on the Action tab, select Browse:


locate the batch file you saved and then select Open:





8. Click the OK button and enter a credentials that can run batch files: