Submit a ticketCall us

AnnouncementsSystem Monitoring for Dummies

Tired of monitoring failures disrupting the system, application, and service? Learn the key monitoring concepts needed to help you create sophisticated monitoring and alerting strategies that can help you save time and money. Read the eBook.

Get your free eBook.

Home > Success Center > Network Performance Monitor (NPM) > NPM - Knowledgebase Articles > JobEngineV2 won't stay running due to Out of Memory Errors

JobEngineV2 won't stay running due to Out of Memory Errors

Updated: February 20, 2019

Overview

 

This article briefly discusses one possible cause as to why the Solarwinds JobEngineV2 Service keeps crashing which results in unstable polling or missed events/alerts.

 

Extract from the C:\ProgramData\SolarWinds\JobEngine.v2\Logs\SolarWinds.JobEngineService_v2_13.log

2018-11-07 16:25:38,485 [47] FATAL SolarWinds.JobEngine.Diagnostics.Log4NetWcfTraceListener - (null)    
System.ServiceModel Critical: 131077 :
<TraceRecord xmlns="http://schemas.microsoft.com/2004/10...nt/TraceRecord" Severity="Critical"><TraceIdentifier>UnhandledException</TraceIdentifier><Description>Unhandled exception</Description><AppDomain>SWJobEngineSvc2.exe</AppDomain><Exception><ExceptionType>System.OutOfMemoryException, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089</ExceptionType><Message>Exception of type 'System.OutOfMemoryException' was thrown.</Message><StackTrace>   at System.Xml.XmlBufferReader.GetCharBuffer(Int32 count)

 

Extract from  C:\ProgramData\SolarWinds\Logs\IPAM\IPAM.Businesslayer.log

**** 2018-11-07 14:59:19,135 [Main] ERROR SolarWinds.IPAM.Storage.Credentials.CredentialInitializer (null) - IPAM was unable to convert 'CredentialName' credential with id:'5365528f-6e2c-4154-b7a3-61a530957c54'; at Exception:System.OutOfMemoryException: Exception of type 'System.OutOfMemoryException' was thrown.
   at System.Convert.ToBase64String(Byte[] inArray, Int32 offset, Int32 length, Base64FormattingOptions options)
   at System.Convert.ToBase64String(Byte[] inArray)
   at System.Security.Cryptography.Xml.CipherData.GetXml(XmlDocument document)
   at System.Security.Cryptography.Xml.EncryptedData.GetXml(XmlDocument document)
   at System.Security.Cryptography.Xml.EncryptedXml.ReplaceElement(XmlElement inputElement, EncryptedData encryptedData, Boolean content)
   at SolarWinds.Orion.Security.CryptoHelper.EncryptXml(String plainText)
   at SolarWinds.Orion.Security.CryptoHelper.Encrypt(String plainText)
   at SolarWinds.IPAM.Storage.DAL.SNMPCredDAL.AddAllValues(SqlCommand command, SNMPCred cred, Boolean IncludeKey)
   at SolarWinds.IPAM.Storage.DAL.SNMPCredDAL.Update(SNMPCred cred)
   at SolarWinds.IPAM.Storage.Credentials.CredentialInitializer.UpdateSNMPCred(SNMPCred[] oldCredentials)

Environment

  • NPM 12.2+
  • IPAM 4.6+

 

Cause 

  • Corrupted IPAM SNMP Credentials. 

 

Resolution

  • As we can see from the second error above, the error is due to the 'CredentialName' credential in the IPAM_SNMPCred table in your Orion Database.
  • Delete the credential from the IPAM SNMP credentails (Settings > IPAM Settings > SNMP credentials).
  • Alternatively, you can delete the credential from the Orion Database by browsing to the IPAM_SNMPcred table using a database management tool and deleting the row.

 

 

You may need to clear SDFs afterwards for the jobsengine v2 and the collector service

 

REPLACE JOB ENGINE v2

1. Stop all Orion services from Orion Service Manager.

2. Go to C:\ProgramData\SolarWinds\JobEngineV2\Data\JobEngine35.sdf file and rename it to JobEngine35_OLD.sdf

3. Then create a copy of <C:\ProgramData\SolarWinds\JobEngineV2\Data\JobEngine35 - Blank.sdf> and rename it to JobEngine35.sdf.

4. This way you have the old one to revert back to and you always have a blank copy in case issue reoccurs.

5. Right click on properties of the JobEngine35.sdf you have created and untick the read only box and click ok.

 

REPLACE POLLING CONTROLLER

1. Stop all Orion services from Orion Service Manager.

2. Go to C:\ProgramData\SolarWinds\Collector\Data\PollingController.sdf file and rename it to PollingController_OLD.sdf

3. Then create a copy of <C:\ProgramData\SolarWinds\Collector\Data\PollingController - Blank.sdf> and rename it to PollingController.sdf.

4. This way you have the old one to revert back to and you always have a blank copy in case issue reoccurs.

5. Right click on properties of the PollingController.sdf you have created and untick the read only box and click ok.

6. Restart all Orion Services

 

 

Note: if the JobEngine service or Collector go to stop state on the Orion Service Manager, that means that you missed the read-only option under Properties on the SDF file, please check this.

Right Click the file PollingController.sdf or JobEngine35.sdf and right-click it > properties and untick read-only

 

 

 

Last modified

Tags

This page has no custom tags.

Classifications

Public