Applications Systems

Realtime process explorer, Service Control Manager and Realtime event log viewer not working on some Additional Pollers

Real-time process explorer, Service Control Manager and Realtime event log viewer does not work on some of the Additional Polling Engines. Real time process explorer in SAM for APE.

First published date

11/1/2019 4:51 AM

Last published date

7/18/2022 8:47 AM

Overview

When clicking Real-time process explorer, Service Control Manager or Real-time Event log viewer on supported nodes on some Additional Pollers, you will see an error message similar to this: 
Server stack trace: at System.ServiceModel.Channels.SecurityChannelFactory`1.SecurityRequestChannel.ProcessReply(Message reply, SecurityProtocolCorrelationState correlationState, TimeSpan timeout) at System.ServiceModel.Channels.SecurityChannelFactory`1.SecurityRequestChannel.Request(Message message, TimeSpan timeout) at System.ServiceModel.Security.SecuritySessionSecurityTokenProvider.DoOperation(SecuritySessionOperation operation, EndpointAddress target, Uri via, SecurityToken currentToken, TimeSpan timeout) at System.ServiceModel.Security.SecuritySessionSecurityTokenProvider.GetTokenCore(TimeSpan timeout) at System.IdentityModel.Selectors.SecurityTokenProvider.GetToken(TimeSpan timeout) at System.ServiceModel.Security.SecuritySessionClientSettings`1.ClientSecuritySessionChannel.OnOpen(TimeSpan timeout) at System.ServiceModel.Channels.CommunicationObject.Open(TimeSpan timeout) at System.ServiceModel.Channels.ServiceChannel.OnOpen(TimeSpan timeout) at System.ServiceModel.Channels.CommunicationObject.Open(TimeSpan timeout) at System.ServiceModel.Channels.ServiceChannel.CallOpenOnce.System.ServiceModel.Channels.ServiceChannel.ICallOnce.Call(ServiceChannel channel, TimeSpan timeout) at System.ServiceModel.Channels.ServiceChannel.CallOnceManager.CallOnce(TimeSpan timeout, CallOnceManager cascade) at System.ServiceModel.Channels.ServiceChannel.Call(String action, Boolean oneway, ProxyOperationRuntime operation, Object[] ins, Object[] outs, TimeSpan timeout) at System.ServiceModel.Channels.ServiceChannelProxy.InvokeService(IMethodCallMessage methodCall, ProxyOperationRuntime operation) at System.ServiceModel.Channels.ServiceChannelProxy.Invoke(IMessage message) Exception rethrown at [0]: at System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(IMessage reqMsg, IMessage retMsg) at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData& msgData, Int32 type) at SolarWinds.JobEngine.IJobScheduler.AddJob(ScheduledJob jobConfiguration) at SolarWinds.JobEngine.Routing.JobRouterWCFProxy.<>c__DisplayClass11_0.b__0(IJobExecutionRouterProxy n) at SolarWinds.JobEngine.WCFUtils.ChannelProxy`1.Invoke[TResult](Func`2 fnc, ChannelProxyBehavior behavior) at SolarWinds.JobEngine.WCFUtils.ChannelProxy`1.Invoke[TResult](Func`2 fnc) at SolarWinds.JobEngine.Routing.JobRouterWCFProxy.AddJob(ScheduledJob jobConfiguration)

When you check the JobEngineV2 log, you will see an interesting error:
  
ERROR SolarWinds.JobEngine.Security.UsernamePasswordValidator - (null)     SUPPORT: Invalid username or password.  Job Engine password files are out of sync.  Remove ucdat1.xml on the MAIN poller and run Configuration Wizard on the MAIN to resolve the issue

 

Product section

Server Application Monitor

Cause

This issue could be due to a corrupted udcdat1.xml file

Resolution

  1. On ALL pollers including additional web servers starting with the main poller, go to C:\ProgramData\SolarWinds\JobEngine.v2\ and move the file ucdat1.xml to another location outside of the Orion folder
  2. Run the Configuration Wizard and complete the process
  3. Check and confirm if the issue still occurs