Orion Platform Troubleshooting - Polling

7/17/2019 6:52 PM

5/27/2020 3:56 PM

Description  

One of the more common SolarWinds customer support calls involve polling failures. There are many factors that come into play so it is important that you understand how to address this issue in a logical manner. 

Customers might not always provide a lot of details but it will almost certainly be described as:

• Not polling
• Polling failed
• Charts are empty
• Orion incorrect
• Orion showing wrong information
• Charts are incorrect
• Nodes not updating
• Device reporting different values to Orion

Scoping the Issue  

What to Ask 

Here are a few questions to ask before jumping into the problem. You don't have to stop at these questions. Different scenarios might need you to probe differently, but this is a good starting point.

• Narrow the focus

  • Are all nodes affected? 
  • Is other information on the node/volume/interface being polled correctly?  
  • Are some nodes returning the same information correctly? Is there a pattern you can clearly define?
  • Is Orion having any other issue at the same time? (This may increase the issue scope beyond a polling issue)
  • How is the node, interface, volume being polled?

• When was this issue first observed?
  • Try to establish what happened before the onset of the problem
  • Were there any changes made before the problem started?
  • If this is affecting just one device, has the device been recently added or recently changed/upgraded?
     
• Reproduce the issue
  • If you can't reproduce the issue, you can't confirm whether it's been fixed
  • If its a once-off occurrence, the most you can do is check the log files, and try to determine what occurred at that time.
  • Can it be easily reproduced or is it intermittent? Is there any pattern?
  • What are the expected results?

• Understand the Environment
  • What types of devices are being monitored
  • What are the Orion server specs? SQL server specs? (are they below recommended requirements?)
  • Check CPU and Memory usage on both servers
  • Is there an Anti-Virus agent on the server? Has it been configured for on-access file scans?
  • In the case of incorrect polling what makes it appear the information is incorrect? (Is there someplace else where the values are otherwise?)

Orion Platform

How Polling Works 

Orion Core Polling Responsibilities 

The Orion core is responsible for polling:

• Nodes
  • Status (ICMP)
  • Response Time & Packet Loss (ICMP or SNMP)
  • System Uptime (SNMP or WMI)
  • System Information (SNMP or WMI)
  • CPU & Memory Statistics (SNMP or WMI)
• Volumes (SNMP or WMI)
  • Status 
  • Volume Information
  • Volume Statistics
• Topology (SNMP)

Services Involved

Service Name	Description	Software Packages	Role
Job Engine v2	Polling Execution Engine for all SolarWinds Orion products. Executes all polling tasks used for Orion products by spawning Job Engine workers. Each module will use this service differently.	SolarWinds Orion Core/Platform	The Job Engine (JE) is a Windows service that is used by Orion-based products to run polling tasks (jobs). Polling in this case means repeated work that gathers data from monitored devices through various protocols (SNMP, WMI, ICMP) and sends this polled data back to the Collector services to be processed and saved into the Orion database. Receives poll results from the job engine worker processes, and forwards the results to the Collector Polling Controller Scheduled jobs are stored persistently in the JobEngine SDF file. If you need to reset the jobs, you can replace these SDF files with the blank version. Job Engine v2 process (on each poller) contains the scheduler internally, so unlike v1, there's no scheduler process Job Engine v2 Workers also perform Network Sonar Discoveries.
Job Engine V2	Polling Execution Engine for older SolarWinds Orion products. Executes all polling tasks used for older Orion products by spawning Job Engine workers.	SolarWinds Orion Core/Platform	Job Engine (JE) is a Windows service that is used by Orion-based products to run polling tasks (jobs). Polling in this case means (mostly) periodic work that gathers data from monitored devices. Receives poll results from the job engine worker processes, and forwards the results to the Business Layer to be written to the database.   the Job Engine and is responsible for scheduling the polling Jobs.
Collector Service	It does any necessary calculations on the data, then stores the results to the database.		The collector takes the polling results from the MSMQ It does any necessary calculations on the data, then stores the results to the database.
Message Queuing	Provides a messaging infrastructure and development tool for creating distributed messaging applications for Windows-based networks and programs. If stopped, distributed messages will be unavailable. If disabled, any services explicitly depending on it will fail.	Microsoft Windows	The Job Engine Workers send polling results to MSMQ.

Other Components

Component	Function
SQL Compact Edition (SQL CE) 32 bit	Allows the job engines and collectors to store data in temporary SDF files.
SQL Compact Edition (SQL CE) 64 bit	Allows the job engines and collectors to store data in temporary SDF files. Requires the 32 bit version be installed first.
SDF files	Job Engine use SDF files.

Single Node Polling Failures 

Single  Status or Statistic Polling Failures 

Verify if a node should be able to be polled by Orion

• Check how the node is being polled (the Polling Method) and which Polling Engine is polling the node. 
  • For Status Polling this is currently only possible to determine by checking the Manage Pollers page in the web console or by checking the pollers table in the database.
    • To check the Manage Pollers in the web console browse to Settings->Manage Pollers, review the polling methods in use 
• If the node is being polled via ICMP: Ping (ip address) using the command prompt on the Orion Server
  • If the ping command fails then confirm with the customer if ICMP connectivity is blocked. It is expected that Orion is not able to detect ICMP status of the node if a ICMP ping from the Orion server also fails due to the device being switched off or network issues or network configuration are preventing ICMP communication. 
  • If the ping command fails and the customer doesn't believe there is a firewall or other ICMP issue on their network perform a packet trace in Wireshark using a capture filter of 

    • host <IP_Address>

    • where <IP_Address> is replaced with the IP address of the node.
• If the Node is being polled for Status via SNMP:
  • Determine the Orion polling engine monitoring the node. The Polling Details Resource in the Web Console displays the polling engine the node is allocated to. This web console resource is the easiest way to tell. This also shows the statistic polling method. 
  • RDP to the polling engine polling the device 
  • Open C:\Program Files (x86)\SolarWinds\Orion and run SNMPWalk.exe
  • If the SNMP Walk tool fails when run or has no results his suggests there is no SNMP connectivity or the SNMP port or credentials are are incorrect and Orion is correctly reporting the device as down.
  • You could also use Wireshark
    • Apply a display filter in Wireshark for snmp and diagnose the results, the expected results when SNMP is working correctly 
    • f the SNMP Walk fails and a Wireshark capture shows requests being send but no replies received then this points to an issue on the device or on the network which the customer must identify and resolve.
    • If the SNMP Walk succeeds confirm Orion is polling the same IP address and perform a packet capture as per above however don't perform a manual SNMP Walk against the device device and wait for Orion to do so normally (usually every 2 minutes).
      • If an SNMP request is sent by Orion and none is received back then you have identified an issue on the device or on the network which the customer must identify and resolve. (Potential causes are misconfigured devices or SNMP being blocked to the device.)
      • If Orion sends no SNMP requests then Orion is likely misconfigured for the device name/IP address and the filtering in place is not detecting where the ICMP requests are being sent. 
• If the node is being polled via WMI
  • Check connection using WMI Test. See Testing WMI Connectivity.
• If the Node is Polled via Agent check:
  • C:\ProgramData\Solarwinds\Logs\AgentManagement\*
  • Download Agent logs from Target device/node if possible if not

    • Located in:      ClientServer  C:\ProgramData\Solarwinds\Logs\Agent
      
      See: How to gather NPM and SAM agent diagnostics

Multiple Polling Failures 

Status and Statistic Polling Failures 

Determine common property and root cause:

• Determine if the nodes affected have a common property, all in one customer site, all polled via one protocol, all of one device type, all polled from one polling engine.
  • This can be done by both asking the customer and checking the details of some of the working and some of the failing nodes.
• Take a sample of one of the failing nodes and perform the above checks for a single node failing.
• Take a second sample and repeat the checks.
• Take a sample of a working device and verify it is polling correctly.
• From this is should be clear where/why polling is failing for some devices and not others and a common property should have been determined.

No common property:

It is possible the Polling Engine or specific polling engines are not able to handle the polling load (too many elements or polling too quickly) and the polling load must be reduced or the environments polling capacity increased (more server resources per poller or more pollers). This can be verified in the web console by browsing to Settings->Polling Engines and reviewing the Polling Completion and the Element counts. The Scalability Guide gives more details on the number of elements which can be polled per polling engine per module.

Verify if any statistic data should be able to be polled by Orion

• Check how the node is being polled (the Polling Method) and which Polling Engine is polling the node. 
  • The Polling Details Resource in the Web Console displays the polling engine the node is allocated to. This web console resource is the easiest way to tell. This also shows the statistic polling method. See the steps Above in single polling failures

If all nodes fail to poll check Polling service are running.
See Orion Platform Troubleshooting MSMQ/Collector/Job Engine (Polling Issues).

Single Node Polling Status Issues 

ICMP Status Failing

Having determined an issue causing ICMP status to fail for a single device a few potential root causes exist.

• The wrong IP address or hostname is used.
• If a device name is used DNS may not have the expected entry for the device or DNS may not be resolving as expected.
• A firewall, IPS, proxy, Anti-virus is blocking ICMP traffic.
• The device is refusing ICMP traffic.
• The ICMP packet size is too large (unlikely).
• Latency or hop count to the remote device is too large and the request or the response are timing out.
• MAC address mismatches (Orion has the incorrect MAC address for the device if on the same subnet).
• IP routing to just the device may be flawed (tracert and route print commands along with Wireshark may help validate).
• The Orion server may have a host file entry for the device affecting DNS resolution.

 

SNMP Status Failing

Having determined an issue causing SNMP status to fail for a single device a few potential root causes exist.

• The wrong IP address or hostname is used.
• If a device name is used DNS may not have the expected entry for the device or DNS may not be resolving as expected.
• A firewall, IPS, proxy, Anti-virus is blocking SNMP traffic.
• The device is refusing SNMP traffic (port 161 not open).
• The SNMP packet size is too large
• Latency to the remote device is too large and the SNMP requests or the responses are timing out.
• MAC address mismatches (Orion has the incorrect MAC address for the device if on the same subnet).
• IP routing to just the device may be flawed (tracert and route print commands along with Wireshark may help validate).
• The Orion server may have a host file entry for the device affecting DNS resolution.
• The wrong SNMP version is being used.
• The wrong SNMP community string or authentication settings are in use.

 

SNMP Statistic Polling Failing

Having determined an issue causing all SNMP statistic polling to fail for one device there are a lot of potential root causes.

• The wrong IP address or hostname is used and Orion is not polling the expected device.
• If a device name is used DNS may not have the expected entry for the device or DNS may not be resolving as expected.
• A firewall, IPS, proxy, Anti-virus is blocking SNMP traffic.
• The device is refusing SNMP traffic (port 161 not open).
• The wrong SNMP version is being used.
• The wrong SNMP community string or authentication settings are in use.
• The SNMP packet size is too large.
• Latency to the remote device is too large and the SNMP requests or the responses are timing out.
• The device isn't returning any or enough information over SNMP
• The device's sysObject OID is not matching one Orion knows and the wrong pollers have been assigned (Poller Checker and Pollers table will identify pollers assigned).

 

Having determined an issue causing one or some SNMP statistic polling to fail or report incorrectly for one device a few potential root causes exist.

• The wrong IP address or hostname is used and Orion is not polling the expected device.
• If a device name is used DNS may not have the expected entry for the device or DNS may not be resolving as expect.
• The device isn't returning any or enough information over SNMP
• The device's sysObject OID is not matching one Orion knows and/or the wrong pollers have been assigned (Poller Checker and Pollers table will identify pollers assigned).
• The required statistic on the device does not have an in-built poller in Orion (Technology Pollers, Custom Pollers, UnDPs can be used to poll the needed data)
• Orion errors on the data returned by the poller due to the data returned being in an unexpected format. 

 

WMI Statistic Polling Failing 

To Determine an issue causing all or some WMI statistic polling to fail for one device there are a number of possible root causes. 

• WMI/RPC ports are blocked (this is very common across WAN connections)
• The wrong IP address or hostname is used and Orion is not polling the expected device.
• If a device name is used DNS may not have the expected entry for the device or DNS may not be resolving as expected.
• A firewall, IPS, proxy, Anti-virus is blocking WMI/RPC traffic.
• The device is refusing WMI traffic (many ports not open).
• Latency to the remote device is too large and the WMI requests or the responses are timing out.
• The device isn't returning any or the correct information over WMI for the information being polled. 
• The wrong pollers have been assigned for some reason (Poller Checker and Pollers table will identify pollers assigned).

 

Agent polling Failing

• Connection Timeouts or Delays 

Network latency, routing issues or congestion can cause problems with the Agents from communicating with the poller it's assigned and vice versa. It's important that the customer be aware that they need to insure that the routing path to the client from the poller is correct, the necessary NICs are bound as the primary and that there is not a significant delay in communication. This follows standard peer to peer connectivity that would be explicit to network routing in the hosts environment and is a foundation that should be in place before ever attempting agent connectivity. If the foundation is unstable, then the performance will be unstable as well.

Such causes can be:

• High latency response time through pings
• High latency response time through hops in a tracert
• Across a WAN link
• NATed network
• Host server has a multi NIC environment with a priority order on communication

 

•  Socket Errors 

The Agent must be able to communicate with the poller it's assigned to through the necessary ports (17778). If these ports are blocked, disabled or held for listening, then it can cause the poller and the agent to not communicate.

Confirm the Agent and Orion server are in communication, using the "Using NetStat" steps above.

Example: check number of ports. See Check the number of ephermal ports in use.

 

When Multiple Nodes Fail To Poll 

Based on the troubleshooting this should be reducible to an issue with all nodes on a single poller or a superset of the single node polling issues. 

It is also possible the Polling Engine is not able to handle the polling load (too many elements or polling too quickly) and the polling load must be reduced or the environments polling capacity increased (more server resources per poller or more pollers).

Database issues may also cause erratic operation for all Orion components which may manifest as erratic polling.

 

When All Nodes Fail to Poll 

There are a large number of root causes for such issues.

• Database communication issues.
• A recent migration hasn't had all nodes moved to the new Polling Engine. (The engine ID nodes are assigned to and the engine details will not match.)
• Anit-Virus issues.
• Permission issues.
• MSMQ not installed.
• Misapplied hotfix or Buddy Drop.