Orion Platform SAM troubleshooting - Components

7/17/2019 2:17 PM

7/17/2019 2:17 PM

 Description  

Generally speaking, SAM Monitors will return one of the following statuses:

• Up
  • Everything is up and no thresholds are being exceeded.
• Down
  • The resource being monitored is not present.  You will typically see this status for monitors that are designed to ensure that something is available.  A few examples:
    • A Windows Service Monitor ensuring that a service is running
    • A Process Monitor ensuring that a process is running in memory.
    • An HTTP Monitor ensuring that a page is served from a web site.
• Warn
  • One of the Warn thresholds within the Monitor's settings has been exceeded.
• Critical
  • One of the Critical thresholds within the Monitor's settings has been exceeded.
• Unknown
  • The Monitor was unable to determine the status of the resource it is attempting to monitor. This can be due to (but not exclusive to) any of the following:
    • Communication issues with the device being monitored.
    • Authentication failures against the device being monitored.
    • Problem with the process responsible with providing information regarding the resource.
    • Simple configuration errors within the Monitor.
    • There is an internal problem with Orion and/or SAM.

The most common issues reported by Solarwinds Customers is that their Application is reporting a status of "Unknown", which is due to the fact that it contains one or more Monitors in an "Unknown" state and their status has rolled up to the Application.  This article is designed to help guide you through getting additional information regarding this "Unknown" status, how to replicate it and possibly how to fix it.

 

Scoping the Issue  

What to Ask 

Here are a few questions to ask before jumping into the problem. You don't have to stop at these questions. Different scenarios might need you to probe differently, but this is a good starting point.

• When was this issue first observed?
  • Were there any changes made before the problem started?
  • Were any new Firewall rules or network policies put in place that might be preventing communication?
  • Were any new Windows updates installed on the target system?

 

• Reproduce the issue
  • Can the error be generated ad-hoc or is it intermittent?  To verify this, open the Application's settings page, click "Poll Now" and wait a few seconds to let the information get updated.  
  • If the error can be generated ad-hoc, can the error be reproduced outside of SAM?  
  • If the error is intermittent, do the Alert Emails provide the reason behind the failure?  Can Application's debug logs be gathered until the error occurs again?  See "Examine the Logs" section below.

• Understand the Environment
  • Some Monitors depend on 3rd party clients to be installed.  Have these been installed?  Are they still installed?  Review the documentation for the specific Monitor type to get its requirements.  All Monitor types are listed here
  • Some Monitors are for a specific version of an application.  Has the application being monitored been upgraded? If so, it is possible that the resource being monitored by the Monitor no longer exists in the newer version of the Application.
  • Some Monitors depend on certain ports to be open.  Are these ports still open?

 

• Narrow the focus
  • Is the issue different when performing a "Poll Now" from the Application's details page vs. a Monitor test from the Application's Settings page?  Typically if both have the same outcome, the problem is due to an environmental issue outside of SAM (communication problem, authentication problem, missing 3rd party client, etc).  On the other hand, if the test works but the poll fails, there is likely an internal problem within Orion and/or SAM.
  • Is the issue isolated to a specific Additional Poller?  Try assigning the Node to a different Poller and see if the Monitors then succeed.  Assuming they do, the Additional Poller may be in an isolated environment that cannot access the resources being monitored.
  • Is the issue isolated to a specific Node?  If so, the Node may be unreachable or may be experiencing problems.

Server Application Monitor

Services Involved:

There are a few services involved in polling for data, analyzing it and storing it to the database.  All of these services are not SAM-specific and are part of the Orion Core Platform.
 

Service Name	Description	Software Package	Role
Collector Service	It does any necessary calculations on the data, then stores the results to the database.	Solarwinds Orion Core /Platform	The collector takes the polling results from the MSMQ It does any necessary calculations on the data, then stores the results to the database.
MSMQ	Provides a messaging infrastructure and development tool for creating distributed messaging applications for Windows-based networks and programs. If stopped, distributed messages will be unavailable. If disabled, any services explicitly depending on it will fail.	Microsoft Windows	The Job Engine Workers send polling results to MSMQ. (depreciated as of Orion Platform release 2020.2.6 and replaced by SQLite)
Information Service/ SWIS	Communication Service for READ ONLY SQL data	Solarwinds Orion Core /Platform	Used for Subscriptions, Notifications, Dependencies, Containers (Groups), Orion Authentication, EOC Communication, NCM Communication and Limitations
Module Engine	Service used to launch module engine plug-ins for all SolarWinds Orion products	Solarwinds Orion Core /Platform	Used for Core Licensing and Module Specific operations. Each Orion Module will use this service differently. Common examples are Topology and Manage Nodes operations
Job Engine V2	Polling Execution Engine for all SolarWinds Orion products		Executes all polling tasks used for Orion products. Each module will use this service differently. Features like Poll Now, Test Credentials and List Resources are common examples of Web console operations using JE2

Examine the Logs:

Applications have an option to enable debug logging. See Turn on application debugging for SAM.

• Once this is done, click "Save and Continue Working" at the bottom.  Then, select the Component that is in an "unknown" state and click "Test", as seen below:

NOTE:  AppInsight

Appinsight applications do not have a "Test" button.  As such, it is not possible to generate logs for a specific Monitor.  To generate the logs, you can either wait until the next poll or click the "Poll Now" button from the Application's Details view. 
 

• After the test has completed and returned the results, note 3 things: 

1. The ID of the Application
   1. To gather this value, open the settings page for the Application and note the last numerical value in the browser's URL field.  
   2. Example: http://.../Orion/APM/Admin/Edit/Edit...ation.aspx?id=20
2. The ID of the Component
   1. To gather this value, open the details page for the Component and note the last numerical value in the browser's URL field.  
   2. Example: http://.../Orion/APM/MonitorDetails.aspx?NetObject=AM:2092
3. The Polling Engine used
   1. To gather this value, open the details page for the Node the Application is assigned to, scroll down to the "Polling Details" web resource and note the "Polling Engine" value.

• Now logon to the system that is hosting the noted Polling Engine and open the following directory:
  • C:\ProgramData\Solarwinds\Logs\APM\ApplicationLogs\AppId[##]\  
    • Where [##] is the noted Application ID.

The log files will contain the Date and Time, the Monitor ID, and the type of entry it is (Debug, Info, Error).  Isolate all entries for the specific Component Monitor and look for ERROR entries pertaining to it. Example [C2092] Error Solarwinds.APM.Probes, C2092 is the component ID.
 

Using Tools to replicate Monitor tasks:

As SAM has many different monitor types, the tools and methods to replicate a Monitor's tasks outside of SAM will differ. This section provides a quick look at the different tools and methods used for the most popular Monitors.  These have been organized as follows:

• WMI-based Monitors - test using Wbemtest. See Testing WMI Connectivity
• Mail-based Monitors -  Test using Telnet. See Test SMTP server connectivity via telnet.
• Database Monitors
• Script Monitors -
  • Unix/Linux script test using SSH
  • Troubleshoot Powershell script 
• Performance Counter Monitors. See Performance counters are not working or missing in PerfMon
• SNMP-based Monitors - test using SNMPwalk. See Run SolarWinds SNMP Walk.

 

Likely Root Causes:

Explore each of the potential root causes below to learn more about...

The required 3rd party client is missing or not installed correctly. 

Things to check:

 1. Verify the documentation for the Monitor in question, to see what is required.  The following page contains the full list of Monitors and links to their documentation:

SAM component monitor types

You can see a full list of out of the box templates and links to their documentation here:

SAM monitor template reference

2. Verify that the platform of the installed client matches the polling platform specified within the Application (x86 vs. x64)

3. Verify that the version of the installed client is compatible with the version of the application being monitored.  An Oracle 9 client is not likely compatible with an Oracle 11 database server.

The Remote device is responding incorrectly. 

The remote device is not providing the expected information.  The Monitor may be receiving an error such as "Invalid WMI Query" or "Performance Counter not found".  

To view the full details of the error, use the steps in the "Examine the Logs" section above and  then use the steps in the "Using tools to replicate Monitor tasks" section above to see if the error can be replicated outside of SAM.

A required port is closed. 

1. Try stopping the Windows Firewalls on the system hosting the SAM Poller and the system being monitored.

2. Ensure that all ports mentioned in the following document have been opened:

SolarWinds Port Requirements