SAM PowerShell Monitors are not working

This is a thorough and comprehensive guide on troubleshooting the most common issues with PowerShell Component monitors in Server & Application Monitor. Some of this information carries over some into other script monitors as well. But this guide focuses primarily on PowerShell. 

For WinRM or "Connecting to remote server failed" errors, please see: 
PowerShell script error. Connecting to remote server failed
Connecting to remote server failed with the message WinRM cannot complete the operation

 

• Script body is incorrect syntax for SAM
• Wrong config settings within the monitor

Note:

-- Scripts are not supported under any SolarWinds support program or service.
-- Scripts are provided AS IS without warranty of any kind. SolarWinds further
-- disclaims all warranties including, without limitation, any implied warranties
-- of merchantability or of fitness for a particular purpose. The risk arising
-- out of the use or performance of the scripts and documentation stays with you.
-- In no event shall SolarWinds or anyone else involved in the creation,
-- production, or delivery of the scripts be liable for any damages whatsoever
-- (including, without limitation, damages for loss of business profits, business
-- interruption, loss of business information, or other pecuniary loss) arising
-- out of the use of or inability to use the scripts or documentation.

The first step in troubleshooting a PowerShell script monitor in SAM is to have it working in PowerShell. PowerShell ISE is a highly recommended script editor for ensuring that the output pairs are expected results. Additionally, SolarWinds support does not assist in the writing, configuration, or implementation of custom scripting. This article serves as educational and troubleshooting purposes only.

Supported Output Format

In SAM, the syntax used to send an output as a statistic result will be the following: 

Write-Host Statistic.StatisticName: $StatValue
Write-Host Message.MessageName: $MsgValue

Where StatisticName and MessageName are optional identifiers for the output pair. You may omit these and the preceding period and SAM will automatically generate a name for this output pair. SAM supports up to 10 output pairs. An output pair consists of 1 statistical output (integer datatype only) and 1 message output (string datatype only).
 

Application Template Settings

Preferred Polling Method: Agent / Agentless
If polling against an Orion agent node - it is usually advised to choose 'Agent' so the agent can execute the script locally. 
If polling against a non-agent node (i.e. snmp/wmi) - then this option does not have an affect. Leave it set to default: Agent.
Platform to run polling job on: x64 / x86 
In most cases, it is recommended to use the x64 bit.
If choosing x86 against a x64 server, it is known to cause issues. Here is an example:
PowerShell script error. Connecting to remote server failed with the following error message: The WS-Management cannot process the request. This user has exceeded the maximum number of concurrent shells allowed for this plugin. Close at least one open shell or raise the plugin quota for this user. For more information, see the about_Remote_TroubleshootingHelp topic.



 

Additional Settings

Execution mode: Local host or Remote host

To understand which to use, you must know the polling method on the assigned node. Also, the polling "Preferred Polling Method"  that the application monitor is set to.All of which play a part on how the script monitor will be executed.

Scenario 1: Local host setting assigned to a non-agent polled node will be executing the script on the Orion server.
Scenario 2: Remote host setting assigned to a non-agent polled node, will be executed from the Orion server, against the remote host the application is assigned to via Enter-PSSession command.
Scenario 3: Local host and remote host settings assigned to an agent polled node, with application Preferred Polling Method set to agent, the script is executed by the Orion agent.
 

Count Statistic as difference: True or False

Used only if you require the statistic returned in your script to be the difference of the last polled value. Nearly all cases, leave this set to false.

Testing Script Output

Click Edit Script followed by Get Script Output to generate an output of the script. This will also generate output pair threshold names. What is meant by 'generate output pair thresholds', is a new section at the bottom of the monitor will appear, like so


This is important that you do not change your script body output pair names after these thresholds have been generated. This has been known to cause discrepancies in the returned results and respective status. 

 

Exit Codes

The use of exit codes are completely optional. But worth noting their value. The use of exit codes can essentially set up thresholds through the use of the script body, rather than using the built-in threshold settings. This can potentially be more customizable to your monitoring needs. For example, you could exit down if a service was not found on the machine. Or maybe just exit as warning/critical. Or exit down only if several conditions must be true.

Exit codes:

For information on passing Script arguments, please see Miscellaneous SAM script components on how to pass arguments and which are supported.
 

Other Errors

Receiving NaN 
In your script body, your statistic return is likely not a number (NaN). Ensure that your statistic output is, in fact, a number. If it is, you may need to explicitly define this such as:

[int]$variable = "0"

This is also often received when there is an error returned by the script. It is advised to run the script on a server in PowerShell to verify the error that is received. Ensure that the script is executed by the correct user.

Another note is that the script is run by the SYSTEM account on the server. That is unless you have the impersonation option enabled. 

Final Notes

It is imperative that you follow any and all SolarWinds updates. Always stay on the latest versions to ensure the highest security and avail the latest features. Some updates to take note of
 · On SAM 2020.2.5 - several PowerShell issues exist. Solution: upgrade to HF1 for that release.