Network Management

Troubleshoot Orion CLI Polling credential test failure

This article describes how to troubleshoot Orion CLI polling credentials and will try to find the root cause through the Orion logs .

First published date

3/12/2020 3:26 PM

Last published date

1/13/2025 5:37 PM

Overview

In this article we will learn how to correct and isolate the issue such as

Delay or Time out with CLI polling test
CLI Connection failed
CLI connection refused
CLI logs location

Product section

Network Performance Monitor

Cause

Bad password
Device failure privileged mode
Device failed to respond to SSH
Device delay response
Device rejecting packets / Connection
Orion failure due to L2/L3 connection failure
Cisco Nexus failed to connect CLI / Connection Time out
Cisco ASA Failed VPN Polling
User is not added to authentication group

Resolution

Let's understand and troubleshoot and know which logs we need to check

All CLI polling logs will be stored in the default location and mainly, we need to check these two logs

CLI.BusinessLayer.log
Session Trace
Collector Polling Logs

C:\ProgramData\SolarWinds\Logs\Orion\CLI
CLI.BusinessLayer.log

C:\ProgramData\SolarWinds\Collector\Logs\Plugins
SolarWinds.Node.ASA.Snmp.log

C:\ProgramData\SolarWinds\Logs\Orion\CLI\Session-Trace
(Device ip address Session Trace)

Please make sure Session Trace is enabled first. Only then will these logs be generated
Settings > All Settings > CLI Settings > Session Trace (Enable)

Once this is done let's test our connection
Example Case: Success connection

When testing CLI Credentials, please make sure you can see Cisco privileged mode #

2020-03-12 15:05:46,781 [147] INFO SolarWinds.Orion.Cli.BusinessLayer.Managers.CliCredentialsManager - ValidateLogin Started HostName:192.168.1.1 UserName:Usman TemplateId:76
2020-03-12 15:05:46,859 [147] INFO SolarWinds.Orion.Core.Common.JobEngine.JobScheduler - Creating default JobEngineV2 binding (netNamePipe)
2020-03-12 15:05:51,703 [235] INFO SolarWinds.Orion.Core.BusinessLayer.OneTimeJobs.OneTimeJobManager - Result of one time job 5dc6223a-4771-41f4-a127-1eef522f5948 received
2020-03-12 15:05:51,703 [147] INFO SolarWinds.Orion.Cli.BusinessLayer.Managers.CliCredentialsManager - ValidateLogin Complete on HostName:192.168.1.1 IsError:False

Example Case: Failure connection

2020-03-12 15:10:17,938 [228] INFO SolarWinds.Orion.Core.Common.JobEngine.JobScheduler - Creating default JobEngineV2 binding (netNamePipe)
2020-03-12 15:10:19,644 [45] INFO SolarWinds.Orion.Core.BusinessLayer.OneTimeJobs.OneTimeJobManager - Result of one time job f284b7ee-a783-42b8-b00e-e6d9ea09c5ef received
2020-03-12 15:10:19,644 [228] INFO SolarWinds.Orion.Cli.BusinessLayer.Managers.CliCredentialsManager - ValidateLogin Complete on HostName:192.168.1.1 IsError:True.

View Device communication through Session Trace logs finding the failure / success logs
C:\ProgramData\SolarWinds\Logs\Orion\CLI\
On every single request device will generate CLI Session Trace

Please make sure Session Trace are enabled first
Settings > All Settings > CLI Settings > Session Trace (Enable)

For example, in case of any failure, the CLI Session Trace will show the reason. Example Cisco ASA

Incorrect Password
Use Keyboard Interactive Authentication is Yes. Set to No.

[4a1c226c] [03/21/2020 13:27:33.100] Device Template: 76  Generic Cisco Adaptive Security Appliance - 1.3.6.1.4.1.9.1 
[4a1c226c] [03/21/2020 13:27:33.100] Try Acquire Connection... 
[4a1c226c] [03/21/2020 13:27:33.100] Connection Acquired 
[850b073c] [03/21/2020 13:27:33.100] -----------------Net. SWTelnet v7.11.0.412------------------- 
[850b073c] [03/21/2020 13:27:33.100] UseCustomMorePromptBehaviour: False 
[850b073c] [03/21/2020 13:27:33.100] MorePromptKey:   
[850b073c] [03/21/2020 13:27:33.100] Custom MorePrompt Pattern:  
[850b073c] [03/21/2020 13:27:33.131] Login Attempts: 1 
[850b073c] [03/21/2020 13:27:33.131] Enable Identifier: #$ 
[850b073c] [03/21/2020 13:27:33.131] UseExactPromptMatch=False 
[850b073c] [03/21/2020 13:27:33.131] Menu-based mode=False 
[850b073c] [03/21/2020 13:27:33.131] FreezeLoginForPreCommands mode= False 
[850b073c] [03/21/2020 13:27:33.131] CommandCharKey = Default 
[850b073c] [03/21/2020 13:27:33.131] FISP mode= False 
[850b073c] [03/21/2020 13:27:33.131] EncryptionAlgorithm= encAny 
[850b073c] [03/21/2020 13:27:33.131] WODSSH Version: 2.7.2.180 
[850b073c] [03/21/2020 13:27:33.131] UseVirtualPromptForCommands False 
[850b073c] [03/21/2020 13:27:33.131] Responce Timeout in seconds: 15 
[850b073c] [03/21/2020 13:27:33.131] Connection Timeout in seconds: 45 
[850b073c] [03/21/2020 13:27:33.131] Resx Culture is: English (United States) 
[850b073c] [03/21/2020 13:27:33.131] Encoding: Unicode (UTF-8) 
[850b073c] [03/21/2020 13:27:33.131] Process Architecture : 32-bit 
[850b073c] [03/21/2020 13:27:33.131] StateChange: Connecting 
[850b073c] [03/21/2020 13:27:33.194] Got HostFingerPrint: System.Byte[] 
[850b073c] [03/21/2020 13:27:33.209] SWTelnet9 Crypto Information Begin  
[850b073c] [03/21/2020 13:27:33.209] Protocol = SSH2 
[850b073c] [03/21/2020 13:27:33.209] RemoteName = 192.168.1.1 
[850b073c] [03/21/2020 13:27:33.209] SCcipher = AES128ctr 
[850b073c] [03/21/2020 13:27:33.209] CSCipher = AES128ctr 
[850b073c] [03/21/2020 13:27:33.209] Keys = ssh-rsa 
[850b073c] [03/21/2020 13:27:33.209] SWTelnet9 Crypto Information End 
[850b073c] [03/21/2020 13:27:33.209] StateChange: Disconnected 
[850b073c] [03/21/2020 13:27:33.209] TimerTick: Killing session timer. 
[850b073c] [03/21/2020 13:27:33.319] Connection Error : WeOnlyDo.Exceptions.SSH.AuthorizationException: Authentication with the server failed.
   at A.B.E(Int32 )
   at A.B.d(Int32 )
   at A.B.A(IAsyncResult , Int32 ) 
[850b073c] [03/21/2020 13:27:33.319] TimerTick: Killing session timer. 
[850b073c] [03/21/2020 13:27:33.319] Force Disconnect 
[850b073c] [03/21/2020 13:27:33.319] ERROR WODSSH: Authentication with the server failed.

Success Connection will show now Error under Session Trace

[7ae23765] [03/21/2020 13:24:55.406] Device Template: 76  Generic Cisco Adaptive Security Appliance - 1.3.6.1.4.1.9.1 
[7ae23765] [03/21/2020 13:24:55.406] Try Acquire Connection... 
[7ae23765] [03/21/2020 13:24:55.422] Connection Acquired 
[6d9e14f5] [03/21/2020 13:24:55.422] -----------------Net. SWTelnet v7.11.0.412------------------- 
[6d9e14f5] [03/21/2020 13:24:55.422] UseCustomMorePromptBehaviour: False 
[6d9e14f5] [03/21/2020 13:24:55.422] MorePromptKey:   
[6d9e14f5] [03/21/2020 13:24:55.422] Custom MorePrompt Pattern:  
[6d9e14f5] [03/21/2020 13:24:55.422] Login Attempts: 1 
[6d9e14f5] [03/21/2020 13:24:55.422] Enable Identifier: #$ 
[6d9e14f5] [03/21/2020 13:24:55.422] UseExactPromptMatch=False 
[6d9e14f5] [03/21/2020 13:24:55.422] Menu-based mode=False 
[6d9e14f5] [03/21/2020 13:24:55.422] FreezeLoginForPreCommands mode= False 
[6d9e14f5] [03/21/2020 13:24:55.422] CommandCharKey = Default 
[6d9e14f5] [03/21/2020 13:24:55.422] FISP mode= False 
[6d9e14f5] [03/21/2020 13:24:55.422] EncryptionAlgorithm= encAny 
[6d9e14f5] [03/21/2020 13:24:55.422] WODSSH Version: 2.7.2.180 
[6d9e14f5] [03/21/2020 13:24:55.422] UseVirtualPromptForCommands False 
[6d9e14f5] [03/21/2020 13:24:55.422] Responce Timeout in seconds: 15 
[6d9e14f5] [03/21/2020 13:24:55.422] Connection Timeout in seconds: 45 
[6d9e14f5] [03/21/2020 13:24:55.422] Resx Culture is: English (United States) 
[6d9e14f5] [03/21/2020 13:24:55.422] Encoding: Unicode (UTF-8) 
[6d9e14f5] [03/21/2020 13:24:55.422] Process Architecture : 32-bit 
[6d9e14f5] [03/21/2020 13:24:55.422] StateChange: Connecting 
[6d9e14f5] [03/21/2020 13:24:55.469] Got HostFingerPrint: System.Byte[] 
[6d9e14f5] [03/21/2020 13:24:55.547] SWTelnet9 Crypto Information Begin  
[6d9e14f5] [03/21/2020 13:24:55.547] Protocol = SSH2 
[6d9e14f5] [03/21/2020 13:24:55.547] RemoteName = 192.168.1.1 
[6d9e14f5] [03/21/2020 13:24:55.547] SCcipher = AES128ctr 
[6d9e14f5] [03/21/2020 13:24:55.547] CSCipher = AES128ctr 
[6d9e14f5] [03/21/2020 13:24:55.547] Keys = ssh-rsa 
[6d9e14f5] [03/21/2020 13:24:55.547] SWTelnet9 Crypto Information End 
[6d9e14f5] [03/21/2020 13:24:55.547] StateChange: Connected 
[6d9e14f5] [03/21/2020 13:24:55.547] Connected! 
[6d9e14f5] [03/21/2020 13:24:55.547] -->Type help or '?' for a list of available commands. 
[6d9e14f5] [03/21/2020 13:24:55.547] --> 
[6d9e14f5] [03/21/2020 13:24:55.547] ProcessLogin State: Banner 
[6d9e14f5] [03/21/2020 13:24:55.547] --> 
[6d9e14f5] [03/21/2020 13:24:55.547] -->ciscoasa>  
[6d9e14f5] [03/21/2020 13:24:55.547] ProcessLogin State: Banner 
[6d9e14f5] [03/21/2020 13:24:57.438] TimerTick: mstrData=<ciscoasa> > State=Connected 
[6d9e14f5] [03/21/2020 13:24:57.438] Pending Disconnect = False 
[6d9e14f5] [03/21/2020 13:24:57.438] Sending something to get a banner 
[6d9e14f5] [03/21/2020 13:24:57.438] <-- 
 
[6d9e14f5] [03/21/2020 13:24:57.438] --> 
[6d9e14f5] [03/21/2020 13:24:57.438] --> 
[6d9e14f5] [03/21/2020 13:24:57.438] ProcessLogin State: Banner 
[6d9e14f5] [03/21/2020 13:24:57.438] --> 
[6d9e14f5] [03/21/2020 13:24:57.438] -->ciscoasa>  
[6d9e14f5] [03/21/2020 13:24:57.438] ProcessLogin State: Banner 
[6d9e14f5] [03/21/2020 13:24:59.453] TimerTick: mstrData=<ciscoasa> > State=Connected 
[6d9e14f5] [03/21/2020 13:24:59.453] Pending Disconnect = False 
[6d9e14f5] [03/21/2020 13:24:59.453] TimerTick: send CRLF to get the prompt 
[6d9e14f5] [03/21/2020 13:24:59.453] <-- 
 
[6d9e14f5] [03/21/2020 13:24:59.453] --> 
[6d9e14f5] [03/21/2020 13:24:59.453] --> 
[6d9e14f5] [03/21/2020 13:24:59.453] ProcessLogin State: Banner 
[6d9e14f5] [03/21/2020 13:24:59.453] --> 
[6d9e14f5] [03/21/2020 13:24:59.453] -->ciscoasa>  
[6d9e14f5] [03/21/2020 13:24:59.453] ProcessLogin State: Banner 
[6d9e14f5] [03/21/2020 13:24:59.453] Custom Prompt detector detected a '>/#' prompt 
[6d9e14f5] [03/21/2020 13:24:59.453] no credentials needed - Command mode prompt detected 
[6d9e14f5] [03/21/2020 13:24:59.453] Prompt being set to: ciscoasa>  
[6d9e14f5] [03/21/2020 13:24:59.453] Logged into device 
[6d9e14f5] [03/21/2020 13:24:59.453] Force Disconnect 
[6d9e14f5] [03/21/2020 13:24:59.453] StateChange: Disconnecting 
[6d9e14f5] [03/21/2020 13:24:59.453] StateChange: Disconnected 
[6d9e14f5] [03/21/2020 13:24:59.453] TimerTick: Killing session timer. 
[6d9e14f5] [03/21/2020 13:24:59.453] Disconnected - From: 192.168.1.1 
[6d9e14f5] [03/21/2020 13:25:55.427] Force Disconnect

Cisco Nexus failing with TimeOut

[e0c2cfaa] [] Pre-Commands: Waiting more than 3 seconds for response start sending pre-command if any... 
[e0c2cfaa] [] TimerTick: send CRLF to get the prompt 
[e0c2cfaa] [ <-- 
 
[e0c2cfaa] [6] TimerTick: Login Timeout 
[e0c2cfaa] [03] ReceiveData Error:WeOnlyDo.Exceptions.SSH.TimeoutException: Timeout occurred due to inactivity.
   at A.B.A(Int32 , Boolean )
   at WeOnlyDo.Client.SSH.ReceiveData(Int32 BytesCount)
   at SolarWinds.Net.SWTelnet.Session.ReceiveData(Int32 bytesCount, Boolean& isError) 
[e0c2cfaa] [] StateChange: Disconnecting 
[e0c2cfaa] [] StateChange: Disconnected

Please check your Cisco Nexus Firmware and make sure this is upgraded earlier than 2016. Only then the Nexus devices will work with CLI polling.

If you need any assistance or already have a SolarWinds support ticket, please provide these logs to the Support Engineer in ZIP format.

Disclaimer: Please note, any content posted herein is provided as a suggestion or recommendation to you for your internal use. This is not part of the SolarWinds software or documentation that you purchased from SolarWinds, and the information set forth herein may come from third parties. Your organization should internally review and assess to what extent, if any, such custom scripts or recommendations will be incorporated into your environment. You elect to use third-party content at your own risk, and you will be solely responsible for the incorporation of the same if any.