Network Management

Troubleshooting DPI/QoE Agent Failed to Deploy/Agent Unknown/Agent Memory Leak

The most common issue with QoE/DPI is where the agent fails to deploy or shows unknown on remote system even the credentials are passing. This article provides a few troubleshooting steps to help you during this process. When checking the packet analysis sensors, it's showing sensor status as unknown and connected.

First published date

10/26/2018 3:33 PM

Last published date

12/21/2022 3:03 PM

Overview

The most common issue with QoE/DPI occurs when the agent fails to deploy or shows unknown on remote system even the credentials are passing. This article provides a few troubleshooting steps to help you during this process. When checking the packet analysis sensors, it shows sensor status as unknown and connected.

Product section

Network Performance Monitor

Resolution

Where do i have to start or What logs i have to check for DPI Agent Deployment failed

Check the Agent Logs "Collect new diagnostics"

C:\ProgramData\Solarwinds\Logs\DPI

DPI.BusinessLayer.Log

Which DB Table i need to Check for Agents Status.

For Agent Deployment status, check AgentManagement_Agents Table within the DB.

Check if the Agent is been deployed previously on the target host Or Current status of the agent.
Check if there is any Firewall Recently been applied (Windows).
Check connects to the server on port 17778. This port must be opened on the server firewall so the Agent can connect to it.
Check the Node have dual NIC binding.
Make sure the ip address associated to the Node added under Orion have higher priority binding.
Node must be on 64 bit O/S.

To edit the network card binding order in Windows Server 2008:

Click Start > Control Panel > Network and Sharing Center.

On the left side, select Manage network connections.

Press Alt+N to display the Advanced menu.

Select Advanced Settings. On the Adapters and Bindings tab, highlight your NIC and use the arrows on the right side to adjust it’s binding order.

Troubleshooting an Example Case DPI / QoE Agent Failed to deploy #1

Checked the DPI BusinessLayer Logs.

2014-10-22 12:05:16,199 [65] INFO SolarWinds.DPI.BusinessLayer.Management.AgentPluginDeployment - Credentials test passed on node 11-EscapeAD
2014-10-22 12:05:16,199 [65] INFO SolarWinds.DPI.BusinessLayer.Management.AgentPluginDeployment - Deploying agent on nodeId: 11, assigned to engine 1 with userName Administrator
2014-10-22 12:05:16,231 [65] ERROR SolarWinds.InformationService.Contract2.InfoServiceProxy - Error executing invoke: SafeCall failed, check fault information.
Orion.AgentManagement.Agent.DeployToNode
2014-10-22 12:05:16,231 [65] ERROR SolarWinds.DPI.BusinessLayer.Management.AgentPluginDeployment - Deployment of Agent failed with message: SafeCall failed, check fault information.
System.ServiceModel.FaultException`1[SolarWinds.InformationService.Contract2.InfoServiceFaultContract]: Invoke failed, check fault information. (Fault Detail is equal to SolarWinds.InformationService.Contract2.InfoServiceFaultContract)

WHERE IP = @ipaddress AND IsActiveAgent = True RETURN XML RAW
2014-10-22 12:00:12,148 [65] ERROR SolarWinds.InformationService.Contract2.InfoServiceProxy -      ipaddress=192.168.1.51
2014-10-22 12:00:13,162 [65] ERROR SolarWinds.InformationService.Contract2.InfoServiceProxy - Error closing exception.
System.ServiceModel.CommunicationObjectFaultedException: The communication object, System.ServiceModel.Channels.ServiceChannel, cannot be used for communication because it is in the Faulted state.

Server stack trace:
   at System.ServiceModel.Channels.CommunicationObject.Close(TimeSpan timeout)
   at System.ServiceModel.Channels.CommunicationObject.Close()

Exception rethrown at [0]:
   at System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(IMessage reqMsg, IMessage retMsg)
   at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData& msgData, Int32 type)
   at System.ServiceModel.ICommunicationObject.Close()
   at SolarWinds.InformationService.Contract2.InfoServiceProxy.Close()
2014-10-22 12:00:14,192 [65] ERROR SolarWinds.InformationService.Contract2.InfoServiceProxy - Error closing exception.
System.ServiceModel.CommunicationObjectFaultedException: The communication object, System.ServiceModel.Channels.ServiceChannel, cannot be used for communication because it is in the Faulted state.

Server stack trace:
   at System.ServiceModel.Channels.CommunicationObject.Close(TimeSpan timeout)
2014-10-22 12:00:14,270 [65] ERROR SolarWinds.DPI.BusinessLayer.Management.AgentPluginDeployment - Deployment of Agent failed with message: SafeCall failed, check fault information.
System.ServiceModel.FaultException`1[SolarWinds.InformationService.Contract2.InfoServiceFaultContract]: Invoke failed, check fault information. (Fault Detail is equal to SolarWinds.InformationService.Contract2.InfoServiceFaultContract).
2014-10-22 12:04:26,135 [68] INFO SolarWinds.DPI.BusinessLayer.DPIBusinessLayerService - DeployRemoteProbe - nodeId:11
2014-10-22 12:04:26,213 [68] INFO SolarWinds.DPI.BusinessLayer.Management.AgentPluginDeployment - Credentials test passed on node EscapeAd.Escape
2014-10-22 12:04:26,213 [68] INFO SolarWinds.DPI.BusinessLayer.Management.AgentPluginDeployment - Deploying agent on nodeId: 11, assigned to engine 1 with userName Administrator
2014-10-22 12:04:26,229 [68] ERROR SolarWinds.InformationService.Contract2.InfoServiceProxy - Error executing invoke: SafeCall failed, check fault information.
Orion.AgentManagement.Agent.DeployToNode

Start Your Troubleshooting

Note the IP Address and the node ID from above error.
Open AgentManagement_Agents table under verify agent was previously deployed for the same node.
Verify the IP Address under the logs and node must be same.
Remove the entry from the table IF it's there.
Find the IP address under the node is showing 192.168.1.50. However, the above Logs showing 192.168.1.51.
Verify that the nodes table IP address is been added for this node. This confirms the node must have more than one NIC assigned.
Jump on the target server.
Remove/uninstall the previously installed agent.
Find assigned two NICs to the server and the above error showing 192.168.1.51 as the NIC have priority binding.
Change the binding to the IP address assigned to the node in Orion.

Resolution:

This confirms the node has been added with 192.168.1.50. However, the other NIC 192.168.1.51 has higher priority binding. Therefore, when agent is been deployed the response is been received from 192.168.1.51.

✔ Re-deploy the agent resolves the issue. This time, there is no error.

Troubleshooting an Example Case DPI / QoE Agent Failed to deploy #2

Deploy agent failed. Error in deploying Packet Analysis Sensor. Deployment of Agent failed with message: StateCall failed check fault information.

When checked the Error showing License Errors. (However 8 Server Licenses Available)

2014-11-15 10:10:23,120 [34] INFO SolarWinds.DPI.BusinessLayer.DPIBusinessLayerService - DeployRemoteProbe - nodeId:9
2014-11-15 10:10:23,216 [34] INFO SolarWinds.DPI.BusinessLayer.DPIBusinessLayerPlugin - Credentials test passed on node 9-DC2008
2014-11-15 10:10:23,217 [34] INFO SolarWinds.DPI.BusinessLayer.DPIBusinessLayerPlugin - Deploying agent on nodeId: 9, assigned to engine 1 with userName webpalm\Administrator
2014-11-15 10:10:23,543 [34] INFO SolarWinds.DPI.BusinessLayer.Management.AgentSwisNotificationSubscriber - Registering for changes on AgentId : 9
2014-11-15 10:10:23,544 [34] INFO SolarWinds.DPI.BusinessLayer.DPIBusinessLayerService - Probe assigned for AgendId 9 hasn't been found, creating new one
2014-11-15 10:10:23,586 [34] INFO SolarWinds.DPI.BusinessLayer.DPIBusinessLayerService - new Probe inserted into DB: DpiProbe {
ID:8
EngineID:1
Name:DC2008
Description:
Mode:LocalTraffic
Status:NotPresent
AgentID:9
Endpoint:00000000-0000-0000-0000-000000000000
Enabled:Truen Version:n NodeID:0
}
2014-11-15 10:10:23,615 [14] INFO SolarWinds.DPI.BusinessLayer.Management.AgentSwisNotificationSubscriber - Agent: 9 is reporting status: Unknown, connection: DeploymentInProgress, message: Unknown
2014-11-15 10:10:56,348 [70] ERROR SolarWinds.Orion.Core.Common.LicenseBasedProductInfoProvider - Unable to load license for DPI v1.0 (QoE:1). System.ArgumentNullException: Value cannot be null.
Parameter name: licenses

2014-11-15 12:38:50,464 [27] INFO SolarWinds.DPI.BusinessLayer.DPIBusinessLayerService - DeployRemoteProbe - nodeId:24
2014-11-15 12:38:50,582 [27] INFO SolarWinds.DPI.BusinessLayer.DPIBusinessLayerPlugin - Credentials test passed on node 24-DC2008
2014-11-15 12:38:50,582 [27] INFO SolarWinds.DPI.BusinessLayer.DPIBusinessLayerPlugin - Deploying agent on nodeId: 24, assigned to engine 1 with userName webpalm\Administrator
2014-11-15 12:38:50,783 [27] ERROR SolarWinds.InformationService.Contract2.InfoServiceProxy - Error executing invoke: SafeCall failed, check fault information.
Orion.AgentManagement.Agent.DeployToNode
2014-11-15 12:38:50,783 [27] ERROR SolarWinds.DPI.BusinessLayer.DPIBusinessLayerPlugin - Deployment of Agent failed with message: SafeCall failed, check fault information.
System.ServiceModel.FaultException`1[SolarWinds.InformationService.Contract2.InfoServiceFaultContract]: Invoke failed, check fault information. (Fault Detail is equal to SolarWinds.InformationService.Contract2.InfoServiceFaultContract).
2014-11-15 12:39:21,390 [9] ERROR SolarWinds.Orion.Core.Common.LicenseBasedProductInfoProvider - Unable to load license for DPI v1.0 (QoE:1). System.ArgumentNullException: Value cannot be null.
Parameter name: licenses
at SolarWinds.Orion.Core.Common.LicensedProductInfo.UpdateLicenses(IProductLicense[] licenses)
at SolarWinds.Orion.Core.Common.LicenseBasedProductInfoProvider.CreateProduct(ILicensedProduct product)
2014-11-15 12:39:21,400 [9] ERROR SolarWinds.Orion.Core.Common.LicenseBasedProductInfoProvider - Unable to load license for SEUM v2.0.1 (WPM:2). System.ArgumentNullException: Value cannot be null.
Parameter name: licenses
at SolarWinds.Orion.Core.Common.LicensedProductInfo.UpdateLicenses(IProductLicense[] licenses)
at SolarWinds.Orion.Core.Common.LicenseBasedProductInfoProvider.CreateProduct(ILicensedProduct product)

When checked the database table found the agent installed and removed however still listed under the DB therefor causing the license issue.

Manually removed the old entries from the table .

This time, deploy without encountering any error. Agent is up and running.

Resolution

Always check the error and DB table.

Remove old installation match from the Web Console.

Check the IP address of the node under the errors and in the nodes table.

Troubleshooting an example aase DPI/QoE agent failed to deploy #3

There seems to be a bug in older versions of .NET Framework 4 that may cause DPI BusinessLayer to crash. In my case, all other modules were working fine, but the DPI BusinessLayer failed to start because it couldn’t communicate with AgentManagement service. You will see many similar errors in Logs\AgentManagement\AgentManagement.Service.log:

Could not load file or assembly 'System, Version=2.0.5.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e, Retargetable=Yes' or one of its dependencies. The given assembly name or codebase was invalid. (Exception from HRESULT: 0x80131047)

Exception Details: System.IO.FileLoadException: Could not load file or assembly 'System, Version=2.0.5.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e, Retargetable=Yes' or one of its dependencies. The given assembly name or codebase was invalid. (Exception from HRESULT: 0x80131047)

Resolution

The solution is to install the following update for .NET Framework 4 and rebooting the server: http://support.microsoft.com/kb/2468871

This can happen with a fresh Windows installation where not all Windows Updates are installed yet or where Windows Updates are disabled for some reason.

Troubleshooting an Example Case DPI / QoE Agent taking High Memory #4

733728 Memory leak Solarwinds Agent Service.exe [Consuming high memory ]
Known issue when the Agent is not deployed to any server and not configured .

http://fogbugz.swdev.local/default.asp?324600

http://fogbugz.swdev.local/default.asp?393115

Troubleshooting an Example Case DPI / QoE Agent Failed to deploy #5

QOE sensor fails to install. QOE uses Pcap drive and if Wireshark is installed on the machine it could be an old version of that driver is interfering with the install for DPI.

Resolution

1. Uninstall Wireshark and PCAP driver.

2. Reboot the Windows server after installing.

3. Uninstall Agent and QOE sensor from the Web Console.

4. Download Agent installer and run this on the Windows server.

5. Deploy QOE sensor.

Disclaimer: Please note, any content posted herein is provided as a suggestion or recommendation to you for your internal use. This is not part of the SolarWinds software or documentation that you purchased from SolarWinds, and the information set forth herein may come from third parties. Your organization should internally review and assess to what extent, if any, such custom scripts or recommendations will be incorporated into your environment. You elect to use third-party content at your own risk, and you will be solely responsible for the incorporation of the same if any.