Submit a ticketCall us

Webinar: Web Help Desk for HR, Facilities and Accounting Departments
This webinar will focus on use cases for HR, Facilities and Accounting.

Having a unified ticketing and asset management system for all the departments in your company can provide end-users with a seamless experience and make things easier for your IT team. Yet, with different business tasks and objectives, many departments don’t fully understand the capabilities of Web Help Desk and how the software can be customized for effective use in their departments.
Register Now.

Home > Success Center > Network Performance Monitor (NPM) > NPM v12 - Poller stops work due TCP Port Exhaustion - Orion Agent Polling Netpath led to Error: Insufficient winsock resources available to complete socket connection initiation

NPM v12 - Poller stops work due TCP Port Exhaustion - Orion Agent Polling Netpath led to Error: Insufficient winsock resources available to complete socket connection initiation

Created by Kevin Twomey, last modified by Harry Griffiths on Jun 08, 2017

Views: 722 Votes: 6 Revisions: 14

Overview

This article provides brief information and steps to resolve the following error: 

ErrorCode: NetPath_60011
Message: Store traceroutePollerResult failed.
Exception: System.InsufficientMemoryException:
Insufficient winsock resources available to complete socket connection initiation.

---> A socket operation could not be performed because the system lacked sufficient buffer space or because a queue was full 100.100.100.111:17777

 

  • Increased amount of Eventlog entries on the Orion Node. 
  • The system has 4 or 5 NetPath probes active, including the default Google.com.
  • It appears that NetPath runs into timeouts on every Winsock port that points to where Orion cannot poll anymore. 
  • Netpath Requirements not met

Environment

  • NPM v12
  • Orion Agent to poll NetPath

 

Cause 

Agent Management is using persistent HTTPS connections to passive agents which may look suspicious to some traffic inspection firewalls because the behavior of this connection does not look like valid HTTPS communication.

  • If something (firewall, load balancer, ...) aborts such connection Agent Management can't properly terminate it and needs to open a new one.
  • If these connection abortions happen frequently there can be growing number of dead hanging connections.

 

 

If a firewall is causing issues releasing ports, switching the agent from Passive to Active should provide a workaround.

This problem might occurs due to some firewalls or load balancers that prevents connection from agent being closed properly. 
 

Agent is doing its best to close connection but sometimes connection is kept pending due to network infrastructure.

  • This results in high CPU, Network Connectivity issues (SQL, Web site) after upgrading to NPM 12. 
  • This is caused by TCP/IP port exhaustion. 
  • Agent Management Service will have thousands of TIME_WAIT ports for 17791 or 17778 open in NETSTAT -BAO. 
  • The problem is caused by Passive Agents traversing traffic shaping firewalls.  
  • Changing the Agent to active will resolve the issue 
  • Once remove or edit Agent to Active, then should no longer seeing the TIME_WAIT ports open for 17790.
  • Netpath port requirements here not being met

Resolution


1. User Disabled NetPaths and issue went away.
Not Ideal as Netpath maybe required to be enabled, so if still issue proceed to Step3.
 

2. Configure Firewall Exceptions.
Setup exceptions for connections originating from Orion server and connecting to port 17791 on agents.
Firewalls should not touch these connections at all.

3. Edit from Server to Agent initiated communication.

  • To make this change
    • Open Orion Web > Settings > Manage Agents > Edit each Agent
    • Edit communication type from Server (passive) to Agent (Active) initiated communication.


4
Orion Platform v2016.1 Hot Fix 4
* SolarWinds Agents deployed in server-initiated mode can cause port exhaustion on the poller.

 

5

Check SolarWinds.Net windows event logs (Note, in NPM 12.1, this is Netpath events are redirected to a new log called SWI)

 

Search our Success Center for the related errors, Examples below.

 

Large_volume_of_WHOIS_requests_generated_by_NetPath

 

/Troubleshoot_Missing_BGP_Whois_Contact_Information



5 If issue persists, contact support.


 


How has the agent been set up to communicate with the Orion server?

Server initiated communication » Learn More

This Agent waits for requests from the server on a specified port.
This port must be opened on the Agent machine firewall so the server can connect to it. No change on the server firewall is required.

 

Server-initiated communication.

  • All communication between your SolarWinds Orion server or additional polling engine and the agent is initiated by the server, and the agent does not initiate communication to your SolarWinds Orion server.
  • You must have a direct route from the server with the agent installed to your SolarWinds Orion server or additional polling engine.
  • To use this communication method, port 17790 must be open on the firewall of the remote host to retrieve information from the agent.

This communication method is also known as a passive agent.
 



Agent-initiated communication » Learn More    

This Agent initiates communication with server. Agent connects to the server on TCP port 17778.
This port must be opened on the server firewall so the Agent can connect to it. No change on the Agent firewall is required.

 

Agent-initiated communication

  • All communication between your Orion server or additional poller and the agent is initiated by the agent, and your SolarWinds Orion server does not initiate communication with the agent.
  • You do not need to have a direct route from the server with the agent installed to your SolarWinds Orion serveror additional poller.
  • To use this communication method, port 17778 and 17791 must be open on theSolarWinds Orion server firewall to receive information from the agent.
  • This communication method is most useful when the agent is installed on a network separated from your Orion server by one or more NAT devices, and you have no easy way to connect the two.
    This communication method is also known as an active agent.

 

 

Last modified
03:39, 8 Jun 2017

Tags

Classifications

Public