Network Management
Ephemeral Port Exhaustion On SolarWinds Platform Servers
Port exhaustion will cause issues with Solarwinds when the server has run out of available ports. Incorrect credentials on WMI polling, DCOM errors causing a rapid usage of available ports or WMI based SAM components being applied to none WMI devices are some of the reasons your system can experience the said issue.
First published date
Last published date
Overview
This article overviews possible solutions to port exhaustion being present on your SolarWinds Platform Server. Some of these solutions are: fixing Improper WMI credentials, WMI templates in SAM applied to none WMI nodes, or windows servers running out of available port connections, DCOM errors, and SAM Templates/components with incorrect credentials.
Symptoms of this issue can include but are not limited to:
- Slow SolarWinds Platform web console requiring a reboot to resume normal polling for a short time
- SolarWinds Platform server requiring reboots on a daily or weekly basis
- General SolarWinds Platform web console (website) issues such as resource errors and pages timing out
- SolarWinds Platform services stopping/starting intermittently
- High CPU/memory usage
- Intermittent "Service could not open a connection to the database" errors
- Multiple DCOM errors across several nodes
Product section
Cause
Port exhaustion occurs when all available ephemeral ports on a machine are in use, preventing new connections from being established (outbound communication).
For Windows Server 2008 and later, the default ephemeral port range is 49152-65535.
To check the current ephemeral port range on your machine, use the following command:
netsh int ipv4 show dynamicport tcp
You should see a Start Port of 49152 and a Number of Ports set to 16384.
To determine how many ephemeral ports are currently in use, you can run the following PowerShell script:
(Get-NetTCPConnection | Where-Object { $_.LocalPort -ge 49152 -and $_.LocalPort -le 65535 } | Group-Object -Property LocalPort).Count
Feel free to adjust the LocalPort range in the script according to your environment.
The output of these commands will show you the current port utilization. If you are experiencing port exhaustion, you will notice that nearly all ephemeral (dynamic) ports are in use. In this example, you would be approaching port exhaustion if you were using around 15,000-16,000 ports.
To extend the dynamic port range, use the following command:
netsh int ipv4 set dynamicport tcp start=40000 num=25536
Feel free to adjust the port range according to your needs.
For more information, refer to the Microsoft documentation on TCP/IP port exhaustion troubleshooting - Windows Client | Microsoft Learn.
You may also see the following Errors in SolarWinds Logs:
Note: Error found in Collector Logs: (Located: C:\ProgramData\Solarwinds\Collector\Logs\Collector.BusinessLayer.log)
FATAL SolarWinds.Orion.Core.Common.RemoteFeatureManager - Unable to load Feature Manager data from SWIS System.InsufficientMemoryException: Insufficient winsock resources available to complete socket connection initiation. ---> System.Net.Sockets.SocketException: An operation on a socket could not be performed because the system lacked sufficient buffer space or because a queue was full 10.95.8.107:17777
The following error message can also appear in Configuration Wizard when running into port exhaustion issues:
Connection Timeout Expired. The timeout period elapsed while attempting to consume the pre-login handshake acknowledgment. This could be because the pre-login handshake failed or the server was unable to respond back in time. The duration spent while attempting to connect to this server was - [Pre-Login] initialization=21032; handshake=0;
Resolution
Below is a list of possible solutions to help you with resolving your Port exhaustion issues. First, we will start with general port exhaustion; at the bottom will be SAM-specific instructions. If you do not have the SAM module installed, then you can skip these steps.
-
Windows Available TCP ports
- Before Windows Server 2019 and Windows 10, Windows followed a fundamental guideline for port availability only, making around 4,000 ports available for dynamic port availability. A Solarwinds polling engine would very quickly overload on this legacy setting. Around December 2018, Microsoft updated Windows to use a higher number of active ports, which has helped to mitigate Solarwinds simply overloading the available dynamic port range on the windows server; however, a full polling engine (around 10,000 elements) will still max out this new range. You can get more information from this Microsoft documentation:
- For further information on how you can resolve this issue by increasing the number of available ports for windows to use please see the following article
-
Upgrade to SolarWinds Platform 2020.2.4 or newer
- Over the years, there have been several product updates to address bugs that caused port exhaustion issues in Solarwinds Platform modules. Please make sure to update your system to version 2020.2.4 or newer to resolve any known bugs with your SolarWinds Platform products that can cause port exhaustion. Also, please see the following article for details on how to upgrade your SolarWinds Platform Deployment:
-
DCOM Errors
- A few different things can cause DCOM errors; for the scope of this article, we talk specifically only about DCOM errors related to SolarWinds Deployments.
- Within SolarWinds, WMI credentials are the most common issue which causes DCOM errors on your SolarWinds servers.
- You may want to use the following Powershell code to assist with getting a list of IPs that are causing DCOM errors on your system:
-
<# *** SolarWinds Technical Support DCOM Error Processor *** -- Scripts are not supported under any SolarWinds support program or service. -- Scripts are provided AS IS without warranty of any kind. SolarWinds further -- disclaims all warranties including, without limitation, any implied warranties -- of merchantability or of fitness for a particular purpose. The risk arising -- out of the use or performance of the scripts and documentation stays with you. -- In no event shall SolarWinds or anyone else involved in the creation, -- production, or delivery of the scripts be liable for any damages whatsoever -- (including, without limitation, damages for loss of business profits, business -- interruption, loss of business information, or other pecuniary loss) arising -- out of the use of or inability to use the scripts or documentation. This script will scan the current computer's system event logs and output to the current users desktop a txt file with a list of all DCOM errors sorted by IP address. #> $date = Get-date -Format "MM-dd_hh_mm" Clear-Host Write-Host "`r`nGathering DCOM ID:10028 errors from your system" $DCOM = Get-winevent system | Where-Object {($_.ProviderName -like "Microsoft-Windows-DistributedCOM") -and ($_.ID -like "10028")} Write-Host "`r`nProcessing DCOM messages" $DCOMHash = New-Object "System.Collections.Generic.List[System.Object]" $DCOM | ForEach-Object -Process{ $DCOMfr =$_.Message.IndexOf("computer",1) $DCOMto =$_.message.IndexOf("using") $DCOMip = "" $DCOMip= $_.message.substring($DCOMfr+9,$DCOMto-$DCOMfr-9) $DCOMhash += $DCOMip -replace '\.$' } Out-File -filepath ([Environment]::GetFolderPath("Desktop")+"\DCOMErrors_$date.txt") "Total DCOM Error Count: " + $DCOMhash.count | Out-File -FilePath ([Environment]::GetFolderPath("Desktop")+"\DCOMErrors_$date.txt") -Append $DCOMhash | Group-Object | Sort-Object count -Descending | format-Table -Property Name,count |Out-File -FilePath ([Environment]::GetFolderPath("Desktop")+"\DCOMErrors_$date.txt") -Append Write-Host "Thanks for using this script to gather the IPs with DCOM errors assosiated with them `r`n There is now a txt file on your desktop with the output, Opening txt file in" Start-Sleep -Seconds 3 Write-Host "... 5 ..." Start-Sleep -Seconds 1 Write-Host "... 4 ..." Start-Sleep -Seconds 1 Write-Host "... 3 ..." Start-Sleep -Seconds 1 Write-Host "... 2 ..." Start-Sleep -Seconds 1 Write-Host "... 1 ..." Start-Sleep -Seconds 1 Start-Process notepad ([Environment]::GetFolderPath("Desktop")+"\DCOMErrors_$date.txt")
-
- Once you have identified which devices are causing your DCOM errors you will want to review them and make sure that they have valid WMI credentials for polling.
- If you have the Server and Application Monitor (SAM) module installed you will also want to do the following steps:
- You will want to make sure that your
components and Application monitors have proper Credentials - You will want to make sure that you only have WMI based
components or monitors applied to windows devices.
- You will want to make sure that your
Completing the above steps should resolve your port exhaustion issue. In summary, the three items are, verify that all your WMI-based polling is working correctly and has valid permissions, confirm you are running SolarWinds 2019.4 or newer, and increase your windows registry settings to allow for more port usage.