Network Management

Orion Server UDP port exhaustion when monitoring WMI windows mount point

This article describes an unusual issue where the Orion polling engine has port exhaustion on UDP Ports. Frequent reboots of the server may be required. Checking in windows event logs you can see that there are WMI query errors that relate to polling windows mount points.

First published date

6/3/2019 8:03 PM

Last published date

6/3/2019 8:03 PM

Overview

The Orion server can experience port exhaustion when monitoring a very large number of WMI windows mount points on a WMI server(s).

Upon investigation, you can see in Windows Logs -> Applications and Services Logs -> Microsoft -> Windows -> WMI-Activity -> Operational, following error:

Id =

{4FBA8EAE-DC81-0018-3A55-BC4F81DCD301}

; ClientMachine = HOSTNAME; User = DOMAIN\USER; ClientProcessId = 43608; Component = Unknown; 
Operation = Start IWbemServices::ExecQuery - root\CIMV2 : associators of

{Win32_PnPEntity.DeviceID='DIGITS\1&7F6B424&0&36304376463030314242423532374534364235353431303030'}

where resultclass = Win32_SCSIController; ResultCode = 0x80041032; 
PossibleCause = Unknown

Used by MountPointDetailsPoller (V.Details.WMI.MountPoint) and NativeVolumeDetailsPoller (V.Details.WMI.MountPoint)

Some related errors from Core.Collector.Jobs logs (C:\ProgramData\SolarWinds\Logs\Orion\Core.Collector.Jobs_[ID])

ERROR SolarWinds.Orion.Pollers.Framework.PollerBase - (null) WMI Poller V.Status.WMI.MountPoint on IPAddress failed: Timed out Scope: IPAddress\root\CIMV2[\DOMAIN\USER] ErrorCode: 0x40004
2018-08-08 04:52:32,758 STP SmartThreadPool Thread #49 ERROR SolarWinds.Orion.Pollers.Framework.PollerBase - (null) Cannot poll mount point performance.
SolarWinds.Net.WMI.WmiResponseException: Scope: IPAddress\root\CIMV2[\DOMAIN\USER] ErrorCode: 0x80070102 ---> System.Runtime.InteropServices.COMException
at System.Runtime.InteropServices.Marshal.ThrowExceptionForHRInternal(Int32 errorCode, IntPtr errorInfo)
at System.Management.ManagementObjectCollection.ManagementObjectEnumerator.MoveNext()
at SolarWinds.Net.WMI.WmiDataReader.Read()
— End of inner exception stack trace —
at SolarWinds.Net.WMI.WmiDataReader.Read()
at SolarWinds.Orion.Core.Pollers.Volumes.WMI.LogicalDiskPerformanceDataHelper.ReadLogicalDiskPerfResult(IDataReader reader, VolumeWmiStatisticsResult result)
at SolarWinds.Orion.Core.Pollers.Volumes.WMI.MountPointStatisticsPoller.PollPerformanceData(WmiVolumePollerSetting pollerSettings, VolumeWmiStatisticsResult result)
at SolarWinds.Orion.Core.Pollers.Volumes.WMI.MountPointStatisticsPoller.DoPoll(WmiVolumePollerSetting pollerSettings)

Product section

Orion Platform

Cause

WMI query times out using up all the available the ports.

Resolution

  • Try instead using the SolarWinds agent when polling many windows mount points on a windows server.
  • If this node has to be monitored via WMI (For Example, if it is Hyper V and monitored using VMAN)
You will need to make a global change on the polling engine level to work around this issue.

On polling engine that polls this node:

  1. Backup SolarWinds.Orion.Core.Pollers.dll.config
  2. Edit and add the following line to increase all WMI request timeouts to 7 minutes:

<add key="WmiQueryTimeout" value="7:00" />

  1. Backup SolarWinds.Orion.Core.Collector.dll.config (to the desktop, outside of the SW directory),
  2. Edit and modify line with WmiVolumeStatisticsJobTimeout to increase all WMI statistics job timeouts to 7 minutes:

<add key="WmiVolumeStatisticsJobTimeout" value="00:07:00" />

  1. Shutdown all Orion services.
  2. Replace SDF on collector:
  1. Browse to: C:\ProgramData\SolarWinds\Collector\Data
  2. Delete PollingController.sdf
  3. Copy and rename the blank to
  4. PollingController.sdf
  5. Right Click and properties
  6. Uncheck read only.
  1. Replace SDF on job engine v2:
  1. Browse to: C:\ProgramData\SolarWinds\JobEngine.v2\Data
  2. Delete JobEngine35.sdf
  3. Copy and rename the blank to
  4. JobEngine35.sdf
  5. Right Click and properties
  6. Uncheck read only.
  1. Start all Orion services.