Network Management
Excessive UDT-monitored ports cause SolarWinds.UDT.Layer2PollingPlan collector queue database file to grow rapidly
This article explains an issue where the SolarWinds.UDT.Layer2PollingPlan.db collector queue database file grows rapidly due to an excessive number of UDT-monitored ports on a polling engine, and how to resolve it by identifying and removing the offending nodes from UDT.
First published date
Last published date
Overview
SolarWinds User Device Tracker (UDT) uses Layer 2 and Layer 3 topology polling to discover and track devices and users across the network. Polling data is temporarily stored in a local SQLite database file (SolarWinds.UDT.Layer2PollingPlan.db) located at C:\ProgramData\SolarWinds\Collector\Queues on each polling engine before being committed to the SolarWinds Orion database.
When the number of UDT-monitored ports on a polling engine is excessively high, each polling cycle generates a large volume of data that floods the collector queue. If the queue cannot process and commit data fast enough, the .db file grows rapidly and may approach its 2 GB maximum size, potentially causing polling failures and data loss.
Additionally, invalid or stale Active Directory (AD) credentials configured in UDT can prevent the Business Layer from fully processing Layer 2 polling results, further contributing to queue buildup.
Relevant Logs and Files
The following logs and files are relevant to diagnosing this issue. These can be found in the SolarWinds Diagnostics collected from the affected polling engine:
|
File |
Location in Diagnostics |
Description |
|---|---|---|
|
|
|
Contains AD lookup errors, data processing failures, and SQL exceptions |
|
|
|
Shows per-node UDT job execution details including duration and failures |
|
|
|
Contains missing entity errors, stale NodeIDs, and SQL connectivity issues |
|
|
|
Same as above but for Layer 3 polling |
|
|
|
Contains queue commit failures, OutOfMemoryException, and general collector errors |
|
|
Root level of diagnostics |
Contains point-in-time collector queue counters including |
|
|
|
Lists all configured credentials with ID, Description, CredentialType, and CredentialOwner — used to map credential IDs in log errors to actual credential names |
On the server (outside diagnostics):
|
File |
Location |
Description |
|---|---|---|
|
|
|
The collector queue SQLite database file — monitor its size for growth |
|
|
|
Layer 3 collector queue SQLite database file |
Note: To enable verbose logging for deeper investigation, use the SolarWinds Log Adjuster (Start Menu → SolarWinds Platform → Log Adjuster) and set UDT Business Layer and UDT Jobs to Debug. Allow 1–2 polling cycles (~1 hour) for the logs to populate before re-collecting diagnostics. Reference: Adjust logging levels with the SolarWinds Log Adjuster
The following errors may appear in the UDT.BusinessLayer.log or Collector\Logs\Plugins\SolarWinds.UDT.Layer2PollingPlan.log when stale AD credentials are configured:
-
The user name or password is incorrect -
The specified domain does not exist or cannot be contacted
Product section
Cause
The collector queue database file grows rapidly when one or more nodes have an abnormally high number of UDT-monitored ports (e.g., hundreds of thousands), generating excessive polling data per cycle. This commonly occurs when server nodes (such as Linux servers with virtual network interfaces) are inadvertently included in UDT topology polling.
Stale or invalid AD credentials in UDT configuration can compound this issue by preventing the UDT Business Layer from processing polling results, causing data to remain in the queue longer.
Resolution
Resolution 1 – Diagnose the root cause
Step 1 – Identify nodes with the highest number of UDT-monitored ports
Run the following SQL query against the SolarWinds Platform database:
-- Scripts are not supported under any SolarWinds support program or service.
-- Scripts are provided AS IS without warranty of any kind. SolarWinds further
-- disclaims all warranties including, without limitation, any implied warranties
-- of merchantability or of fitness for a particular purpose. The risk arising
-- out of the use or performance of the scripts and documentation stays with you.
-- In no event shall SolarWinds or anyone else involved in the creation,
-- production, or delivery of the scripts be liable for any damages whatsoever
-- (including, without limitation, damages for loss of business profits, business
-- interruption, loss of business information, or other pecuniary loss) arising
-- out of the use of or inability to use the scripts or documentation.
SELECT COUNT(p.PortID) AS PortCount, n.NodeID, n.Caption, n.IP_Address, e.EngineID, e.ServerName
FROM [dbo].[UDT_Port] p
JOIN Nodes n ON p.NodeID = n.NodeID
JOIN Engines e ON n.EngineID = e.EngineID
GROUP BY n.NodeID, n.Caption, n.IP_Address, e.EngineID, e.ServerName
ORDER BY PortCount DESC
Review the results and identify any nodes with an unusually high port count. Server nodes (e.g., Linux servers, virtual hosts) that expose thousands of virtual interfaces should not be included in UDT topology polling.
Step 2 – Check for failing UDT jobs
Run the following SQL query to identify nodes with UDT job failures:
-- Scripts are not supported under any SolarWinds support program or service.
-- Scripts are provided AS IS without warranty of any kind. SolarWinds further
-- disclaims all warranties including, without limitation, any implied warranties
-- of merchantability or of fitness for a particular purpose. The risk arising
-- out of the use or performance of the scripts and documentation stays with you.
-- In no event shall SolarWinds or anyone else involved in the creation,
-- production, or delivery of the scripts be liable for any damages whatsoever
-- (including, without limitation, damages for loss of business profits, business
-- interruption, loss of business information, or other pecuniary loss) arising
-- out of the use of or inability to use the scripts or documentation.
SELECT TOP 20 n.NodeID, n.Caption, n.IP_Address, COUNT(*) AS FailedJobs
FROM [dbo].[UDT_Job] j
JOIN Nodes n ON j.NodeID = n.NodeID
WHERE j.JobLastResult = 0
GROUP BY n.NodeID, n.Caption, n.IP_Address
ORDER BY FailedJobs DESC
Step 3 – Check for stale/missing NodeIDs still referenced by UDT
Run the following SQL query to identify orphaned nodes:
-- Scripts are not supported under any SolarWinds support program or service.
-- Scripts are provided AS IS without warranty of any kind. SolarWinds further
-- disclaims all warranties including, without limitation, any implied warranties
-- of merchantability or of fitness for a particular purpose. The risk arising
-- out of the use or performance of the scripts and documentation stays with you.
-- In no event shall SolarWinds or anyone else involved in the creation,
-- production, or delivery of the scripts be liable for any damages whatsoever
-- (including, without limitation, damages for loss of business profits, business
-- interruption, loss of business information, or other pecuniary loss) arising
-- out of the use of or inability to use the scripts or documentation.
SELECT DISTINCT p.NodeID
FROM [dbo].[UDT_Port] p
LEFT JOIN Nodes n ON p.NodeID = n.NodeID
WHERE n.NodeID IS NULL
If results are returned, UDT is still polling nodes that no longer exist in the SolarWinds Platform database.
Step 4 – Review the UDT.BusinessLayer.log for AD credential errors
Open LogFiles\Orion\UDT.BusinessLayer.log from the diagnostics and search for the following errors:
-
The user name or password is incorrect— indicates an incorrect password on a configured AD credential. -
The specified domain does not exist or cannot be contacted— indicates the domain is unreachable from the polling engine.
The credential ID referenced in the error can be mapped to its description using the Credential.csv file located at DB\Credential.csv in the diagnostics.
Step 5 – Review the PerformanceCounters_merged.csv for queue size
Open PerformanceCounters_merged.csv from the root level of the diagnostics and filter for the following counters:
|
Counter Name |
What It Indicates |
|---|---|
|
|
Current size of the UDT Layer 2 queue |
|
|
Current size of the UDT Layer 3 queue |
|
|
Number of items waiting to be committed |
|
|
How long each commit takes — indicates SQL performance |
Note: These counters are a point-in-time snapshot. The
.dbfile on disk may retain its size even after the queue has been processed, as SQLite does not automatically shrink the file after data is committed.
Resolution 2 – Remove offending nodes from UDT
-
Back up the SolarWinds Platform database before making any changes.
-
Reference: Back up the SolarWinds Platform database
-
-
Navigate to Settings → All Settings → UDT Settings → Manage Ports.
-
Change the Show dropdown from Ports to Nodes.
-
Select the offending node.
-
Click Delete and choose "Delete node and data from UDT only".
Note: This action removes all UDT-related data for the selected node from the following database tables:
UDT_Port,UDT_NodeCapability,UDT_PortToPortCurrent,UDT_PortToEndpointCurrent,UDT_VLAN, andUDT_IPAddressCurrent. The node will remain in NPM and other modules — only UDT data is removed.Note: Deleting individual ports via Port Management only flags the rows (
IsMonitored=0,Flag=1) but does not remove them from theUDT_Porttable. The node-level deletion is required to fully remove the rows from the database. -
Repeat steps 4–5 for any other offending nodes.
Resolution 3 – Fix or remove stale AD credentials
-
Navigate to Settings → All Settings → UDT Settings → Credentials Management.
-
Review all configured AD credentials and verify:
-
Passwords are correct.
-
Domain names are valid and reachable from the polling engine.
-
-
Update any credentials with incorrect passwords, and remove credentials pointing to domains that are no longer in use.
Resolution 4 – Clear the collector queue and validate
-
Stop all SolarWinds services on the affected polling engine.
-
Navigate to
C:\ProgramData\SolarWinds\Collector\Queues. -
Delete the following files:
-
SolarWinds.UDT.Layer2PollingPlan.db -
SolarWinds.UDT.Layer3PollingPlan.db -
Any related
.db-journalfiles
-
-
Start all SolarWinds services.
-
Monitor the
.dbfile size over the next 24–48 hours to confirm it does not grow back.
Resolution 5 – Disable UDT topology polling (optional)
If UDT is no longer being used in the environment, topology polling can be disabled entirely:
-
Navigate to Settings → All Settings → UDT Settings → Polling Settings.
-
Disable Layer 2 topology polling.
-
Disable Layer 3 topology polling.
-
Click Submit to save the changes.
-
Clear the collector queue files as described in Resolution 4.