Network Management
Agent Nodes Down After Additional Polling Engine Reinstall Due to PollingEngineId Mismatch in the SolarWinds Platform
After uninstalling and reinstalling an Additional Polling Engine (APE), agent-polled nodes on that APE show as Down because the AgentManagement_Agents table retains the old EngineID that no longer exists in the Engines table.
First published date
Last published date
Overview
Error:
In Manage Agents (Settings > Manage Agents), agents on the reinstalled APE show ConnectionStatus: Unknown and AgentStatus: Unknown.
In the AgentManagement_Agents table, the affected agents reference a PollingEngineId that does not exist in the Engines table.
In the NodesData table, the same nodes have their EngineID correctly updated to the new APE EngineID.
Symptoms:
-
After a fresh uninstall and reinstall of an Additional Polling Engine, all agent-polled nodes assigned to that APE show as Down.
-
ICMP and SNMP polling on the same APE works correctly.
-
List Resources and Poll Now may work from the web console (if DNS/port 17777 communication between Main and APE is healthy).
-
The Manage Agents page shows all agents on the reinstalled APE with ConnectionStatus and AgentStatus as Unknown.
-
The reinstalled APE shows 0 agents registered to it even though nodes are assigned to it.
Product section
Cause
When an Additional Polling Engine is uninstalled and reinstalled, it receives a new EngineID in the Engines table. The node assignments in the Nodes/NodesData table are updated to point to the new EngineID, but the agent registrations in the AgentManagement_Agents table are not updated. They continue to reference the old EngineID, which no longer exists after the reinstall.
Because the current APE (new EngineID) has 0 agents registered to it, it cannot manage or send polling jobs to any of the agent-polled nodes, causing all agent nodes to show as Down while ICMP and SNMP polling continues to work normally.
Resolution
Resolution:
-
Run the following query against the SolarWinds database to identify the mismatch. Compare the PollingEngineId values in the AgentManagement_Agents table against the Engines table:
-- Scripts are not supported under any SolarWinds support program or service.
-- Scripts are provided AS IS without warranty of any kind. SolarWinds further
-- disclaims all warranties including, without limitation, any implied warranties
-- of merchantability or of fitness for a particular purpose. The risk arising
-- out of the use or performance of the scripts and documentation stays with you.
-- In no event shall SolarWinds or anyone else involved in the creation,
-- production, or delivery of the scripts be liable for any damages whatsoever
-- (including, without limitation, damages for loss of business profits, business
-- interruption, loss of business information, or other pecuniary loss) arising
-- out of the use of or inability to use the scripts or documentation.
SELECT
A.AgentId,
A.Hostname,
A.NodeId,
A.PollingEngineId AS Agent_EngineId,
N.EngineID AS Node_EngineId,
E_Agent.ServerName AS Agent_EngineName,
E_Node.ServerName AS Node_EngineName
FROM AgentManagement_Agents A
LEFT JOIN NodesData N ON A.NodeId = N.NodeID
LEFT JOIN Engines E_Agent ON A.PollingEngineId = E_Agent.EngineID
LEFT JOIN Engines E_Node ON N.EngineID = E_Node.EngineID
WHERE A.PollingEngineId <> N.EngineID
If the query returns results where Agent_EngineName is NULL (the old EngineID no longer exists) and Node_EngineName shows the current APE, this confirms the mismatch.
-
Identify the old (non-existent) EngineID and the new (current) EngineID. Run the following query to see which EngineIDs exist in the Engines table:
-- Scripts are not supported under any SolarWinds support program or service.
-- Scripts are provided AS IS without warranty of any kind. SolarWinds further
-- disclaims all warranties including, without limitation, any implied warranties
-- of merchantability or of fitness for a particular purpose. The risk arising
-- out of the use or performance of the scripts and documentation stays with you.
-- In no event shall SolarWinds or anyone else involved in the creation,
-- production, or delivery of the scripts be liable for any damages whatsoever
-- (including, without limitation, damages for loss of business profits, business
-- interruption, loss of business information, or other pecuniary loss) arising
-- out of the use of or inability to use the scripts or documentation.
SELECT EngineID, ServerName, IP, ServerType
FROM Engines
ORDER BY EngineID
-
Run the following query to confirm how many agents are affected:
-- Scripts are not supported under any SolarWinds support program or service.
-- Scripts are provided AS IS without warranty of any kind. SolarWinds further
-- disclaims all warranties including, without limitation, any implied warranties
-- of merchantability or of fitness for a particular purpose. The risk arising
-- out of the use or performance of the scripts and documentation stays with you.
-- In no event shall SolarWinds or anyone else involved in the creation,
-- production, or delivery of the scripts be liable for any damages whatsoever
-- (including, without limitation, damages for loss of business profits, business
-- interruption, loss of business information, or other pecuniary loss) arising
-- out of the use of or inability to use the scripts or documentation.
SELECT AgentId, Hostname, NodeId, PollingEngineId, ConnectionStatusMessage, AgentStatusMessage
FROM AgentManagement_Agents
WHERE PollingEngineId = <OLD_ENGINE_ID>
Replace <OLD_ENGINE_ID> with the non-existent EngineID from step 1. Note the number of rows returned.
-
Take a full backup of the SolarWinds database before making any changes.
-
Stop SolarWinds services on the reinstalled APE using the SolarWinds Platform Service Manager.
-
Run the following update within a transaction to reassign the agents from the old EngineID to the current EngineID:
-- Scripts are not supported under any SolarWinds support program or service.
-- Scripts are provided AS IS without warranty of any kind. SolarWinds further
-- disclaims all warranties including, without limitation, any implied warranties
-- of merchantability or of fitness for a particular purpose. The risk arising
-- out of the use or performance of the scripts and documentation stays with you.
-- In no event shall SolarWinds or anyone else involved in the creation,
-- production, or delivery of the scripts be liable for any damages whatsoever
-- (including, without limitation, damages for loss of business profits, business
-- interruption, loss of business information, or other pecuniary loss) arising
-- out of the use of or inability to use the scripts or documentation.
BEGIN TRANSACTION
UPDATE AgentManagement_Agents
SET PollingEngineId = <NEW_ENGINE_ID>
WHERE PollingEngineId = <OLD_ENGINE_ID>
Replace <OLD_ENGINE_ID> with the non-existent EngineID and <NEW_ENGINE_ID> with the current APE EngineID from step 2. Verify the number of rows affected matches the count from step 3. If correct, run COMMIT TRANSACTION. If the count does not match, run ROLLBACK TRANSACTION and investigate further.
-
Run the verification query to confirm the update:
-- Scripts are not supported under any SolarWinds support program or service.
-- Scripts are provided AS IS without warranty of any kind. SolarWinds further
-- disclaims all warranties including, without limitation, any implied warranties
-- of merchantability or of fitness for a particular purpose. The risk arising
-- out of the use or performance of the scripts and documentation stays with you.
-- In no event shall SolarWinds or anyone else involved in the creation,
-- production, or delivery of the scripts be liable for any damages whatsoever
-- (including, without limitation, damages for loss of business profits, business
-- interruption, loss of business information, or other pecuniary loss) arising
-- out of the use of or inability to use the scripts or documentation.
SELECT AgentId, Hostname, NodeId, PollingEngineId, ConnectionStatusMessage, AgentStatusMessage
FROM AgentManagement_Agents
WHERE PollingEngineId = <NEW_ENGINE_ID>
-
Start SolarWinds services on the APE using the SolarWinds Platform Service Manager.
-
Navigate to Settings > Manage Agents and verify the agent status for the affected nodes. The agents should begin reconnecting to the APE.
-
Test by running Poll Now and List Resources on one of the agent-polled nodes assigned to the APE.
This update only changes the engine pointer in the agent registration table. Historical polling data is not affected as it is tied to the NodeID, not the AgentID.