Resolve SEM agent issues related to InternalUnknownAgent and InternalDuplicateConnection

11/29/2018 10:34 PM

11/29/2018 10:34 PM

You are receiving InternalUnknownAgent or  InternalDuplicateConnection events under the Security Event Manager (formerly Log & Event Manager) Internal Events filter. Some or all of these agents will either not appear in the Nodes list, or appear as offline in the Nodes list in the SEM console (Manage > Nodes).

Security Event Manager

Agents appear as InternalDuplicateConnection because of two possibilities:

1. The agent goes offline and the SEM manager does not know. This can happen because of loss of network connectivity or port closure on the agent computer.
2. The agent computer was created by imaging. Imaged computers (with a new hostname) can be using the same certificate issued to the original image. These new imaged computers need the agent certificate cleared and the agent re-started.

Agents appear as InternalUnknownAgent because of two possibilities:

1. Agent was removed from the SEM console (Manage > Nodes), but agent software was never un-installed from the Windows/Linux/Unix computer itself, and it is still trying to connect.
2. Agent was installed without the proper permissions to connect to SEM manager. Be sure agents are installed with the "runas administrator", or because of increased security restrictions on Windows 8.1, 10, 2012-R2, you may need to include "win-7compatibility" mode, or even use the Local Agent Installer instead of the Remote Installer. Windows 2016 agent install currently requires the Remote Agent Installer before the agent will connect to the SEM. Further security restrictions either through GPO's or network firewalls can prevent any agent connection.

Generate InternalDuplicateConnection and InternalUnknownAgent reports

• If the SEM reports application has been installed, run the Agent Maintenance report for the last day or two, and then export it in PDF or Excel format.
• If the reports application has not been installed, the same information can be generated from the SEM Web console.

Generate a list of Duplicate agents from the SEM Web console

• Use the SEM Web (or Adobe Air) console to search for InternalDuplicateConnection.
  1. On the SEM console menu bar, navigate to Explore > nDepth.
  2. Expand Events in the middle/left column, and then search for  InternalDuplicateConnection.
  3. Drag the retrieved event type to the search bar at the top, select the timeframe for 1 day, and then start the search.
  4. Expand the Refine Fields section in the top-left corner, and then expand the DetectionIP section to view the hostnames of affected machines.
  5. If needed, export the results in CSV format from the gear on the far right.

Generate a list of InternalUnknownAgent from GUI/Web-console

• Use the SEM Web (or Adobe Air) Console to search for InternalUnknownAgent.
  1. On the SEM console menu bar, navigate to Explore > nDepth.
  2. Expand Events in the middle/left column, and then search for InternalUnknownAgent.
  3. Drag the retrieved event type to the search bar at the top, select the timeframe for 1 day, and then start the search.
  4. Expand the Refine Fields section in the top-left corner, and then expand the DetectionIP section to view the hostnames of affected machines.
  5. If needed, export the results in CSV format from the gear on the far right

Clear the security certificates for agents showing InternalDuplicateConnection

1. Use the CSV document (spreadsheet list)  showing InternalDuplicateConnection and prepare to clear the certificate and restart the agent.
2. Connect with each agent individually through Remote Desktop, or by using "Windows Explorer" to delete files/folders and "Services.msc" or "SC.exe" command to restart the agent service.
3. To clear the certificate, delete the 6 files (3- *.xml and 3- *.trigeo) from one of the following directories:
   C:\Windows\SysWOW64\ContegoSPOP\spop\ (64-bit)
   C:\Windows\system32\ContegoSPOP\spop\ (32-bit)
4. Open the Windows Services menu (services.msc), scroll down the services list to the SolarWinds Security Event Manager Agent service, and then restart it. (or connect remotely using services.msc, or use the "sc" command, ie "sc \\server1 stop contego_spop" & "sc \\server1 start contego_spop" )
5. The six certificate files will reappear within a couple of minutes, which indicates agent connection is complete.
6. The agent will appear in the Nodes list, and remove any disconnected duplicates 'of this agent node' from the list.

Note: if this is a new agent install, and minimal data has been received, you can do the following:

• Stop the agent service
• Delete the entire spop sub-folder "C:\Windows\SysWOW64\ContegoSPOP\spop\"
• Start the agent service
• Watch for this agent connection, and remove any disconnected duplicates from the agent list.

Note: If the node does not appear after several minutes, please contact SolarWinds Technical Support to take a closer look and see if the agent is having other issues.
 

Re-install the agents showing up as InternalUnknownAgent  

1. Remove the existing agent from Program and Features. (or download and use the Remote Agent Un-installer from the customer portal, which also deletes the agent directory.)  
2. Remove the agent directory (C:\Windows\SysWOW64\ContegoSPOP\or C:\Windows\system32\ContegoSPOP\ ) if still present.
3. Remove the agent and any duplicate nodes from the agent Nodes list in the SEM Web console (Manage > Nodes).
4. If the agent won't uninstall, try reinstalling over the top, and then try another uninstall.
5. Copy either the Local Agent installer or the Remote Agent installer from the customer portal.

   Copy the agent (remote or local) to the local hard drive in order to "Run as administrator" (will not work over a network share).

6. Right-click to select "Run as administrator" for the install on Windows 7, 8, 2008, 2012, or later.
   (If installing on Windows 2012-R2, 8.1, or 10, use only the local installer from the local hard drive, select both "Windows 7" compatibility and runas-administrator.)
   (If installing on Windows 2016, use the Remote Agent Installer from the local hard drive, and select both "Windows 7" compatibility and runas-administrator.)
7. Configure the connectors for this agent (SEM Web console, Manage > Nodes, or place the agent into a Connector Profile, Build > Groups.
8. Verify (or delete) any duplicates in the agent list.
9. Open the SEM Web console (Explore > nDepth) and perform an nDepth search, selecting Alert-Groups, select AnyAlert, drag the DetectionIP to the top search, enter server1* (hostname). You should see data received into the database.

Citrix VDI Images
If you have XenApp images with nightly image rebuild (any changes made during the day not saved), then the following method automatically adds the SEM agent cert during image re-creation.
 
Stage 1 - Renew & capture all Agent certificates - One time task
For each XA server:

1. Stop SWSEM service.
2. Delete C:\Windows\SysWOW64\ContegoSPOP\spop\.
3. Start SWSEM service.
4. Copy C:\Windows\SysWOW64\ContegoSPOP\spop\ to <SWLEM_CERT_REPO>\<XA_HOSTNAME\spop\.

 Stage 2 - Startup script installed on image to run when each XA servers boot - (or run from another server? Not as reliable?)

1. Stop SWSEM service.
2. Delete C:\Windows\SysWOW64\ContegoSPOP\spop\.
3. Copy <SWSEM_CERT_REPO>\<XA_HOSTNAME>\spop\to C:\Windows\SysWOW64\ContegoSPOP\spop\.
4. Start SWSEM service.

Note: If the node does not appear after several minutes, please contact SolarWinds Technical Support to take a closer look and see if the agent is having other issues.