Troubleshooting SEM agents

• SEM Appliance IP or hostname changed
• Agent Certificate is corrupted
• Network / Firewall / AV blocking port ranges 37891-37892
• Agent service was not running
• SEM agent node is closed
• Upgrade to 2023.2 Certificate SAN error
• SEM License is expired

Initial checks:

1. Verify that the computer is still in your environment.
2. Verify that the computer is turned on.
3. Verify you have available Licenses for Workstation and Universal nodes.
4. Verify that the SEM agent service is running on the node.
5. Verify the IP, Hostname or FQDN SAN configuration in SEM security settings
6. Ensure the SEM License is active and not expired.

SEM License Activation

1. Activate or Update SEM License.  Activate the SEM license

Windows host

1. Open the Control Panel > Administrative Tools > Services.
2. Search for SolarWinds Security Event Manager agent.
3. If the SEM agent is not running, click Start (green Play button).

Linux host

1. Run ps ax | grep contego  in a CLI terminal.
2. Search for ContegoSPOP directory, which by default is installed under /usr/local/contego/.
3. If the SEM agent is not running, try one of the following commands:

   run sudo /etc/init.d/swlem-agent start (enter your root password if necessary)
   
   (or)
   
   /usr/local/contego/ContegoSPOP/swlemagent start

   If the commands above do not work for newer agent versions, try the following:

   /bin/systemctl start swlem-agent.service

Mac host

1. In a CLI terminal, run ps ax | grep -i trigeo

2. Search for SWLEMAgent.
3. If the LEM  agent is not loaded, run launchctl load /Library/LaunchDaemons/com.trigeo.trigeoagent.plist.

SEM Agent Connection Ports:

If the service is running, but the SEM agent is still not showing, or is showing as disconnected, verify a firewall is not blocking your connection. The SEM agent relies on the following ports to communicate with the SEM appliance. Refer: How to verify SEM agent service and port status

Ensure you have the proper exceptions in place for any firewall between a SEM agent and the SEM appliance.    

• Verify the SEM agent is running the current version of the software. The agent version typically follows the SEM version, with some exceptions.  Example: (SEM 2024.2 uses 2024.2 Agent version )
• 37890-37892: Traffic from SEM agents to the SEM appliance.
  • Use Powershell to Test via the Test-NetConnection command:
    • 37890..37892 | ForEach-Object {Test-NetConnection <SEMIP> -Port $_}
    • 
• Upon Failure test:
  • Disable the anti-virus and re-install the agent per the un-install/re-install above.
  • Disable the Windows firewall for all 3 profiles (domain, public, private).

    If the public or private profile is enabled, event though the domain is disabled, communications may be blocked.

  • Look into the software installed on this computer for any possible resource conflict.

To check your SEM agent version:

1. Open the most recent copy of spoplog.txt.

• Windows: C:\Windows\SysWOW64\ContegoSPOP\
• Linux: /usr/local/contego/ContegoSPOP/
• Mac: /Applications/TriGeoAgent/

2. In a text editor from the installation folder:

• Search for Release.
• The most recent entry should reflect the current version running on your system. For example, SolarWinds Log and Event Agent (Release 6.3.1).

Agents failing to communicate to SEM Manager

Check the error in spop.log

10:48:58 SGT 2024) EE:ERR [ComNetworkParent] {ComModuleSpop:32} Failed to acquire install channel connection, status: 3
javax.net.ssl.SSLHandshakeException: Remote host terminated the handshake

1. Reset the SEM Agent Certificate (See the steps below on how to reset)
2. Check the SEM Settings > Security > Agents > Certificate > Subject Alternative Names (SAN).  Ensure the FQDN, Hostname and IP are configured.
      a - Log into the SEM Webui
      b - Navigate to Settings > Security > Agents > Certificate
       c - Edit or add FQDN, Hostname or IP

3. Increase the Channel Timeout in the SEM Manager (via cmc CLI)

Login as cmc
manager → rcc
set cnf;manager;CommandChannelTimeout 180000
manager → restart
Wait for manager to start
Restart the agent service (because it could gave up after reaching max install attempts)
Wait for installation (3 minutes) and check logs

Reset the SEM agent certificate

This procedure corrects connection issues caused by the following:

• Intermittent connectivity
• Inability to upgrade the SEM agent software
• General failure to connect

If you have verified all previous conditions, and you are experiencing any of these symptoms:

Windows host

1. Stop SolarWinds Security Event Manager service in Control Panel > Administrative Tools > Services.
2. Delete only the files (.truststore, .keystore & kvStorage) located in C:\Windows\SysWOW64\ContegoSPOP\spop\

   Warning: Do not delete the ContegoSPOP folder or spop folder

3. If the SEM appliance hostname has changed update the name in C:\Windows\SysWOW64\ContegoSPOP\spop.conf (skip if this if there was no change here)
4. Delete the entry for the affected SEM agent in the Manage > Nodes pane in the SEM console. Click the gear icon, next to the agent's entry, and then click Delete.
5. Restart the SEM agent service (SolarWinds Security Event Manager Agent).

Linux host

1. Stop the swlem-agent service: 

   /etc/init.d/swlem-agent stop (or  /usr/local/contego/ContegoSPOP/swlemagent stop)

     If the command above does not work for newer agent versions, try the following:

   /bin/systemctl stop swlem-agent.service

2. Delete the three (.truststore, .keystore & kvStorage if on latest ver on SEM) ) files in /usr/local/contego/ContegoSPOP/spop
3. Delete the entry for the affected SEM agent in Manage > Nodes pane in the SEM console. Click the gear icon next to the agent's entry, and then click Delete.
4. Restart the swlem-agent service: /etc/init.d/swlem-agent start (or  /usr/local/contego/ContegoSPOP/swlemagent start).

   If the command above does not work for newer agent versions, try the following:

   /bin/systemctl start swlem-agent.service

Mac host

1. Unload swlemagent.plist: launchctl unload /Library/LaunchDaemons/com.swlem.swlemagent.plist.
2. Delete the three .xml and three .trigeo files in /Applications/TriGeoAgent/spop.
3. Delete the entry for the affected SEM agent in Manage > Nodes pane in the SEM console. Click the gear icon next to the agent's entry, and then click Delete. 
4. Reload swlemagent.plist: launchctl load /Library/LaunchDaemons/com.swlem.swlemagent.plist.

Duplicated Nodes Appearing in the Console

If you are seeing an extra nodes in the nodes list, whether they are agent nodes or syslog nodes, this could be an issue with the SEM detecting multiple network addresses for the device. The steps in this article can help merge these nodes. Refer: Some agent nodes are duplicated and show as a non-agent node

SEM Hostname or IP Address changed or renamed

1. Uninstall the agent from Add/remove or using the agent uninstaller available from https://customerportal.solarwinds.com/  - applicable to windows hosts only
2. And re-install as per Install the SEM Agent on Linux and Unix

A large amount of nodes aren't connecting to the appliance or connections are intermittent:
1.Check the DNS  and gateway settings on the SEM appliance- If the settings are invalid, correct the settings and see if the issue is resolved. 
2. The virtual appliance network interface is corrupted on SEM- Delete the existing Virtual NIC  from the SEM appliance. Recreate the network interface in VMWare or Hyper V host and re add to the appliance.
3. Check the SEM Settings > Security > Agents > Certificate > Subject Alternative Names (SAN).  Ensure the FQDN, Hostname and IP are configured.  

Contacting Support

If you are unable to resolve your issue using this article, open a ticket with SolarWinds Customer Support for further assistance. Please be prepared to provide the following once you are in touch with a representative:

Gathering logs from the agent

Locate the file C:\windows\syswow64\ContegoSPOP\collectLogs.bat, right-click, and then select Run as administrator.
Grab the file created for SolarWinds support:  "C:\windows\syswow64\ContegoSPOP\lemAgentLogs.cab."

• Windows: C:\Windows\SysWOW64\ContegoSPOP\
• Linux: /usr/local/contego/ContegoSPOP/
• Mac: /Applications/TriGeoAgent/