Troubleshoot SEM agent connections, 32-bit

This article provides troubleshooting steps to help you work around the most common causes when a Security Event Manager (formerly Log & Event Manager) agent cannot connect to your SEM appliance.
Note: as of SEM version 6.7, 32-bit is no longer supported.

• Verify that the computer is still in your environment.
• Verify that the computer is turned on.
• Verify that the SEM Agent service is running:

The SEM agent runs as a service on the host operating system. Ensure the service is running on the host using one of the following (or similar) procedures:

Windows host

1. Open the Control Panel > Administrative Tools > Services.
2. Search for SolarWinds Log and Event Manager Agent.
3. If the SEM agent is not running, click Start (green Play button).

Linux host

1. Run ps ax | grep contego  in a CLI terminal.
2. Search for ContegoSPOP directory, which by default is installed under /usr/local/contego/.
3. If the SEM Agent is not running, try one of the following commands:
   run sudo /etc/init.d/swlem-agent start(enter your root password if necessary)
   -or-

   /usr/local/contego/ContegoSPOP/swlemagent start)
   If the commands above do not work for newer agent versions, try the following:

   /bin/systemctl start swlem-agent.service

Mac host

1. Run ps ax | grep -i trigeo in a CLI terminal.
2. Search for SWLEMAgent.
3. If the SEM  Agent is not loaded, run launchctl load /Library/LaunchDaemons/com.trigeo.trigeoagent.plist.

Verify a firewall is not blocking your connection

The SEM agent relies on the following ports to communicate with the SEM appliance . Ensure you have the proper exceptions in place for any firewall between a SEM Agent and the SEM appliance.    

• 37890-37892: Traffic from SEM agents to the SEM appliance.
• Ephemeral ports: Traffic from SEM appliance communications to LEM Agent (6.3.1)
• 37893-37896: Traffic from the SEM appliance to LEM Aaents on LEM Agent (6.2.1 & earlier).

Verify the SEM agent is running the current version of the software

Agent version typically follows the SEM version, with some exceptions (example: LEM 6.3.1-hotfix-6 uses the Agent version 6.3.1-hotfix-5).

To check your SEM agent version:

1. Open the most recent copy of spoplog.txt.

• Windows: C:\Windows\System32\ContegoSPOP\
• Linux: /usr/local/contego/ContegoSPOP/
• Mac: /Applications/TriGeoAgent/

2. ​In a text editor from the installation folder: 

• Search for Release.
• The most recent entry should reflect the current version running on your system. For example, SolarWinds Log and Event Agent (Release 6.3.1).

Agents failing to communicate properly can be caused from the following:

Agent does not show in SEM console

This procedure corrects connection issues caused by the following:

• Agent does not display in Manage > Nodes.
• An agent re-install is in order, but it needs runas-admin permissions to avoid being an 'unknownAgent' (unknown agents result in performance loss with constant communications attempts, but fail to show or send logs).

1. Remove the existing agent from Program and Features. (or use the Remote Agent Un-installer, which also deletes the agent directory.)
2. Remove the agent directory (C:\Windows\system32\ContegoSPOP\).
3. (Should be already done) Stop the agent service if running (Solarwinds Log and Event Manager Agent).
4. Remove the agent and duplicates of this node from SEM console, Manage > Nodes, (node-list).
5. If there are any issues un-installing, try the latest remote agent un-installer from the customer portal.
6. On rare occasions, you may need to install over the top of existing agent, to be able to un-install.
7. Install the "new" agent applicable to the version of the SEM from the customer portal.
8. Copy the agent installer (remote or local) to the local hard drive from which it will be run (runas-admin will not work when the installer file is located on a network share).
9. Right-click to select "runas administrator" for the install (If for some reason the agent is not communicating to SEM because of Windows security settings, try the local installer, & select both "Windows 7" compatibility and runas-administrator for the install).
10. Configure the connectors for this agent, or place the agent into a Connector Profile (default connectors are Application, System, and the Security Event logs).
11. Verify that the new agent does not have any duplicates in the agent list.
12. Do nDepth search, Alert-Groups select AnyAlert, drag the DetectionIP to the top search, enter the hostname*.

Reset the SEM agent certificate

This procedure corrects connection issues caused by the following:

• Intermittent connectivity
• Inability to upgrade the SEM Agent software
• General failure to connect

If you have verified all previous conditions and you are experiencing any of the symptoms: 

Windows host

1. Stop SolarWinds Security Event Manager service in Control Panel > Administrative Tools > Services.
2. Delete the three .xml and three .trigeo files in C:\Windows\System32\ContegoSPOP\.

       Do not delete the ContegoSPOP folder.

3. Delete the entry for the affected SEM Agent in the Manage > Nodes pane in the LEM console. Click the gear icon, next to the agent's entry, and click Delete.
4. Restart the SEM Agent service (Solarwinds Security Event Manager Agent).

Linux host

1. Stop the swlem-agent service: /etc/init.d/swlem-agent stop (or  /usr/local/contego/ContegoSPOP/swlemagent stop).

   If the command above does not work for newer agent versions, try the following: /bin/systemctl stop swlem-agent.service.

2. Delete the three .xml and three .trigeo files in /usr/local/contego/ContegoSPOP/spop.
3. Delete the entry for the affected SEM Agent in Manage > Nodes pane in the SEM console. Click the gear icon, next to the agent's entry, and then click Delete.
4. Restart the swlem-agent service: /etc/init.d/swlem-agent start (or  /usr/local/contego/ContegoSPOP/swlemagent start).

   If the command above does not work for newer agent versions, try the following: /bin/systemctl start swlem-agent.service.

1. Unload swlemagent.plist: launchctl unload /Library/LaunchDaemons/com.swlem.swlemagent.plist
2. Delete the three .xml and three .trigeo files in /Applications/TriGeoAgent/spop.
3. Delete the entry for the affected SEM Agent in Manage > Nodes pane in the SEM console. Click the gear icon, next to the agent's entry, and then click Delete. 
4. Reload swlemagent.plist: launchctl load /Library/LaunchDaemons/com.swlem.swlemagent.plist

SEM agent port communications issues

Check the following if you are experiencing communications issues:

1. Be sure that the SEM is updated to the latest version and the agent is updated to the latest version.
2. From a command line, telnet from the Agent to port 37890 on SEM ("telnet <lem-ip> 37890," then try port 37891 & port 37892. All 3 ports are needed for communications).

3. Run a netstat command and verify if another service or process has taken ownership of the ports.
4. Run a netstat command and note the source port used by the telnet client.

On pre-6.3.1 agents, agent source was 37893->37896. 6.3.1 agents use ephemeral ports for source.

5. Disable the anti-virus and re-install the agent per the un-install/re-install above.
6. Disable the Windows firewall for all 3 profiles (domain, public, private). if the public or private profile is enabled, event though the domain is disabled, communications may be blocked.

7. Look into the software installed on this computer for any possible resource conflict.

Duplicated Nodes Appearing in the Console

• By design, the SEM will register devices sending data, either by IP address, or hostname if it can resolve the IP.
• Only use one of the following in the 'spop.conf' file, and no more than one.
• Remove any duplicates of this agent from the node-list. If fixed, the additional nodes will not return.  

If the agent is a web server, multiple hostnames or IP address may register in the Node List.
Two steps to help resolve this:

1. Open the"C:\windows\system32\ContegoSPOP\spop.conf", add the following line, & restart the agent. UseLocalEnvironmentVariableForLocalHost=true.
2. Open the "C:\windows\system32\drivers\etc\hosts"  file and add the windows server itself as the first entry. For example: 10.1.2.3   server1     server1.contoso.com.

If the agent is a laptop, communicating with ethernet, and the wireless grabbed an APIPA address (169.254.x.x), then follow this step: Open the "C:\windows\system32\ContegoSPOP\spop.conf", add the following line, & restart the agent.
ForcedLocalAddress=<hostname>

Use the actual hostname of Windows server/workstation, not <hostname>.

If one of the above two does not resolve the issue, try the following: Open the "C:\windows\system32\ContegoSPOP\spop.conf", add the following line, & restart the agent.
UseManagerDetectionOfLocalHost=true

Gathering logs from the agent

Locate the file "C:\windows\system32\ContegoSPOP\collectLogs.bat", right-click and select runas-administrator.
Grab the resultant file created for SolarWinds support:  "C:\windows\syswow64\ContegoSPOP\lemAgentLogs.cab"

Contacting Support

If you are unable to resolve your issue using this article, open a ticket with SolarWinds Support for further assistance. Please be prepared to provide the following once you are in touch with a representative:

• The exact operating system the host computer is running
• The version of your SEM components:
  • Agent installer
  • SEM appliance
  • SEM Console
• The most recent copy of spop.log from the Agent installation folder.