Submit a ticketCall us

AnnouncementsFace your biggest database issues head-on

Our new eCourse helps you navigate SQL Server performance blocks by teaching you how to recognize and deal with the three DBA Disruptors: Performance Hog, Blame Shifter, and Query Blocker. Register today to learn how to defend your environment and fend off menacing disruptions.

Register for your free eCourse.

Home > Success Center > Log & Event Manager (LEM) > LEM - Knowledgebase Articles > Troubleshoot LEM agent connections, 64-bit

Troubleshoot LEM agent connections, 64-bit

Updated: September 7, 2018

Overview

This article provides troubleshooting steps to help you work around the most common causes when a LEM Agent cannot connect to your LEM appliance.

Environment

All LEM versions

Steps

  • Verify that the computer is still in your environment
  • Verify that the computer is turned on
  • Verify you are not running our of Licenses for Workstation and Universal nodes
  • Verify that the LEM Agent service is running on the node. Ensure the service is running on the host using one of the following (or similar)  procedures:

Windows host

  1. Open the Control Panel > Administrative Tools > Services.
  2. Search for SolarWinds Log and Event Manager Agent.
  3. If the LEM Agent is not running, click Start (green Play button).

Linux host

  1. Run ps ax | grep contego  in a CLI terminal.
  2. Search for ContegoSPOP directory, which by default is installed under /usr/local/contego/.
  3. If the LEM Agent is not running, try one of the following commands:
    run sudo /etc/init.d/swlem-agent start (enter your root password if necessary)

       -or-
       /usr/local/contego/ContegoSPOP/swlemagent start

Mac host

  1. Run ps ax | grep -i trigeo in a CLI terminal.
  2. Search for SWLEMAgent.
  3. If the LEM  Agent is not laoded, run launchctl load /Library/LaunchDaemons/com.trigeo.trigeoagent.plist.

If the service is running, but the LEM Agent is still not showing, or is showing as disconnected, verify a firewall is not blocking your connection. The LEM Agent relies on the following ports to communicate with the LEM appliance. Ensure you have the proper exceptions in place for any firewall between a LEM Agent and the LEM appliance.    

  • 37890-37892: Traffic from LEM Agents to the LEM appliance.
  • Ephemeral ports: Traffic from LEM appliance communications to LEM Agent (6.3.1)
  • 37893-37896: Traffic from the LEM appliance to LEM Agents on LEM Agent (6.2.1 & earlier).
  • Verify the LEM Agent is running the current version of the software. Agent version typically follows the LEM version, with some exceptions.  Example: LEM 6.3.1-hotfix-6 uses the Agent version 6.3.1-hotfix5)

To check your LEM Agent version:

  1. Open the most recent copy of spoplog.txt.
  • Windows: C:\Windows\SysWOW64\ContegoSPOP\
  • Linux: /usr/local/contego/ContegoSPOP/
  • Mac: /Applications/TriGeoAgent/
  1. In a text editor from the installation folder:
  • Search for Release.
  • The most recent entry should reflect the current version running on your system. For example, Solarwinds Log and Event Agent (Release 6.3.1).

Agents failing to communicate properly can be caused by the following:

 

Agent does not show in the LEM console

This procedure corrects connection issues caused by the following:

  • Agent does not display in Manage > Nodes.
  • An agent re-install/update is in order, but it needs to run as-admin permissions to avoid being an unknown agent.
  • Remove the existing agent from Program and Features (or use the Remote Agent Un-installer, which also deletes the agent directory).
  1. Remove the agent directory (C:\Windows\sysWOW64\ContegoSPOP. It should be already deleted. If not, stop the LEM agent service if running via service manager, and then try deleting again. 
  2. To remove the duplicates of this agent node from the LEM console, navigate to Manage > Nodes
  3. If there are any issues un-installing, try the latest remote agent un-installer from the customer portal. On rare occasions you may need to install over the top of an existing agent to uninstall. 
  4. Download the new agent applicable to the version of LEM from the Solarwinds customer portal. Copy the agent installer (remote or local) to the local hard drive from which it will be run.  

    Run as admin will not work when the installer file is located on a network share.

  5. Right-click and select Run as administrator for the install.  If for some reason the agent is not communicating to LEM because of Windows security settings, try the local installer and select both Windows 7 compatibility and Run as administrator for the install.
  6. Configure the connectors for this agent, or place the agent into a Connector Profile.
  7. Verify the new agent does not have any duplicates in the agent list.
  8. Conduct an nDepth search.  In Alert-Groups, select AnyAlert, drag the DetectionIP to the top search, and then enter the hostname*.

 

Reset the LEM Agent certificate

This procedure corrects connection issues caused by the following:

  • Intermittent connectivity
  • Inability to upgrade the LEM Agent software
  • General failure to connect

If you have verified all previous conditions and you are experiencing any of the symptoms: 

Windows host

  1. Stop SolarWinds Log and Event Manager service in Control Panel > Administrative Tools > Services.
  2. Delete the three .xml and three .trigeo files in C:\Windows\SysWOW64\ContegoSPOP\.

    Do not delete the ContegoSPOP folder.

  1. Delete the entry for the affected LEM Agent in the Manage > Nodes pane in the LEM console. Click the gear icon, next to the agent's entry, and then click Delete.
  2. Restart the LEM Agent service (SolarWinds Log and Event Manager Agent).

Linux host

  1. Stop the swlem-agent service: /etc/init.d/swlem-agent stop (or  /usr/local/contego/ContegoSPOP/swlemagent stop)
  2. Delete the three .xml and three .trigeo files in /usr/local/contego/ContegoSPOP/spop.
  3. Delete the entry for the affected LEM Agent in Manage > Nodes pane in the LEM console. Click the gear icon, next to the agent's entry, and then click Delete.
  4. Restart the swlem-agent service: /etc/init.d/swlem-agent start (or  /usr/local/contego/ContegoSPOP/swlemagent start)

Mac host

  1. Unload swlemagent.plistlaunchctl unload /Library/LaunchDaemons/com.swlem.swlemagent.plist
  2. Delete the three .xml and three .trigeo files in /Applications/TriGeoAgent/spop.
  3. Delete the entry for the affected LEM Agent in Manage > Nodes pane in the LEM console. Click the gear icon, next to the agent's entry, and then click Delete. 
  4. Reload swlemagent.plist: launchctl load /Library/LaunchDaemons/com.swlem.swlemagent.plist

 

LEM Agent port communications issues

Check the following if you are experiencing communications issues:

  1. Be sure that the LEM is updated to the latest version and the agent is updated to the latest version.
  2. From a command line, telnet from the Agent to port 37890 on LEM. "telnet <lem-ip> 37890"

    Then try port 37891 & port 37892. All 3 ports are needed for communications.

  1. Run a netstat command and verify if another service or process has taken ownership of the ports.
  2. Run a netstat command and note the source port used by the telnet client.

    On pre-6.3.1 agents, agent source was 37893->37896. 6.3.1 agents use ephemeral ports for the source.

  1. Disable the anti-virus and re-install the agent per the un-install/re-install above.
  2. Disable the Windows firewall for all 3 profiles (domain, public, private).

    If the public or private profile is enabled, event though the domain is disabled, communications may be blocked.

  3. Look into the software installed on this computer for any possible resource conflict.

 

Duplicated Nodes Appearing in the Console

By design, the LEM will register devices sending data, either by IP address, or hostname if it can resolve the IP.

  •  Only use one of the following in the 'spop.conf' file, and no more than one.
  •  Remove any duplicates of this agent from the node-list. If fixed, the additional nodes will not return.

If the agent is a web server, multiple hostnames or IP address may register in the Node List.
To resolve this:

  1. Open the"C:\windows\syswow64\ContegoSPOP\spop.conf", add the following line, and then restart the agent.
  • UseLocalEnvironmentVariableForLocalHost=true
  1. Open the "C:\windows\system32\drivers\etc\hosts"  file and add the windows server itself as the first entry. 
  • For example: 10.1.2.3 server1 server1.contoso.com

If the agent is a laptop communicating through Ethernet, and the wireless grabbed an APIPA address (169.254.x.x), then follow this step:

  • Open C:\windows\syswow64\ContegoSPOP\spop.conf, add the following line, and then restart the agent:
    • ForcedLocalAddress=<hostname>  (use the actual hostname of Windows server/workstation, not <hostname>.)

If one of the above two does not resolve the issue, try the following: 

  • Open C:\windows\syswow64\ContegoSPOP\spop.conf, add the following line, and then restart the agent.
    • UseManagerDetectionOfLocalHost=true

 

Gathering logs from the agent

Locate the file "C:\windows\syswow64\ContegoSPOP\collectLogs.bat", right-click and select runas-administrator.
Grab the resultant file created for Solarwinds support:  "C:\windows\syswow64\ContegoSPOP\lemAgentLogs.cab"

 

Contacting Support

If you are unable to resolve your issue using this article, open a ticket with SolarWinds Support for further assistance. Please be prepared to provide the following once you are in touch with a representative:

  • The exact operating system the host computer is running
  • The version of your LEM components:
    • Agent installer
    • LEM appliance
    • LEM Console
  • The most recent copy of spop.log from the Agent installation folder.

 

Last modified

Tags

Classifications

Public