Submit a ticketCall us

AnnouncementUpgrading SolarWinds Orion Platform products has never been easier!

The SolarWinds Orion Installer is an all-in-one application for installations and upgrades. You use one installer to install or upgrade multiple Orion Platform products and install or upgrade Additional Polling Engines, Additional Web Servers, and High Availability Servers.

Learn more in the SolarWinds Orion Installer.

Home > Success Center > Log & Event Manager (LEM) > LEM - Knowledgebase Articles > Troubleshooting LEM Rules and Email Responses

Troubleshooting LEM Rules and Email Responses

Overview

The following scenarios and resolutions apply to LEM rules that are not firing as expected or sending the expected notifications. For additional information about any of the procedures referenced in these scenarios, see the associated footnotes.

 

Environment

  • LEM all versions

Cause 

The cause of this is because in a rule you can use any email template you want but Best Practices is to use the name of the rule as the Subject of the Email when your building your email template under Build -> Groups > Email Templates.

SCENARIOS:

  1. I don't know which rule triggered the email.
  2. The rule triggers but I don't get an email.
  3. I see the alerts, but my rule doesn't fire.
  4. My rule doesn't fire, and I don't see the expected alerts.

Resolution

To find out which rule triggered the email

  1. Get the time that's on the email that the alert came in.
  2. Log into the LEM web console or Adobe Air Console.
  3. On the LEM toolbar, navigate to Explore > nDepth.
  4. On the far right, click the drop-down list (default value is set to Last 10 Min) and select Custom range.
  5. Enter one minute before and after the date and time that you got the email.
  6. In the List pane, click Events.
  7. Then in the search box, type INTERNALRULEFIRED, and then drag it to search bar at the top.
  8. To the far right of the search bar, click the blue Play button.
  9. In the List pane, click Refine Fields (blue filter icon).
  10. To show all the rules that fired, click InferenceRule.
  11. On the nDepth toolbar below, click Result Details .
  12. Verify that the ExtraneousInfo field of the InternalRuleFired alert shows the associated email action in Email [recipient] format.
  13. When you see a rule that fired an email, see what events actually caused that rule to fire.
  14. To see if the event that triggered that rule matches your email, click the rule that the fired email. 
  15. On the upper right of the nDepth pane, click the Explore drop-down list, and then select Event.
  16. On the upper left, click Event Details.
  17. In the bottom box, click the event to get details for the event and not the rule. 

 

The rule triggers but I don't get an email

  1. If that action is not present, add the Send Email Message action to the rule.
  2. Verify that the intended recipient has an email address associated with his LEM user account:
    1. On the LEM toolbar, navigate to Build > Users.
    2. Click the LEM user account associated with the intended recipient.
  3. If the Contact Information box is blank in the User Information pane, edit the user to add an email address.

    If you are unable to add an email address to an AD user, you may need to create a separate user and add the email to that user account, and then select that user in the email template.

  1. Verify that the Email Active Response connector is configured on your LEM Manager:
    1. On the LEM toolbar, navigate to Manage > Appliances.
    2. Next to your LEM Manager, click the gear icon, and then select Connectors.
    3. On the Connector Configuration window, select Configured on the Refine Results pane.
  2. If Email Active Response is not in the list, clear the Configured check box, and then configure the missing connector.

 

You do not see the expected InternalRuleFired alerts in the default SolarWinds Alerts or Rule Activity filters under the Monitor in the LEM console, nor do you see the alerts needed to fire your rule anywhere in your LEM console.

To determine whether the requisite alerts are in your LEM console, create a filter or nDepth search that matches the correlations in your rule. If the alerts are not present, complete the following procedure:

 

  1. Review the network devices that are sending syslog data to LEM, and validate the configurations on that network device to send data. Verify that one of your devices is logging the events you want to capture. For example:

    • Remote logging devices, such as firewalls and web filters, should be logging your web traffic events.
    • Domain controllers and end-user computers should be logging domain-level and local authentication and change management events.

      If you have multiple domain controllers, they will not all replicate every domain event. Each server only logs the events they execute.

    • Other servers, such as database servers and web servers, should be logging events associated with their particular functions.
  1. Validate if data is received by the LEM.
    • Validate if the LEM icons show syslog/agent connection:
      1. Syslog device IPs will appear in the LEM web console in the Manage > Nodes list as a pipe-Y symbol.
      2. Agent host names and IP addresses will appear in the LEM web console in the Manage  > Nodes list as a green plug icon.
    • Validate if data is being received by syslog facility or by the agent.
      1. If a network syslog device is sending syslog data to LEM, you should be able to view the LEM syslog files for that data.
      2. Perform the following:
        • Open the vSphere/Hyper-V console to access LEM.

          : You may also use a PuTTY session, port 32022, cmc user.

        • Enter the appliance menu, and enter the checklogs command.
        • View the syslog that was chosen by the network device. All of the data received in this area is UDP traffic received on port 514.
      3. Agent data is encrypted and more difficult to tell if it is received by the LEM.
  2. If your device is not in the Nodes list, configure computers by installing a LEM Agent, or configure other devices, such as firewalls, to log to your LEM appliance. After your device is in the list, continue to the next step.
  3. If your device is in the Nodes list, configure the appropriate connectors:
    1. To configure syslog connectors (manager connectors) on your LEM Manager for remote logging devices, navigate to Manage > Appliances on the LEM toolbar.
    2. Next to the Agent or Manager on which you want to configure the new connectors, click the gear icon, and then select Connectors.
    3. Use the Search box at the top of the Refine Results pane to locate the appropriate connectors.
    4. Configure the connector according to your needs.
    5. To configure agent connectors, navigate to Manage > Nodes, select the gear icon next to the agent, and then edit the connectors

 

I see the alerts but my rule doesn't fire

  1. Problem:

    You see the alerts required to fire your rule in the LEM Console, but your rule still doesn't fire.

    Steps to resolve:

  2. Verify that all of your rules have been activated in all open LEM Consoles:
    1. Click the Build tab, and then select Rules.
    2. If the Activate Rules button is not greyed out, click it. This synchronizes all of the changes you have made to your rules in the Console with your LEM Manager.
    3. Repeat these steps for all open LEM Consoles in your environment.
  3. Compare the InsertionTime and DetectionTime values in the alerts you expected to fire your rule.
  4. If the time is off by more than five minutes, verify and correct the time settings on your LEM appliance and any remote logging devices as necessary. See To view and modify the time on your LEM appliance.
  5. If none of the previous troubleshooting steps help, restart the Manager service on your LEM appliance. In general, consider doing this once every six months:
    1. Connect to your LEM virtual appliance using either the vSphere console view, or an SSH client like PuTTY.
    2. If you are using an SSH client, log in to your LEM virtual appliance using your CMC credentials.
    3. At the cmc> prompt, enter manager.
    4. At the cmc::cmm prompt, enter restart.
    5. Press Enter to confirm your entry.
      Note: Restarting the Manager service will make your LEM Manager unavailable for about one minute. However, no data is lost during this process.
    6. Enter exit twice to leave the CMC interface.

 

The rule fires but the email is blank

Problem:

You receive an email notification for the alert, but the fields in the custom email template are blank.

Steps to resolve:

  1. Click the Build tab, and then select Rules.
  2. Locate your rule, click the gear icon on the left and select Edit. You will notice that the fields in the Actions box are blank.
  3. Copy the event assigned to this rule. This is the string before the dot in the Correlation box.
  4. Click Events on the left pane and type the event in the search field.
  5. Drag the fields required in your rule from the Fields pane to populate the blank fields in the Actions box.
  6. Click Save to close the Rule Creation window.
  7. Click Activate Rules on the Rules window.

 

You have set up the rule correctly, and data matches in nDepth, but the rule still doesn't fire

The primary issue is the time needs to be correct in LEM in order for the response time frame in the rule to work correctly.

To view and modify the time on your LEM appliance:

  1. Connect to your LEM virtual appliance using either the vSphere console view, or an SSH client like PuTTY.
  2. If you are using an SSH client, log in to your LEM virtual appliance using your CMC credentials.
  3. At the cmc> prompt, enter appliance.
  4. At the cmc::acm prompt, enter dateconfig.
  5. Press Enter through all of the prompts to view the current date and time settings on your LEM applaince.
  6. By default, the LEM receives a time synchronization from the VM host computer. Without this, the time in LEM will be off and rules may not fire. You will need to disable the time sync on the VM host computer, and then enable LEM to get time from an NTP server:
    1. At the cmc::acm prompt, enter ntpconfig.
    2. To start the configuration script, press Enter.
    3. Enter the IP addresses of your NTP servers, separated by spaces.
    4. To verify your entry, enter y.
  7. To exit the CMC interface, enter exit twice.

 

You can also check that the connector is turned on for the device that is sending the data. For example, if it is a syslog device like Websense, navigate to Manage > Appliances, click gear beside name of the appliance, and then select Connectors. In the refine results pane, select the Configured check box, and then make sure there is a green status icon indicating the connector is running.

Additional Information

 

 

 

Last modified

Tags

Classifications

Public