Submit a ticketCall us

Have You Auto Renewed? If not, you're missing out.
The SolarWinds Renewal Program comes with a host of benefits including the most recent product updates, 24/7 technical support, virtual instructor-led training and more. Experience all of this with the convenience of Auto Renewal, and never worry about missing any of these great benefits. Learn More.

Home > Success Center > Log & Event Manager (LEM) > LEM Administrator's Guide > Troubleshoot LEM rules and email responses

Troubleshoot LEM rules and email responses

Updated: September 15, 2017

This page provides troubleshooting steps to try if your LEM rules that are not firing as expected or if your rules are not sending the expected notifications.

General rule troubleshooting

If you created a rule that generates unexpected results, verify the following to track down the root cause:

If you do not see an InternalRuleFired event for your rule, verify that:

  1. Click the Monitor tab and check for the requisite events.

    For example, if your rule is based on the NewGroupMember event, locate a requisite event in the All Events or default Change Management filter.

  2. If you cannot view the requisite events, troubleshoot your devices and connectors to move the events into LEM.

  3. Check for an InternalRuleFired event in the SolarWinds Events filter.

    If you see an InternalRuleFired event for your rule, go to the next step.

    • The rule is enabled.
    • The Correlation Time or Response Window in your rule was not modified.
    • You did not click Activate Rules after saving your rule.
    • The time on your device is not more than five minutes off from the time on your LEM appliance.
  1. If you see an InternalRuleFired event for your rule but LEM does not respond to the rule as expected, check the following:

    • Send Email Message
      Verify you configured and started the Email Active Response connector on the LEM Manager. Additionally, verify you associated an email address for your selected LEM user as your email account.
    • Agent-based Actions
      Verify you installed the LEM Agent on a computer that will respond to LEM.
    • Block IP
      If using the Block IP active response, verify that you configured the active response connector for the targeted firewall that will respond to this action. The active response connector is separate from the data-gathering connector.

The rule fires but you do not receive an email

Problem statement: You see the expected InternalRuleFired alerts in the default SolarWinds Alerts and Rule Activity filters in the LEM console, but you are not getting the expected email notification.

To resolve this issue:

  1. Verify that the ExtraneousInfo field of the InternalRuleFired alert shows the associated email action in Email [recipient] format.

  2. If this action is not present, add the Send Email Message action to the rule.

  3. Verify that the intended recipient has an email address associated with his LEM user account:

    1. Open the LEM console. See Log in to the LEM web console or Log in to the LEM desktop console for steps.

      Click Build > Users.

    2. Click the LEM user account associated with the intended recipient.

  4. If the Contact Information box is blank in the User Information pane, edit the user to add an email address.

    If you cannot add an email address to an Active Directory user, create a separate user, add the email to that user account, and then select that user in the email template.

  5. Verify that the Email Active Response connector is configured on your LEM Manager.
    1. Click Manage > Appliances.

    2. Click button-gear.png next to your LEM Manager and select Connectors.

    3. In the Connector Configuration window, select the Configured check box.

  6. If Email Active Response is not in the list, clear the Configured check box configure the missing connector.

The rule does not fire and expected alerts do not display

Problem statement: You cannot see the expected InternalRuleFired alerts in the default SolarWinds Alerts or Rule Activity filters in the LEM console or the alerts needed to fire your rule anywhere in your LEM console.

To determine if the requisite alerts are in your LEM console, create a filter or nDepth search that matches the correlations in your rule.

If the alerts are not present, complete the following procedure:

  1. Review the network devices sending syslog data to the LEM and validate the configurations on that network device to send data. Verify that one of your devices is logging the events you want to capture.

    For example:

    • Remote logging devices, such as firewalls and web filters, should be logging your web traffic events
    • Domain controllers and end-user computers should be logging domain-level and local authentication and change management events

      If you have multiple domain controllers, they will not all replicate every domain event. Each server only logs the events they execute.

    • Other servers, such as database servers and web servers, should be logging events associated with their particular functions.
  2. Verify that the LEM is receiving data.

    Verify that the LEM icons display a syslog or Agent connection. Syslog device IPs display with the File:Success_Center/Reusable_content_-_InfoDev/LEM/Log_and_Event_Manager_Administrator's_Guide/LEM_Administrator's_Guide/1370-Troubleshoot_LEM_rules_and_email_responses/icon-y-pipe.png icon in the Manage > Nodes grid. Agent host names and IP addresses appear in the Manage > Nodes list with the File:Success_Center/Reusable_content_-_InfoDev/LEM/Log_and_Event_Manager_Administrator's_Guide/LEM_Administrator's_Guide/1370-Troubleshoot_LEM_rules_and_email_responses/icon-connected_18x18.jpg icon.

    Next, verify that the syslog facility or Agent is receiving data. If a network syslog device is sending syslog data to the LEM, you can view the LEM syslog files for that data.

    1. Open the CMC command line. See Log in to the LEM CMC command line interface for steps.

    2. Type appliance , and enter the checklogs command.

      You can also open a PuTTY session on port 32022 as a cmc user.

    3. View the syslog that was chosen by the network device. All of the data received in this area is UDP traffic received on port 514.

  3. If your device is not in the Nodes list, configure your computers by installing a LEM Agent or configure other devices (such as firewalls) to log to your LEM VM or appliance. After your device is in the list, continue to the next step.

  4. If your device is in the Nodes list, configure the appropriate connectors:

    1. Open the LEM console. See Log in to the LEM web console or Log in to the LEM desktop console for steps.

      Click Manage > Appliances.

    2. Click  next to the Agent or LEM Manager and select Connectors.

      Use the Search box at the top of the Refine Results pane to locate the appropriate connectors.

    3. Configure the syslog connector according to your needs.

    4. Click Manage > Nodes.

    5. Click next to the Agent.

    6. Configure the Agent connector as required.

Alerts display but the rule does not fire

Problem statement: You see the alerts required to fire your rule in the LEM console, but your rule still doesn't fire.

To resolve this issue:

  1. Verify that all of your rules are activated in all open LEM consoles:

    1. Open the LEM console. See Log in to the LEM web console or Log in to the LEM desktop console for steps.

      Click Build > Rules.

    2. Click Activity Rules.

      All rule changes you implemented in your LEM Manager are synchronized.

    3. Repeat these steps for all open LEM consoles in your environment.

  2. Compare the InsertionTime and DetectionTime values in the alerts you expected to fire your rule.

    If the time is off by more than five minutes, verify and correct the time settings on your LEM VM or appliance, and any remote logging devices as necessary.

  3. If your rules will not fire, restart the Manager service on your LEM VM/appliance.

    In general, consider doing this once every six months:

    1. Open the CMC command line. See Log in to the LEM CMC command line interface for steps.

    2. At the cmc> prompt, enter manager and press Enter.

    3. At the cmc::manager> prompt, type restart and press Enter.

    4. Press Enter to confirm your entry.

      Restarting the LEM Manager service disconnects the Manager for a few seconds. No data is lost during this process.

    5. Enter exit and press enter twice to leave the CMC interface.

The rule fires but the email is blank

Problem statement: You receive an email notification for the alert, but the fields in the custom email template are blank.

To resolve this issue:

  1. Open the LEM console. See Log in to the LEM web console or Log in to the LEM desktop console for steps.

  2. Click Build > Rules.

  3. Locate your rule in the Rules grid.

  4. Click next to your targeted rule and select Edit. Notice that the files in the Action box are blank.

  5. Copy the event assigned to this rule.

    This is the string before the dot in the Correlation box.

  6. Click Events and enter the event in the search field.

  7. Drag the event fields required for your rule into the Actions box.

  8. Click Save to close the Rule Creation window.

  9. Click Activate Rules.

View and modify the time on your LEM appliance

  1. Open the CMC command line. See Log in to the LEM CMC command line interface for steps.

  2. At the cmc> prompt, enter appliance.

  3. At the cmc::appliance> prompt, enter dateconfig.

  4. Press Enter through all of the prompts to view the current date and time settings on your LEM appliance.

    By default, LEM receives a time synchronization from the VM host computer. Without the synchronization, the LEM time is not correct and the rules may not trigger when required.

  5. Disable the time sync on the VM host computer and enable LEM to receive time information from an NTP server.
    1. At the cmc::appliance> prompt, enter ntpconfig and press Enter.

    2. Press Enter to start the configuration script.

    3. Enter the IP addresses of your NTP servers separated by spaces.

    4. Enter y and press Enter to verify your entry.

  6. Enter exit and press Enter twice to leave the CMC interface.

The rule is not triggered when it should be

Check your rule logic and timestamps. The LEM VM host layer may need to be configured for NTP. By default, rules will not fire when incoming data drifts more than five minutes from the LEM VM's clock.

  1. Open the CMC command line. See Log in to the LEM CMC command line interface for steps.

  2. Type appliance to enter the appliance menu.

  3. Enter the dateconfig command, and confirm the date and time. You can change the time with this command, but when the vSphere/Hyper-V time sync pushes the time to LEM, this will change.
Last modified

Tags

Classifications

Public