Submit a ticketCall us

Looking to compare latest NPM features with previous versions of NPM?
The NPM new feature summary offers a comparison of new features and improvements offered with this release.

 

Home > Success Center > Log & Event Manager (LEM) > Troubleshooting LEM Rules and Email Responses

Troubleshooting LEM Rules and Email Responses

Overview

The following scenarios to troubleshoot LEM rules that are not firing as expected or sending the expected notifications. For additional information about any of the procedures referenced in these scenarios, see the associated footnotes.

My rule fires, but I don't get an email.

Environment

  • LEM 6.2

Cause 

The cause of this is because in a rule you can use any email template you want but Best Practices is to use the name of the rule as the Subject of the Email when your building your email template under Build -> Groups > Email Templates.

SCENARIOS:

1.To find out what rule trigger email do the following

2. I see the alerts, but my rule doesn't fire.

3. My rule doesn't fire, and I don't see the expected alerts.

Resolution

To find out which rule triggered the email:

  1. Get the Time that's on the Email that the alert came in.
  2. Log into the LEM web or Adobe Air Console.
  3. Go to Explore tab -> then click on nDepth.
  4. On the far right there's a drop down for the default value is set to Last 10 Minutes.
  5. Click on the down arrow at the end and Select Custom Range.
  6. Put in 1 Minute before and after the Date and Time that you got the email.
  7. Then click on Events.
  8. Then in the search box type in "INTERNALRULEFIRED" and drag it to the top search bar beside the letter A on LEM.
  9. Then click on the Play button in blue on the Far Right of the Search bar.
  10. Then click on the Refine Fields that has a blue filter.
  11. Then click on InferenceRule to show all the rules that fired.
  12. Below the results there is an icon with 3 platters with lines on it that says Result Details click on that.
  13. Verify that the ExtraneousInfo field of the InternalRuleFired alert shows the associated email action in Email [recipient] format.

14. When you see a rule that fired an email to see what Events actually caused that rule to fire.

15. To see if the Event that triggered that rule matches your email click on the rule that fired email 

16. Click on the little Explore right above the time frame box that says Custom Range and Select Event

17. Then Click on Event Details on the top left of the window

18. In the bottom box click on the Event to get details for the Event and not the rule

 

The Rule triggers but I don't get an email:

  1. If that action is not present, add the Send Email Message action to the rule.
  2. Verify that the intended recipient has an email address associated with his LEM user account:
    1. Click the Build tab, and then select Users.
    2. Click the LEM user account associated with the intended recipient.
  3. If the Contact Information box is blank in the User Information pane, edit the user to add an email address.

Note: If you are unable to add an email address to an AD user, you may need to create a separate user and add the email to that user account, and then select that user in the email template.

  1. Verify that the Email Active Response connector is configured on your LEM Manager:
    1. Click the Manage tab, and then select Appliances.
    2. Click the gear icon next to your LEM Manager, and then select Connectors.
    3. On the Connector Configuration window, select Configured on the Refine Results pane.
  2. If Email Active Response is not in the list, clear the Configured check box, and then configure the missing connector.

 

 

You do not see the expected InternalRuleFired alerts in the default SolarWinds Alerts or Rule Activity filters under the Monitor in the LEM Console, nor do you see the alerts needed to fire your rule anywhere in your LEM Console.

To determine whether the requisite alerts are in your LEM Console, create a filter or nDepth search that matches the correlations in your rule. If the alerts are not present, complete the following procedure:

 

  1. Review the network devices that are sending syslog data to the LEM, and validate the configurations on that network device to send data. Verify that one of your devices is logging the events you want to capture. For example:

    • Remote logging devices, such as firewalls and web filters, should be logging your web traffic events.
    • Domain controllers and end-user computers should be logging domain-level and local authentication and change management events.
      Note: If you have multiple domain controllers, they will not all replicate every domain event. Each server only logs the events they execute.
    • Other servers, such as database servers and web servers, should be logging events associated with their particular functions.

 

  1. Validate if data is received by the LEM.
    • Validate if the LEM icons show syslog/agent connection:
      1. Syslog device IPs will appear in the GUI-console Manage > Nodes list as a pipe-Y symbol.
      2. Agent host names and IP addresses will appear in the GUI-console Manage  > Nodes list as a green plug icon.
    • Validate if data is being received by syslog facility or by the agent.
      1. If a network syslog device is sending syslog data to the LEM, you should be able to view the LEM syslog files for that data.
      2. Perform the following:
        • Open the vSphere/Hyper-V console to access the LEM.
          Note: You may also use a PuTTY session, port 32022, cmc user.
        • Enter the appliance menu, and enter the checklogs command.
        • View the syslog that was chosen by the network device. All of the data received in this area is UDP traffic received on port 514.
      3. Agent data is encrypted and more difficult to tell if it is received by the LEM.
  2. If your device is not in the Nodes list, configure computers by installing a LEM Agent, or configure other devices, such as firewalls, to log to your LEM appliance.After your device is in the list, continue to the next step.
  3. If your device is in the Nodes list, configure the appropriate connectors:
    1. To configure syslog connectors (manager connectors) on your LEM Manager for remote logging devices, click the Manage tab, and then click Appliances.
    2. Click the gear icon next to the Agent or Manager on which you want to configure the new connectors, and then select Connectors.
    3. Use the Search box at the top of the Refine Results pane to locate the appropriate connectors.
    4. Configure the connector according to your needs.
    5. To configure agent connectors, go to Manage > Nodes, select the gear icon next to the agent and edit the connectors

 

I SEE THE ALERTS BUT MY RULE DOESN'T FIRE

  1. Problem:

    You see the alerts required to fire your rule in the LEM Console, but your rule still doesn't fire.

    Steps to resolve:

  2. Verify that all of your rules have been activated in all open LEM Consoles:
    1. Click the Build tab, and then select Rules.
    2. If the Activate Rules button is not greyed out, click it. This synchronizes all of the changes you have made to your rules in the Console with your LEM Manager.
    3. Repeat these steps for all open LEM Consoles in your environment.
  3. Compare the InsertionTime and DetectionTime values in the alerts you expected to fire your rule.
  4. If the time is off by more than five minutes, verify and correct the time settings on your LEM appliance and any remote logging devices as necessary. See To view and modify the time on your LEM appliance.
  5. If none of the previous troubleshooting steps help, restart the Manager service on your LEM appliance. In general, consider doing this once every six months:
    1. Connect to your LEM virtual appliance using either the vSphere console view, or an SSH client like PuTTY.
    2. If you are using an SSH client, log in to your LEM virtual appliance using your CMC credentials.
    3. At the cmc> prompt, enter manager.
    4. At the cmc::cmm prompt, enter restart.
    5. Press Enter to confirm your entry.
      Note: Restarting the Manager service will make your LEM Manager unavailable for about one minute. However, no data is lost during this process.
    6. Enter exit twice to leave the CMC interface.

 

RULE FIRES BUT EMAIL IS BLANK

 

Problem:

You receive an email notification for the alert, but the fields in the custom email template are blank.

Steps to resolve:

  1. Click the Build tab, and then select Rules.
  2. Locate your rule, click the gear icon on the left and select Edit. You will notice that the fields in the Actions box are blank.
  3. Copy the event assigned to this rule. This is the string before the dot in the Correlation box.
  4. Click Events on the left pane and type the event in the search field.
  5. Drag the fields required in your rule from the Fields pane to populate the blank fields in the Actions box.
  6. Click Save to close the Rule Creation window.
  7. Click Activate Rules on the Rules window.

 

YOU HAVE SETUP RULE CORRECTLY DATA MATCHES NDEPTH BUT YOUR RULE STILL DOESN'T FIRE 

   The biggest issue here is Time needs to be correct on the LEM in order for the Response Time Frame in the Rule to Work         Correctly, so to view and modify the time on your LEM appliance:

  1. Connect to your LEM virtual appliance using either the vSphere console view, or an SSH client like PuTTY.
  2. If you are using an SSH client, log in to your LEM virtual appliance using your CMC credentials.
  3. At the cmc> prompt, enter appliance.
  4. At the cmc::acm prompt, enter dateconfig.
  5. Press Enter through all of the prompts to view the current date and time settings on your LEM applaince.
  6. By default, the LEM receives a time synchronization from the VM host computer. Without this, time on the LEM wil be off and rules may not fire. You will need to disable the time sync on the VM host computer, and enable the LEM to get time from an NTP server:
    1. At the cmc::acm prompt, enter ntpconfig.
    2. Press Enter to start the configuration script.
    3. Enter the IP addresses of your NTP servers separated by spaces.
    4. Enter y to verify your entry.
  7. Enter exit twice to leave the CMC interface.

 

***Might also check that the connector is turned on for the device that is sending the data, like if it is a Syslog device like Websense go to Manage > Appliance > gear beside name of LEM > check mark the configured box and make sure there is a green play button***

Additional Information

 

 

 

 

 

Last modified
05:32, 26 May 2017

Tags

Classifications

Public