Troubleshooting SEM Rules and Email Responses

10/17/2018 10:27 PM

5/30/2019 9:36 PM

The following scenarios and resolutions apply to Security Event Manager (formerly Log & Event Manager) rules that are not firing as expected or sending the expected notifications. For additional information about any of the procedures referenced in these scenarios, see the associated footnotes.

 

Security Event Manager

The cause of this is because in a rule you can use any email template you want but best practices is to use the name of the rule as the subject of the email when your building your email template under Build > Groups > Email Templates.

SCENARIOS:

1. I don't know which rule triggered the email.
2. The rule triggers but I don't get an email.
3. I see the alerts, but my rule doesn't fire.
4. My rule doesn't fire, and I don't see the expected alerts.

To find out which rule triggered the email

1. Get the time that's on the email that the alert came in.
2. Log in to the SEM web console or Adobe Air Console.
3. On the SEM toolbar, navigate to Explore > nDepth.
4. On the far right, click the drop-down list (default value is set to Last 10 Min) and select Custom range.
5. Enter one minute before and after the date and time that you received the email.
6. In the List pane, click Events.
7. In the search box, type INTERNALRULEFIRED, and then drag it to search bar at the top.
8. To the far right of the search bar, click the blue Play button.
9. In the List pane, click Refine Fields (blue filter icon).
10. To show all the rules that fired, click InferenceRule.
11. On the nDepth toolbar below, click Result Details . 
12. Verify that the ExtraneousInfo field of the InternalRuleFired alert shows the associated email action in Email [recipient] format.
13. When you see a rule that fired an email, see what events actually caused that rule to fire.
14. To see if the event that triggered that rule matches your email, click the rule that fired the email. 
15. On the upper right of the nDepth pane, click the Explore drop-down list, and then select Event.
16. On the upper left, click Event Details.
17. In the bottom box, click the event to get details for the event and not the rule. 

 

The rule triggers but I don't get an email

1. If that action is not present, add the Send Email Message action to the rule.
2. Verify that the intended recipient has an email address associated with his SEM user account:
   1. On the SEM toolbar, navigate to Build > Users.
   2. Click the SEM user account associated with the intended recipient.
3. If the Contact Information box is blank in the User Information pane, edit the user to add an email address.

   If you are unable to add an email address to an AD user, you may need to create a separate user and add the email to that user account, and then select that user in the email template.

4. Verify that the Email Active Response connector is configured on your SEM Manager:
   1. On the SEM toolbar, navigate to Manage > Appliances.
   2. Next to your SEM Manager, click the gear icon, and then select Connectors.
   3. On the Connector Configuration window, select Configured on the Refine Results pane.
5. If Email Active Response is not in the list, clear the Configured check box, and then configure the missing connector.

 

You do not see the expected InternalRuleFired alerts in the default SolarWinds Alerts or Rule Activity filters under Monitor in the SEM console, nor do you see the alerts needed to fire your rule anywhere in your SEM console.

To determine whether the requisite alerts are in your SEM console, create a filter or nDepth search that matches the correlations in your rule. If the alerts are not present, complete the following procedure:

 

1. Review the network devices that are sending syslog data to SEM, and validate the configurations on that network device to send data. Verify that one of your devices is logging the events you want to capture. For example:

   • Remote logging devices, such as firewalls and web filters, should be logging your web traffic events.
   • Domain controllers and end-user computers should be logging domain-level and local authentication and change management events.

     If you have multiple domain controllers, they will not all replicate every domain event. Each server only logs the events they execute.

   • Other servers, such as database servers and web servers, should be logging events associated with their particular functions.
2. Verify data is received by SEM.
   • Verify that SEM icons show syslog/agent connections:
     1. Syslog device IPs will appear in the SEM web console in the Manage > Nodes list as a pipe-Y symbol.
     2. Agent host names and IP addresses will appear in the SEM web console in the Manage  > Nodes list as a green plug icon.
   • Validate if data is being received by syslog facility or by the agent.
     1. If a network syslog device is sending syslog data to SEM, you should be able to view the SEM syslog files for that data.
     2. Perform the following:
        • Open the vSphere/Hyper-V console to access SEM.

          You may also use a PuTTY session, port 32022, cmc user.

        • Enter the appliance menu, and enter the checklogs command.
        • View the syslog that was chosen by the network device. All of the data received in this area is UDP traffic received on port 514.
     3. Agent data is encrypted and more difficult to tell if it is received by SEM.
3. If your device is not in the Nodes list, configure computers by installing a SEM agent, or configure other devices, such as firewalls, to log to your SEM appliance. After your device is in the list, continue to the next step.
4. If your device is in the Nodes list, configure the appropriate connectors:
   1. To configure syslog connectors (manager connectors) on your SEM manager for remote logging devices, navigate to Manage > Appliances on the SEM toolbar.
   2. Next to the agent or manager on which you want to configure the new connectors, click the gear icon, and then select Connectors.
   3. Use the search box at the top of the Refine Results pane to locate the appropriate connectors.
   4. Configure the connector according to your needs.
   5. To configure agent connectors, navigate to Manage > Nodes, select the gear icon next to the agent, and then edit the connectors

 

I see the alerts but my rule doesn't fire

Problem:

You see the alerts required to fire your rule in the SEM console, but your rule still doesn't fire.

Steps to resolve:

1. Verify that all of your rules have been activated in all open SEM consoles:
   1. Click the Build tab, and then select Rules.
   2. If the Activate Rules button is not greyed out, click it. This synchronizes all of the changes you have made to your rules in the console with your SEM manager.
   3. Repeat these steps for all open SEM consoles in your environment.
2. Compare the InsertionTime and DetectionTime values in the alerts you expected to fire your rule.
3. If the time is off by more than five minutes, verify and correct the time settings on your SEM appliance and any remote logging devices as necessary. See View and modify the time on your SEM appliance.
4. If none of the previous troubleshooting steps help, restart the manager service on your SEM appliance. In general, consider doing this once every six months:
   1. Connect to your SEM virtual appliance using either the vSphere console view, or an SSH client like PuTTY.
   2. If you are using an SSH client, log in to your SEM virtual appliance using your CMC credentials.
   3. At the cmc> prompt, enter manager.
   4. At the cmc::cmm prompt, enter restart.
   5. Press Enter to confirm your entry.
      Note: Restarting the Manager service will make your SEM manager unavailable for about one minute. However, no data is lost during this process.
   6. Enter exit twice to leave the CMC interface.

 

The rule fires but the email is blank

Problem:

You receive an email notification for the alert, but the fields in the custom email template are blank.

Steps to resolve:

1. Click the Build tab, and then select Rules.
2. Locate your rule, click the gear icon on the left and select Edit. You will notice that the fields in the Actions box are blank.
3. Copy the event assigned to this rule. This is the string before the dot in the Correlation box.
4. Click Events on the left pane and type the event in the search field.
5. Drag the fields required in your rule from the Fields pane to populate the blank fields in the Actions box.
6. Click Save to close the Rule Creation window.
7. Click Activate Rules on the Rules window.

 

You have set up the rule correctly, and data matches in nDepth, but the rule still doesn't fire

The primary issue is the time needs to be correct in SEM in order for the response time frame in the rule to work correctly.

To view and modify the time on your SEM appliance:

1. Connect to your SEM virtual appliance using either the vSphere console view, or an SSH client like PuTTY.
2. If you are using an SSH client, log in to your SEM virtual appliance using your CMC credentials.
3. At the cmc> prompt, enter appliance.
4. At the cmc::acm prompt, enter dateconfig.
5. Press Enter through all of the prompts to view the current date and time settings on your SEM appliance.
6. By default, SEM receives a time synchronization from the VM host computer. Without this, the time in SEM will be off and rules may not fire. You will need to disable the time sync on the VM host computer, and then enable SEM to get time from an NTP server:
   1. At the cmc::acm prompt, enter ntpconfig.
   2. To start the configuration script, press Enter.
   3. Enter the IP addresses of your NTP servers, separated by spaces.
   4. To verify your entry, enter y.
7. To exit the CMC interface, enter exit twice.

You can also check that the connector is turned on for the device that is sending the data. For example, if it is a syslog device like Websense, navigate to Manage > Appliances, click the gear icon beside name of the appliance, and then select Connectors. In the refine results pane, select the Configured check box, and then make sure there is a green status icon indicating the connector is running.

Additional Information 

• For general instructions for working with SEM rules, see Creating Rules from your SEM Console to Take Automated Action. For additional information about the specific procedures discussed in this article, see the following related articles according to your needs.
• For additional information about using email notifications in SEM rules, see Using the Send Email Message Action in Rule Creation.
• For additional information about creating SEM user accounts, see Create SEM Console users with domain credentials.
• For additional information about configuring the Email Active Response connector, see Configure the Email Active Response Connector.
• For additional information about installing SEM agents on Windows computers, see Using the SolarWinds SEM Remote Agent Installer. For articles related to installing SEM agents on other operating systems, browse or search the Agents category of the SEM knowledge base.
• For additional information about configuring remote logging devices to log to your SEM appliance, search the Connectors category of the SEM knowledge base.