Tips and tricks for managing traps and syslog in Orion NPM

10/30/2018 2:29 PM

10/25/2022 10:16 PM

This article provides recommended settings in environments with large syslog / traps tables that have a direct impact on database size. This post is part of Quick Orion database health check guide. SolarWinds strongly recommends viewing this guide before you proceed. 
 

Orion Log Viewer is the new and improved way to collect, search, filter, and alert on syslogs and traps from within your Orion Web Console. It uses the same alert engine as all Orion Platform products, allows you to correlate logs with performance metrics, and will reduce the strain on your Orion database by sending them to a separate database specifically designed for log data.

Orion Log Viewer is available to users of NPM 12.3 or newer for free by downloading and installing a trial of SolarWinds Log Analyzer . Even if your Log Analyzer trial expires, the Orion Log Viewer functionality stays activated in your NPM instance. The Log Analyzer feature comparison lists the features available with the legacy Syslog and Trap Viewers, Orion Log Viewer, and Log Analyzer. 

Requirements to receive Syslog and Traps from a Node:
Node must be added in Orion first to receive the Traps or Syslog Minimum ICMP and better with SNMP or Agent 
 

• Log Manager Licensing

  • By default, when log data is received by a node, that node will consume an LA license. 

    • However, this can be overwritten and you can exclude nodes from automatically consuming a license if log data is received. 

  • You can also add/remove nodes from the LA license pool, in a similar manner to NCM. It is not possible to go over the license. 

    • Once you hit your license limit you will get a notification to inform you that you have reached the limit and it will not be possible to add additional nodes at that point.

    • Therefore logs will be automatically discarded until you free up some licenses or upgrade to a higher tier.

  • If a node is excluded from consuming a license, any log data transmitted to LA from that node will be discarded until it is added to the LA license pool.

    • The logs will not be processed under basic functionality.

  • As part of the licensing framework, LA receives messages from all nodes the Orion Platform manages.

    • However, messages sent from an unknown node prompt a dropped message notification in LM, which requires user action to map the node to the Orion Platform.

Orion Platform

n/a

Traps Filter Plan 

The best way to maintain the size of traps tables is to change the retention settings. This can be set in the Trap Viewer settings. By default, traps are retained for seven days. Reduce the time to keep the size of the database smaller.

SolarWinds suggests checking the Trap Viewer for the types of traps being received. If you receive a lot of info/debug severity messages from a device, the device itself can be set up to only send higher severity messages. Your vendor should be able to provide configuration commands for sending traps on the device.
 

It would help if you also looked for traps being received by the Trap Viewer that you do not intend to keep. You can create a new rule to discard traps by right-clicking and choosing Add Rule. It should automatically fill out all tabs of the new rule to match that trap. Use wildcards ( * ) as appropriate to expand what the rule will match to. Add the actions to the following: Discard the Trap Message and Stop Processing Trap Rules. 

When using Log Analyzer Core 2019.4, You can create a rule to Discard all the Traps as below. 
If you select 'Alerts and Activity'>'Traps'>'Configure Rules', you will be able to create new rules through the Orion RuleBuilder page by clicking on the option 'Create a Rule.'

1. Give a name to your new rule
2. configure the conditions according to your needs
3. and add the action 'Flag for discard.'

Trap rules are checked in order from top to bottom. Place these discard rules at the top of the list to ensure that these messages are discarded first and that no other rules are checked against those messages.

Syslog Filter Plan 

Option 1

An easy solution is to stop the Orion Syslog Service. This stops the Syslog table from growing again.
 

Option 2

1. Edit your Syslog Retention settings to keep syslogs for x days.
2. Tune the Severity levels for the syslog output on your devices to Warning or above.
3. Launch the Syslog view on the server and go to Server Settings.
4. On the first tab, reduce the number of days data is kept.

Option 3

Configure a device to stop sending some or all syslog messages.
 

Option 4

Syslog Message comes to Orion through the Syslog Service. Use Rules from the Syslog Viewer to determine whether you want to store the syslog message in the database or discard the message.

If you have a definite need for level 5 (notice) or above, review the data retention settings in the syslog application within the Orion Platform. Alternatively, you could use filter rules so that the ones that filter and discard messages are at the top of the list. This ensures that they are processed first.

SolarWinds recommends making sure that all rules that are set up to discard messages (such as the Discard Syslog Message rule) contain the line Stop processing syslog rules.

The syslog and traps filter/rules function differently compared to the Orion alerting engine. Each time a syslog message or trap is received, it work through each rule--beginning from the top--until it either reaches the end or hits a rule that specifically tells it to stop processing further rules (such as Stop Processing Syslog Rules). 

Discard Syslog Message

1. Choose Start > Program Files > Solarwinds > Orion > Syslog Viewer.
2. From this tool, go to File > Syslog Server settings > Alert/Filter Rules Tab.
3. Filter using various methods, such as IP address, by Message Type Patterns, Syslog Message Patterns, Severity, and so on.
4. Add the following alert actions to your rules: Discard Syslog Messages and Stop processing syslog rules.  
5. Rearrange the syslog rules so that the ones that filter and discard messages are at the top of the list. This ensures that they are processed first.

Example Rule Screenshots

Syslog / Traps message retention  

Do not change your Default database retention settings for the Orion Platform. Any change can cause database size to increase, causing traps / syslogs to grow quickly and impact your database performance. 

Storing large log, traps, and syslog files from several devices for auditing

SolarWinds Log and Event Manager is a SIEM tool that supports more demanding environments. This tool stores event logs for security, compliance, and troubleshooting that you can use later on for auditing purposes.

SEM is bench marked to 200 million events per day but we regularly see customer with 500 millions plus events per day,
This article discusses the amount of traffic that SEM can process per day.
SEM traffic limitations (solarwinds.com)

90->130 million events/day a 64GB of RAM & 10-CPUs
130->200 million events/day a128GB of RAM & 12-CPUs

In terms of retention, its based on disk size and not time based. It uses intensive compression ratios of 40:1 to 60:1.
This article answers common questions about SEM data storage such as, "How long is data stored in SEM?" and "Can I increase the size of the SEM database?"
Live data storage retention in SEM (solarwinds.com)

SEM reports: Create reports for regulatory and compliance purposes
SEM reports: Create reports for regulatory and compliance purposes (solarwinds.com)

The Security Event Manager (formerly Log & Event Manager) is a virtual appliance that is an all-in-one SIEM tool IT and security pros used to simplify detecting and investigating security issues using event log data. The SEM appliance makes use of various methods to secure the event data stored within to prevent tampering and ensure strict compliance.
SEM Appliance Security Information (solarwinds.com)

Use the intuitive search builder to create custom search queries. To conduct custom searches, navigate to Historical Events in the SEM console.
Create a search query (solarwinds.com)

SolarWinds Hybrid Cloud Observability security integration
SolarWinds Hybrid Cloud Observability security integration

For advanced alerting, see the following video: 

Alertapalooza: Syslogs, Traps, and Advanced Alerting - SolarWinds® Lab #3 - YouTube (© 2018 YouTube, available at https://www.youtube.com/watch?v=y-EW1XoP7dE, obtained on October 19, 2018.)

 

Disclaimer: Please note, any content posted herein is provided as a suggestion or recommendation to you for your internal use. This is not part of the SolarWinds software or documentation that you purchased from SolarWinds, and the information set forth herein may come from third parties. Your organization should internally review and assess to what extent, if any, such custom scripts or recommendations will be incorporated into your environment.  You elect to use third-party content at your own risk, and you will be solely responsible for the incorporation of the same if any.