Submit a ticketCall us

Have You Auto Renewed? If not, you're missing out.
The SolarWinds Renewal Program comes with a host of benefits including the most recent product updates, 24/7 technical support, virtual instructor-led training and more. Experience all of this with the convenience of Auto Renewal, and never worry about missing any of these great benefits. Learn More.

Home > Success Center > Log & Event Manager (LEM) > LEM Administrator's Guide > Manage LEM data storage

Manage LEM data storage

Updated: September 15, 2017

This topic addresses LEM database management.

About the three LEM data stores

By default, the LEM database is allowed 230 GB of the 250 GB allocated to the LEM virtual appliance. This partition consists of three data stores:

  • Syslog store
  • Events store
  • Original or raw log data store (optional)

The syslog storeconsists of all syslog or SNMP log data sent to the LEM VM. LEM reads and processes the data in real time, and then sends it to the event store for long-term storage. LEM stores the original data for 50 days in its original format (in case you need to review it). The data in the syslog store is compressed and rotated daily to maintain a consistent 50-days worth of data. The amount of data stored here should level off at around the 50-day mark.

The event store (the second store) contains all normalized events generated by the LEM Manager and LEM Agents. Data in this store is compressed at ratios of 40:1 to 60:1, which equates to an average compression rate of 95–98%. Both nDepth and the LEM reports application query the event store for event data when they run.

The original log store (the third store) is an optional store for original or raw log messages that can be searched using Log Message queries in nDepth. The data in this store can come from LEM Agents or other devices logging to the LEM appliance. You can configure if data is sent to this store at the connector level, so not all devices have to store raw log messages in this manner.

For more information about storing original log messages, see Configure LEM to store original log messages (nDepth log retention).

Strategies for managing your LEM data storage needs

Depending on the needs of your environment, you can use one or more of the alternate storage methods listed below.

  • Back up the LEM VM on a regular basis. This will provide offline storage for your LEM data stores.
  • Decrease the number of days that syslog/SNMP data is stored in LEM.
  • Deploy another LEM VM to be used as a syslog server.
  • Deploy another LEM VM to be used as a database server.
  • Increase the space allocated to your LEM VM.

To get help with any of these methods, submit a ticket to Customer Support:

https://customerportal.solarwinds.com/support/submit-a-ticket

Viewing LEM database usage numbers

There are three locations to find metrics that indicate how the LEM database is used:

  • Disk Usage summary in the CMC
  • Database maintenance report
  • Log storage maintenance report

View the Disk usage summary

When you use the command line to log in to LEM, LEM automatically generates a Disk Usage summary. You can also generate an ad hoc disk usage summary by running the diskusage command from the cmc >appliance prompt. The two lines to note here are Logs/Data and Logs.

  • The Logs/Data figure represents the total space being utilized by the LEM database. This value is presented in the percent % (usedG/allocatedG) format, where percent is the percent of the allocated space currently being used, and allocated is the total amount of space currently allocated to the LEM database.
  • The Logs figure represents the amount of space used by the syslog store. This figure is included in the used figure noted above. To figure out how much space is currently being used by the Event store, subtract the Logs value from the used value. If you are storing original log messages in the LEM database, the above calculation shows the combined space utilized by both your Event and original log stores.

View the Database maintenance report

Run the Database Maintenance Report in LEM reports to view a snapshot of your current database usage. The report includes the following values:

  • Disk Usage Summary – provides disk usage values in terms of the percentage of space allocated to the LEM database
  • Disk Usage Details – provides disk usage values in terms of physical file size
  • Database Time Span (days) – shows how many days worth of live event data is currently stored in the LEM database
  • Other Files – represents the amount of space used by the syslog store

For more information, see the following KB article in the Customer Success Center:
"Use the LEM Database Maintenance Report to See Retention and Volume of Traffic"


For more information, see the following KB article in the Customer Success Center:
"Use the LEM Database Maintenance Report to See Retention and Volume of Traffic." /Success_Center/Log_Event_Manager_(LEM)/Use_the_LEM_Database_Maintenance_Report_to_See_Retention_and_Volume_of_Traffic

View the Log storage maintenance report

Run the log storage maintenance report in LEM reports to get detailed information about the original log store. If you have not enabled LEM to store original log messages, this report will be blank.

For more information, see the following KB article in the Customer Success Center: "Live Data Storage Retention in LEM."

Create a disk usage alert in LEM to warn you when a disk reaches a set limit

You can create a disk usage alert from the CMC command line to warn you when a disk partition reaches a preselected use limit. When the limit is reached, an InternalWarning event displays in the Monitor view.

You can define the disk use limit by the percentage of unavailable disk space (such as 75%), or by the amount of free disk space (such as 58G).

To create the disk usage alert:

  1. Open the CMC command line. See Log in to the LEM CMC command line interface for steps.

  2. At the cmc> prompt, enter appliance to access the Appliance menu.

    File:Success_Center/Reusable_content_-_InfoDev/LEM/Log_and_Event_Manager_Administrator's_Guide/LEM_Administrator's_Guide/0310-Set_up_single_sign-on_SSO_in_LEM/lem_qsg_putty_log_in4_400x225.png

  3. At the cmc::appliance> prompt, enter diskusage to view the disk use of each partition. For example:

    cmc::appliance > diskusage
    Checking Disk Usage (this could take a moment)
    ... ....00.00.00.00.00.00.00.
    Partition Disk Usage:
    LEM:             35% (991M/3.0G)
    OS:              45% (1.3G/3.0G)
    Logs/Data:        1% (901M/234G)
    Temp:             2% (252M/5.9G)
    
    Database Queue(s): 4.0K (No alerts queued, 0 alerts waiting in memory)
    Rules Queue: 2.1M (0 alerts queued, 0 alerts waiting in memory)
    Console Queue: 2.1M (0 alerts queued, 0 alerts waiting in memory)
    DataCenter Queue: 2.1M (0 alerts queued, unknown number of alerts waiting in memory)
    EPIC Rules Queue: 2.1M (0 alerts queued, 0 alerts waiting in memory)
    Forensic Database Queue: 2.1M (0 data queued, unknown number of data items waiting in memory)
    Logs: 1.3M
    Tool Profiles Message Queue: 2.1M (0 alerts queued, unknown number of alerts waiting in memory)					
    
    cmc::appliance > 
  4. At the cmc::appliance> prompt, enter diskusageconfig.

    Each partition and corresponding disk use limit displays on your screen. For example:

    cmc::appliance > diskusageconfig
    Current Disk Usage Configuration:
    # | Partition (filesystem) | Configured limit
    ===============================================
    1 |    LEM (/usr/local)    |        90%
    2 |    OS (/)              |        90%
    3 |    Logs/Data (/var/)   |        10G
    4 |    Temp (/tmp)         |        90%
    You can define your disk use limit by the percentage of unavailable disk
    space (such as 75%) or the amount of free disk space (such as 58G). 
    Enter the partition number you want to change (enter 'exit' and press 
    <Enter> to quit): 
  5. Enter the partition number you want to change, and then press Enter.
  6. Enter the disk usage limit value in percentage (such as 75%) or size (such as 58G), and then press Enter.

    For example, to change the OS disk partition limit in step 3 from 45% to 40%, enter 40%. To change the OS disk partition limit from 1.3 GB to 2.0 GB, enter 2GB.

    Disk usage limit [90%, sizeK, sizeM, sizeG, sizeT] (default 90%): 40%
    Limit '40%' for the 'OS' partition is set.
    Press <Enter> to set the next partition. Enter 'exit' and press <Enter>
    to quit: 
  7. Press Enter to set the next partition and repeat step 6 (if required).

    See the next section (Change the Logs/Data partition setting) for additional information.

  8. When you are finished, type exit, and then press Enter to quit.

Change the Logs/Data partition setting

When you set the Logs/Data partition (3), a message prompts you to consider changing the database disk configuration using the dbdiskconfig command. SolarWinds recommends setting the Logs/Data partition and the database disk configuration to the same value.

 

To change your database disk configuration:

  1. Finish configuring your partitions.
  2. At the cmc::appliance> prompt, enter dbdiskconfig.

    The following message displays:

    Current configuration:
    DoNotExceedPercentage = 90%
     
    The Manager will restart and apply your changes. To exit, enter 'exit' and press Enter.
    Enter a new value for DoNotExceedPercentage (default 90):
    Please enter an inter number 0-100 or 'exit'
  3. At the prompt, enter a usage limit value between 0 and 100, and then press Enter.

    If you enter a value less than 25, the partition will be deleted when this value is reached.

    The database disk configuration value is saved, and the appliance restarts the Manager Service.

View a disk usage event

Log in to your LEM console as an administrator and click to open Monitor view. The event displays in the All Events grid.

For example, if you set the OS disk partition limit as a percentage, the following event displays in the All Events grid when the limit is reached:

File:Success_Center/Reusable_content_-_InfoDev/LEM/Log_and_Event_Manager_Administrator's_Guide/LEM_Administrator's_Guide/0240-Manage_LEM_data_storage/lem-ug-set-the-disk-use-limit4_412x39.png

If you set the OS disk partition limit as a file size, the following event displays in the All Events grid when the limit is reached.

File:Success_Center/Reusable_content_-_InfoDev/LEM/Log_and_Event_Manager_Administrator's_Guide/LEM_Administrator's_Guide/0240-Manage_LEM_data_storage/lem-ug-set-the-disk-use-limit5_414x38.png

Select the event in the grid and review the content in the Event Details tab for additional information.

Last modified

Tags

Classifications

Public