Submit a ticketCall us

AnnouncementsFace your biggest database issues head-on

Our new eCourse helps you navigate SQL Server performance blocks by teaching you how to recognize and deal with the three DBA Disruptors: Performance Hog, Blame Shifter, and Query Blocker. Register today to learn how to defend your environment and fend off menacing disruptions.

Register for your free eCourse.

Home > Success Center > Log & Event Manager (LEM) > LEM - Knowledgebase Articles > LEM Logs/Data Partition has reached 100%

LEM Logs/Data Partition has reached 100%

Created by Craig O’ Neill, last modified by Tim Rush on Feb 12, 2018

Views: 1,318 Votes: 2 Revisions: 8

Updated 8/25/2016

Overview

This article provides brief information and steps to resolve the issue when you are unable to log in to LEM. 

After recently adding a device or devices you find that your LEM Logs/Data partition has reached 100%.

Environment

  • All LEM versions
  • Receiving syslog from Enterprise Firewall(s)
  • Recently added syslog devices

 

Cause 

Just for clarity, LEM receives syslog data from various network devices, but can only push that data into LEM's internal database if a connector is configured to read/interpret (normalize) the incoming data. Without a connector, data is received into the syslog, and dropped depending on the syslog rotation/retention. This means syslog data (withput a connector) will not make it to the LEM database.

LEM 6.3.1 (and before) have default settings to collect syslog into temporary files (referred to as syslog facilities), and by default, the log files rotate daily. In most deployments, default works fine. If a firewall was set into debug mode, or in larger environments with high volume syslog, we need to rotate the syslog more often, like hourly. This prevents the size of syslog from growing too large and displacing the database (based on overall size of LEM).

Any of the following can cause the issue:

  • You are receiving a quantity of single-device syslog data which exceeds 8GB in 1 day.
  • One day is the default rotation time for these log files.
  • LEM (being a Linux-based appliance) cannot reliably handle text (syslog) files which exceeds 8GB in size.

Resolution

  1. If unsure how to proceed, open a ticket with SolarWinds Support to help identify the log files collecting this data, or the IP addresses sending the data. Support can reduce or clear out large syslog files, and assist with connector configurations.
  2. To view the syslog, open a Vshere/Hyper-V console (or putty session), enter the "checklogs" command under the "appliance" menu to view the current size of the syslog files since the default log rotation at 6:30am each day. If already changed to hourly, the log rotation should happen at 17 minutes after each hour of the day. It will require Solarwinds support to clear out the large syslog files, or just just wait a couple of days for the hourly rotation to clear out some of the log files automatically.
  3. Use the cmc → appliance setlogrotate command and change the syslog rotation to hourly. This will prevent recurrence by rotating the logs every hour instead of every day so they aren't able to grow to an excessive size.
  4. By default we do limit the number of syslog files to 50, but this can be set from 1 to 100 to keep up the 100 days worth of syslog, or if set to hourly rotation, up to 4 days (+4hours) worth of syslog data. To limit the number of syslog files, use the cmc → appliance limitsyslog command.

 

RELATED ARTICLE: https://support.solarwinds.com/Succe...log_data_files

 

 

Last modified

Tags

Classifications

Public