Submit a ticketCall us

Don’t fall victim to a ransomware attack
Backups are helpful, but sometimes that’s not enough to protect your business against ransomware. At our live webcast we will discuss how to protect against ransomware attacks with SolarWinds® Patch Manager and how to leverage log data to detect ransomware. Register now for our live webcast.

Home > Success Center > Kiwi Syslog Server > Summarize data for the Web Console and Events.sdf

Summarize data for the Web Console and Events.sdf

Table of contents
Created by marvin m tomelden, last modified by MindTouch on Jun 23, 2016

Views: 39 Votes: 0 Revisions: 5

Overview

This article provides information about summarizing data for the Web Console and Events.sdf.

Environment

Kiwi Syslog version 9.x

Detail

Events.sdf

  • Stores all Syslog messages for the Web Console
  • The web client only stores and displays 4 GB of data/events.
  • If you remove/rename the events.sdf, it only affects the data in the Kiwi Syslog Web Access Console.

 

Events.sdf in previous versions

Kiwi Syslog v9.1 and earlier

  • There was only one SDF File
  • Max Size 4GB.
  • Once it reached the 4GB Limit, the oldest 10% of messages were removed on the fly.

Issue:

This is acceptable for most user, however, for large environments or where a very large number of syslogs were being received, the rolling 10% summarization was unable to keep up. SDF files have a 4GB file size limit, and when the summarization was unable to keep up, this lead to the 4GB file size limit being breached, corrupting the file. The only way to resolve this issue was to start with a fresh Events.sdf file.

 

Kiwi Syslog v9.2

  • The summarization issue in 9.1 was handled, so this couldn't occur - the 4GB file limit cannot be breached.
  • New DB Maintenance Feature “Max Message Age” was introduced.

 

Summarizing data for the Web Console

How the Events.sdf file works (applies to both v9.1 and earlier, and v9.2 and later).

  • The Events.sdf Database is implemented as a set of sub-files.
  • 64MB max each and up to 64 files 
  • Management of DB size is done by dropping one or more of these sub-files files when limit condition is met, ether size or time.
  • This allows the maintenance to remove messages in chunks of 64MB rather than removing them on a message-by-message basis.

 

 

 

Handling the Time Limit

  • In a case of time limit the last message written to the file is checked.
  • If the difference between "now" and the timestamp of the message is more than the configured time limit the sub-file file is marked for deletion. Otherwise it stays in DB. 
  • Because of how this is handled, if there is very little messages being stored in the Events.sdf file, there may be a number of messages in the sub-file much older than the 'maximum event age'. This is normal and expected behaviour, as the sub-file will only be deleted once the newest message in that subfile is older than the 'maximum event age' setting - essentially, all messages in that subfile must be older than the Maximum Event Age before it is marked for deletion.
  • The goal of the time limit setting is to ensure that any messages newer than that time setting remain in the database, but that the database size is maintained and managed. This should not be confused with filtering - if a user only wants to see messages from the last 24 hours, they should use filters instead.

 

Last modified
19:51, 22 Jun 2016

Tags

Classifications

Public