Kiwi Syslog service Crashing after 1.Gb System.OutOfMemoryException

4/22/2020 10:00 AM

6/21/2022 10:12 AM

You are running Kiwi Syslog and the Service is crashing frequently consuming above 1 GB of memory. 

You have collected Kiwi Syslog Tech Support Logs and found the following Error in the Kiwi Syslog Errorlog.txt file.

2020-04-08 14:34:48    *** INTERNAL PROGRAM ERROR - Please contact http://www.kiwisyslog.com/support/ ***
2020-04-08 14:34:48    Service Version =9.6.7.1 | Error Number: -2147024882 | Description: Exception of type 'System.OutOfMemoryException' was thrown. | Module Name: Syslogd.frm | Procedure Name: MessageCheckTimer_Timer | Line Number: 200 | Date and time: 08/04/2020 2:34:48 PM
2020-04-08 14:37:32    *** INTERNAL PROGRAM ERROR - Please contact http://www.kiwisyslog.com/support/ ***
2020-04-08 14:37:32    Service Version =9.6.7.1 | Error Number: -2147024882 | Description: Exception of type 'System.OutOfMemoryException' was thrown. | Module Name: Syslogd.frm | Procedure Name: MessageCheckTimer_Timer | Line Number: 200 | Date and time: 08/04/2020 2:37:32 PM
2020-04-10 20:12:45    *** INTERNAL PROGRAM ERROR - Please contact http://www.kiwisyslog.com/support/ ***

Kiwi Syslog Server

You will see 'System.OutOfMemoryException' messages in the log files because Kiwi Syslog can only support a certain number of events coming into it. So, the real issue for now is a single 32-bit process under a 64-bit OS is limited to 2GB. In fact, according to the load test, the service crashed after RAM usage was 1.4 G more or less around which we tested with the number of messages. 

The licensed version has been increased to have a default 500,000 message buffer, while the free version has a maximum 500 message buffer. The licensed version allows you to set a maximum buffer of 10,000,000 messages.

Refer: Kiwi Syslog Server Maximum number of messages for Kiwi Syslog Daemon

This above article explains the maximum number of messages that the Kiwi Syslog Daemon can handle.

• The licensed version of the software can handle around 2 million messages per hour and the free version handles about 300,000 per hour. The licensed version has been regularly tested to handle 400-600 messages per second while logging into a file.
• The licensed version has been increased to have a default 500,000 message buffer, while the free version has a maximum 500 message buffer. The licensed version allows you to set a maximum buffer of 10,000,000 messages.
• If you suspect that you may be losing messages, Go to the File > Debug options > View message buffer option to check that the Message Queue overflow value is always 0. This indicates the number of messages that have been dropped.
• If you are running the Service version then this same information can be found from the Manage > Debug options menu.
• To decrease the number of messages being displayed, modify your device configurations to only send messages that meet a set level.

So, it is all about the number of messages coming into Kiwi Syslog which is causing the issue where the service cannot handle the amount. This article talks about the registry max value - Refer: MsgBufferSize

 

However, in Kiwi, there is a max number of messages it can handle, for example, 2 Million per hour and the burst is handled by a message buffer. However, it is limited to 1.4GB as it is a 32-bit application.
 

Workarounds that you can do here are as follows:

1. Reduce the number of network devices sending Syslog traffic to the Kiwi Syslog server. Only configure the core network devices.

2. Configure your network device to stop sending Syslog messages with a level of "Notice," "Information," and Debug. A message with levels of "Warning" and above are usually enough.

3. Logging Severity 4 and above only, not to include 5-7 these messages are chatty and provide too much detail on non-severity, the lower the value, the higher the severity.

   • Integer	Severity
     0	Emergency: System is unusable.
     1	Alert: Action must be taken immediately.
     2	Critical: Critical conditions.
     3	Error: Error conditions.
     4	Warning: Warning conditions.

4. Refer Load balance the Kiwi Syslog Server KB to add additional Kiwi Syslog Server for load balancing :

How to Collect Standard Kiwi Syslog  Diagnostic

• Open Kiwi Syslog Service manager.
• Go to File > Create Tech-Support File (Zip).
• The .zip file is generated automatically. The location of the file is indicated on the message window that is displayed once the file is successfully created.
• You should also gather the following information and include them in the packet passed to SolarWinds support:
  • Physical\Virtual Machine.
  • CPU/Memory usage, OS version, and others. Run msinfo32 command to get the system information.
  • Check EventLogs for issues and, if possible, export Eventlogs on that System.
  • Screenshot(s) of the issue and/or any error messages.​
  • Kiwi Web Access Logs
  • C:\Program Files\UltiDev\Cassini Web Server for ASP.NET\2.0\UltiDevCassinWebServer2a.trace.log
  • C:\Program Files (x86)\SolarWinds\Kiwi Syslog Web Access\html\KiwiSyslogWebAccess.log
  • Additional Info: See Start-up Debug for other debug logs information.

Refer to the Syslog_Diagnostics.txt log file to check the amount of Syslogs events sent by each device and to optimize your setup and try to determine ways to reduce message per minute.

Other SolarWinds Products for Log Monitoring: 

 

• SolarWinds Security Events Manager (Formerly LEM) might be the correct product for you to handle your firewalls scaling as high as 400M events per day.
• Refer; SEM 2022.2 System Requirements for more details quotation contact SolwarWinds customer services here Working with Support | SolarWinds Success Center
• SEM Demo Link