Network Management
OrionLog database LogEntryMessage dump table grows due to disk full partition creation failure
The OrionLog_LogEntryMessage_dump table in the SolarWindsOrionLog database grows indefinitely when LogMaintenancePlugin fails to create partitions due to insufficient disk space (ErrorNumber 1101), causing a self-reinforcing cycle where the dump table consumes all available disk space and prevents partition creation.
First published date
Last published date
Overview
Environment: SolarWinds Platform 2022.4 and later
Applies To: Log Analyzer, Orion Log Viewer (OLV), SolarWinds Platform
Symptom
The SolarWindsOrionLog database grows to hundreds of gigabytes or over 1 TB. The OrionLog_LogEntryMessage_dump table contains the majority of the data, with rows dating back months or years. Database Maintenance reports a failure for LogMaintenancePlugin every night.
The following errors appear in C:\ProgramData\SolarWinds\Logs\Orion\swdebugMaintenance.log:
ERROR SolarWinds.Orion.LogMgmt.Maintenance.LogMaintenancePlugin - Failed to create partitions in Log , Attempt #1, ErrorNumber 1101
System.Data.SqlClient.SqlException (0x80131904): Could not allocate a new page for database 'SolarWindsOrionLog' because the 'PRIMARY' filegroup is full due to lack of storage space or database files reaching the maximum allowed size.
Followed by:
WARN SolarWinds.Data.DatabaseMaintenance.OrionNotificationHelper - Detected that operation [Plugin action]-[LogMaintenancePlugin->PreMaintenance for Orion.LogMgmt] failed 3 times in a row.
The SolarWinds web console displays the notification: "There was a Log Management database maintenance failure. Future partitions haven't been prepared correctly. Please collect diagnostics immediately and contact SolarWinds Technical support."
Product section
Cause
Starting in SolarWinds Platform 2022.4, Log Analyzer uses application-level table-based partitioning in the SolarWindsOrionLog database. Each day and log source combination gets its own OrionLog_LogEntryMessage_table. The Database Maintenance job (LogMaintenancePlugin) pre-creates these partition tables 30 days in advance.
The OrionLog_LogEntryMessage_dump table is a fallback table. When partition creation fails, incoming Syslog, Trap, and other log messages have no prepared partition table to write to. Instead of dropping those messages, Log Analyzer writes them into the dump table. The dump table is not subject to the normal retention cleanup cycle.
When partition creation fails due to ErrorNumber 1101 (insufficient disk space in the PRIMARY filegroup), the following self-reinforcing cycle occurs:
-
LogMaintenancePlugin cannot create new partition tables because the MDF has no available space.
-
All incoming log messages are written to OrionLog_LogEntryMessage_dump.
-
The dump table is not subject to retention cleanup, so it grows indefinitely.
-
The dump table consumes more disk space, preventing partition creation.
-
The cycle repeats.
Resolution
Resolution
NOTE: The steps can only be performed for version 2022.4 and newer. The SQL scripts are designed to be run using Microsoft SQL Server Management Studio (SSMS). This will not work via SolarWinds Database Manager.
NOTE: This procedure deletes all historical Syslog, Trap, Windows Event, Flat File, and VMware Event log messages stored in the SolarWindsOrionLog database. Node status, performance data, alerting, polling, NCM configurations, IPAM data, and all other SolarWinds data stored in the SolarWindsOrion database are not affected.
Step 1 — Confirm the dump table contains data
Run the following script in SSMS against the SolarWindsOrionLog database:
-- Scripts are not supported under any SolarWinds support program or service.
-- Scripts are provided AS IS without warranty of any kind. SolarWinds further
-- disclaims all warranties including, without limitation, any implied warranties
-- of merchantability or of fitness for a particular purpose. The risk arising
-- out of the use or performance of the scripts and documentation stays with you.
-- In no event shall SolarWinds or anyone else involved in the creation,
-- production, or delivery of the scripts be liable for any damages whatsoever
-- (including, without limitation, damages for loss of business profits, business
-- interruption, loss of business information, or other pecuniary loss) arising
-- out of the use of or inability to use the scripts or documentation.
DECLARE @logEntryCount AS bigint = 0
SELECT @logEntryCount = COUNT_BIG(*)
FROM OrionLog_LogEntryMessage_dump WITH(NOLOCK)
IF @logEntryCount = 0
PRINT N'No Log Entries stored in OrionLog_LogEntryMessage_dump table.'
ELSE
BEGIN
DECLARE @discSpaceInGB AS float = @logEntryCount * 1.13 / CONVERT(float, 86400000) * 130
PRINT N'There are ' + CONVERT(nvarchar, @logEntryCount)
+ N' Log Entries (approx. ' + CONVERT(nvarchar, @discSpaceInGB)
+ N' GB disk space) in OrionLog_LogEntryMessage_dump table.'
END
Step 2 — Truncate the SolarWindsOrionLog tables
-
Take a FULL backup of the SolarWindsOrionLog database.
-
Stop all SolarWinds services on ALL servers (MPE and all APEs) using the SolarWinds Service Manager.
-
In SSMS, connected to the SolarWindsOrionLog database, run the following query to generate the TRUNCATE statements for all OrionLog_LogEntryMessage tables:
-- Scripts are not supported under any SolarWinds support program or service.
-- Scripts are provided AS IS without warranty of any kind. SolarWinds further
-- disclaims all warranties including, without limitation, any implied warranties
-- of merchantability or of fitness for a particular purpose. The risk arising
-- out of the use or performance of the scripts and documentation stays with you.
-- In no event shall SolarWinds or anyone else involved in the creation,
-- production, or delivery of the scripts be liable for any damages whatsoever
-- (including, without limitation, damages for loss of business profits, business
-- interruption, loss of business information, or other pecuniary loss) arising
-- out of the use of or inability to use the scripts or documentation.
SELECT 'Truncate Table [dbo].[' + table_name +']'
FROM Information_Schema.tables
WHERE table_name LIKE 'OrionLog_LogEntryMessage_%'
AND table_name NOT LIKE 'OrionLog_LogEntryMessageSource'
AND table_type = 'BASE TABLE'
-
Copy the query results to a Notepad and add the following statements:
-- Scripts are not supported under any SolarWinds support program or service.
-- Scripts are provided AS IS without warranty of any kind. SolarWinds further
-- disclaims all warranties including, without limitation, any implied warranties
-- of merchantability or of fitness for a particular purpose. The risk arising
-- out of the use or performance of the scripts and documentation stays with you.
-- In no event shall SolarWinds or anyone else involved in the creation,
-- production, or delivery of the scripts be liable for any damages whatsoever
-- (including, without limitation, damages for loss of business profits, business
-- interruption, loss of business information, or other pecuniary loss) arising
-- out of the use of or inability to use the scripts or documentation.
TRUNCATE TABLE OrionLog_LogEntry WITH (PARTITIONS(1));
TRUNCATE TABLE OrionLog_LogEntryFieldValue WITH (PARTITIONS(1));
TRUNCATE TABLE OrionLog_LogEntrySecondarySourceAssignment WITH (PARTITIONS(1));
TRUNCATE TABLE OrionLog_LogEntryTagAssignment WITH (PARTITIONS(1));
GO
-
Execute all the statements together in SSMS. TRUNCATE is a minimally-logged operation that deallocates data pages — it completes within seconds regardless of the table size.
-
After the truncation completes, verify the space has been freed inside the database:
-- Scripts are not supported under any SolarWinds support program or service.
-- Scripts are provided AS IS without warranty of any kind. SolarWinds further
-- disclaims all warranties including, without limitation, any implied warranties
-- of merchantability or of fitness for a particular purpose. The risk arising
-- out of the use or performance of the scripts and documentation stays with you.
-- In no event shall SolarWinds or anyone else involved in the creation,
-- production, or delivery of the scripts be liable for any damages whatsoever
-- (including, without limitation, damages for loss of business profits, business
-- interruption, loss of business information, or other pecuniary loss) arising
-- out of the use of or inability to use the scripts or documentation.
USE [SolarWindsOrionLog];
GO
EXEC sp_spaceused;
GO
These steps do not reduce the size of the main database file. After the truncation, the MDF file will remain the same size on disk, but internally the space will be mostly empty and available for SQL Server to reuse for new data and partition tables. This is sufficient to resolve the Error 1101 (filegroup full) condition.
Reference: Truncate the SolarWindsOrionLog tables in OLV/LA
-
Start all SolarWinds services on all servers.
Step 3 — Run Database Maintenance and verify partition creation
-
Run Database Maintenance manually from Start > SolarWinds Platform > Database Maintenance (DatabaseMaint.exe) on the main SolarWinds server. Confirm it completes without errors.
-
After Database Maintenance completes, run the following query in SSMS against the SolarWindsOrionLog database to confirm that partition tables have been created:
-- Scripts are not supported under any SolarWinds support program or service.
-- Scripts are provided AS IS without warranty of any kind. SolarWinds further
-- disclaims all warranties including, without limitation, any implied warranties
-- of merchantability or of fitness for a particular purpose. The risk arising
-- out of the use or performance of the scripts and documentation stays with you.
-- In no event shall SolarWinds or anyone else involved in the creation,
-- production, or delivery of the scripts be liable for any damages whatsoever
-- (including, without limitation, damages for loss of business profits, business
-- interruption, loss of business information, or other pecuniary loss) arising
-- out of the use of or inability to use the scripts or documentation.
SELECT table_name
FROM Information_Schema.tables
WHERE table_name LIKE 'OrionLog_LogEntryMessage_%'
AND table_name NOT LIKE 'OrionLog_LogEntryMessageSource'
AND table_type = 'BASE TABLE'
ORDER BY table_name
Multiple OrionLog_LogEntryMessage_<PartitionID> tables should be listed alongside the existing OrionLog_LogEntryMessage_dump table. If the only result is OrionLog_LogEntryMessage_dump with no partition tables, LogMaintenancePlugin is still failing — review swdebugMaintenance.log for the specific error.
Note
-
The "Table is partitioned" property in SSMS Table Properties > Storage will show "False" for all OrionLog tables. This is correct and expected. SolarWinds Log Analyzer uses application-level table-based partitioning (separate physical tables per partition), not SQL Server native table partitioning (partition functions/schemes).