Observability

RabbitMQ queues are experiencing crashes after upgrading to SolarWinds Platform versions 2024.4, 2024.4.1, and 2025.1

After the upgrade to SolarWinds Platform 2024.4, 2024.4.1, and 2025.1, RabbitMQ queues are crashing.

First published date

11/13/2024 10:01 PM

Last published date

8/14/2025 2:07 PM

Overview

In 2024.4, 2024.4.1, or 2025.1, RabbitMQ queues to classic v2 from v1.  After this change, rabbitmq starts to crash after attempting to restarted crashed queues with the following crash dump in the log file (C:\ProgramData\SolarWinds\Orion\RabbitMQ\log\rabbit@SERVERNAME.log):

 

[error] <0.4133252.0>   crasher:
[error] <0.4133252.0>     initial call: rabbit_channel:init/1
[error] <0.4133252.0>     pid: <0.4133252.0>
[error] <0.4133252.0>     registered_name: []
[error] <0.4133252.0>     exception exit: {{{{badmatch,true},
[error] <0.4133252.0>                        [{rabbit_classic_queue_index_v2,init,3,
[error] <0.4133252.0>                             [{file,"rabbit_classic_queue_index_v2.erl"},
[error] <0.4133252.0>                              {line,183}]},

 

 

The symptoms are the following, but not limited to:

  • Alerts are delayed or do not appear in a timely manner until a report occurs
  • Consistent performance problems where RabbitMQ queues are slow to process messages
  • RabbitMQ queues are not clearing and they are growing
  • Traps are not creating alerts

Product section

Hybrid Cloud Observability

Cause

There are some issues with the Classic v2 queues in some environments, causing it to crash. This is a known issue that impacts the following versions:

  • SolarWinds Platform 2024.4
  • SolarWinds Platform 2024.4.1
  • SolarWinds Platform 2025.1

Resolution

Note:

The fix for this issue was included in the 2025.1.1 release. In this version release, the queues are the Classic V1. Upgrade to the latest release to ensure the issue is resolved. 

 

If you are unable to upgrade to this version, you can try the below workarounds:

Recommended Workaround: Revert queues to Classic v1 through command prompt

If you are using this workaround, upgrading to version 2025.1.1 is not strictly necessary. 

If you are on SolarWinds Platform 2024.1, then perform the below steps:

  1. Revert to Classic Queues v1 by executing the command below in an administrative command prompt. 
"%ProgramFiles%\SolarWinds\Orion\RabbitMQ\sbin\rabbitmqctl.bat" set_policy orion-v1 ".*" "{\"ha-mode\":\"all\",\"ha-sync-mode\":\"automatic\",\"message-ttl\":86400000,\"expires\":86400000,\"queue-version\":1}" --priority 100 --apply-to queues

"%ProgramFiles%\SolarWinds\Orion\RabbitMQ\sbin\rabbitmqctl.bat" set_policy orion-shortliving-v1 "(.*@.*/w3wp_.*)|(.*_Database-Maint\.exe)" "{\"ha-mode\":\"all\",\"ha-sync-mode\":\"automatic\",\"message-ttl\":86400000,\"expires\":3600000,\"queue-version\":1}" --priority 110 --apply-to queues

net stop rabbitmq

rd "%ProgramData%\SolarWinds\Orion\RabbitMQ\db\rabbit@%COMPUTERNAME%-mnesia\msg_stores" /s /q

net start rabbitmq

 

  1. Run the commands on the Primary Polling Engine with High Availability (HA) disabled or the pool removed. 
  2. You can then run the commands on the HA standby. 
  3. Restart all SolarWinds services on all polling engines and web servers. This will ensure that the queues have consumers correctly.

Note:

The command location assumes the C:\ drive in the default installation path. You may need to adjust the path in the command as needed.

 

After running these commands, the queues will be Classic v1 and should no longer crash.

 

Alternate Workaround: Upgrade to 2025.1.1 to revert queues to Classic v1

If you are on 2025.1, we recommend upgrading to 2025.1.1 to revert the queues to v1.  The service release is now available for download in the Customer Portal.