Troubleshooting RabbitMQ for SolarWinds Platform

The Pub/Sub delivery system has been moved from MSMQ to RabbitMQ.

RabbitMQ has many advantages over MSMQ such as better performance, more storage, and clustering support, which is integral to the High Availability Solution. This change is ONLY for Pub/Sub - the Collector still uses MSMQ in 2020.2.5 and older, and SQLite in 2020.2.6 and newer.

Potential symptoms of this issue can include but are not limited to:

• Slowness in the SolarWinds Web Console
• Web resource errors/no data available charts
• Intermittent SolarWinds service stopping/starting issues
• False positive alerts
• Intermittent crashing issues requiring reboot or restart of services
• Broker connection errors in Information Service V3.0 logs
• Website stuck loading after login page.
• Agent nodes intermittently showing wrong status (down even though node is up and agent connected).

• RabbitMQ is a Windows service installed ONLY ON THE PRIMARY/MAIN POLLER (and standby server if in High Availability).
• RabbitMQ is not branded "SolarWinds" like the other services. It appears in the Orion Service Manager in the Dependencies section along with MSMQ.

• RabbitMQ failure during startup in Orion

Troubleshooting topics covered in this article:

1. Check ports used by different RabbitMQ
2. View RabbitMQ logs and message queues
3. Test the RabbitMQ connectivity
4. Corrupted vhost files in the RabbitMQ folder.
5. Disable RabbitMQ without removing it
6. Address the inconsistent cluster error
7. Purge a RabbitMQ queue
8. Delete RabbitMQ completely (blow it away)
9. Address RabbitMQ issues occurring even after the RabbitMQ has been deleted
10. Troubleshoot the application not starting
11. Recreate the certificate

1. Ports used by RabbitMQ

• For encrypted RabbitMQ messaging (AMQP/TLS) into the main polling engine from all Orion servers: 5671 | TCP | Bidirectional
• If you have SolarWinds HA: TCP ports 4369 and 25672 must be open between the primary and secondary servers to allow RabbitMQ clustering between the two servers. These ports exchange EPMD and Erlang distribution protocol messages for RabbbitMQ. They do not need to be open in additional polling engine pools.
• Required for RabbitMQ messaging (Erlang distribution): 25672 | TCP | Bidirectional
   

2. View logs and the message queues using the RabbitMQ Management Console

(Screenshot property of © 2020 Microsoft)

• Openrabbitmqconsole.exe fetches it from the database, decrypts it, then copies it to the clipboard and opens the browser.
• Then you must type "orion" into the username box and hit paste/ctrl+v in the password box.

(Screenshot property of © 2020 VMware)
 

(Screenshot property of © 2020 VMware)

3. Test RabbitMQ connectivity

INFO  SolarWinds.Orion.Swis.PubSub.MessageBus.MessageBusConnectionProvider - 
Message bus on WIN-OH918KEMKMT is reachable.

INFO  SolarWinds.Orion.Swis.PubSub.MessageBus.MessageBusConnectionProvider - 
Exception verifying connectivity to message bus. Assuming not reachable. System.TimeoutException: 
The operation requested on PersistentChannel timed out.

4. Corrupted vhost files in the RabbitMQ folder.

You may see below error when trying to start Vhost service on Rabbitmq via CLI command rabbitmqctl.bat restart_vhost
 

Trying to restart vhost '/' on node 'rabbit@myhost0005' ...
Failed to start vhost '/' on node 'rabbit@myhost0005'
Reason: {:shutdown, {:failed_to_start_child, :rabbit_vhost_process, 
{:error, {{{:function_clause, [{:rabbit_queue_index, :journal_minus_segment1, 
[{:no_pub, :no_del, :ack}, {{true, <<104, 19, 13, 23, 41, 254, 17, 253, 102, 
23, 140, 247, 16, 245, 255, 167, 0, 5, 200, 57, 59, 190, 31, 128, 0, 0, 4, 94>>, 
<<131, 104, 6, 100, 0, 13, 98, 97, 115, 105, 99, 95, 109, 101, 115, 115, 97, 
103, 101, 104, 4, 100, 0, 8, 114, 101, 115, 111, 117, ...>>}, :no_del, :no_ack}], 
[file: 'src/rabbit_queue_index.erl', line: 1231]}, 
{:rabbit_queue_index, :"-journal_minus_segment/3-fun-0-", 4, 
[file: 'src/rabbit_queue_index.erl', line: 1208]}, 
{:array, :sparse_foldl_3, 7, [file: 'array.erl', line: 1684]}, 
{:array, :sparse_foldl_2, 9, [file: 'array.erl', line: 1678]}, 
{:rabbit_queue_index, :"-recover_journal/1-fun-0-", 1, 
[file: 'src/rabbit_queue_index.erl', line: 915]}, {:lists, :map, 2, [file: 'l

And the following errors in Rabbitmq Logs:
 

<0.492.0> CRASH REPORT Process <0.492.0> with 0 neighbours crashed with reason: 
no match of right hand value {error,
{not_a_dets_file,"c:/PROGRA~3/SOLARW~1/Orion/RabbitMQ/db/RABBIT~1/msg_stores/vhosts/628WB79CIFDYO9LJI6DKMI09L/recovery.dets"}} in 
rabbit_recovery_terms:open_table/1 line 197
[error] <0.491.0> Unable to recover vhost <<"/">> data. 
Reason {badmatch,{error,{{{badmatch,{error,
{not_a_dets_file,"c:/PROGRA~3/SOLARW~1/Orion/RabbitMQ/db/RABBIT~1/msg_stores/vhosts/628WB79CIFDYO9LJI6DKMI09L/recovery.dets"}}},
[{rabbit_recovery_terms,open_table,1,[{file,"src/rabbit_recovery_terms.erl"},{line,197}]},{rabbit_recovery_terms,init,1,
[{file,"src/rabbit_recovery_terms.erl"},{line,177}]},{gen_server,init_it,2,[{file,"gen_server.erl"},{line,374}]},{gen_server,init_it,6,
[{file,"gen_server.erl"},{line,342}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,249}]}]},{child,undefined,rabbit_recovery_terms,
{rabbit_recovery_terms,start_link,[<<"/">>]},transient,30000,worker,[rabbit_recovery_terms]}}}}
Stacktrace [{rabbit_recovery_terms,start,1,
[error] <0.4669.0> Error on AMQP connection <0.4669.0> ([::1]:60381 -> [::1]:5671, vhost: 'none', user: 'orion', state: opening), channel 0:
{handshake_error,opening,
{amqp_error,internal_error,
"access to vhost '/' refused for user 'orion': vhost '/' is down",
'connection.open'}}

To fix this, follow the below steps:

• Stop the RabbitMQ service using the Orion Service Manager
• Navigate to “C:\ProgramData\SolarWinds\Orion\RabbitMQ” and delete “.erlang.cookie” (create a back-up in case needed)
• Navigate to “C:\ProgramData\SolarWinds\Orion\RabbitMQ\db\rabbit@<hostname>mnesia\msg_stores\vhosts\628WB79CIFDYO9LJI6DKMI09L” and delete “recovery.dets” if size is not 6 KB
• Navigate to C:\ProgramData\Solarwinds\Orion\RabbitMQ\db\<hostname>\msg_stores\vhosts
• Click "queues" and delete the folder's content (not the folder)
• Restart the RabbitMQ service

Please note that this folder and subsequent files will be recreated when the RabbitMQ service is restarted.

5. Disable RabbitMQ usage without removing it

2017-09-28 10:11:18,002 [15] ERROR SolarWinds.Orion.CortexMessageTranslator.MessageReader - 
(null) (null)  Error opening connection and opening queue 'CortexEvents'.
System.ApplicationException: Orion.PubSub.PubSubOverMessageBusEnabled == false, so not opening connection. 
(Change in /orion/admin/advancedconfiguration/global.aspx)

6. Troubleshoot the inconsistent_cluster error

{:inconsistent_cluster, 'Node rabbit@<LocalRabbitNode> thinks it\'s clustered with node rabbit@<RemoteRabbitNode>, but rabbit@<RemoteRabbitNode> disagrees'}

1. Disable the HA pool on the main polling engine.
2. Stop the RabbitMQ service on both main polling engine Active and StandBy servers.
3. Open an administrative command prompt on the server where you see the error, like in the example above. and run the following code.
   Replace < and with the corresponding names from the error message.

   C:\Program Files (x86)\SolarWinds\Orion\RabbitMQ\sbin\rabbitmqctl.bat forget_cluster_node -n rabbit@<LocalRabbitNode> --offline rabbit@<RemoteRabbitNode>

4. Start the RabbitMQ service on both the Active and StandBy servers on the main polling engine. RabbitMQ should keep running on both Active and StandBy.
5. Open an administrative command prompt on each server and run the following code to verify that the output lists only a single rabbit@<local host name> node name instance.
   Note that the cluster_name can be either the name of the Active server or of the StandBy server. The StandBy server name is shown if the local node was a member of the Active HA Pool at the time when the pool was created.

    C:\Program Files (x86)\SolarWinds\Orion\RabbitMQ\sbin\rabbitmqctl.bat cluster_status 

6. Enable the HA pool on the main polling engine.

7. How can I purge a queue full of garbage messages?

1. Use OpenRabbitMQConsole.exe to get to the RabbitMQ Management website.
2. On the Queues tab, click on the queue that needs a flush.
3. At the bottom right is a "Purge" button. Click it, and all messages in the queue will be deleted.

8. What if I need to blow RabbitMQ away? Please note that these steps apply only to 2020.2.6 and older. In 2022.2 and newer, if you need to reinstall RabbitMQ, you must repair SolarWinds Observability from the control panel and run the Configuration Wizard.
 

1. If there is an enabled High Availability pool for the Active and Standby servers on the main polling engine, disable the HA Pool.
2. Uninstall "RabbitMQ Server (SolarWinds Distribution)" from Programs and Features.
3. Delete C:\ProgramData\Solarwinds\Orion\RabbitMQ.
4. Delete the service control entry with sc delete rabbitmq. If previous steps were successful, this step should fail. It just checks that the previous steps were successful.
5. Delete the HKEY_LOCAL_MACHINE\SOFTWARE\Ericsson from the registry.
6. Install C:\ProgramData\Solarwinds\Installers\SolarWinds.RabbitMQ.Install.msi
   • For 2018.4 or higher, the installer is now called RABBITMQ-XXXX version.
7. Register RabbitMQ as a service. Open an administrative command prompt and run the following command:

   C:\Program Files (x86)\SolarWinds\Orion\RabbitMQ\sbin\rabbitmq-service.bat install

8. Run the database and services sections of the Configuration wizard. If the Configuration wizard fails with a .dll error, reboot the polling engine and run the Configuration wizard again.
9. If you use High Availability for Active and Standby servers on the main polling engine, restart the RabbitMQ service on the StandBy polling engine in the pool. If RabbitMQ on the StandBy polling engine stays up after restart, than you are done. If RabbitMQ on the StandBy polling engine does not stay running, re-install RabbitMQ, complete the whole procedure on the StandBy server (step 1-8).
10. Enable High Availability pool for Active and StandBy servers on the main polling engine if it was disabled for RabbitMQ re-installation.

• If you get stuck at registering RabbitMQ as a service, the RabbitMQ folder can be installed on another drive (D instead of the default C).

  D:\Solarwinds\RabbitMQ\sbin>rabbitmq-service.bat
  INFO: Could not find files for the given pattern(s).
  
  ******************************
  ERLANG_HOME not set correctly.
  ******************************
  Please either set ERLANG_HOME to point to your Erlang installation or place the
  RabbitMQ server distribution in the Erlang lib folder.

  Delete D:\Program Files (x86)\SolarWinds\RabbitMQ folder and return to Step 1.
• If it failed with the erlang home not set correctly error, run step 7 from the elevated PowerShell and NOT CMD.
• You could also try to reboot and re-run step 7 after rebooting.

9. What if I blew RabbitMQ away but still seeing issues with RabbitMQ?

1. Run the following commands:

   rabbitmqctl stop_app
   rabbitmqctl reset
   rabbitmqctl start_app

2. Run the Database and Services section of the Configuration wizard.

10. What if the application is installed but does not start?

1. Open an elevated command prompt and navigate to C:\Program Files (x86)\SolarWinds\Orion\RabbitMQ\sbin
2. Run the "rabbitmq-service.bat install" and make sure that the service. 
3. If after running you still do not see the application in either services.msc or in the Orion Service Manager, run "rabbitmq-service.bat start" and "rabbitmq-service.bat enable". This will start and enable the services.

11. What if I need to recreate the certificate?