Full RabbitMQ Reset in SolarWinds Platform products

RabbitMQ is a message queuing service used by the SolarWinds Platform, as such if it is experiencing issues in functionality it can manifest a wide variety of issues. It is only hosted by Main Poller and Main Poller Standby (in the case of HA deployment) servers, and Additional Poller or Additional Web Server should not have RabbitMQ service installed on them. If they do, please contact SolarWinds Technical Support.

This article provides detailed information on resetting/reinstalling the RabbitMQ service on the SolarWinds server.

Please note these steps are only valid for SolarWinds Orion Versions 2020.2.6 or Below. Do not reset RabbitMQ if you are on 2022.x or later SolarWinds Platform.

Please note the following for all scripts and commands found below:

# Scripts are not supported under any SolarWinds support program or service.
# Scripts are provided AS IS without warranty of any kind. SolarWinds further
# disclaims all warranties including, without limitation, any implied warranties
# of merchantability or of fitness for a particular purpose. The risk arising
# out of the use or performance of the scripts and documentation stays with you.
# In no event shall SolarWinds or anyone else involved in the creation,
# production, or delivery of the scripts be liable for any damages whatsoever
# (including, without limitation, damages for loss of business profits, business
# interruption, loss of business information, or other pecuniary loss) arising
# out of the use of or inability to use the scripts or documentation. 

If your environment has High Availability for the Main Poller please make sure to disable it. Here are articles that will help you disable the High Availability:

• Disable or delete HA pools, force a failover, or update an HA pool for SolarWinds Platform products

In case you are unable to access the SolarWinds Platform Web Console to disable the High Availablity follow the steps from this article: 

• Disable HA in the command prompt

Option 1: Verify the PATH and ERLANG_HOME values

To check the variable execute the following command in PowerShell: 

[System.Environment]::GetEnvironmentVariable('ERLANG_HOME',[System.EnvironmentVariableTarget]::Machine)

The response should match the response from this PowerShell command:

Get-ItemPropertyValue -Path 'Registry::HKEY_LOCAL_MACHINE\SOFTWARE\WOW6432Node\Ericsson\Erlang\9.0' -Name '(Default)'

If this is not the case please follow these steps: 

1. Execute the following command in PowerShell to remove the ERLANG_HOME environmental variable:

   [System.Environment]::SetEnvironmentVariable('ERLANG_HOME', '',[System.EnvironmentVariableTarget]::Machine)

2. Execute the following command in PowerShell to set ERLANG_HOME environmental variable:

   [System.Environment]::SetEnvironmentVariable('ERLANG_HOME',(Join-Path (Get-ItemPropertyValue -Path Registry::HKEY_LOCAL_MACHINE\SOFTWARE\WOW6432Node\Solarwinds\Orion\Core -Name "InstallPath") "\Erlang"),[System.EnvironmentVariableTarget]::Machine)

If the value is still not present please restart the server.

Option 2: Reset RabbitMQ App simple method

1. Execute the following command in PowerShell:

   Set-Location (Join-Path (Get-ItemPropertyValue -Path Registry::HKEY_LOCAL_MACHINE\SOFTWARE\WOW6432Node\Solarwinds\Orion\Core -Name "InstallPath") "\RabbitMQ\sbin\") .\rabbitmqctl.bat stop_app .\rabbitmqctl.bat reset .\rabbitmqctl.bat start_app

2. Run the Configuration Wizard for Database, Services, and Website.

Option 3 (Full Reset of RabbitMQ):

Follow these steps to fully remove the RabbitMQ service from the server and reinstall it:

1. Uninstall "RabbitMQ Server (SolarWinds Distribution)" from Programs and Features
   Note: Make note of the version of the RabbitMQ you are uninstalling as the same version needs to be installed back.

2. Execute the following command in PowerShell to delete the RabbitMQ configuration folder:

   if(Test-Path -Path C:\ProgramData\Solarwinds\Orion\RabbitMQ)
   {
       Remove-Item -Path C:\ProgramData\Solarwinds\Orion\RabbitMQ -Recurse
   }

3. Execute the following command in PowerShell or Command Prompt to delete the service control entry:

   sc delete rabbitmq

   Note: This often returns an error because the service entry is normally deleted during step 1.

4. Execute the following command in PowerShell to delete the registry folder:

   Remove-Item -Path Registry::HKEY_LOCAL_MACHINE\SOFTWARE\Ericsson -Recurse

5. Execute the following command in PowerShell to delete the RabbitMQ installation folder:

   if(Test-Path -Path (Join-Path (Get-ItemPropertyValue -Path Registry::HKEY_LOCAL_MACHINE\SOFTWARE\WOW6432Node\Solarwinds\Orion\Core -Name "InstallPath") "\RabbitMQ"))
   {
       Remove-Item -Path (Join-Path (Get-ItemPropertyValue -Path Registry::HKEY_LOCAL_MACHINE\SOFTWARE\WOW6432Node\Solarwinds\Orion\Core -Name "InstallPath") "\RabbitMQ") -Recurse
   }

   NOTE: This might require a restart.
    

6. Install RabbitMQ. Confirm that you are installing the correct version from step 1.
    

   C:\ProgramData\Solarwinds\Installers\RABBITMQ-1.X.XXXX.XXX-SolarWinds.RabbitMQ.Install.msi
    

7. Execute the following command in PowerShell to install the service:

   Set-Location (Join-Path (Get-ItemPropertyValue -Path Registry::HKEY_LOCAL_MACHINE\SOFTWARE\WOW6432Node\Solarwinds\Orion\Core -Name "InstallPath") "\RabbitMQ\sbin\")
   .\rabbitmq-service.bat install 

******************************
ERLANG_HOME not set correctly.
******************************

8. Delete the 'Erlang cookie' and 'RabbitMQ user account' from the Credential and CredentialProperty tables.
   
   NOTE: This is an optional step (Skip this if you have HA and RabbitMQ on the Active server is working correctly). However, in many cases, the credentials have issues and actually are the cause of the issue. If RabbitMQ is not working at all on both Active and Standby Main server in case of HA or on Primary Server without HA this step is recommended. 

   SELECT *
   FROM [CredentialProperty]
   WHERE [CredentialID] IN (
   SELECT DISTINCT [ID]
   FROM [Credential]
   WHERE [Name] = 'Erlang cookie' OR [Name] = 'RabbitMQ user account'
   )
   SELECT *
   FROM [Credential]
   WHERE [Name] = 'Erlang cookie' OR [Name] = 'RabbitMQ user account'
   --Delete versions of the above.
   --NOTE: It is advised to run this one at a time because the first query is dependent on what the second query is deleting.
   DELETE
   FROM [CredentialProperty]
   WHERE [CredentialID] IN (
   SELECT DISTINCT [ID]
   FROM [Credential]
   WHERE [Name] = 'Erlang cookie' OR [Name] = 'RabbitMQ user account'
   )
   DELETE
   FROM [Credential]
   WHERE [Name] = 'Erlang cookie' OR [Name] = 'RabbitMQ user account'

9. Run the Configuration Wizard on the server. If you did not skip step 8 above, run the Configuration Wizard on both Active and Standby Main Poller servers. 
10. A restart of all SolarWinds services on all Additional Pollers and Additional Web Servers in the SolarWinds environment is required if the credentials were removed in step 8.

After the above has been performed and confirmed that can access the SolarWinds Platform Web Console, can re-enable HA and make sure HA pools re enabled and working.