RabbitMQ Failed to connect to Broker in NPM

This issue could have different root causes. The Orion.Informationservice.log error looks as follows:

<Timestamp> [5] ERROR SolarWinds.MessageBus.RabbitMQ.EasyNetQLogger - (null) (null)  Failed to connect to Broker: 'SOLARWINDSHOSTNAME', Port: 5671 VHost: '/'. ExceptionMessage: 'None of the specified endpoints were reachable'
<Timestamp> [5] ERROR SolarWinds.MessageBus.RabbitMQ.EasyNetQLogger - (null) (null)  Failed to connect to any Broker. Retrying in 5000 ms 

1. RabbitMQ failed to install or upgrade properly

If you open Orion Service Manager on main polling engine and do not see RabbitMQ in the list of services you need to reinstall RabbitMQ; you can do so by following this procedure.

1. Run the below-referenced installer with local administrator rights.
 

• C:\ProgramData\Solarwinds\Installers\SolarWinds.RabbitMQ.Install.msi

2. Run Configuration Wizard. 

3. Restart SolarWinds Orion Module Engine from the SolarWinds Platform Service Manager.

Problems Installing RabbitMQ or taking the above steps?

Then please see article below:

Troubleshooting RabbitMQ

2. RabbitMQ not starting up  

Please see the  articles below:

RabbitMQ failure during startup

RabbitMQ failure during startup

3. RabbitMQ installed but service missing  

Please see the article below:

RabbitMQ is installed but the service is missing

4. Inaccurate or old references in DB Tables  

Please see the solution in this article: 
Clear Information Service Subscriptions

5. Message Queues are filling up or idle  

To clear the queue follow steps in below article:

RabbitMQ troubleshooting and purging SwisPubSub Queue

6. Incorrect TLS/SSL Configuration

See this article for more information:

Rabbit MQ Failed to connect to Broker in NPM


NOTE:

• Before making any changes kindly make sure that you have any of the applicable following a recent backup of the database, a snapshot of the VM, or a restore point of the physical server.
• Kindly work with your DBA for any database-related concerns.
  
  Rebuild RabbitMQ on version 2020.2.6 and below
  
  a. RDP to the MPE
  b. Stop the Services via the Orion Service Manager including the Dependencies. Stop the SolarWinds High Availability Service via Services.msc.
  c. Check RabbitMQ Version:
• C:\ProgramData\Solarwinds\Installers and the Control Panel. Make sure that they are on the same version.

d. Open Powershell and run as administrator.

[System.Environment]::GetEnvironmentVariable('ERLANG_HOME',[System.EnvironmentVariableTarget]::Machine)  

Get-ItemPropertyValue -Path 'Registry::HKEY_LOCAL_MACHINE\SOFTWARE\WOW6432Node\Ericsson\Erlang\9.0' -Name '(Default)'

• Make sure that the results are the same for the 2 scripts.

e. Uninstall "RabbitMQ Server (SolarWinds Distribution)" from Programs and Features

• Note: Make note of the version of the RabbitMQ you are uninstalling as the same version needs to be installed back.

f. Execute the following command in PowerShell to delete the RabbitMQ configuration folder:
 

if(Test-Path -Path C:\ProgramData\Solarwinds\Orion\RabbitMQ) 
{ 
    Remove-Item -Path C:\ProgramData\Solarwinds\Orion\RabbitMQ -Recurse 
}

g. Execute the following command in Command Prompt to delete the service control entry:   

sc delete rabbitmq

h. Execute the following command in PowerShell to delete the registry folder:

Remove-Item -Path Registry::HKEY_LOCAL_MACHINE\SOFTWARE\Ericsson -Recurse

i. Execute the following command in PowerShell to delete the RabbitMQ installation folder:

if(Test-Path -Path (Join-Path (Get-ItemPropertyValue -Path Registry::HKEY_LOCAL_MACHINE\SOFTWARE\WOW6432Node\Solarwinds\Orion\Core -Name "InstallPath") "\RabbitMQ"))
{
    Remove-Item -Path (Join-Path (Get-ItemPropertyValue -Path Registry::HKEY_LOCAL_MACHINE\SOFTWARE\WOW6432Node\Solarwinds\Orion\Core -Name "InstallPath") "\RabbitMQ") -Recurse
}

• Reboot the server.

 
j. Install RabbitMQ. Confirm that you are installing the correct version from step e.
 

• C:\ProgramData\Solarwinds\Installers\RABBITMQ-1.X.XXXX.XXX-SolarWinds.RabbitMQ.Install.msi

 
k. Execute the following command in PowerShell to install the service:

Set-Location (Join-Path (Get-ItemPropertyValue -Path Registry::HKEY_LOCAL_MACHINE\SOFTWARE\WOW6432Node\Solarwinds\Orion\Core -Name "InstallPath") "\RabbitMQ\sbin\")
.\rabbitmq-service.bat install 

l. Delete the 'Erlang cookie' and 'RabbitMQ user account' from the Credential and CredentialProperty tables.

• NOTE: This is an optional step (Skip this if you have HA and RabbitMQ on the Active server working correctly). However, in many cases, the credentials have issues and are the cause of the issue. If RabbitMQ is not working at all on both Active and Standby Main server in case of HA or on Primary Server without HA this step is recommended.  

SELECT * 
FROM [CredentialProperty] 
WHERE [CredentialID] IN ( 
SELECT DISTINCT [ID] 
FROM [Credential] 
WHERE [Name] = 'Erlang cookie' OR [Name] = 'RabbitMQ user account' 
) 
SELECT * 
FROM [Credential] 
WHERE [Name] = 'Erlang cookie' OR [Name] = 'RabbitMQ user account' 

--Delete versions of the above.

--NOTE: It is advised to run this one at a time because the first query is dependent on what the second query is deleting.

DELETE 
FROM [CredentialProperty] 
WHERE [CredentialID] IN ( 
SELECT DISTINCT [ID] 
FROM [Credential] 
WHERE [Name] = 'Erlang cookie' OR [Name] = 'RabbitMQ user account' 
)
DELETE
FROM [Credential]
WHERE [Name] = 'Erlang cookie' OR [Name] = 'RabbitMQ user account'

Scripts are not supported under any SolarWinds support program or service. Scripts are provided AS IS without warranty of any kind. SolarWinds further disclaims all warranties including, without limitation, any implied warranties of merchantability or fitness for a particular purpose. The risk arising out of the use or performance of the scripts and documentation stays with you. In no event shall SolarWinds or anyone else involved in the creation, production, or delivery of the scripts be liable for any damages whatsoever (including, without limitation, damages for loss of business profits, business interruption, loss of business information, or other pecuniary loss) arising out of the use of or inability to use the scripts or documentation.


3. Run a repair on SolarWinds Orion Core Services.

4. Run the Configuration Wizard 

5. Rebuild RabbitMQ on Main Poller Standby
a. Perform steps a. and b. from Rebuild RabbitMQ on Main Poller
b. Run the Configuration Wizard 

7. RabbitMQ Manual reset steps 

If steps outlined in the article from section 5 above did not have the desired effect you may proceed with below steps.

1. Go to C:\ProgramData\Solarwinds\Orion\RabbitMQ\db\rabbit@HOSTNAME-mnesia
   1. Note: (replace "HOSTNAME" with correct value )
2. Delete recovery.dets
3. Delete all the content of the subfolder "queues"
4. Open SolarWinds Platform Service Manager and restart RabbitMQ service. 

 General RabbitMQ information and troubleshooting steps available here - Rabbit MQ