RabbitMQ - SWIS Issues - Failed to connect to Broker for the Orion Platform

10/23/2018 8:15 PM

8/25/2022 5:31 PM

You may have found issues with SWISV3 connecting to the RabbitMQ.

After investigating the usual steps, there is no apparent reason for the impossibility of interconnecting SWISv3 to local RMQ. RMQ logs look fine, ports are open, netstat -and | find "5671" DNS resolution works nslookup ORION SERVER HOSTNAME, but SWIS is still unable to connect at times and will disconnect itself. Some queues, such as the SWISPUBSUB queue, are filling up in a ready state. You also may be able to connect to OpenRabbitMQConsole.exe; however, you may not see any SwisPubSub queues.
 

Error 1:

ERROR SolarWinds.MessageBus.RabbitMQ.EasyNetQLogger - (null) (null)
Failed to connect to broker <ORIONSERVER hostname>, Port: 5671 VHost: '/'.
ExceptionMessage: 'None of the specified endpoints were reachable'
2017-01-17 20:37:47,584 [11] ERROR SolarWinds.MessageBus.RabbitMQ.EasyNetQLogger 
- (null) (null) Failed to connect to any Broker. Retrying in 5000 ms

There is "A fatal error occurred while creating an SSL client credential. 
The internal error state is 10013."

Logged in System Event Log every 5 seconds which is also our connection retry interval.

Error 2:

 

In some cases, a slightly different error may appear during Config Wizard run on APE's or in SWIS logs on APE's; below is an example that occurred when the APP FW or Proxy between APE and MPE rejecting TLS traffic on port 5671 or AMQP port. 

 

NOTE: Telnet to port 5671 was working fine from APE to MPE 

with chain System.Security.Cryptography.X509Certificates.X509Chain
with errors RemoteCertificateNameMismatch, RemoteCertificateChainErrors
2021-06-18 11:16:59,739 [49] ERROR EasyNetQLogger - Failed to connect to broker <ORIONSERVER hostname>, port 5671, vhost / -> RabbitMQ.Client.Exceptions.BrokerUnreachableException: None of the specified endpoints were reachable ---> System.IO.IOException: Unable to write data to the transport connection: An existing connection was forcibly closed by the remote host. ---> System.Net.Sockets.SocketException: An existing connection was forcibly closed by the remote host

 

Orion Platform

Error 1 could be caused by,

• TLS 1.2 not enabled
• RabbitMQ needs reset
• TCP port 5671 is blocked

for Error 2

• A proxy/Firewall Rejecting TLS traffic based on Certificate validation policy.

Before doing any of the below steps please make sure port 5671 is open bidirectional.

Enable TLS 1.2 on all Orion servers (after first ensuring the cert is SHA). IIS Crypto Tool can be used for this: Nartac Software - IIS Crypto (© 2017 Nartac Software, available at https://www.nartac.com, obtained on June 1, 2017.)
 

A simple enable of TLS 1.2 and reboot may not resolve the issue as the: 'C:\ProgramData\Solarwinds\Orion\RabbitMQ\.erlang.cookie' file and the: 'C:\ProgramData\Solarwinds\Orion\RabbitMQ\db' database may have to get recreated per the new TLS settings.
 

Reset RabbitMQ

NOTE: This step no longer works in Orion Core Platform 2022.X and newer

1. Uninstall "RabbitMQ Server (SolarWinds Distribution)" from Programs and Features (take note of the version of RabbitMQ you are uninstalling as you must reinstall the same or the latest version).

2. Delete:

• C:\ProgramData\Solarwinds\Orion\RabbitMQ

3. Delete the service control entry in command prompt/PowerShell:

• sc delete rabbitmq (often errors because the service entry normally is deleted during installation).

4. Delete registry folder:

• HKEY_LOCAL_MACHINE\SOFTWARE\Ericsson

5. Delete directory:

• C:\Program Files (x86)\SolarWinds\RabbitMQ (NOTE: may require a restart)

6. Delete 'Erlang cookie' and 'RabbitMQ user account' from Credential and CredentialProperty tables

NOTE: optional step, however, in many cases the credentials have issues and actually are the cause of your issue.
 

-- Scripts are not supported under any SolarWinds support program or service.
-- Scripts are provided AS IS without warranty of any kind. SolarWinds further
-- disclaims all warranties including, without limitation, any implied warranties
-- of merchantability or of fitness for a particular purpose. The risk arising
-- out of the use or performance of the scripts and documentation stays with you.
-- In no event shall SolarWinds or anyone else involved in the creation,
-- production, or delivery of the scripts be liable for any damages whatsoever
-- (including, without limitation, damages for loss of business profits, business
-- interruption, loss of business information, or other pecuniary loss) arising
-- out of the use of or inability to use the scripts or documentation.

SELECT * FROM [CredentialProperty] WHERE CredentialID IN (SELECT ID FROM [Credential] WHERE [Name] = 'Erlang cookie' OR [Name] = 'RabbitMQ user account')
SELECT * FROM [Credential] WHERE [Name] = 'Erlang cookie' OR [Name] = 'RabbitMQ user account'


--Delete versions of the above.
--NOTE: it is advised to run this one at a time because the first query is dependent on what the second query is deleting.

DELETE FROM [CredentialProperty] WHERE CredentialID IN (SELECT ID FROM [Credential] WHERE [Name] = 'Erlang cookie' OR [Name] = 'RabbitMQ user account')
DELETE FROM [Credential] WHERE [Name] = 'Erlang cookie' OR [Name] = 'RabbitMQ user account'

7. Install:C:\ProgramData\Solarwinds\Installers\RABBITMQ-1.X.XXXX.XXX-SolarWinds.RabbitMQ.Install.msi (confirm correct version in step 1)

8. Execute in PowerShell or CMD prompt, if PS fails, try cmd prompt or vice versa:

• C:\Program Files (x86)\SolarWinds\Orion\RabbitMQ\sbin\rabbitmq-service.bat install 

NOTE: if the installation is NOT on C drive, you must cd to the correct Orion installation drive.

9. Run the Database and Services section of the Configuration Wizard.

NOTE: If the configuration wizard failed to try to reboot the server and run the configuration wizard again.

Solution For Error 2:

- Enable the proxy / Firewall to allow AMQP traffic over TLS