Submit a ticketCall us

Webinar: Web Help Desk for HR, Facilities and Accounting Departments
This webinar will focus on use cases for HR, Facilities and Accounting.

Having a unified ticketing and asset management system for all the departments in your company can provide end-users with a seamless experience and make things easier for your IT team. Yet, with different business tasks and objectives, many departments don’t fully understand the capabilities of Web Help Desk and how the software can be customized for effective use in their departments.
Register Now.

Home > Success Center > Network Performance Monitor (NPM) > Rabbit MQ

Rabbit MQ

Table of contents

Updated November 10, 2016

Overview

The Pub/Sub delivery system has been moved from MSMQ to RabbitMQ.

RabbitMQ has many advantages over MSMQ such as, better performance, larger storage, and it supports clustering, which is integral to the High Availability Solution. This change is ONLY for Pub/Sub - the Collector still uses MSMQ as always. 

Environment

NPM 12

Detail

RabbitMQ is installed ONLY ON THE PRIMARY POLLER as a regular service:
Note that it is not branded "SolarWinds" like the other services. It appears in the Orion Service Manager in the Dependencies section along with MSMQ.

Ports used by RabbitMQ     

For encrypted RabbitMQ messaging (AMQP/TLS) into the main polling engine from all Orion servers 5671 TCP Bidirectional

Required for RabbitMQ messaging (Erlang distribution) 25672   TCP   bidirectional

 

Troubleshooting/Issues

Logs for RabbitMQ can be found here:
C:\ProgramData\SolarWinds\Orion\RabbitMQ\log. 

 

You can view the message queues by using the RabbitMQ Management Console. This is a simple website that is served by RabbitMQ itself (IIS is not involved).

On the main polling engine, navigate to the installation directory (C:\Program Files (x86)\SolarWinds\Orion by default) and find OpenRabbitMQConsole.exe

 

This will read the RabbitMQ password from the Orion database, decrypt it, and copy it to the clipboard, and then open the system default browser to http://localhost:15672
Log in with username "orion" and paste in the password to visit the RabbitMQ management console. (By configuration, this console is not accessible remote - only on localhost.)

  • Openrabbitmqconsole.exe fetches it from the database, decrypts it, then copies it to the clipboard and opens the browser.
  • Then you have to type "orion" into the username box and hit paste/ctrl+v in the password box.

 

Note: The username is case sensitive. Only orion will work.

On the Queues tab, the number of messages in the "ready" state should stay at zero. If one of these numbers is hanging out above zero, that means that some process is not receiving its messages properly. The "SwisPubSub" queue is used by Orion components to send publish pubsub messages. SWISv3 on the main polling engine reads this queue and forwards messages to subscribers. Other queues are used by subscribers to receive notifications forwarded by SWIS.

 

 

In this release, not all pubsub subscribers have been updated to use RabbitMQ. You can see which subscriptions are using RabbitMQ by looking in the Subscriptions table in the database. Rows where Binding is "MessageBus" are using RabbitMQ. For these subscriptions, the EndpointAddress column holds the queue name, even if it looks like a WCF Uri.

 

Testing for RabbitMQ connectivity

The various Orion services test for connectivity to RabbitMQ on startup and choose whether to subscribe using RabbitMQ or WCF accordingly. If the test succeeds, you will see a message like this in the service's log:

INFO  SolarWinds.Orion.Swis.PubSub.MessageBus.MessageBusConnectionProvider - Message bus on WIN-OH918KEMKMT is reachable.

 

If the service cannot reach RabbitMQ for any reason (port 5671 blocked by firewall, problem with the RabbitMQ "orion" user password, RabbitMQ not running, etc.), then you will see a message like this in the service's log. The specific exception will tell you something about why the test failed, but the key part is "Assuming not reachable".

 

INFO  SolarWinds.Orion.Swis.PubSub.MessageBus.MessageBusConnectionProvider - Exception verifying connectivity to message bus. Assuming not reachable. System.TimeoutException: The operation requested on PersistentChannel timed out.

 

Disabling RabbitMQ usage without removing it

You can block access to RabbitMQ by turning off the setting "PubSubOverMessageBusEnabled" in the Orion.PubSub section of Centralized settings feature, either globally or per-server. 

When you turn off that setting and restart Orion sevices, the following message will be displayed in the Orion.InformationService log file of your Main Polling Engine:

"2017-09-28 10:11:18,002 [15] ERROR SolarWinds.Orion.CortexMessageTranslator.MessageReader - (null) (null)  Error opening connection and opening queue 'CortexEvents'.
System.ApplicationException: Orion.PubSub.PubSubOverMessageBusEnabled == false, so not opening connection. (Change in /orion/admin/advancedconfiguration/global.aspx)"

 

And in the same log file of your Additional Polling Engine:

"2017-09-28 10:12:05,934 [25] INFO  SolarWinds.Orion.Swis.PubSub.MessageBus.MessageBusConnectionProvider - (null) (null) Orion.PubSub.PubSubOverMessageBusEnabled == false, so not attempting message bus connection. (Change in /orion/admin/advancedconfiguration/global.aspx)"

 

How can I purge a queue that is full of garbage messages?

Use OpenRabbitMQConsole.exe to get to the RabbitMQ Management website. On the Queues tab, click on the queue that needs a flush. At the bottom right is a "Purge" button. Click this and all messages in the queue will be deleted.

 

What if I need to totally blow RabbitMQ away?
Any scenario in which this would be necessary is HIGHLY unlikely, but just in case: 

  1. Uninstall "RabbitMQ Server (SolarWinds Distribution)" from Programs and Features.
  2. Delete C:\ProgramData\Solarwinds\Orion\RabbitMQ.
  3. Delete the service control entry with sc delete rabbitmq.
  4. Delete HKEY_LOCAL_MACHINE\SOFTWARE\Ericsson from the registry.
  5. Install C:\ProgramData\Solarwinds\Installers\SolarWinds.RabbitMQ.Install.msi.
  6. Delete 'Erlang cookie' and 'RabbitMQ user account' from Credential and CredentialProperty tables
  7. Run the database and services sections of the Configuration Wizard.

 

What if the application is installed but does not start?

  1. Open an elevated command prompt and navigate to c:\Program Files (x86)\SolarWinds\Orion\RabbitMQ\sbin.
  2. Run the "rabbitmq-service.bat install" and make sure that the service. 
  3. Once the above has been run and you still do not see it in either services.msc or in the orion service manager you can run "rabbitmq-service.bat start" and "rabbitmq-service.bat enable"
  4. This will start and enable the services and you should be ok.

 

What if I need to recreate the certificate?

Orion configures RabbitMQ to use TLS for queue clients (which use the AMQP protocol on port 5671) using the CN=SolarWinds-Orion self-signed certificate. This is stored in a file at C:\ProgramData\Solarwinds\Orion\RabbitMQ\SolarWinds-Orion.pem. It needs to be readable by Local System. If it gets corrupted, just delete it and run the services portion of the Configuration Wizard.

 

Last modified
09:33, 28 Sep 2017

Tags

Classifications

Public