HA Standby Server's Status becomes Critical after enabling the HA Pool due to inability to join the RabbitMQ Cluster

9/1/2022 11:57 AM

6/10/2026 4:39 PM

When enabling the HA Pool, the Standby server's HA status becomes critical

From the HA_PoolMembers Table, It is due to a RabbitMQ clustering issue

An error occurred in 'rabbitmqctl.bat join_cluster': 
 
Error: unable to perform an operation on node 'rabbit@<node_name>'. Please see diagnostics information and suggestions below.
 
Most common reasons for this are:
 
 * Target node is unreachable (e.g. due to hostname resolution, TCP connection or firewall issues)
 * CLI tool fails to authenticate with the server (e.g. due to CLI tool's Erlang cookie not matching that of the server)
 * Target node is not running
 
In addition to the diagnostics info below:
 
 * See the CLI, clustering and networking guides on https://rabbitmq.com/documentation.html to learn more
 * Consult server logs on node rabbit@W6V-SWAS01
 * If target node is configured to use long node names, don't forget to use --longnames with CLI tools
 
DIAGNOSTICS
===========
 
attempted to contact: ['rabbit@<node_name>]
 
rabbit@W6V-SWAS01:
  * connected to epmd (port 4369) on <Primary Server>
  * epmd reports node 'rabbit' uses port 25672 for inter-node and CLI tool traffic 
  * TCP connection succeeded but Erlang distribution failed 
  * suggestion: check if the Erlang cookie identical for all server nodes and CLI tools
  * suggestion: check if all server nodes and CLI tools use consistent hostnames when addressing each other
  * suggestion: check if inter-node connections may be configured to use TLS. If so, all nodes and CLI tools must do that
   * suggestion: see the CLI, clustering and networking guides on https://rabbitmq.com/documentation.html to learn more
 
 
Current node details:
 * node name: 'rabbitmqcli-412-rabbit@<node_name>'
 * effective user's home directory: C:\ProgramData\SolarWinds\Orion\RabbitMQ
 * Erlang cookie hash: ZQ1ys9ow8CaLgIIPJ0WnVw==

This article also applies when the standby server fails to join the RabbitMQ cluster with the following variant during rabbitmqctl join_cluster:

00:24:08.399 [error] Feature flags: error while running:
Feature flags: rabbit_ff_controller:running_nodes]
Feature flags: on node rabbit@RABBITMQ01P:
Feature flags: exception error: {erpc,noconnection}
Feature flags: in function erpc:call/5 (erpc.erl, line 710)
Feature flags: in call from rabbit_ff_controller:rpc_call/5 (rabbit_ff_controller.erl, line 1384)
Feature flags: in call from rabbit_ff_controller:list_nodes_clustered_with/1 (rabbit_ff_controller.erl, line 486)
Feature flags: in call from rabbit_ff_controller:check_node_compatibility_task/2 (rabbit_ff_controller.erl, line 398)
Feature flags: in call from rabbit_db_cluster:can_join/1 (rabbit_db_cluster.erl, line 59)
Feature flags: in call from rabbit_db_cluster:join/2 (rabbit_db_cluster.erl, line 91)
Feature flags: in call from erpc:execute_call/4 (erpc.erl, line 589)

Network Performance Monitor

The HA standby RabbitMQ node fails to join the cluster because its Erlang cookie (cluster authentication secret) does not match the cookie used by the active main poller.

Common contributing factors:

• Required RabbitMQ HA ports (4369, 25672, 5671) blocked between main and standby.

• Corrupted or mismatched .erlang.cookie file in C:\ProgramData\SolarWinds\Orion\RabbitMQ.

• Stale cluster metadata on the standby server

A. Please verify if all required ports for HA are allowed between the MPE HA servers, especially RabbitMQ ports including 4369.

If not, please allow the ports and then verify the status. 

Refer to this document for HA port requirements:  https://documentation.solarwinds.com/en/success_center/orionplatform/content/ha_requirements.htm

B. If the ports are open, Confirm RabbitMQ health on the active main poller

1. On the active main poller, open Orion/SolarWinds Platform Service Manager and confirm:

   • RabbitMQ is Running.

   • Other core services (Information Service, Job Engine, etc.) are healthy.

2. If RabbitMQ is not stable on the active server, resolve that first using Troubleshooting RabbitMQ for SolarWinds Platform.

Only proceed once active RabbitMQ is healthy.

C. Fix Erlang cookie mismatch

Perform on the standby first:

1. Stop RabbitMQ on the standby via SolarWinds Platform Service Manager.

2. Navigate on the standby Server C:\ProgramData\SolarWinds\Orion\RabbitMQ to:

   1. Back up and delete the Erlang cookie file:

      • Make a copy of .erlang.cookie for reference.

      • Delete the original .erlang.cookie file from this folder.

   2. Start RabbitMQ using SolarWinds Platform Service Manager

3. Recreate the HA Pool.

If the standby still cannot join the cluster after this, continue to the next section.

D. Reset RabbitMQ cluster membership on the standby server using  simple method.

On the standby server:

1. Open an elevated command prompt and change to the RabbitMQ sbin directory:

cd "C:\Program Files (x86)\SolarWinds\Orion\RabbitMQ\sbin"

   

rabbitmqctl stop_app
rabbitmqctl reset
rabbitmqctl start_app

Alternative: PowerShell Command:

Set-Location (Join-Path (Get-ItemPropertyValue -Path Registry::HKEY_LOCAL_MACHINE\SOFTWARE\WOW6432Node\Solarwinds\Orion\Core -Name "InstallPath") "\RabbitMQ\sbin\")
.\rabbitmqctl.bat stop_app
.\rabbitmqctl.bat reset
.\rabbitmqctl.bat start_app

    3. Restart the SolarWinds High Availability service

  Recreate the HA Pool.

E.  If the issue persists, continue with a platform Full RabbitMQ reset repair on the standby server.

     RabbitMQ failure with the error message "Node rabbit@[standby server] thinks it's clustered with node rabbit@[active server], but rabbit@[active server] disagrees"

F.  Recreate the HA Pool Then Enable the HA Pool. 

 

If the error specifically shows {:aborted_feature_flags_compat_check, {:error, {:erpc, :noconnection}}}, prioritize bidirectional port validation and hostname resolution checks before proceeding to cookie or full RabbitMQ reset steps, because this variant commonly indicates failed Erlang RPC communication between the standby and active nodes during the compatibility check.