How to rebuild SolarWinds High Availability

10/25/2018 12:42 PM

11/19/2024 3:27 PM

Here is a quick run-through on the steps to rebuild SolarWinds High Availability (HA). Some examples of when you might need to rebuild HA:

• failovers not occurring
• incorrect service member types
• problems making members active
• server status is critical on HA deployment summary page

Orion Platform

Inconsistencies in the HA entries in the database.
Tables:

• HA_Audit
• HA_FacilitiesInstances
• HA_PoolMemberInterfacesInfo
• HA_PoolMembers
• HA_Pools
• HA_ResourcesInstances

***IMPORTANT***
This procedure involves updating the values in the Orion database. SolarWinds recommends backing up your database before proceeding. See How to Backup the Orion SQL database.

SolarWinds strongly recommends that you back up your registry before making any edits to your system registry. You should only edit the registry if you are experienced and confident in doing so. Using a registry editor incorrectly can cause serious issues with your operating system, which could require you to reinstall your operating system to correct them. SolarWinds cannot guarantee resolutions to any damage resulting from making registry edits.

Steps to Rebuild HA - Do not skip any steps and follow the order

1. RDP to Main Poller
2. Open the Command Prompt, and set the current directory to the Orion installation directory.

3. Execute the following command in Command Prompt to disable the High Availablity across all Orion servers: 

   HAEnableDisable /DisableHA

4. Create a backup of the Windows Registry. 
5. Open Services.msc
6. Stop the SolarWinds High Availability service to prevent it from writing to the database while High Availablity is being rebuilt. 
7. Stop all services using the Orion Service Manager

8. Repeat steps from step 4  to step 7 on the Standby server. 
9. Use the Database Manager to run the following SQL query and truncate HA tables:

   -- Scripts are not supported under any SolarWinds support program or service.
   -- Scripts are provided AS IS without warranty of any kind. SolarWinds further
   -- disclaims all warranties including, without limitation, any implied warranties
   -- of merchantability or of fitness for a particular purpose. The risk arising
   -- out of the use or performance of the scripts and documentation stays with you.
   -- In no event shall SolarWinds or anyone else involved in the creation,
   -- production, or delivery of the scripts be liable for any damages whatsoever
   -- (including, without limitation, damages for loss of business profits, business
   -- interruption, loss of business information, or other pecuniary loss) arising
   -- out of the use of or inability to use the scripts or documentation.
   
   TRUNCATE TABLE  HA_Pools
   TRUNCATE TABLE  HA_PoolMembers
   TRUNCATE TABLE  HA_PoolMemberInterfacesInfo
   TRUNCATE TABLE  HA_FacilitiesInstances
   TRUNCATE TABLE  HA_Audit
   TRUNCATE TABLE  HA_ResourcesInstances

   ***NOTE***
   For High Availability 1.0, change HA_Audit to HA_ResourcesTypes
   
   Note: In the event that Orion is with multiple Polling Engines
              To fix issues on Specific Servers refer to the Article: Remove orphaned standby High Availability servers
    

10. Check the High Availability registry on the Active Main Poller, Standby, and all Additional Pollers.
    1. Open the Registry Editor (regedit.exe)
    2. Navigate to:

       HKEY_LOCAL_MACHINE\SOFTWARE\WOW6432Node\Solarwinds\HighAvailability

    3. Examine the InstallType and RunType values and double-click them to update (if necessary). Verify the following:
       • On the active primary polling engine, both InstallType and RunType values must be MainPoller.
       • On the Standby primary polling engine, both InstallType and RunType values must be MainPollerStandby.
       • On an active APE, both InstallType and RunType values must be AdditionalPoller.
       • On a Standby APE, both InstallType and RunType values must be AdditionalPollerStandby.
       • Start the SolarWinds High Availability Service on the primary server using the Services.MSC.
11. Make sure to set Startup Type to the following On Active Poller(s):

Name	Status	Startup Type
SolarWinds Administration Service	Running	Automatic
SolarWinds Agent Service	Running	Automatic
SolarWinds Alerting Service V2	Running	Manual
SolarWinds Collector Service	Running	Manual
SolarWinds Cortex Service	Running	Manual
SolarWinds High Availability Service	Running	Automatic
SolarWinds Information Service	Running	Manual
SolarWinds Information Service V3	Running	Manual
SolarWinds Job Engine v2	Running	Manual
SolarWinds Orion Module Engine	Running	Automatic
SolarWinds Recommendations	Running	Manual
SolarWinds Syslog Service	Running	Manual
SolarWinds Trap Service	Running	Manual

 

12. On the MainPoller and MainPollerStandby check the registry to ensure it displays the proper values for its respected member type of the pool (in this case, MainPoller, MainPollerStandby).
13. Restart the SolarWinds High Availability service on the Standby server.
14. Make sure to set Startup Type to the following on Standby Poller(s):

Name	Status	Startup Type
SolarWinds Administration Service	Running	Automatic
SolarWinds Agent Service	Running	Automatic
SolarWinds Alerting Service V2		Disabled
SolarWinds Collector Service		Disabled
SolarWinds Cortex Service		Disabled
SolarWinds High Availability Service	Running	Automatic
SolarWinds Information Service		Disabled
SolarWinds Information Service V3		Disabled
SolarWinds Job Engine v2		Disabled
SolarWinds Orion Module Engine	Running	Automatic
SolarWinds Recommendations		Disabled
SolarWinds Syslog Service		Disabled
SolarWinds Trap Service		Disabled

 
These steps only apply if you have HA Pool for the Additional Polling Engines, customers who only have Main Poller in HA pool can skip to verification section:

1. On the Additional Poller and Additional Poller, Standby servers, check the registry to ensure it displays the proper values for its respected member type of the pool (AdditionalPoller, AdditionalPollerStandby)
2. If you have servers beyond the Main and Standby servers, repeat these steps to restart the SolarWinds High Availability service and check the register for each server.
3. Open the Command Prompt, and set the current directory to the Orion installation directory. (Default location: C:\Program Files (x86)\SolarWinds\Orion)
4. Execute the following command in the Command Prompt to re-enable High Availablity globally: 

5. HAEnableDisable /EnableHA

6. In the Orion Web Console, navigate to the High Availability Deployment Summary page.
7. Check the status of the pool member in the Orion Web Console. The servers should be up(Green) and you should have option to setup HA Pools again.

Verify that HA is running

1. Use the Database Manager to execute the following query: 

   -- Scripts are not supported under any SolarWinds support program or service.
   -- Scripts are provided AS IS without warranty of any kind. SolarWinds further
   -- disclaims all warranties including, without limitation, any implied warranties
   -- of merchantability or of fitness for a particular purpose. The risk arising
   -- out of the use or performance of the scripts and documentation stays with you.
   -- In no event shall SolarWinds or anyone else involved in the creation,
   -- production, or delivery of the scripts be liable for any damages whatsoever
   -- (including, without limitation, damages for loss of business profits, business
   -- interruption, loss of business information, or other pecuniary loss) arising
   -- out of the use of or inability to use the scripts or documentation.
   
   -- Select rows from a Table 'HA_PoolMembers'
   SELECT * FROM HA_PoolMembers

2. Make sure the servers exist.
3. Verify that in PoolMemberType column, there is both MainPoller and MainPollerStandby and if you have APE pools you should also see AdditionalPoller and AdditionalPollerStandby pairs.

If you encounter any issues while performing these steps, please contact SolarWinds Technical support for assistance.