Both high availability (HA) servers are Standby and Orion Module Engine Services are stopped

Prior to HA failover occurring, the poller will be in the following state:

• Active poller will have all services started and should be running.
• Standby poller will have Orion Module Engine and High Availability running and all other services stopped 

In some scenarios, the HA environment 

Note:  For the main poller or additional web server (AWS) standby, the web console will be turned off and not accessible.

HA failover may occur for the following reasons:

1. Some services on the active poller are detected as not running.
2. Heartbeat from active poller to the database is lost or not properly updated.  Standby poller will checked the last heartbeat entry in the SolarWinds database from the active poller.  If this is greater than 32 seconds (default setting for IntervalMemberDown), the standby will determine the active poller is down and perform a failover.
3. Connectivity between the poller in the pool is lost.

When a failover occurs, it is important to not disable the HA service until all services are up and running on the new poller.  In some scenario, HA failover can take several minutes (up to 5 minutes) due to standby is waiting for all services to be stopped prior to properly failover.

In the HighAvailability.Service.log, the following WARN message may occur:

WARN  SolarWinds.Orion.HighAvailability.Facilities.ServiceController - Service SolarWinds Information Service V3 didn't get status Stopped during period 95000. Current status is still StopPending.

This issue may occur for many reasons given above.

1. When a manual failover is triggered, it is not recommended to disable HA service until failover is completed and all services are up and running on the standby.
2. If HA services is not disabled and taking more than 5 minutes, review of the HighAvailability.Service.log may be necessary to understand the cause.

To immediate resolve the problem, perfrom one of the methods below.

If HA does not properly failover, please open a support ticket for further investigation to help determine the root cause.

METHOD 1

If the pool is disabled and both pollers are in standby, the recommendation is to enable the HA pool and allow the application complete the failover process.  This process should not take longer then 5 minutes.  If it take longer than 5 minutes, please open a Technical Support case for investigation on the cause of the issue.

1. Enable the HA pool that has both servers set as standby.
2. Wait for 10 minutes for services to start.
3. RDP to the active server.
4. Using Orion Service manager stop all services and start all services.
5. Check-in Orion web console → Settings → All Settings (in some versions, this one does not exist, so click on Settings) → Polling Engines if the engine in questions has green on ENGINE STATUS.

If the issue persists, please get in touch with our Technical support team to resolve this issue.

METHOD 2

Run an HA repair in the background using the command line tool.

1. On the SolarWinds Platform server, open a command prompt as an administrator.
2. Navigate to the SolarWinds directory (by default, C:\Program Files (x86)\SolarWinds\Orion).
3. Enter the following command, replacing "PoolName" with the name of the pool you want to repair:

HAEnableDisable.exe /RepairPool "PoolName"

4. Press Enter.
5. A confirmation message will appear in the CLI once the process is finished.

METHOD 3

If the HA pool cannot be enabled for any reason, perform the manual steps below:

1. Disable HA Globally:
   • Open Command Prompt as admin (right-click > run as administrator)
   • Use CD command to move to the install location of Solarwinds, e.g. C:\Program Files (x86)\SolarWinds\Orion
   • Once in Volume:\Program Files (x86)\SolarWinds\Orion type HAEnableDisable.exe /DisableHA and execute
   • A message should show that HA was successfully disabled
2. In Services shut down the Solarwids HA Service and Solarwinds Administration Service on all pollers, Main and Additional
3. Stop all services in Orion Service Manager on all pollers, Main and Additional
4. On the Main Active poller, Open Database Manager
5. Click add default server
6. Select your server
7. Expand the database (click on the +)
8. Double click any of the tables and replace the original query with these truncate queries:

-- Scripts are not supported under any SolarWinds support program or service.
-- Scripts are provided AS IS without warranty of any kind. SolarWinds further
-- disclaims all warranties including, without limitation, any implied warranties
-- of merchantability or of fitness for a particular purpose. The risk arising
-- out of the use or performance of the scripts and documentation stays with you.
-- In no event shall SolarWinds or anyone else involved in the creation,
-- production, or delivery of the scripts be liable for any damages whatsoever
-- (including, without limitation, damages for loss of business profits, business
-- interruption, loss of business information, or other pecuniary loss) arising
-- out of the use of or inability to use the scripts or documentation.

Truncate Table [dbo].[HA_Audit]
Truncate Table [dbo].[HA_FacilitiesInstances]
Truncate Table [dbo].[HA_PoolMemberInterfacesInfo]
Truncate Table [dbo].[HA_PoolMembers]
Truncate Table [dbo].[HA_Pools]
Truncate Table [dbo].[HA_ResourcesInstances]

9. Once you have them put in, click Execute Query.
10. Double-click and select Execute Query on the Engines and Websites tables in Database Manager to make sure the correct server is named. If not, update the correct row in each table.
11. Again, on the Main Active poller, open Registry Editor (Regedit)
    • Navigate to HKEY_LOCAL_MACHINE > SOFTWARE > Wow6432Node > SolarWinds > High Availability
    • Edit the RunType key and enter value MainPoller
    • Save the changes
12. In CMD, use the command to enable HA Once in Volume:\Program Files (x86)\SolarWinds\Orion type HAEnableDisable.exe /EnableHA
13. On the Main Active poller, use the Windows services and change the Disabled Solarwinds service to Manual.
14. In Services start the Solarwinds HA Service on all pollers, Main and Additional.
15. Start all services in Orion Service Manager on all pollers, Main and Additional.
16. Check the registry to make sure the corrected key has value MainPoller
17. Check the Engines table to make sure the Correct Main server (Active) entry should be there. Only 1 Primary Pollers must be there, and we should not have a Standby Main poller in that table. 
18. Open Orion Web Console and recreate the HA pool

**On the Active Primary polling engine, both InstallType and RunType values must be MainPoller.
**On the Standby Primary polling engine, both InstallType and RunType values must be MainPollerStandby.
**On the Active Additional polling engine, both InstallType and RunType values must be AdditionalPoller.
**On the Standby Additional polling engine, both InstallType and RunType values must be AdditionalPollerStandby.

Note: In the event that SolarWinds Platform has multiple Polling Engines
To fix issue on Specific Server refer to this article: Remove orphaned standby High Availability servers

Related Article:

• How to rebuild SolarWinds High Availability