Group created on Main Poller is not visible on Additional Web Server

This article is built to help determine the root cause of an issue that is known to occur but is very hard to reproduce. The customer is most often not able to replicate the issue until it happens naturally next time. As the issue is known to occur on very rare occasions, this article was built to provide steps that can be used when the issue is experienced to capture relevant information so we can effectively root cause the issue and address the issue.

How do we identify the issue?

Ways the issue can represent itself is when the environment has:

• Main Poller and Additional Web Server deployed - If a group is created or edited on the Main Poller and the change is not visible on the Additional Web Server for more than a 2-minute time window.
• Main Poller and Additional Polling Engine -  If a group is created or edited on the Main Poller and the change is not visible while running a SWIS API call over the Additional Poller for over a 2-minute time window.

The root cause has not been identified as the logging level required for it is DEBUG, and this is not recommended for a long-time running SolarWinds Information v3 service.

Please note the following for all scripts below:

-- Scripts are not supported under any SolarWinds support program or service.
-- Scripts are provided AS IS without warranty of any kind. SolarWinds further
-- disclaims all warranties including, without limitation, any implied warranties
-- of merchantability or of fitness for a particular purpose. The risk arising
-- out of the use or performance of the scripts and documentation stays with you.
-- In no event shall SolarWinds or anyone else involved in the creation,
-- production, or delivery of the scripts be liable for any damages whatsoever
-- (including, without limitation, damages for loss of business profits, business
-- interruption, loss of business information, or other pecuniary loss) arising
-- out of the use of or inability to use the scripts or documentation.

1. RDP to Main Poller and Additional Web/Poller Server
2. Open Database Manager (Default location: C:\Program Files (x86)\SolarWinds\Orion\DatabaseManager.exe) on Main Poller
3. Execute the following set of queries and save the results by pressing the Export to CSV button.
   

   SELECT * FROM [dbo].[Containers]

   SELECT * FROM [dbo].[Events]

   SELECT * FROM [dbo].[ContainerMemberSnapshots]

4. Open Orion Web Console on Main Poller and log in with Admin type account.
5. Navigate to http(s)://{MainPoller}/Orion/Admin/swis.aspx
6. Open Orion Web Console on Additional Web/Poller Server and log in with Admin type account.
7. Navigate to http(s)://{Web/Poller}/Orion/Admin/swis.aspx
8. Execute the following queries on Main Poller and Additional Web/Poller and save the results by Right Click -> Save As -> Save as type -> Webpage, HTML only (*.html;*.htm) -> Save
   

   SELECT 
       ContainerMemberSnapshotID, 
       ContainerID, 
       Name, 
       FullName, 
       EntityType, 
       EntityDisplayName, 
       EntityDisplayNamePlural, 
       MemberUri, 
       Status, 
       EntityID, 
       MemberAncestorDisplayNames, 
       MemberAncestorDetailsUrls, 
       Description
   FROM Orion.ContainerMemberSnapshots

   SELECT 
       ContainerID, 
       Name, 
       Owner, 
       Frequency, 
       StatusCalculator, 
       RollupType, 
       IsDeleted, 
       PollingEnabled, 
       LastChanged, 
       UnManageFrom, 
       UnManageUntil, 
       DetailsUrl, 
       ModernIcon
   FROM Orion.Container

   If you are testing this on the Additional Polling Engine, you will need to use SWQL Studio to connect to the Additional Polling Engine and run the same queries. Make sure the same account is used for the Main Poller and Additional Polling Engine.

9. Open RabbitMQ Web Console (Default location: C:\Program Files\SolarWinds\Orion\OpenRabbitMQConsole.exe)
10. Navigate to the Queues and Streams tab and look for a queue that starts with the following Name and confirm if there is a consumer. Take a full-screen screenshot and save it in jpg format (you can use Paint):
    

    OrionSwis.SolarWinds.Data.Providers.Orion.Containers.StorageUpdateService.StorageUpdateServicePlugin

11. Open Log Adjuster (Default location: C:\Program Files (x86)\SolarWinds\Orion\LogAdjuster.exe) on Main Poller and Additional Web/Polling Server
12. Search for Information Service 3.0 and set the logging level to DEBUG
13. Click Apply
14. On the Main Poller Web Console navigate to the group that was identified as problematic and edit the name of the group by adding one number.
15. Check the RabbitMQ console to confirm if messages have not been consumed and are still in the queue identified in step 10.
16. Reset the log level with Log Adjuster to default on Main Poller and Additional Web/Poller Server.
17. Collect diagnostics from the Main Poller and Additional Web/Poller Server and all related High Availability pairs.
18. On Additional Web/Poller Server, open SolarWInds Platform Service Manager (Default location: C:\Program Files x86)\SolarWinds\Orion\StopNetPerfMon.exe)
19. If High Availablity is enabled on this server, disable it.
20. Restart SolarWinds Information Service V3
21. Let the environment run for ~10 minutes and check if the value matches the value seen on the Main Poller.

If the Group is still not seen the same way in the Additional Web/Poller Server as on the Main Poller, enable the DEBUG logging for Information Service 3.0 on the Additional Web/Poller Server and restart the SolarWinds Information Service V3 service. Reset the Logging level back to default and collect an additional set of diagnostics.