SolarWinds service account locked after changing password on Active Directory (AD) on SAM

The following issue can occur with the following symptoms:

1. After removing the SolarWinds service account from the Windows credential menu and the account has been deactivated in AD. But Orion continues to send authentication requests for the deactivated account.
2. After changing the password associated with the SolarWinds service account, the account keeps getting locked out due to failed authentication attempts.

In the Cortex.DataCollection.log (under LogFiles > Cortex), the following error message will appear.

<Timestamp> [15] ERROR DataCollection.SolarWinds.Cortex.DataCollection.Internal.Protocols.WMI.WmiPollingService - QueryAsync exception for DOMAIN\\ServiceAccount at X.X.X.X with query select AvgDiskQueueLength, AvgDiskSecPerTransfer, AvgDiskSecPerTransfer_Base, DiskReadsPerSec, DiskWritesPerSec, Frequency_PerfTime, Timestamp_PerfTime, Timestamp_Sys100NS from Win32_PerfRawData_PerfDisk_LogicalDisk where Name = "E:".SolarWinds.Cortex.Monitoring.WMI.Exceptions.WmiSecurityException: Access is denied. (Exception from HRESULT: 0x80070005 (E_ACCESSDENIED)) Scope: X.X.X.X\root\cimv2[\DOMAINI\SERVICEACCOUNT] ErrorCode: 0x80070005 ---> System.UnauthorizedAccessException: Access is denied. (Exception from HRESULT: 0x80070005 (E_ACCESSDENIED))

This issue occurs because the SolarWinds service account is stored in the Cortex_Document table.  If the account is deactivated or replaced, or the password is changed, the record in the Cortex_Document was not updated.

This issue is resolved in SolarWinds Platform 2023.2 and above. If a workaround is needed, perform the following steps below:

Resolution

1. Perform a snapshot of the Orion DB or Database backup.
2. Open Orion Database Manager (Start > Search Database Manager)
3. Click Add Orion Server
4. Right-click the Orion database
5. Click New Query
6. Paste the query below:

-- Scripts are not supported under any SolarWinds support program or service.
-- Scripts are provided AS IS without warranty of any kind. SolarWinds further
-- disclaims all warranties including, without limitation, any implied warranties
-- of merchantability or of fitness for a particular purpose. The risk arising
-- out of the use or performance of the scripts and documentation stays with you.
-- In no event shall SolarWinds or anyone else involved in the creation,
-- production, or delivery of the scripts be liable for any damages whatsoever
-- (including, without limitation, damages for loss of business profits, business
-- interruption, loss of business information, or other pecuniary loss) arising
-- out of the use of or inability to use the scripts or documentation.

truncate table Cortex_Documents

If deleting the whole table is a concern, the following query can be used:

-- Scripts are not supported under any SolarWinds support program or service.
-- Scripts are provided AS IS without warranty of any kind. SolarWinds further
-- disclaims all warranties, including, without limitation, any implied warranties
-- of merchantability or of fitness for a particular purpose. The risk arising
-- out of the use or performance of the scripts and documentation stays with you.
-- In no event shall SolarWinds or anyone else involved in the creation,
-- production or delivery of the scripts be liable for any damages whatsoever
-- (including, without limitation, damages for loss of business profits, business
-- interruption, loss of business information, or other pecuniary loss) arising
-- out of the use or inability to use the scripts or documentation.

DELETE FROM Cortex_Documents
WHERE DeletedDate IS NOT NULL
OR Data LIKE '%"ModelType": "Orion.Volume"%' OR Data LIKE '%"Type": "Orion.NodeToVolumes"%'
OR Data LIKE '%"ModelType": "Orion.Cpu"%' OR Data LIKE '%"Type": "Orion.NodeToCpus"%'
OR Data LIKE '%"ModelType": "Orion.Interface"%' OR Data LIKE '%"Type": "Orion.NodeToInterfaces"%'