Disk Space Fills Up Due to Excessive Key Files in MachineKeys Directory on SolarWinds Platform Additional Polling Engine Server

Some users may report that the disk space on Additional Polling Engines is being consumed rapidly due to the constant creation of .key files in the following directory:

C:\ProgramData\Microsoft\Crypto\RSA\MachineKeys

Upon investigation, it was found that these files are linked to the CN=SolarWinds-Orion certificate. If the number of files associated with this certificate is excessive, the issue may be related to a bug fixed in SolarWinds 2025.2.

No error is logged during this behavior, making it harder to detect without monitoring the file system.

The SWIS service performs a periodic certificate check every 10 minutes. If discrepancies are found between the certificate in the database and the one in the Windows Certificate Store, a new key file is created and stored in the MachineKeys directory. These files are never deleted, even when the services are restarted. This behavior is caused by a bug that is fixed in SolarWinds version 2025.2.

Please note the following for the scripts below:

-- Scripts are not supported under any SolarWinds support program or service.
-- Scripts are provided AS IS without warranty of any kind. SolarWinds further
-- disclaims all warranties including, without limitation, any implied warranties
-- of merchantability or of fitness for a particular purpose. The risk arising
-- out of the use or performance of the scripts and documentation stays with you.
-- In no event shall SolarWinds or anyone else involved in the creation,
-- production, or delivery of the scripts be liable for any damages whatsoever
-- (including, without limitation, damages for loss of business profits, business
-- interruption, loss of business information, or other pecuniary loss) arising
-- out of the use of or inability to use the scripts or documentation.

Workaround: Modify SWIS Config to Stop Further File Generation

1. Connect to the SolarWinds server.
2. Navigate to: C:\Program Files\SolarWinds\Orion.
3. Backup SolarWinds.InformationService.Service.exe.config outside the Orion folder (e.g., Desktop).
4. Edit the file and add/update the following settings under <appSettings>:

<add key="ServerCertificates-CheckEnabled" value="false" />
<add key="ServerCertificates-CheckInterval" value="3600" />
<add key="ServerCertificates-CheckShortInterval" value="3600" />

5. Save the file and restart the SolarWinds Information Service.

Script: Identify Number of Key Files Matching Each Certificate

Use this PowerShell script to determine how many key files correspond to the CN=SolarWinds-Orion certificate:

$certificatePath = "Cert:\LocalMachine\My"
$machineKeysPath = "C:\ProgramData\Microsoft\Crypto\RSA\MachineKeys\"

function LoadParametersFromKeyFile {
    param ([string]$filename)
    $providerType = 1
    $providerName = "Microsoft Enhanced Cryptographic Provider v1.0"
    $containerNameOffset = 0x28
    $containerName = New-Object System.Text.StringBuilder
    try {
        $keyFile = [System.IO.File]::OpenRead($filename)
        $keyFile.Position = $containerNameOffset
        while ($true) {
            $byte = $keyFile.ReadByte()
            if ($byte -eq 0 -or $byte -eq -1) { break }
            $containerName.Append([char]$byte) | Out-Null
        }
        $keyFile.Close()
    } catch {
        Write-Error "Failed to read key container name from file: $_"
        return
    }

$cspParameters = New-Object System.Security.Cryptography.CspParameters($providerType, $providerName)
    $cspParameters.Flags = [System.Security.Cryptography.CspProviderFlags]::UseMachineKeyStore
    $cspParameters.KeyContainerName = $containerName.ToString()

try {
        $rsaKey = New-Object System.Security.Cryptography.RSACryptoServiceProvider($cspParameters)
        $rsaParameters = $rsaKey.ExportParameters($false)
        $rsaKey.Dispose()
    } catch {
        Write-Error "Failed to load RSA parameters: $_"
        return
    }
    return $rsaParameters
}

$certificates = Get-ChildItem -Path $certificatePath
$keyModulusList = foreach ($keyFile in Get-ChildItem -Path $machineKeysPath) {
    $rsaParameters = LoadParametersFromKeyFile($keyFile.FullName)
    [PSCustomObject]@{
        Modulus = [System.Convert]::ToBase64String($rsaParameters.Modulus)
    }
}

$result = foreach ($certificate in $certificates) {
    $rsaPublicKey = $certificate.PublicKey.Key
    $modulus = $rsaPublicKey.ExportParameters($false).Modulus
    [PSCustomObject]@{
        Subject = $certificate.Subject
        Modulus = [System.Convert]::ToBase64String($modulus)
        NumberOfKeyFiles = 0
    }
}

foreach ($key in $keyModulusList) {
    foreach ($certificate in $result) {
        if ($certificate.Modulus -eq $key.Modulus) {
            $certificate.NumberOfKeyFiles++
        }
    }
}

$result | Select-Object Subject, NumberOfKeyFiles

Optional: Backup All MachineKey Files Before Cleanup

$machineKeysPath = "C:\ProgramData\Microsoft\Crypto\RSA\MachineKeys\"
$timestamp = Get-Date -Format "yyyyMMdd_HHmmss"
Compress-Archive -Path $machineKeysPath\* -DestinationPath $machineKeysPath\backup_$timestamp.zip

Optional: Script to Delete Redundant Key Files Matching Orion Certificate

Ensure you back up the directory before using this script. Use with extreme caution.

Running the script as Administrator may not be sufficient, since the files in that directory are protected by stricter system-level permissions. The script needs to be executed under the SYSTEM user context, which has the necessary access rights. To do this, you can use the PsExec tool from Microsoft Sysinternals.

$certificatePath = "Cert:\LocalMachine\My"
$machineKeysPath = "C:\ProgramData\Microsoft\Crypto\RSA\MachineKeys\"

function LoadParametersFromKeyFile 
{
    param (
        [string]$filename
    )

    $providerType = 1 #PROV_RSA_FULL
    $providerName = "Microsoft Enhanced Cryptographic Provider v1.0"
    $containerNameOffset = 0x28

    $containerName = New-Object System.Text.StringBuilder
    try {
        $keyFile = [System.IO.File]::OpenRead($filename)
        $keyFile.Position = $containerNameOffset

        while ($true) {
            $byte = $keyFile.ReadByte()
            if ($byte -eq 0 -or $byte -eq -1) { break } # Stop at null or EOF
            $containerName.Append([char]$byte) | Out-Null
        }

        $keyFile.Close()
    }
    catch {
        Write-Error "Failed to read key container name from file: $_"
        return
    }

    $cspParameters = New-Object System.Security.Cryptography.CspParameters($providerType, $providerName)
    $cspParameters.Flags = [System.Security.Cryptography.CspProviderFlags]::UseMachineKeyStore
    $cspParameters.KeyContainerName = $containerName.ToString()

    # Load RSA parameters
    try {
        $rsaKey = New-Object System.Security.Cryptography.RSACryptoServiceProvider($cspParameters)
        $rsaParameters = $rsaKey.ExportParameters($false) # false to not include private key
        $rsaKey.Dispose()
    }
    catch {
        Write-Error "Failed to load RSA parameters: $_"
        return
    }

    return $rsaParameters
}

# Get SolarWinds-Orion certificate from cert store
$orionCertificate = Get-ChildItem -Path $certificatePath | Where-Object { $_.Subject -like "CN=SolarWinds-Orion" }

# Calculate modulus of SolarWinds-Orion key
$orionCertificateModulus = [System.Convert]::ToBase64String($orionCertificate.PublicKey.Key.ExportParameters($false).Modulus)

# Get key container name from the SolarWinds-Orion
$orionKeyFile = $orionCertificate.PrivateKey.CspKeyContainerInfo.UniqueKeyContainerName

$keyFiles = Get-ChildItem -Path $machineKeysPath

$deletedFilesCount = 0
# Loop through key files and extract Modulus
foreach ($keyFile in $keyFiles) 
{
    $rsaParameters = LoadParametersFromKeyFile($keyFile.FullName)

    $keyPath = $keyFile.FullName
    $keyModulus = [System.Convert]::ToBase64String($rsaParameters.Modulus)

    if ([string]$keyFile -ne [string]$orionKeyFile) {
        # Compare modulus of the key from file and SolarWinds-Orion key modulus
        if ($orionCertificateModulus -eq $keyModulus) 
        {
            try 
            {
                Remove-Item -Path $keyPath -Force
                $deletedFilesCount += 1
            }
            catch 
            {
                Write-Host "Failed to delete: $keyFile"
            }
        }
    }
}

Write-Host "`nDeleted $deletedFilesCount key files."

Note: The powershell scripts may take a long time to finish depending on the number of files in the MachineKeys directory and the performance of the server.