Job Engine v3 service flapping in the SolarWinds Platform after an upgrade due to BITS Proxy

The Job Engine v3 process cannot establish a healthy connection to the SolarWinds Certificate Management Service (CMS) because local gRPC traffic is being sent through a proxy configured on the BITS (Background Intelligent Transfer Service). The proxy returns HTTP 403 responses, causing Job Engine v3 to exit and be repeatedly restarted by Windows Service Recovery.

Clearing/correcting the BITS proxy configuration on the APE resolves the issue, and Job Engine v3 remains stable.

On the affected APE:

* SolarWinds Job Engine v3:

- Service repeatedly starts and stops (flapping).
- Windows Services console shows the service running briefly before stopping again.

* SolarWinds Web Console may show:

- APE in a degraded state.
- Data gaps from elements polled by that APE.

* System/Application logs and platform logs show:

- Errors around Certificate Management and gRPC connectivity.
- Intermittent SQL connectivity errors from the APE to the SolarWinds Platform database server.

The Job Engine v3 process cannot establish a healthy connection to the SolarWinds Certificate Management Service (CMS) because local gRPC traffic is being sent through a proxy configured on the BITS (Background Intelligent Transfer Service).

Resolution:

1. Remove or correct BITS proxy on the APE

On the affected APE:

1. Open an elevated Command Prompt or PowerShell.

2. Check BITS/IE proxy mapping for LocalSystem (example):

   bitsadmin /util /getieproxy localsystem
   

3. If a proxy is defined, reset or correct it so that localhost and local SolarWinds servers are not proxied. For example, to remove proxy inheritance and avoid forced proxy for LocalSystem:

   bitsadmin /util /setieproxy localsystem NO_PROXY
   

   or adjust per your corporate standard to ensure local addresses are bypassed.

Note: Exact proxy policy may be controlled via GPO. Coordinate with the system/network team to ensure LocalSystem and BITS traffic on SolarWinds servers either has no proxy or properly bypasses localhost, 127.0.0.1, and SolarWinds core/APE hostnames.

2. Ensure CMS is running

On the APE:

1. Open Services.msc.
2. Confirm SolarWinds Certificate Management Service:
   • Startup type: Automatic
   • Status: Running
3. If not running, start the service.

3. Restart Job Engine v3

After correcting the proxy configuration:

1. Restart SolarWinds Certificate Management Service (if proxy was changed).
2. Restart SolarWinds Job Engine v3 service.
3. Monitor:
   • Services.msc: ensure Job Engine v3 remains in Running state.
   • SolarWinds.JobEngine.Service_v3_*.log: verify no new gRPC/HTTP 403/proxy errors and no “CMS Client is not available” fatal entries.

Once the BITS proxy was removed in the real case, Job Engine v3 stopped flapping and ran normally.

Additional Recommendations

Logs may also show repeated SQL connectivity issues from the APE to the SolarWinds Platform database server.

1. Validate SQL connectivity from the APE

From the APE:

Test-NetConnection <SQL_Server_IP_or_FQDN> -Port 1433

• If this fails:
  • Check firewalls between the APE and SQL server.
  • Verify SQL Server is listening on the correct port and configured to allow remote connections.

2. Confirm SolarWinds DB account status

On the SQL instance:

• Validate the SolarWindsOrionDatabaseUser login:
  • Correct password (not changed unexpectedly after upgrade).
  • Not locked, disabled, or denied access to the SolarWindsOrion database.
  • Required permissions are intact.

3. General best practices for proxy configuration on SolarWinds servers

For all SolarWinds core and APE servers:

• Ensure local traffic is never forced through a proxy, including:
  • localhost
  • 127.0.0.1
  • SolarWinds core and APE hostnames/FQDNs
• Review and, if needed, standardize:
  • WinINET / IE proxy settings
  • WinHTTP proxy:

    netsh winhttp show proxy
    netsh winhttp reset proxy
    

  • BITS proxy or IE proxy inheritance for LocalSystem:

    bitsadmin /util /getieproxy localsystem
    bitsadmin /util /setieproxy localsystem NO_PROXY
    

  • Global environment variables (HTTP_PROXY, HTTPS_PROXY, ALL_PROXY) so they do not intercept localhost or internal SolarWinds traffic.