SAM Monitors show Unknown and the web console is slow after upgrading to 2025.4

Most SAM monitors are displaying as unknown on the web console, and web performance is significantly slow after upgrading to 2025.4.

Windows Application Event log:

ERROR .NET Runtime:

Application: SolarWinds.BusinessLayerHost.exe
Framework Version: v4.0.30319
Description: The process was terminated due to an unhandled exception.
Exception Info: System.OutOfMemoryException
   at System.Threading.Monitor.ReliableEnter(System.Object, Boolean ByRef)
   at System.Runtime.AsyncResult.Complete(Boolean)
   at System.ServiceModel.Channels.StreamSecurityUpgradeAcceptorAsyncResult.OnAuthenticateAsServer(System.IAsyncResult)
   at System.Runtime.Fx+AsyncThunk.UnhandledExceptionFrame(System.IAsyncResult)
   at System.Net.LazyAsyncResult.Complete(IntPtr)
   at System.Net.LazyAsyncResult.ProtectedInvokeCallback(System.Object, IntPtr)
   at System.Net.Security.SslState.FinishHandshake(System.Exception, System.Net.AsyncProtocolRequest)
   at System.Net.Security.SslState.WriteCallback(System.IAsyncResult)
   at System.Runtime.AsyncResult.Complete(Boolean)

Different iterations of Orion.InformationService.log:

2025-11-07 15:37:28,618 [213] ERROR ApiProxy<IAPMBusinessLayer, ServiceDirectoryQueryInternal> - w3wp| /LM/W3SVC/1/ROOT-1-134069625041214668| |  (null)     All #15 retries calling an API channel from a pool 'RootPool' failed due to a transient error detected.
2025-11-07 15:37:28,640 [213] WARN  SolarWinds.Orion.Core.Common.Licensing.LicenseSaturationLogic - w3wp| /LM/W3SVC/1/ROOT-1-134069625041214668| |  (null)     LicenseSaturationQuery: SELECT IsNull(COUNT(DistinctNodes)*(CountedFeatureElementFactor),0) AS Nodes FROM ( SELECT DISTINCT n.NodeID AS DistinctNodes FROM Orion.APM.Application AS a INNER JOIN Orion.Nodes n ON n.NodeID = a.NodeID UNION( SELECT DISTINCT n.NodeID as DistinctNodes FROM Orion.APIPoller.ValueToMonitor as vtm INNER JOIN Orion.APIPoller.ApiPoller as ap ON vtm.RequestDetails.ApiPollerId = ap.Id INNER JOIN Orion.Nodes n ON ap.RelatedEntityId = n.NodeID AND ap.RelatedEntityType = 'Orion.Nodes') ) INNER JOIN Orion.APM.LicenseInfo ON 1=1 Exception: SolarWinds.ApiProxyFactory.TransientApiProxyException: Exception of type 'SolarWinds.ApiProxyFactory.TransientApiProxyException' was thrown. ---> System.ServiceModel.ServerTooBusyException: Server 'net.tcp://SERVERNAME:17777/orion/apm/businesslayer' sent back a fault indicating it is too busy to process the request. Please retry later. Please see the inner exception for fault details. ---> System.ServiceModel.FaultException: There are too many pending secure conversations on the server. Please retry later.

APM.Businesslayer.log

2025-11-11 23:29:56,323 [53] ERROR SolarWinds.Orion.Common.SWEventLogging - Unhandled exception: Exception of type 'System.OutOfMemoryException' was thrown.
2025-11-11 23:29:56,401 [83] ERROR SolarWinds.APM.BusinessLayer.ApmBusinessLayerPlugin - Unhandled exception.
System.OutOfMemoryException: Exception of type 'System.OutOfMemoryException' was thrown.
   at System.Threading.Monitor.ReliableEnter(Object obj, Boolean& lockTaken)
   at System.Runtime.AsyncResult.Complete(Boolean completedSynchronously)
   at System.ServiceModel.Channels.SessionChannelDemuxer`2.PeekAsyncResult.OnOpenCompleteStatic(IAsyncResult result)
   at System.Runtime.Fx.AsyncThunk.UnhandledExceptionFrame(IAsyncResult result)
   at System.Runtime.AsyncResult.Complete(Boolean completedSynchronously)
   at System.Runtime.AsyncResult.AsyncCompletionWrapperCallback(IAsyncResult result)
   at System.Runtime.Fx.AsyncThunk.UnhandledExceptionFrame(IAsyncResult result)
   at System.Runtime.AsyncResult.Complete(Boolean completedSynchronously)

Current analysis indicates an overload of communication with the APM BusinessLayer that escalates into an OutOfMemoryException and repeated failures to establish secure sessions (WCF secure conversations), which then blocks new requests and cascades into polling and UI issues. Root cause is under investigation by Engineering.

To restore stability, adjust the SolarWinds.APM.BusinessLayer.dll.config file to better handle the load that it's under to prevent it crashing and being throttled by too many connections.

These changes should be done on the Primary Poller and all Polling Engines. This is not required on Additional Web Servers.

NOTE: If you have HA, please disable it globally before proceeding to avoid unnecessary failovers. To minimize downtime with HA, follow these steps on the standby servers and then fail over to them once you've completed the adjustments. Fail over the Primary first, then the Additional pollers. After they come up, you can make the config file changes on the new standby servers.

1.) Stop services on all pollers.

2.) Open Notepad as an Administrator and edit this file "C:\Program Files\SolarWinds\Orion\SolarWinds.APM.Businesslayer.dll.config"

NOTE: This is the default install location. Your file might be on a different drive if you chose one during installation.

3.) Find the line:

<SolarWinds.BusinessLayerHost LoadPlugins="true" RequiredFeature="SAM" />

And replace it with:

<SolarWinds.BusinessLayerHost LoadPlugins="true" RequiredFeature="SAM" Support64="true" SeparateProcess="true" />

4.) Find the line:

<service name="SolarWinds.APM.BusinessLayer.APMBusinessLayerService">

And replace it with:

<service name="SolarWinds.APM.BusinessLayer.APMBusinessLayerService" behaviorConfiguration="APMBusinessLayerServiceBehavior">

5.) Find the line:

 <serviceBehaviors>

And add the following section right below it one line:

<behavior name="APMBusinessLayerServiceBehavior">
      <serviceThrottling maxConcurrentCalls="250" maxConcurrentSessions="250" maxConcurrentInstances="500" />
</behavior>

Calculating the correct numbers for Throttling:

Default values:

maxConcurrentCalls = 16 * processor count

maxConcurrentSesions = 100 * processor count

maxConcurrentInstances = maxConcurrentCalls + maxConcurrentSesions

Example:

Poller has 24 processor cores

maxConcurrentCalls = 16*24 = 384 *1.5(to add 50% more overhead)= 576 -->Round up to 600

maxConcurrentSessions = 100*2400*1.5 = 3600

maxConcurrentInstances = 600 + 3600= 4200

Final Result:

<behavior name="APMBusinessLayerServiceBehavior">
      <serviceThrottling maxConcurrentCalls="600" maxConcurrentSessions="3600" maxConcurrentInstances="4200" />
</behavior>

6.) Save the file.

7.) Start the services on the Primary polling engine. Wait 2 minutes.

8.) Start the services on the rest of the Additional polling engines and web servers.

Afterwards, the system should remain stable. If an upgrade is performed, these changes will be overwritten and you will need to reapply them.