Network Management

NCM Compliance Report Stuck In Caching State

A compliance report stuck in a caching state prevents the compliance update from completing or causes the CPU on the SolarWinds Platform Server to spike towards 100%.

First published date

10/22/2018 8:47 PM

Last published date

7/31/2025 9:35 PM

Overview

A compliance report stuck in a caching state

Product section

Network Configuration Manager

Cause

There are several reasons for a compliance report to be stuck.

  • Bad Regex Inside a Rule
  • Indexing Process Is Stuck
  • Too many configs in the config archive table
  • Really large configs to check in the policy
  • Too many rules applied to too may devices in the Report
  • Banner checks

We will discuss how to stop a caching report and the how to resolve the two most common causes.

Resolution

First, we must stop the report from caching:

  1. Restart SW services to stop the policy from caching.
  2. If that does not work, You can navigate to the NCM_PolicyReports Table in the Database Manager on the SolarWinds Platform Server.
  3. Click Execute Query and locate the report in question. Click Enable Table Editing at the top.
  4. In the column titled CacheStatus enter the number 3 and press Enter. 
  5. Review the rules that were in that report to modify them and recreate rules as necessary.
  6. Restart SW services.
  7. Update the policy reports and see if they are able to cache. 

Next you can narrow down the cause.

In the NCMBusinessLayerPlugin.log in the following folder: 

C:\ProgramData\Solarwinds\Logs\Orion\NCM

There will be a reference to the cause of the hanging report. If it is a Regex configuration issue, it will reference something about not being able to process the Regex. It should also state which rule the Regex is in. If the issue is related to indexing, you will see a reference to an out of memory error.

Bad Regex 

Out Of Memory (usually related to indexing) This could prove challenging. You will need to go through each Regex within each rule, within each policy, within the report. Unless the NCMBusinessLayer.log specifies the rule in which it is hanging on. This can be best handled by using a process of deduction and removing policies and or rules one at a time until the bad rule is located. Then it is about making sure you have a valid Regex. 

If there is a reference to an Out Of Memory Exception in the NCM BusinessLayer.log, this is likely due to indexing being in a hung state. In the SolarWinds Platform Web Console, navigate to Settings>NCM Settings>Search Settings. If indexing does not say 100% complete, please perform the following:

  1. Re-Index the Configuration Search
    1. Click Settings > NCM settings > Search Settings and disable and then enable the Search.
    2. Try the polling report. If still no results show,
    3. Re-index the configuration search:
      1. go to Settings > NCM Settings > Search Settings
      2. locate the "Enter a path for the index catalogue on the main NCM server" field
      3. click "Apply" force a re-indexing of the configuration search (note: allow sufficient time for re-indexing to complete)
      4. optionally you may need to restart SolarWinds services if re-indexing does not start 
    4. If indexing still fails:
      1. stop SolarWinds Module Engine
      2. go to the following path: C:\ProgramData\SolarWinds\Cirrus\Lucene
      3. delete content (all files)
  2. Run the Configuration Wizard to restart the indexing process

Note: You may need to stop services in order to delete the contents

Monitor the indexing process on the Search Settings Page mentioned above for the next few hours to see if it hangs at any point during the indexing. If indexing hangs or fails to complete again, you will need to start purging configs from the database. We suggest 30 configs per device in the database. Start by lowering the retention in the Default Purge Config Job in NCM, and then perform the same process above to restart the indexing process. If it hangs again, continue to trim the config archive. The number of configs that one customer can retain depends on their environment. Some will be successful with hundreds while larger environments will need to be closer to the recommended 30 configs. 

*Make sure that the Policy Report cache update does not conflict with scheduled NCM jobs, execute scripts, and Database maintenance since this would affect performance significantly.

It is best practice to distribute the jobs/schedules to different times without conflicts and Policy Reports should be completing correctly.