Submit a ticketCall us

Announcing NCM 7.7
With NCM 7.7, you can examine the rules that make up an access control list for a Cisco ASA device. Then you can apply filters to display only rules that meet the specified criteria, order the rules by line number or by the hit count, and much more.
See new features and improvements.

Home > Success Center > Network Performance Monitor (NPM) > Module Engine (Businesslayer) crashing and low disk space issues

Module Engine (Businesslayer) crashing and low disk space issues

Created by Seamus.Enright, last modified by MindTouch on Jun 23, 2016

Views: 63 Votes: 0 Revisions: 14

Overview

This article addresses the issue where the Module Engine or other Orion services crash or stop responding due to low disk space in the Orion server.

Environment

All Orion products

Cause

The Orion server may experience low disk spaced due to an excessive growth of Windows Error Reporting logs which can include the following:

  • NCM Configs are stored on SW Programdata folder.
  • NCM Session tracing is left on.
  • DEBUG Logs are not turned off.
  • VERBOSE Logs are not turned off.
  • Windows Error Reporting (WER) files are growing excessively.  
  • NCM v7.3 stores downloaded Configs in NCM_ConfigArchive Table, but users often also save a copy of Config to Disk ( ..Solarwinds\Orion\NCM\Config-Archive). This leads to thousands or millions of downloaded Configs stored locally within the Orion Install directories.

 

Resolution

Notes: 

  • When troubleshooting issues in Orion, it is beneficial to use LogAdjuster to elevate the amount of information collected in our logs.  Unless our Support team has specifically asked to leave logs in an elevated state, they need to be returned to their default state. 
  • The Program Files & Inetpub folders should remain static and can be configured during first installation to point to other disk drives.
  • ProgramData & MSMQ locations should remain where they are by default.C:\Program Files (x86)\SolarWinds\
  • Orion has 4 main folder locations: 
    • C:\Inetpub\SolarWinds
    • C:\Windows\System32\msmq\storage\
    • C:\Program Files (x86)\SolarWinds\ 
      • C:\Program Files (x86)\SolarWinds\ could contain copy of NCM Config-Archive, if so move it to its own location.
    • C:\ProgramData\Solarwinds\C:\ProgramData\Solarwinds\ can take up to approximately 2-4 GB as it contains the following:
      • MIB Database: 1500MB MIB Database 
      • 1GB of installer files that should not be removed
      • 0-1GB of SDF Files Collector and JE
      • Solarwinds rolling logs
  1. Go to the following location
    C:\ProgramData\Microsoft\Windows\WER\ReportQueue
    Note: You can use a third-party tools, such as Treesize to help identify where the large folders reside..image2015-5-1 11-9-12.png
  2. Clean up orphaned log files. Delete any files named .LOG or .LOG.1-5 logs but skip files that are still in use.
  3. Edit Diagnostics Config Files not to pull back large tables. ​Diagnostics - How do we gather diagnostics and how you can edit defaults?
  4. Reset all logging to default using Log Adjuster. Launch LogAdjuster using one of following methods:
  5. Run c:\Program Files (x86)\SolarWinds\Orion\LogAdjuster.exe or go to to Start > > All Programs > Solarwinds Orion > Documentation and Support > Orion Log Adjuster.
  6. Click Reset to Defaults.
  7. Click Apply.
  8. Replace Orion services:
    1. Log in to your server as a Local Administrator.
    2. Verify that User Account Control (UAC) is turned off. This requires a reboot.

    3. Log back in as a Local Administrator.

    4. Open Orion Service Manager and stop all services.

    5. Click Control Panel > Programs and Features.

    6. Uninstall the following:

      • SolarWinds Job Engine v1

      • SolarWinds Job Engine v2

      • SolarWinds Collector

    7. Go to to C:\ProgramData\Solarwinds\Installers
    8. Run the following msi files:

      • CollectorInstall.msi

      • JobEngine.msi

      • JobEngine.v2.msi

    9. Stop and then start all services from the Orion Service Manager  

  9. If your  environment site has NCM installed
    1. Verify that NCM Config files are not stored in SolarWinds folders. Otherwise, move them to another drive and set up a default location from NCM Web settings pages. 
      Note: NCM SESSION-TRACING can be disabled and those files can be deleted as they can grow quite large.
    2. Setup NCM download configjobs to only store configs files that have changed since last download. However, this may cause the following issues:
    3. Verify that Session Trace is turned off once the necessary traces have been collected. SwTelnetDebug.log and Session Traces files will be created when Session Tracing is enabled. 
    4. Delete session trace files from the following location:
      C:\Program Files\Solarwinds\Configuration Management\Session-Trace\
      Note: You can edit the storage location of these files through NCM Web Settings.
  10. Clear out VIM DEBUG LOGS
    1. Go to Start > All Programs > Solarwinds Orion > Documentation & Support > LogAdjuster.
    2. Click Reset All to Default and Save.
    3. Clear out DEBUG logs. 
    4. For VIM 1.5 and later:
      1. Run c:\Program Files (x86)\SolarWinds\Orion\LogAdjuster.exe
      2. Set the following options:
        • Data Processor log level: DEBUG
        • File size: 50 MB. 
      3. Set the same options for VIM VMware Jobs and VIM VMware Jobs - Results generator.
      4. In the VIM Section set all to DEBUG and 50MB. 
      5. Click Poll now on the affected ESX\vCenter Server. 
      6. Open DataProcessor.log and locate the following string 
      7. VMwarePollingResultsProcessor - VMwareJobResults saved to file
      8. In this directory you can find VMware poll results where they reside and send the zipped directory to SolarWinds with VMware results is either in:
        • c:\Windows\SysWOW64\VMwareJobResultsOutput\ or
        • C:\ProgramData\Solarwinds\Collector\Logs\VMwareJobResultsOutput
Last modified
22:42, 22 Jun 2016

Tags

Classifications

Public