Network Management

Charts - Percent Memory (Node) vs Memory Diskspace (Volume) Utilization

Percent Memory (Node) vs Diskspace (Volume) Utilization.

First published date

11/6/2018 3:18 AM

Last published date

3/21/2025 3:09 PM

Overview

The following diagram shows one of the issues in Memory Utilization.

 

If you are seeing mismatches then the issue here is with your SNMP Agent.
We are simply polling memory OIDs on 2 fronts:

  1. Polling Volumes (RAM is also a volume)
  2. Polling Memory Directly

Product section

Network Performance Monitor

Resolution

  • Two different OIDs sets and calculations are used as shown below.
  • While both should match up, when they don't, it is due to the SNMP Agent returning incorrect data and that is for the Vendor to rectify.

To further troubleshoot: 

  • Use PollerChecker tool to see which Polling Job is assigned and Supported.
  • Check what Values it brings back and whether values are correct or not.
  • Use SNMP Walk and SNMP responder to see values returned from a particular device.
  • Use DEVICE Studio manage pollers to overwrite default memory poller used for troublesome node.

 

Useful Links 

*********************************************************************************************************************************

Further Information 

 

There can be also due to oddness of the SNMP results from the system.


The results for the two statistics are gathered from entirely different sets of MIBs.

  • For the statistics posted under CPU Load and Memory underlying information is gathered from a combination of MIBs. We look at the process run table and the performance table (hrSWRunTable and hrSWRunPerfTable) to add up the memory commits for each process running and then use that to determine the percent of memory used.
  • This is a bit laborious and usually results in very high memory usage stats.
  • For the stats under the Volumes section these are gathered from a single table and are entirely based on the results of this table, hrStorageTable. I suspect in this particular instance, good information might not be got from the hrSWRunPerfTable, and sometimes memory usage is LOW or HIGH .
     

The "Memory" Usage and the "Physical Memory" Utilization are calculated differently by Orion NPM.


Method 1: Memory: 
The Memory is the sum of the memory used by the processes listed in the HOST-RESOURCES-MIB:hrSWRunPerfTable (1.3.6.1.2.1.25.5.1).
The OID used is the hrSWRunPerfMem (1.3.6.1.2.1.25.5.1.1.2).

Method 2: Physical Memory:
The Physical Memory is calculated based on the values populated in the HOST-RESOURCES-MIB:hrStorageTable (1.3.6.1.2.1.25.2.2.3).
The OIDs used are hrStorageSize (1.3.6.1.2.1.25.2.3.1.5), hrStorageUsed (1.3.6.1.2.1.25.2.3.1.6) and hrStorageAllocationUnits (1.3.6.1.2.1.25.2.3.1.4) .

Difference between the 2 methods:

  • Depending on how the SNMP agent on the host is reporting values, it can lead to some discrepancies between the two results. I would advise users to use the most accurate between both and to unmanage the other element.
  • Results will vary depending on the platforms.

 

Method 1: Memory Polling (CPU\Memory Pollers)

Memory.SNMP.MemoryHrSwRunPerf
N.Memory.SNMP.HrSwRunPerf		Memory poller for devices which support HOST-RESOURCES-MIB (Windows SNMP agent) 
Poller utilizes GET-REQUEST to gather total memory size using OIDs:
hrMemorySize = 1.3.6.1.2.1.25.2.2.0

Poller utilizes GetSubtree to gather memory used by running processes using OID:
hrSwRunPerfMem = 1.3.6.1.2.1.25.5.1.1.2.

Poller computes used memory as SUM of memory used by all enumerated processes.
This polling method might produce different values then values displayed in Windows Task Manager,
because it does not count with memory used by system itself (kernel).

Method 2: Memory Polling via Volumes (Volume Pollers)

Volume and memory calculations are separate. Here is what the poller does for volumes and then for memory:

  • VolumeSize = OID_hrStorageSize * OID_hrStorageAllocationUnits
  • VolumeUsed = OID_hrStorageUsed * OID_hrStorageAllocationUnits
  • VolumePercentUsed = VolumeUsed / VolumeSize * 100
     
  • TotalMemory = OID_hrMemorySize * 1024
  • UsedMemory = OID_hrSWRunPerfMem (sum of all returned values)
  • MemoryUtilization = 100 * UsedMemory / TotalMemory
     
  1. The Orion server monitors the amount of total committed memory including swap space. 
  2. Percent Memory Utilization within Orion NPM is based on real memory AND swap space. 

*********************************************************************************************************************************

Memory over 100% on Windows Servers 

This is due to Microsoft combining physical and virtual memory as one figure for usage. 

  • Percent Memory Utilization within Orion NPM is based on real memory AND swap space.
  • If a device has 100MB of memory and is using 90MB, then the percent memory used will be reported as 90%.
  • If it is using 100MB, then percent memory used will be reported as 100%.
  • However, if the device is using 100MB of memory (all of it) AND 50MB of swap space, then the percent memory used will be reported as 150%.
     

So if your getting Reports/Alerts of 100% Memory Utilization then you have to decide if this is a problem for your particular application/device or not. For example, a server running a set credit card processing applications may run just fine with 2GB of physical memory and 2GB of committed swap,but a SQL Server may run terribly with 2GB of physical memory and 2GB of swap.  

*********************************************************************************************************************************

Memory Usage Graphs Explanation 


When looking at the Node Details page for a Windows server. It is possible to see values on two resources that seem to report different values for Memory Used.

  • One of the resources that displays this information is CPU Load & Memory Utilization which displays a radial dial (a speedometer) labeled Memory Used.
  • The other is the Disk Volumes resource and it shows a table of values for Physical Memory,
  • Virtual Memory and other values that are not relevant to this discussion. See the attached files for screen shots taken from the same Node Details page.

Each of the values shown in these two screen shots is reported from a separate MIB.

  • This means that Orion is not performing any calculations on these values.
  • The values reported in Disk Volumes are raw numbers. 

So in this example, the server being monitored has a maximum of 383 MB of physical RAM and 1.5 GB of virtual memory. 
The discrepancy arises in how Windows calculates the value of Memory Used.
Percentage usage of any quantity is a ratio of the amount used to the amount available
.But the MIB which reports Memory Used divides the Physical Memory being used (327MB) by a combination of the Physical Memory available (383MB) and the Virtual Memory available (1.5Gb).


Thus, the MIB reports 17.3659% memory used instead of 85% Physical Memory Used or 98% Virtual Memory Used.
This error in reporting is understood industry-wide as an error in the way that Windows calculates this value. 
SolarWinds is confident that Microsoft will update their code in a future release which will correct this reporting issue and therefore the CPU Load & Memory Utilization resource has not been removed from Orion.

*********************************************************************************************************************************

Linux Memory Usage 

In 10.2 we did change and started polling hrStorage for Windows boxed
(In case hrStorage doesn't report data - then we poll hrStorageUsed) 
and special NetSNMP OIDs for NetSNMP (Linux/Unix boxes). 

  •     /// memTotalReal = 1.3.6.1.4.1.2021.4.5.
  •     /// memAvailReal = 1.3.6.1.4.1.2021.4.6.
  •     /// memCached = 1.3.6.1.4.1.2021.4.15.
  •     /// memBuffer = 1.3.6.1.4.1.2021.4.14.
  •     /// UsedMemory = (totalMem - availMem - cachedMem - bufferMem) * 1024.0


There are currently three ways how the 'Memory Utilization' (in the CPU Load > and Memory Utilization resource) value is calculated.
The three ways are:

1) OIDs: hrMemorySize and hrSWRunPerfMem (1.3.6.1.2.1.25.2.2 and > 1.3.6.1.2.1.25.5.1.1.2)

  • For example: Common on Windows Hosts running the MS SNMP service.
  • Assuming the GNU/Linux box is running Net-SNMP these OIDs are probably not relevant to this case.



2) OIDs: memTotalReal and memAvailReal (1.3.6.1.4.1.2021.4.5 and > 1.3.6.1.4.1.2021.4.6)

  • Deducting memAvailReal from memToTalReal should give you how much of memory is used. (i.e. 'Absolute' memory utilization)
  • These OIDs are commonly polled for Memory Utilization on many GNU/Linux systems (running Net-SNMP).
  • The value obtained via these OIDs pretty much represents how much of the physical memory is occupied with data.
  • What should be kept in mind is that some systems can make use of the 'free' memory by using it as cache.
  • Such memory is practically free/available, should an application need it, however from an 'Absolute' point of view it's taken.
  • On such systems memory utilization based on these two OIDs is therefore pretty much all the time reported as almost 100% even though there's alot of memory actually available.
     

3) OIDs: memTotalReal, memAvailReal, memBuffer and memCached (1.3.6.1.4.1.2021.4.5, 1.3.6.1.4.1.2021.4.6, 1.3.6.1.4.1.2021.4.14 and 1.3.6.1.4.1.2021.4.15)

  • This method is similar to method number 2), however instead of just deducting memAvailReal (from memTotalReal)
  • It also deducts memBuffer and memCache in order to solve the 'issue' with systems reporting 99-100% memory utilization all the time (as described above).
  • The resulting value should therefore represent more accurately the real  memory usage.


Note: the way 3) takes precedence over 2). 
(i.e. if memBuffer and memCached are available they are deducted.)

As for the Physical Memory (or Real Memory etc.) Volume and why it can show a different value than the Node Memory Utilization graph:

  • Although the value in Volumes is polled from different OIDs, and therefore can vary based on SNMP implementation, the value shown is typically what method number 2) would show (i.e. how much of memory is taken, no matter if whatever is taking up the memory is actual data or just buffers/cache).
  • The Volume's Real Memory is showing the how much of memory is used absolutely, i.e. including the cache etc. (And because of the nature of the system it might be close to 100% all the time.)

 

CPU usage is reporting incorrectly for Linux servers using Net-SNMP 

Issue:

CPU usage is reporting incorrectly for Linux servers using Net-SNMP

Resolution:

This issue has been experienced when monitoring Linux servers using Net-SNMP versions 4.0 through 5.7. 
These versions of Net-SNMP do not properly handle counter rollovers for the ssCpuRawIdle OID. For more information, see the following related Red Hat bugs:


To correct this issue:

Upgrade all monitored Linux servers to the latest version of Net-SNMP. For more information, including packages for download, see Net-SNMP - Download.
This article applies to all SolarWinds Orion products 

*********************************************************************************************************************************

Useful Tools to help troubleshoot and resolve Poller jobs issues: 


Solarwinds SNMP Walk Tool 


Our latest version of NPM Ships with a MIB walk tool.

Located in Directory:

 

This is also a useful tool for testing and confirming SNMP connectivity to a device.
The SolarWinds SNMP Walk tool is available at: SolarWinds SNMP Walk.

**************************************************************************************************

Poller Checker Tool 

Use the Poller Checker tool

Poller Checker tool is shipped with along with NPM v 10.2 and above. 
You can find this in:
c:\Program Files (x86)\SolarWinds\Orion\PollerCheckerTool.exe


You can can memory Pollers against your device to see which ones are supported.

Note: After you run a scan, results pane returns a lot of information:

  •    Light red        - Poller is Not Supported on this device, No error Returned
  •    Dark red        - Poller is Not Supported on this device, Error returned See Error message
  •    Dark green    - Poller is Supported on this device, Already assigned to this Device
  •    Light green    - Poller is Supported on this device, But is not Assigned to this Device

**************************************************************************************************

Device Studio Documentation & Videos 

Template for helping Customer on how to Poll non-supported OIDs using UnDP or Device Studio.

Device Studio Videos:

*********************************************************************************************************************************