NTA Charts - NTA Top Talker/Sampling/Flow Export Timeouts (NPM versus NTA Charts)

A Customer asks WHY an Interface with 90% Utilization doesn't match NTA Charts which show 60%?
Charts do not appear to agree with each other.

Below are a few troubleshooting steps to worth investigating:

To begin with it is important when comparing NPM and NTA charts to consider the following: 

• Be sure to compare same interface with same time period. 
• Be sure that NTA resource is set to display Rate(kbps) and check flow direction (use 'Both' for example). Check that the NTA view is not filtered.
• Use NTA resource which shows all traffic - Top Protocol is ideal in most cases. 
  • If you use Top 10 Conversations for example, there is no guarantee that all traffic is displayed - there are probably much more other conversations, so only fraction of traffic is shown.

*****************************************************

(1) Netflow Flow Export Timeouts 

Maybe worth adjusting likes of below to reduce amount NTA being sent to us and still get accurate results? 
https://supportforums.cisco.com/docs/DOC-6113 (© 2018 Cisco Systems Inc, available at https://www.cisco.com/c/en/us/support/index.html, obtained on October 19, 2018)

flow-export template timeout-rate 1                   

More Info:
An active flow timer or inactive flow timer limit is reached. 
The last in order of precedence are the flow timers. 

Breaks up long-lived flows into 1-minute fragments. You can choose any number of minutes between 1 and 60.
If you leave it at the default of 30 minutes your traffic reports will have spikes.
It is important to set this value to 1 minute in order to generate alerts and view troubleshooting data  (© 2019 Zoho Corp., available at https://www.manageengine.com, obtained on March 23, 2019).

  Ensure that flows that have finished are periodically exported. The default value is 15 seconds.
  You can choose any number of seconds between 10 and 600.
  However, if you choose a value greater than 250 seconds, NetFlow Analyzer may report traffic levels that are too low.

*****************************************************
(2) Netflow Sampling 

flow-sampler-map random_one-out-of-100

Samples is configuring NetFlow to act like sFlow. It decreases the amount of data available. 
This is good because it’s less impact on the database, but it’s bad because you aren’t collecting all of the flows. 
As long as the customer understands the tradeoffs, I don’t see a problem. 
Usually these routers are sending so much traffic that it would be a burden for the device, the network, and the collector to parse it all. 
Also, we will still use SNMP to determine how much traffic is going in and out of the interfaces.

I understand that the transmit/receive utilization will be correct as it’s from NPM via SNMP, however will the types of figures below also be correct?
I have seen 1 of 100 used, and compared to a 20 out of 100 test. Both showed the same relative amount of traffic.

To Clarify, by relative amounts of traffic  I mean the 1% and 20% samples showed that same patterns in the traffic relative to each port being graphed. 
Of course the 1 was 1/20 the total traffic of the 20% sample.

Sampling, which makes it netflow data less accurate. In this case NTA seems to receive flows correctly, 
but the device sends sampled approximation of communication. If you want accurate receive bps numbers, he’s better to use NPM. 

*****************************************************

(3) Solarwinds Setting - NTA Top Talker 

This is a feature we implemented, and its great in most cases but where 100% netflow is needed, 
 

Top Talker may be better for you when set to 100%, just be sure have good SQl Server, RAID 10 etc..
While working on Orion NetFlow Traffic Analyzer (NTA) v3.5, our dev staff spent a lot of time analyzing your packet captures.What they found was really interesting!   
On many networks, up to 95% of all network traffic (i.e. Top Talkers) can be captured and analyzed from as little as 4% of the total amount of Flow data received from monitored Flow sources. 
 

When NTA receives very fragmented data and TOP Talker optimization (TTO) is enabled, big amount of those data can be dropped.

Solutions:

1. Router: Change MTU Stack to produce bigger packets.
2. NTA: Set TOP Talker optimizations from default 95% to 100%.

Because when traffic is very fragmented all fragments have similar size and it is a problem for TTO. 
TTO needs to have differences in packet sizes to drop only the small ones. 
Otherwise it randomly drops some packets (everything has same size) and it can be for instance whole egress traffic.

We had couple of those issues and users usually decided to set TTO to 100% and problem was solved.
 

• • PDU Dropped: By Limiter
  • PDU Dropped: Unmanaged Interfaces
  • PDU Dropped: Unmonitored Interface
  • PDU Dropped: Unmonitored Port
  • PDU Dropped: Unmonitored Protocol

*****************************************************

(4) NPM SNMP Traffic Polling - How differs to NTA and lead different results
NTA gives breakdown of the Traffic, What Type of Traffic where who, what application, ports, etc…
But NPM is will simply report Traffic on the Interfaces as shown below.

If one wants to compare NPM and NTA, then one has to store as much data in NTA as possible. 
This goes against saving some disk space (Sometimes Customers will disable storing of flows on unmonitored ports if concerns about disk space/performance)
So you need to have NTA Settings enabled in order for all Flows to be stored in NTA Database. 

• Allow monitoring of flows from unmanaged Interfaces
• Enable data retention for traffic on unmonitored flows
• In some case Top Talker needs to be set from 95% to 100%.

How NPM Gathers Traffic statistics:
32bit Counter based

64bit Counter based

OID = ifInOctets 1.3.6.1.2.1.2.2.1.10.
Total Transmitted and Received Bytes
                In_TotalBytes = InOctets - prev_InOctets
                Out_TotalBytes = OutOctets - prev_OutOctets
Bps and Percent Utilizations

Out_Bps = (Out_TotalBytes * 8) / Seconds
Out_PercentUtil = Out_bps / OutBandwidth * 100
In_Bps = (In_TotalBytes * 8) / Seconds

                In_PercentUtil = (In_bps / InBandwidth * 100)

What does NPM show for these interfaces? 
Is it accurate? “If you want accurate receive bps numbers, he’s better to use NPM.”

*****************************************************

(5) Data inconsistency between NPM and NTA - Troubleshooting Guide

Data inconsistency between NPM and NTA
Sometimes customers complain about inconsistent data between NPM and NTA resources. 
These issues may be caused by various reasons:

• If the difference is small and charts correlate, there maybe no issue at all.  Both products acquire data through different methods. 
  NPM uses total bytes counters, polled by SNMP with 9 minutes interval by default. 
  NTA uses flows exported from routers, and adds bytes in each flow to get rates. 
  So it's dependent on what a particular device exports - some devices may not export small flows, or do some sort of aggregation. 
  Also flows are exported much more often, so the data have more detailed granularity.
   
• Be sure to compare same interface with same time period. 
  Be sure that NTA resource is set to display Rate(kbps) and check flow direction (use 'Both' for example). Check that the NTA view is not filtered.
   
• NPM charts have separate curve for received and transmitted bytes. 
  NTA has this combined when direction is 'Both'.
   
• Use NTA resource which shows all traffic - Top Protocol is ideal in most cases. 
  If you use Top 10 Conversations for example, there is no guarantee that all traffic is displayed - there are probably much more other conversations, so only fraction of traffic is shown.
   

• If NTA shows less: 

  • The point here is to verify that NTA stores all flows to database and correctly displays data in charts
  • Check NTA business layer log and NetFlow performance counters - look for dropped packets, failed packet parsing, unmanaged interface warning etc.
  • Check Top Talker Optimization settings - changing to 100% might help.
  • Check 'Allow monitoring of flows from unmanaged interfaces' option (enabling might help) or check if all required interfaces are managed by Orion and set as NetFlow source.
  • Check monitored protocols. "PDU Dropped: Unmonitored Protocol" counter increases if some protocol is disabled and flows with this protocol are received
  • Check monitored applications. "PDU Dropped: Unmonitored Port" counter increases if some application is disabled and flows with this protocol are received
  • mls portion of device configuration is missing
  • Check device configuration - verify that NetFlow is generated for all traffic (all interfaces). Active flow export rate should be 1 minute. Ingress/egress data should be exported, etc.
  • Check packet capture - check that it contains flows according to configuration.
  • If expecting to see multicast traffic, make sure you use v9. Multicast traffic is not reported in v5 flows.
  • NTA v3.11 - device is sending sampled netflow v9, make sure it is exporting SAMPLING_INTERVAL (34) and SAMPLING_ALGORITHM (35) fields in template.

• If NTA shows more: 

  • In packet capture look for flow duplication 
    - this is device configuration problem in most cases (NetFlow is exported on input and output from interface or more interfaces).
     
• Because NTA shows averaged rates it can be easily identified as bug that rates for several dates differ from NPM rates. 
  This is caused by fact that when some Endpoints are talking for example 10 minutes at speed 10Mbps than the same data shown with day granularity will be much lower than 10Mbps..
   
• Generally this is about explanation or device configuration. 
  When investigating, packet capture (~15 min) helps, and also detailed explanation with screenshots.