Submit a ticketCall us

Looking to compare latest NPM features with previous versions of NPM?
The NPM new feature summary offers a comparison of new features and improvements offered with this release.

 

Home > Success Center > Netflow Traffic Analyzer (NTA) > NTA Tables and NTA Summarisation Explained

NTA Tables and NTA Summarisation Explained

Overview

An Explanation of NTA Tables and Summarisation

 

Environment

NTA - All Versions

 

Detail

Netflow Tables Overview

Large tables in Netflow is the Uncompressed for 1 hour by default. These are the DETAILED NTA Tables.

What is the difference is between uncompressed and compressed data for the Netflow data management ?

Answer:
Uncompressed data is the as-collected flows. These are marked to the nearest minute and stored in the NetFlow detail tables.
This is set to 60 minutes by default but is user settable and can be increased for up to 4 hours.
When the 4 hours is reached, the data is summarized into summary tables.

The summary tables are 15 min intervals for 1 day, 1 hr intervals for 1 week
and daily intervals for as long as the “retain compressed data” setting allows (up to 3650 days).

 

So if you open Orion Website, Admin Tab and NTA Settings, you will see these settings:

Keep uncompressed data for 4 hours  - (15 mins to 240 mins) Min and Max setting for UnCompressed Data

http://thwack.com/forums/p/13673/56150.aspx#56150
Now it gets very Technical because there is Auto Shrink for Netflow going on in Background as in Netflow 3.1 added Netflow their own NDF Files in SQL

NDF File is same MDF file, just where there is more than one MDF file = DATA File. LDF File is Transaction Logs.

Now have 4 Netflow ndf files ...
FG1 - Stores the FlowCorrelation Pre/Post table
FG2 - SearchBy Tables
FG3 - Summary 1,2,3 Tables
FG4 - Details tables


If you open SQL Server Mgt Studio, FG4 = Detailed uncompress Data (but possible shrinked),
you might see correlation to these and amount Traffic being received.


Detailed NTA Tables in Database are stored in FG4 NDF...
Numbers corresponding to NodeID and InterfacesID...
NetFlowDetail_1459_1373340                              
NetFlowDetail_1459_1373341                              

 

Which get Summarized into Summary Tables

Also Note, after upgrading Netflow, the DB Migration may continue for few days in Background:
http://thwack.com/forums/t/13660.aspx?PageIndex=1


"In NTA 3.0, we would never perform shrinks on your database. 
In NTA 3.1, we do.  In SQL Server, when disk space is freed up from deletions in the database, the space is not recovered by the filesystem. 
Doing so, would take time for SQL Server and so they leave the decision to the user as to when to "shrink" the database and let the filesystem reclaim that space. 

In NTA 3.1, we perform the shrinks a few times during and once after the migration process. 
Furthermore, there is now a nightly maintenance that can be setup in the NTA admin pages, which also performs shrinks."
"FG4 stores only your uncompressed data.  If you reduce your 4 hour uncompressed down to 1 hour,
and restart the service, then when the next night maintenance occurs and a shrink is performed on the database, then that disk will reclaim the space."

**************************************************

Netflow Summary Tables explained


NTA - 3.x - Customer facing template on data retention and compression


Q:           How is data groomed? Number of days data is retained in each table?

NetflowDetails/NetflowSummary1/NetflowSummary2/NetflowSummary3

 

A:            We keep as-received data for the setting of “Uncompressed data”.       

We roll up as-received data from 1 min segments to 15 minute segments each 15 minutes and put it to NetFlowSummary1 table
We then roll up 15 minute segments every X hours to hourly data

 

NetFlowSummary1 –

This table holds the summarized historical data for the first collapse level. 
The data are collapsed and moved to the NetFlowSummary2 table after certain number of hours.
The data in this table summarizes a 24 hours traffic by default. (CollapseTrigger2InHours option in NetFlowGlobalSettings = 24)

NetFlowSummary2 –

This table holds the summarized historical data for the second collapse level. 
The data are collapsed and moved to the NetFlowSummary3 table after certain number of days.
The data in this table summarizes a 3 days traffic by default. (CollapseTrigger3InDays option in NetFlowGlobalSettings = 3)

NetFlowSummary3 –

This table holds the summarized historical data for the third collapse level.  The data are deleted after certain number of days.
The data in this table summarizes a 30 days traffic by default. (RetainCompressedDataInDays option in NetFlowGlobalSettings = 30)
NetFlowSummary is just logical union of this 3 table to refer them all in once.

 

Netflow and DNS

Q: What is the process for DNS resolution?

A: There several tables involved in that process:

  •                 dbo.NetFlowAddressToResolve keeps IP Addresses need to be resolved (new one or expired that needs to be renewed)
  •                 dbo.FlowCorrelationPostDNS keeps information about resolved IP addresses and their expiration time
  •                 dbo.NetFlowEndpoints keeps all unique IP addresses that were captured by NTA

 

********************************************************************************************************

DNS Resolution: Persistent/On Demand/Disabled

 

There are several important options in Admin->NTA Settings->Global settings that impact DNS resolution:
(more here http://www.solarwinds.com/NetPerfMon/SolarWinds/default.htm?context=SolarWinds&file=OrionNetFlowPHDNSResolutionOptions.htm)

Default number of days to wait until next DNS lookup – determine the period after which DNS information will be refreshed for existing IP addresses
Default number of days to wait until next DNS lookup for unresolved IP Addreses -  Default number of days to wait until next DNS lookup for unresolved IP Addreses.
Enable NetBIOS resolution of endpoints – determine to use or not NetBios resolution method

We resolve IP addresses continuously if there are records in NetFlowAddressToResolve.
If number of record in high (use this SQL query: select count(*) from dbo.NetFlowAddressToResolve with(nolock))
it means that DNS resolution is slow or there too many unique IP addresses, so it’s recommended to use On Demand DNS feature instead of Persisting resolving.

********************************************************************************************************

NetflowGlobalSettings 

 

Q:           NetflowGlobalSettings table, which important information does it have?

A:            This table keeps setting common for all NTA receivers (main and additional).

Most of this options can be changed through UI at Admin page. In description column you can find short description of each option.
 

********************************************************************************************************

Check how tables are distributed among filegroups in SQL with NTA v4

 

See how tables are distributed among filegroups in SQL with NTA 4.0 installation. 
We should consider redefining filegroups layout, e.g. group Endpoint related tables into single filegroup.
http://fogbugz.swdev.local/default.asp?295358

SELECT * from (
SELECT
FILEGROUP_NAME(AU.data_space_id) AS FileGroupName,
OBJECT_NAME(Parti.object_id) AS TableName,
SUM(AU.total_pages/128) AS TotalTableSizeInMB,
SUM(AU.used_pages/128) AS UsedSizeInMB,
SUM(AU.data_pages/128) AS DataSizeInMB
FROM sys.allocation_units AS AU
INNER JOIN sys.partitions AS Parti ON AU.container_id = CASE WHEN AU.type in(1,3) THEN Parti.hobt_id ELSE Parti.partition_id END
LEFT JOIN sys.indexes AS ind ON ind.object_id = Parti.object_id AND ind.index_id = Parti.index_id
Group by AU.data_space_id, parti.object_id
)  A
WHERE TableName like '%NetFlow%' ORDER BY FileGroupName, TotalTableSizeInMB DESC

Last modified
20:57, 22 Jun 2016

Tags

Classifications

Public