Submit a ticketCall us

Webinar: Web Help Desk for HR, Facilities and Accounting Departments
This webinar will focus on use cases for HR, Facilities and Accounting.

Having a unified ticketing and asset management system for all the departments in your company can provide end-users with a seamless experience and make things easier for your IT team. Yet, with different business tasks and objectives, many departments don’t fully understand the capabilities of Web Help Desk and how the software can be customized for effective use in their departments.
Register Now.

Home > Success Center > Netflow Traffic Analyzer (NTA) > NTA Flow Storage service error: Failed to index new data

NTA Flow Storage service error: Failed to index new data

Updated: March 14, 2017

Issue

The NTA Flow Storage Service is crashing with the following error in a log: 2017-03-02 08:57:46,747 [30] ERROR SolarWinds.Netflow.FastBit.Server.Service.Program - NetFlowService will be abnormally terminated - UnhandledException was caught : System.AccessViolationException: Attempted to read or write protected memory. This is often an indication that other memory is corrupt.

Environment

NTA 4.2.x and later

Cause 

The issue can happen when a NetFlow enabled node is receiving Cisco Wireless LAN Controller (WLC) or Cisco Next Generation Network-Based Application Recognition (NBAR2) data. The issue is caused by a corrupted column index of the NTA Flow Storage database (FSDB) partition when WLC and non-WLC data or NBAR2 and non-NBAR2 data are mixed into the same data partition. The NTA Service crashes when the corrupted columns are queried to show NBAR2 or WLC related resource or when backup is triggered as it queries all columns.

The following error can be seen in a log: ERROR FastbitCLR.FastbitIndex - appendInternal: Failed to index new data. Partition: TN#2016-10-23@11_0, column: AdvancedApplicationID. The column AdvancedApplicationID indicates NBAR2 related data issue and columns WtpMAC, ClientIP or PostDSCP indicate WLC related data issue.

Resolution

To prevent further corruption of the NBAR2 related column indexes upgrade to NTA 4.2.1 HotFix 1 or later. For NTA version 4.2.0 and 4.2.1 without any HotFix, manual steps need to be done:

  1. On all NetFlow pollers:
    1. Navigate to the NTA installation directory. The default location is C:\Program Files (x86)\SolarWinds\Orion\NetFlowTrafficAnalysis\
    2. Edit the file NetFlowService.exe.config. Make backup first if needed.
    3. Change setting from enableUnsavedNullableColumns=”true” to enableUnsavedNullableColumns=”false” at a line beginning with flowStorageConfig.

    4. Restart the NetFlow Service.
  2. On the NTA Flow Storage server:
    1. Navigate to the FSDB configuration directory. The default location is C:\ProgramData\SolarWinds\NTA\FlowStorage\
    2. Edit the file ServerConfig.xml. Make backup first if needed.
    3. Change setting from <enableUnsavedNullableColumns>true</enableUnsavedNullableColumns> to <enableUnsavedNullableColumns>false</enableUnsavedNullableColumns>.

    4. Restart the Flow Storage Service.

 

To prevent further corruption of the WLC related column indexes manual steps need to be done on all NetFlow pollers.

  1. Navigate to the NTA installation directory. The default location is C:\Program Files (x86)\SolarWinds\Orion\NetFlowTrafficAnalysis\
  2. Edit the file NetFlowService.exe.config. Make backup first if needed.
  3. Change setting from sendAllColumns=”false” to sendAllColumns=”true” at line beginning with flowStorageConfig.
  4. Restart the NetFlow Service.

There are some downsides of having this setting sendAllColumns set to true. One is higher network utilization as larger amount of data are sent from the NetFlow pollers to the Flow Storage. Other is higher disk space requirements for the Flow Storage database as larger amount of data are stored. The difference is typically about 15 percent.

The setting sendAllColumns is rewritten to the default value of "false" after each upgrade or repair of NTA installation and needs to be set again.

 

The FSDB partitions with already corrupted indexes need to be deleted manually. This has to be done for all partitions listed in the log files as failed to index new data. The NTA FastBit Studio command CheckIntegrity can be also used to identify corrupted partitions. Running the command BuildIndexes in the NTA FastBit Studio does not help to fix the corrupted indexes. Deletion of corrupted partitions should be done after index corruption measures are in place.

To delete FSDB partitions with corrupted indexes:

  1. Stop the NTA Flow Storage Service.
  2. Navigate to FSDB Flows folder. The default location is C:\ProgramData\SolarWinds\NTA\FlowStorage\Data\Flows\ The current location can be found by running the Flow Storage Configurator.
  3. Delete folders with corrupted partitions. For example TN#2016-10-23@11_0 in this instance.
  4. Start the NTA Flow Storage Service.

 

 

Last modified
06:42, 15 Mar 2017

Tags

Classifications

Public