Submit a ticketCall us

Putting Your Logs Where They Belong with the New SolarWinds Log Manager for Orion

The new SolarWinds® Log Manager for Orion® finally puts your log data right where it belongs, in the heart of your Orion console. Gain insight into the performance of your infrastructure by monitoring your logs in a unified console allowing you to see a wealth of information about the health and performance of your network and servers.

Reserve a Seat for Wednesday May 23rd 11am CDT | Reserve a Seat for Tuesday May 22nd 10:30am GMT | Reserve a Seat for Tuesday May 22nd 1pm SGT / 3pm AEST

Home > Success Center > Database Performance Analyzer (DPA) > Storage I/O polling is disabled because the file limit was exceeded

Storage I/O polling is disabled because the file limit was exceeded

Updated March 11th, 2016

Issue

The Storage I/O feature in DPA frequently collects file statistics from a monitored database instance. If the instance includes a large number of files, collecting I/O data can incur excessive overhead for both the monitored instance and for DPA. To avoid performance issues, DPA automatically disables storage I/O polling when the number of files exceeds a specified limit. The default limit is 250.

 

If storage I/O polling is disabled, you will see a message similar to the one below when you attempt to access Storage I/O:

Storage I/O has been automatically disabled for this database instance due to the large number of files in this database instance. This was done because gathering I/O data for a large number of files may negatively affect the performance of both DPA and the database instance.
353 files detected. Current limit is 250.

Additionally, the idc.log includes a log entry similar to the one below:

IOCollectionServiceImpl - Disabling IO stats collection for database [id=3] because the number of files 315 exceeds the IO stats poll file limit.

Considerations

If you choose to re-enable storage I/O polling for a database instance with a large number of files, be aware of the following options and their effect on performance.

Option

Description

IO_STATS_POLL_ENABLED
(Automatic vs. manual enablement mode)

Determines whether storage I/O polling is enabled, and whether this value automatically changes based on the file limit. The following values are available:

  • AUTO_ON: Polling is currently enabled. It will be automatically disabled if the number of files exceed the file limit.
  • AUTO_OFF: Polling is currently disabled because the file limit is exceeded. It will be automatically re-enabled if the number of files drops below the file limit.
  • MANUAL_ON: Polling is manually enabled. DPA ignores the file limit, and storage I/O data is gathered regardless of the number of files. (Selecting this value can negatively affect performance.)
  • MANUAL_OFF: Polling is manually disabled.

If you see the error described in the previous section, DPA has changed this option to AUTO_OFF.

IO_STATS_POLL_FILE_LIMIT
(File limit)

Specifies the maximum number of files that can reside on a database instance before DPA automatically disables storage I/O polling. Applies only if the enablement mode is AUTO_ON or AUTO_OFF.

IO_STATS_POLL_INTERVAL
(Polling frequency)

Specifies how often storage I/O data is collected. Polling too frequently with a large number of files can negatively affect performance.

Enabling the Storage I/O Feature

To re-enable storage I/O polling for a monitored instance with a large number of files, complete the following steps to increase the file limit and (if necessary) decrease the polling frequency.

Note: Alternatively, you can tell DPA to ignore the file limit and always enable storage I/O polling. To do so, set IO_STATS_POLL_ENABLED (on the DB Instance Options tab) to MANUAL_ON. However, be aware that MANUAL_ON is not recommended because it can negatively affect performance.

We recommend that you follow these instructions under the guidance of SolarWinds Support.

  1. Increase the file limit:
    1. Click Options.
    2. Click the Administration tab.
    3. Click Advanced Options.
    4. Select Support Options in the upper right corner.
    5. On the System Options tab, click IO_STATS_POLL_FILE_LIMIT and enter a value that is larger than the number of files in the error message.
    6. Restart the monitor.

      If the number of files is below the file limit, IO_STATS_POLL_ENABLED automatically changes to AUTO_ON. Storage I/O polling is re-enabled.
       
  2. On a regular basis, review the idc.log file for errors associated with I/O data collection. Examples:
    IOCollectionServiceImpl - Error collecting IO Stats for QASQL2K12. IO Stat collection for this entity will be retried in 5 minutes. Retries will continue indefinitely, however, this error will only be logged after every 10th failed attempt. This error can be ignored if sporadic.

    IOCollectionServiceImpl - IO Stats query gap detected for dbId [41] last collection: 03/14/2015 08:55:11, current collection: 03/14/2015 08:56:31, difference [79995] > 30000

    IOCollectionServiceImpl - Added file to blacklist [dbId:35] [dbId=2, fileId=5 [2|5]] [expires: 03/16/2015 12:05:30] Missing stats detected.
     
  3. If errors occur, increase the polling interval:
    1. Click Options.
    2. Click the Administration tab.
    3. Click Advanced Options.
    4. Select the Support Options checkbox in the upper right corner.
    5. Click the DB Instance Options tab.
    6. Select the database instance.
    7. Click IO_STATS_POLL_INTERVAL and increase the value. We recommend 30 or 60 (seconds).
    8. Restart the monitor.
Last modified

Tags

Classifications

Public