Storage I/O polling in DPA is disabled because the file limit was exceeded

10/22/2018 12:54 PM

8/25/2022 2:16 PM

The Storage I/O feature in DPA frequently collects file statistics from a monitored database instance. If the instance includes a large number of files, collecting I/O data can incur excessive overhead for both the monitored instance and for DPA. To avoid performance issues, DPA automatically disables storage I/O polling when the number of files exceeds a specified limit. The default limit is 250.

 

If storage I/O polling is disabled, you will see a message similar to the one below when you attempt to access Storage I/O:

Storage I/O has been automatically disabled for this database instance due to the large number of files in this database instance. This was done because gathering I/O data for a large number of files may negatively affect the performance of both DPA and the database instance.
353 files detected. Current limit is 250.

Additionally, the idc.log includes a log entry similar to the one below:

IOCollectionServiceImpl - Disabling IO stats collection for database [id=3] because the number of files 315 exceeds the IO stats poll file limit.

Database Performance Analyzer

Considerations 

If you choose to re-enable storage I/O polling for a database instance with a large number of files, be aware of the following options and their effect on performance.

Option	Description
IO_STATS_POLL_ENABLED (Automatic vs. manual enablement mode)	Determines whether storage I/O polling is enabled, and whether this value automatically changes based on the file limit. The following values are available: AUTO_ON: Polling is currently enabled. It will be automatically disabled if the number of files exceed the file limit. AUTO_OFF: Polling is currently disabled because the file limit is exceeded. It will be automatically re-enabled if the number of files drops below the file limit. MANUAL_ON: Polling is manually enabled. DPA ignores the file limit, and storage I/O data is gathered regardless of the number of files. (Selecting this value can negatively affect performance.) MANUAL_OFF: Polling is manually disabled. If you see the error described in the previous section, DPA has changed this option to AUTO_OFF.
IO_STATS_POLL_FILE_LIMIT (File limit)	Specifies the maximum number of files that can reside on a database instance before DPA automatically disables storage I/O polling. Applies only if the enablement mode is AUTO_ON or AUTO_OFF.
IO_STATS_POLL_INTERVAL (Polling frequency)	Specifies how often storage I/O data is collected. Polling too frequently with a large number of files can negatively affect performance.

Enabling the Storage I/O Feature 

To re-enable storage I/O polling for a monitored instance with a large number of files, complete the following steps to increase the file limit and (if necessary) decrease the polling frequency.

Note: Alternatively, you can tell DPA to ignore the file limit and always enable storage I/O polling. To do so, set IO_STATS_POLL_ENABLED (on the DB Instance Options tab) to MANUAL_ON. However, be aware that MANUAL_ON is not recommended because it can negatively affect performance.

We recommend that you follow these instructions under the guidance of SolarWinds Support.

1. Increase the file limit:
   1. Click Options.
   2. Click the Administration tab.
   3. Click Advanced Options.
   4. Select Support Options in the upper right corner.
   5. On the System Options tab, click IO_STATS_POLL_FILE_LIMIT and enter a value that is larger than the number of files in the error message.
   6. Restart the monitor.
      
      If the number of files is below the file limit, IO_STATS_POLL_ENABLED automatically changes to AUTO_ON. Storage I/O polling is re-enabled.
       
2. On a regular basis, review the idc.log file for errors associated with I/O data collection. Examples:
   IOCollectionServiceImpl - Error collecting IO Stats for QASQL2K12. IO Stat collection for this entity will be retried in 5 minutes. Retries will continue indefinitely, however, this error will only be logged after every 10th failed attempt. This error can be ignored if sporadic.
   
   IOCollectionServiceImpl - IO Stats query gap detected for dbId [41] last collection: 03/14/2015 08:55:11, current collection: 03/14/2015 08:56:31, difference [79995] > 30000

IOCollectionServiceImpl - Added file to blacklist [dbId:35] [dbId=2, fileId=5 [2|5]] [expires: 03/16/2015 12:05:30] Missing stats detected.
    
3. If errors occur, increase the polling interval:
   1. Click Options.
   2. Click the Administration tab.
   3. Click Advanced Options.
   4. Select the Support Options checkbox in the upper right corner.
   5. Click the DB Instance Options tab.
   6. Select the database instance.
   7. Click IO_STATS_POLL_INTERVAL and increase the value. We recommend 30 or 60 (seconds).
   8. Restart the monitor.