Use SAM MIN/MAX Average Statistic charts to set thresholds based on 95th percentile data for component monitors

10/10/2018 4:28 PM

5/16/2019 4:08 PM

SAM provides many ready-to-use templates that include component monitors with thresholds that you can adjust to trigger alerts. Some component monitors include recommended threshold values, while other threshold values need to be provided for several reasons for this, including:

• The component monitor does specific things and results (statistic data) can vary in different environments.
• The component monitor executes custom scripts specific to your environment.

In general, you need to monitor applications and counters for several weeks to collect enough baseline data so you can make an educated guess about Warning and Critical thresholds. You can either define thresholds manually or use SAM's built-in functionality to calculate thresholds, as described in Understand application monitor thresholds .

This article describes how to calculate threshold settings based on 95th percentile data displayed in SAM's MIN/MAX Average Statistic charts, and then experiment with threshold values in your specific environment.

Server Application Monitor

Pre-conditions:

1. Choose a desired time interval to collect enough data to use as a baseline. This interval should be long enough to get correct minimum/maximum values for the monitor being investigated. SolarWinds recommends a one-week interval (“Last 7 Days”) for most cases.
2. Make sure the target system hosting the applications/components being monitored is stable for the desired time period. The target system needs to operate normally to provide valid statistic results for the entire time interval selected.

Methods to calculate thresholds:

1. For counters that should be as low as possible:

• Warning: Should be the 95^th percentile and above (which you can read from MIN/MAX Average Statistic Data historical charts, as described below).
• Critical: Should be the highest sampled piece of data.

2. For counters that should be as high as possible:

• Warning: Should be the 5^th percentile or lower. Calculate this setting manually, as described in the example below.
• Critical: should be the lowest sampled piece of data.

Use Case Example:

Consider the following performance counter monitor example. Every Windows performance counter monitor (as included in SAM's Active Directory and IIS templates, for example) has its own “MIN/MAX Average Statistic Data” chart, as displayed on the Component Details view for the application monitor. To change the time interval for the chart, click Edit in the chart and set the Time Period for the chart to Last 7 Days, as shown here:

You can also set a sample interval to specify how often samples are collected. When finished editing the chart settings, click Submit.

The following example shows the 95th percentile line for a day's worth of collected data for today, but you'll get better results after gathering seven days' worth of data. 

Note that the red arrows mark the information about the 95th percentile for the chart. The top arrow marks a line on the chart that has the name “95th”. This line shows the statistic value that represents 95% of the maximum on a graph. If you follow the line across the graph to the left axis, you can see that the value for the 95th percentile on the graph is 39.00.
Beneath the graph, the bottom arrow marks the caption: “95^th Percentile: 39”, which conveniently lists the 95th Percentile information so you do not need to read it from the chart. Notice the variation in the samples collected over each interval. The blue line on the chart shows the average of the values read.

Using the 95th Percentile information to calculate your thresholds

For counters that should remain as low as possible:

1. Find the 95th percentile information. In our example above, this is 39.
2. Since counters should remain as low as possible, you set the thresholds as described above:

• Warning: Should be the 95^th percentile and above (which you can read from the chart and is indicated by the 95th percentile line as 39).
• Critical: Should be the highest sampled piece of data (which you can read from the chart as 41 and is indicated by the two highest bars).

For counters that should remain as high as possible, you'll need to perform some manual calculations to get the 5^th percentile line value, because SAM charts don’t calculate and display this line. 

1. Open the extended chart view by clicking the desired chart in the Application/Component Details view:

   

2. Click Raw Data to view the actual chart data in Excel format:

   

3. Next, click the AZ button to sort data in ascending order so you can calculate the 5^th percentile line.

4. See Understanding 95^th Percentile Calculations  to learn about calculating the percentile values. 
5. Following this technique, we calculate 5% of the values to drop: 29 values * 5% = 1.45, which we round to 2 values to drop. That leaves the third value 3799 as the value for the 5th percentile. So, the warning threshold is the 5^th percentile line value (3799).
6. The critical threshold is the minimum available value, which you can find in row 2 near the top of the spreadsheet (2979).

These threshold settings will warn the user when the counter goes below the 5^th percentile, and will go critical when it goes below the MIN value ever previously seen in the selected data set. You can update the critical and warning thresholds based on the new data, assuming that the system was still performing normally when the new values were read.

Conclusions 

The procedure described here cannot be considered as an absolutely foolproof method for determining thresholds in all possible situations. However it can be useful when you do not have any other ways to determine thresholds for some particular monitors. Also it should work in cases when some monitor in your system behaves differently than described in the Microsoft documentation, but which you may consider as correct behavior.