Troubleshooting Delayed Alert Actions and MaxConcurrentActions Tuning

4/14/2026 10:33 PM

4/14/2026 10:33 PM

If alert actions are not executing at all, this article does not apply; follow the relevant “no alert actions executed” troubleshooting instead.

Symptoms

You are likely affected if you observe one or more of the following:

• Alert actions eventually run, but with a significant delay (often 10–15 minutes or more).

• Notifications (email, HTTP requests, scripts) arrive late relative to when alert conditions were met.

• The queue of pending actions grows over time and rarely returns to a low, stable level.

Orion Platform

There are two fundamental reasons why alert actions can be delayed even when they are eventually executed:

1. High volume of actions

   • Example scenarios:

     • Alerts triggered for every syslog or SNMP trap message (no or poor filtering).

     • Large environments generating bursty alert activity.

2. Slow action execution

   • Common contributors:

     • I/O‑bound actions (file writes, external scripts, REST API calls).

     • Slow or overloaded email servers.

     • Network latency and external dependency slowness.

     • Database‑related delays (slow queries, heavy or inefficient macros).

A common pattern is that throughput (actions completed per minute) is lower than the incoming action rate, causing a constantly growing queue.

Step 1 – Check the Alerting Queue

1. On the SolarWinds server, open Performance Monitor.

2. Add the following performance counter:

   • Object: SolarWinds Alerting Service V2

   • Counter: Currently Scheduled Actions

3. Observe the behavior over time:

   • Low, stable value: System is generally healthy.

   • Continuously growing or consistently high (hundreds+):

     • Indicates a backlog of pending actions.

     • Proceed with the optimization and tuning steps below.

---

Step 2 – Optimize Before Changing MaxConcurrentActions

Important: Do not increase MaxConcurrentActions until you have optimized the alert configuration and action execution path.

Focus on reducing load and execution time:

2.1 Reduce Alert Noise

• Avoid triggering alerts for every syslog or trap message where possible.

• Use filtering rules:

  • At the Syslog/Trap level.

  • Within the alert definition (e.g., conditions that ignore flapping or low‑value events).

• Remove or consolidate unnecessary alerts and actions, especially when many actions are nearly identical.

2.2 Optimize Macros / Database Access

• Avoid macros that run queries like SELECT * FROM ..., which can force full table scans and slow down action execution.

• Instead:

  • Filter by specific IDs (such as NodeID, InterfaceID), and return only required columns.

  • Example:

    • Poor:

      • SELECT * FROM Interfaces

    • Better:

      • SELECT TOP 1 Caption FROM Interfaces WHERE InterfaceID = @InterfaceID

2.3 Reduce External Latency

Identify and address slow external dependencies that block worker threads:

• REST APIs used in web actions:

  • Measure API response times.

  • Implement retries with backoff and/or reduce payloads if possible.

• Email servers:

  • Check load and performance; resolve any queueing or throttling.

• Scripts:

  • Review PowerShell or external scripts that perform I/O or network calls; optimize or offload heavy work.

• Where supported, shorten or tune action timeouts so that failed or stalled actions do not block threads indefinitely.

---

Step 3 – Measure Action Throughput

Before adjusting MaxConcurrentActions, estimate current throughput:

1. Determine average action execution time (per action instance).

2. Use the following model:

   • If execution time ≈ 1 second and MaxConcurrentActions = 4

     • Throughput ≈ 4 × 60 = 240 actions per minute.

   • If execution time ≈ 15 seconds with the same MaxConcurrentActions = 4

     • Throughput ≈ 240 ÷ 15 ≈ 15 actions per minute.

3. Compare incoming action rate vs estimated throughput:

   • If throughput < incoming rate, the queue will grow and delays will increase.

This calculation helps determine whether increasing MaxConcurrentActions is justified or whether action optimization is still insufficient.

---

Step 4 – Analyze Execution Logs

To understand which actions are slow and how long they take, review both database history and log files:

4.1 Alert History

• Use the AlertHistoryView to:

  • Identify alerts with frequent or heavy actions.

  • Correlate action execution timestamps with observed delays.

4.2 Alerting Logs

Key logs to review on the SolarWinds server:

• Alerting.Service.V2.log

• ActionsExecutionAlert.log

• Core.BusinessLayer.log

Look for entries similar to:

• DEBUG AlertingLogger - Action [ID: …] started

• DEBUG AlertingLogger - Action [ID: …] finished successfully

Use the same Action ID and thread ID, then compare timestamps to determine how long each action took. Focus on actions and dependencies with the longest durations.

---

Step 5 – Understand MaxConcurrentActions and Parallelism

MaxConcurrentActions controls how many alert actions can execute in parallel (roughly the number of worker threads).

Key points:

• Increasing MaxConcurrentActions:

  • Does not make individual actions faster.

  • Does allow more actions to run simultaneously.

• If an action takes 15 seconds, it will still take 15 seconds after tuning; only parallelism changes.

CPU‑Bound vs I/O‑Bound Workloads

1. CPU‑bound actions (less common):

   • Heavy scripts or processing that use a lot of CPU.

   • Recommended:

     • MaxConcurrentActions ≈ number of CPU cores.

2. I/O‑bound actions (most common):

   • HTTP requests, email sending, DB queries, file I/O.

   • Recommended:

     • MaxConcurrentActions can be higher than CPU core count, because threads spend time waiting on I/O.

---

Step 6 – Recommended MaxConcurrentActions Values

Let X = number of CPU cores on the SolarWinds server.

6.1 Starting Point and Ranges

• Default value: MaxConcurrentActions = 4.

• Recommended starting guidelines:

  • Conservative: MaxConcurrentActions = X

  • Balanced: MaxConcurrentActions = 2 × X

  • Aggressive (I/O‑heavy): MaxConcurrentActions = 3–4 × X

• Typical real‑world ranges:

  • Small systems: 4 → 8

  • Medium systems: 8 → 16

  • Large / I/O‑heavy systems: 16 → 24+

6.2 Safe Scaling Strategy

1. Optimize alert configuration and actions first.

2. Measure average execution time and calculate throughput (see Step 3).

3. Increase MaxConcurrentActions gradually, for example:

   • 4 → 8 → 12 → 16 (and possibly higher for very I/O‑heavy environments).

4. Validate after each step:

   • SolarWinds Alerting Service V2 → Currently Scheduled Actions value and trend.

   • Database performance (CPU, I/O, query latency, blocking).

   • External systems (API response times, error and throttling rates, email delivery performance).

---

Step 7 – When to Stop Increasing MaxConcurrentActions

Stop increasing MaxConcurrentActions and reassess if any of the following occur:

• The alert action queue does not decrease despite higher concurrency.

• Database load increases significantly (high CPU, long‑running queries, blocking).

• External systems (email servers, APIs) become slower or start rejecting/throttling requests.

• You see more errors or timeouts in logs.

These symptoms usually indicate that you have hit another bottleneck (database, network, or external service) and that further concurrency increases will harm overall stability.

---

Common Anti‑Patterns

The following configurations frequently cause delayed actions and queue growth:

• Alert per Syslog / Trap (no filtering):

  • Every event triggers an alert and one or more actions.

  • Results in bursts that easily saturate the queue.

• Heavy or unbounded macros:

  • SELECT * FROM Interfaces, SELECT * FROM Nodes or similar whole‑table queries in macros.

  • Causes full table scans and slows down each action.

• Slow external dependencies:

  • REST APIs with high latency or strict rate limits.

  • Slow/overloaded email infrastructure.

• Blocking scripts:

  • PowerShell or external scripts performing heavy I/O or long remote calls.

• Too many similar actions per alert:

  • Multiple notifications or integrations per event without clear need, multiplying system load.

• No deduplication or suppression:

  • The same issue generates repeated alerts and actions instead of being grouped.

• Scaling concurrency without optimization:

  • Increasing MaxConcurrentActions before addressing any of the above.

  • Typically just moves the bottleneck and can destabilize other components.

---

Best Practice Summary

To handle delayed alert actions safely and effectively:

1. Confirm the scenario:

   • This guidance applies only if actions execute but are delayed, not when they fail to execute altogether.

2. Measure the queue and throughput:

   • Use Currently Scheduled Actions performance counter.

   • Estimate actions per minute based on execution time and MaxConcurrentActions.

3. Optimize first:

   • Reduce alert noise and eliminate unnecessary actions.

   • Optimize macros and SQL access patterns.

   • Fix or mitigate slow external dependencies and scripts.

4. Tune MaxConcurrentActions carefully:

   • Start from conservative values based on CPU cores.

   • Increase gradually while monitoring database and external systems.

5. Stop increasing when other components become the bottleneck:

   • Database, APIs, or email infrastructure should remain healthy—if they degrade, roll back and re‑optimize.

Following this process helps improve alerting responsiveness while preserving overall platform stability and avoiding overloading dependent systems.