Why Most Alerts Fail and How to Design Actionable Monitoring

Nature of Alerts

Few system alerts are well‑designed. Good alert design is hard; a low‑quality alert is one you dismiss immediately. Typical CPU‑usage‑over‑90% alerts rarely provide useful information.

High‑quality alerts should let you assess impact instantly and require a graded response; they must be actionable.

Alert Targets

Alerts can monitor business rules or system reliability.

Business‑rule monitoring checks whether software behaves according to its specifications (e.g., game damage limits, win‑rate caps) and can reveal cheating or bugs.

System‑reliability monitoring watches hardware and service health (e.g., server crashes, overloads). A typical dependency diagram shows services built on databases, RPCs, and hardware.

Metrics and Strategies

Effective reliability monitoring focuses on three questions:

Is the work getting done?

Is the user having a good experience?

Where is the problem or bottleneck?

For databases, the most informative metrics are request volume and the proportion of successful responses, rather than raw CPU usage.

Because services are layered, a single metric rarely explains a failure; you must consider the whole dependency chain.

Theory vs. Reality

Simple static thresholds often suffice if you collect the right metrics, but algorithms become necessary when:

You cannot directly count errors (need log classification).

You cannot directly measure success rate (need anomaly detection on request counts).

You only have aggregate totals and need to infer component ratios (requires fitting).

Anomaly Detection

Four basic approaches are described:

Curve smoothness: detect sudden deviation from recent trend using moving average or regression.

Absolute value periodicity: compare current value to a time‑of‑day baseline derived from historical minima.

Amplitude periodicity: examine changes between consecutive points to capture relative drops or spikes.

Curve rebound: confirm a fault when the metric later returns toward normal.

Each method has strengths and weaknesses; combining several signals yields more reliable alerts.

Key Takeaways

High‑quality alerts must be actionable.

Do not let data‑collection difficulty dictate metric choice.

Avoid default CPU‑usage alerts; focus on work‑completion metrics.

Measure request count and success rate to answer “is the work getting done?”.

Measure response latency for user experience.

When the right metrics are collected, complex algorithms are rarely needed.

Simple anomaly‑detection algorithms are easy to implement when necessary.