How to Design Actionable Alerts and Effective Monitoring Strategies

告警的本质

Most system alerts are not well designed; a good alert should be actionable, allowing immediate assessment of impact and a graded response.

High‑quality alerts enable you to evaluate the scope of an issue instantly and require a clear, actionable response.

告警对象

Alerts can be divided into two categories:

Business rule monitoring – checks whether software behaves according to business constraints (e.g., game damage limits, win‑rate caps) and detects cheating or logic errors.

System reliability monitoring – detects hardware or service failures such as server crashes or overloads.

监控的指标和策略

Effective monitoring should answer three questions:

Is the work getting done?

Is the user having a good experience?

Where is the problem or bottleneck?

For databases, key metrics include request count and the proportion of successful responses; similar request‑and‑success ratios apply to other services.

理论与现实

Simple static thresholds often suffice if the right metrics are collected, but algorithms become necessary when:

Errors cannot be directly counted and need log classification.

Success rates are unavailable and anomaly detection on raw counts is required.

Only aggregate totals are available, requiring factor‑based fitting.

异常检测

Four intuitive approaches to detect anomalies in time‑series data:

Curve smoothness – sudden loss of smoothness indicates a fault.

Absolute value periodicity – compare current values against a historical baseline for the same time of day.

Amplitude periodicity – examine changes (Δx) rather than raw values to capture relative shifts.

Curve rebound – a clear recovery after a dip confirms a fault occurred.

基于曲线的平滑性检测

Use a recent window (e.g., 1 hour) and compute an exponentially weighted moving average (EWMA); deviations beyond a multiple of the variance trigger alerts.

基于绝对值的时间周期性

Calculate a dynamic threshold per time‑of‑day using the minimum of the past 14 days multiplied by a factor (e.g., 0.6); alert when the current value falls below this line.

基于振幅的时间周期性

Analyze the difference between consecutive points (or points spaced further apart) to detect abnormal speed of change, adjusting for both relative and absolute amplitude.

基于曲线回升的异常判断

Detect faults by confirming a clear rebound after a dip; this helps validate alerts and build historical fault datasets for more advanced models.

总结

High‑quality alerts must be actionable.

Do not choose metrics based on collection ease; prioritize usefulness.

Avoid relying solely on CPU usage thresholds.

Work‑completion metrics: request count + success rate.

User‑experience metric: response latency.

With proper metrics, complex algorithms are rarely needed.

Simple anomaly‑detection algorithms are sufficient when required.