Google SRE Weekly Alert Limits and Practical Strategies for Reducing Alert Fatigue

Google SRE Weekly Alert Limits

Operations interviewers often ask if on‑call duty is required; the author shares personal experience of handling massive alert volumes during a period of simultaneous system upgrades, leading to high turnover and burnout.

Two screenshots illustrate a contrast: one team experiences a peak of 55,348 daily alerts, while another peaks at only 34, a 1,600‑fold difference, reflecting the reality for many Chinese internet operations teams.

Google SRE aims for no more than ten alerts per week for services with a 99.99% SLA. The author's team targets at most two nights of alerts per week and no more than 50 alerts per day, a much looser threshold.

By controlling daily alert counts, strictly limiting night‑time alerts, and ensuring at least four people share on‑call duties, the team has maintained low turnover and attracted top talent.

On‑Call and Alert Escalation

Two engineers are assigned daily on‑call; if an alert is not acknowledged within 5 minutes or not resolved within 15 minutes, it is escalated to the broader team. If the monitoring system lacks on‑call scheduling, manual rotation or script‑based recipient changes can be used.

On‑call staff should carry laptops for rapid response, and compensation should be provided for holiday duty.

Severity‑Based Response

Alerts should be categorized: critical alerts require immediate action, while lower‑severity notifications can be handled later. Google SRE classifies outputs as alerts, tickets, or records, with only true fault‑level alerts demanding immediate response.

Self‑Healing and Automation

Simple issues often resolve with a restart, covering at least 50% of alerts. Automation should handle repeatable, well‑defined alerts, but must respect service redundancy and avoid excessive automated restarts.

Key automation considerations include:

Do not exceed service redundancy.

Avoid repeated automation on the same issue in a short period.

Provide a global kill‑switch for automation.

Avoid high‑risk operations.

Ensure each action’s result is collected before proceeding.

Alert Dashboard Optimization

Top‑3 alerts often account for 30% of total alerts; assigning owners to these can quickly reduce volume. Further analysis by module, machine, time window, and type enables fine‑grained optimization.

Time‑Based Alert Throttling

During peak traffic, a higher alert threshold (e.g., 20% of instances) may be required, while off‑peak can tolerate higher instance failure rates before alerting, reducing night‑time alerts.

Preventing Spurious Alerts

Configure alerts to trigger only after sustained metric breaches (e.g., 5 out of 10 consecutive samples) to avoid transient spikes causing noise.

Proactive Warning

Trend‑based early warnings (e.g., disk usage approaching 5% free) allow non‑urgent remediation during business hours.

Routine Inspection

Regular inspections catch irregular issues that alerts miss, such as occasional CPU spikes that don't meet alert thresholds.

Metric‑Based Monitoring Standards

CPU_IDLE < 10

MEM_USED_PERCENT > 90

NET_MAX_NIC_INOUT_PERCENT > 80

CPU_SERVER_LOADAVG_5 > 15

DISK_MAX_PARTITION_USED_PERCENT > 95

DISK_TOTAL_WRITE_KB (optional)

DISK_TOTAL_READ_KB (optional)

CPU_WAIT_IO (optional)

DISK_TOTAL_IO_UTIL (optional)

NET_TCP_CURR_ESTAB (optional)

NET_TCP_RETRANS (optional)

Addressing On‑Call Aversion

Policies include maintaining at least four on‑call engineers, offering temporary relief for senior staff who reduce alert volume, and imposing penalties for excessive daily alerts.

Conclusion

After reducing alert volume, focus shifts to alert accuracy and recall, ensuring that every alert corresponds to a real impact and that all impacts generate alerts.