From Firefighting to Fire‑Starting: Mastering Operations for System Reliability

Operations’ Original Intent

Availability is the foundation of operations; when a service is unavailable, any effort is wasted.

The availability capability evolves through three stages:

Firefighting Stage – Keep MTTR of core modules under 20 minutes. Process: receive alert, connect VPN, locate fault, fix. Speed depends on understanding service dependencies and experience.

Fire Prevention Stage – Focus on runbooks, high‑availability design, disaster recovery, automated alerts, service degradation. Faults can be identified before full investigation, allowing pre‑emptive mitigation or graceful degradation, resulting in much lower MTTR.

Fire‑Starting Stage – Aim to keep services stable while deliberately injecting failures to discover hidden “black‑swans”. Requires having passed the previous stages and established operational procedures.

How to Practice the Fire‑Starting Stage

Conduct controlled fault injection drills: manually create failures, run through response procedures, and review outcomes. Avoid the “blue‑team/red‑team” misconception where teams cannot communicate; collaboration is essential.

Typical fault‑drill workflow: divert most traffic → inject failure → intervene → recover → restore traffic → post‑mortem.

Long‑term approach: use platforms to inject random failures without prior notice (e.g., Netflix’s Chaos Monkey) to build system antifragility.

Continuously Improving Availability

Availability can be woven into many valuable activities such as:

Offline environments (development, testing, pre‑release).

Release strategies (canary, staged rollout).

Rapid loss mitigation.

All incidents stem from changes—code, environment, network, hardware, or runtime metrics. The goal is to handle changes quickly.

Relying on intuition to locate faults creates two problems: inexperienced staff cannot find issues fast, and luck dominates, making MTTR unpredictable.

Solution : Real‑time system dashboards that visualize operational data, standardize procedures, and expose current and historical metrics (e.g., Grafana + Prometheus), making every change instantly visible.

The Purpose of Operations

Tools, automation, and platforms are means, not ends.

The aim is to continuously enhance product value throughout its lifecycle, thereby increasing the operational team’s contribution.

Operations staff should highlight the value they create, not just the effort, by linking work to tangible benefits and future potential.

Technical metrics (QPS, load) must be translated into business outcomes such as revenue, page views, or brand impact.

Ensuring availability is the core identity of an operations engineer; coupling it with product value defines a competent internet operations professional.