Mastering Incident Response: Principles and Methods for Effective Operations

1. Fault‑Handling Principles

The two core principles are:

Prioritize business recovery.

Escalate promptly.

1.1 Business‑Recovery First

Regardless of the fault level, the immediate goal is to restore service, not to locate the root cause. For example, when Application A fails to call Application B:

Method 1 : Diagnose the path between A and B, identify the failing component (e.g., HA connection), then restart or scale it.

Method 2 : From A’s server, ping B. If the network and port are reachable, bind B’s address in the hosts file.

Method 2 is usually faster, but if A and B span data‑centers, Method 1 may take longer yet still restores service quickly.

1.2 Timely Escalation

When a fault occurs, its impact can only be roughly predicted, so inform leadership immediately. Escalate without delay if any of the following apply:

Clear business impact (e.g., PV, UV, cart, order, payment metrics).

Critical‑business alerts (core services, core components).

Processing time exceeds defined thresholds.

Senior leaders, monitoring centers, or customer‑service have noticed the issue.

The problem exceeds the responder’s capability.

Note: Operations leaders must be the first to know about any incident; learning about it from another team indicates a handling failure.

2. Fault‑Handling Methodology

Incident handling is divided into three phases: pre‑incident (analysis), during‑incident (resolution), and post‑incident (review). The focus here is on the “during” phase.

2.1 Service‑Centric Methods

The three most important actions for restoring service are restart, isolation, and downgrade .

Restart : Includes service restart and OS restart. The typical order is the faulty object → its upstream → its downstream. Example: for a RabbitMQ failure, restart RabbitMQ first, then the producer if needed, and finally the consumer.

Do not skip a restart simply because metrics look normal; the goal is to restore service, not to locate the root cause.

Isolation : Remove the faulty component from the cluster so it no longer provides service. Common approaches:

Set upstream weight to zero or stop the service if health checks exist.

Bind hosts or adjust routing to bypass the faulty component.

Isolation helps prevent avalanche effects.

Downgrade : Deploy a fallback plan to avoid larger failures. Downgrade is never the optimal user experience but may be necessary (e.g., alternative payment flow). It requires coordination with development to ensure idempotent, stateless services.

2.2 Impact‑Centric Methods

Incidents affect two groups: external users and internal users.

2.2.1 External Users

Goal: convert external‑user problems into internal‑user problems when possible.

Reproduce the issue locally; if reproducible, it is an internal problem.

If not reproducible, involve additional internal users, try hosts binding or DNS checks, and gather external information (IP, client version) for further analysis.

2.2.2 Internal Users

Includes internal service calls and staff‑reported issues; handle them using the same methods described in 2.1.

2.3 Organizational Structure During an Incident

Three roles typically act simultaneously:

Incident Responder – restores service as quickly as possible.

Incident Investigator – finds the root cause when the responder’s methods fail.

Communicator – shares accurate status internally and externally.

In practice, not all roles are present at once (e.g., night‑shift may only have a responder). Roles can be reused as needed.

3. Post‑Incident Review

Every incident requires a thorough summary to address root causes, prevent recurrence, and apply PDCA for continuous improvement.

When documenting, consider responsibility attribution and handle difficult stakeholders carefully. Downgrade should be viewed as the last line of defence.