How to Ensure System Stability and High Availability: An SRE Playbook

1. Deep Understanding of Stability and High Availability

Stability and high availability are frequently discussed concepts. Improving these metrics makes systems healthier and enhances user experience. To define them precisely, we refer to Wikipedia:

Stability is a term in mathematics or engineering that determines whether a system produces bounded output for bounded input. If it does, the system is stable; otherwise, it is unstable.

High availability (HA) is an IT term describing a system's ability to operate without interruption, representing the degree of availability and the ability of components to run for longer periods.

In practice, an application can be seen as a system, service requests as inputs, and responses as outputs. When responses meet expectations, the system is stable. Extending this, a product is stable when user requests (inputs) produce correct product behavior (outputs).

Thus, a system is stable if it consistently generates correct, expected output for given input; otherwise, it is unstable.

High availability, unlike stability, is a quantifiable metric often expressed as "nines" (e.g., 99.9%). Its formula compares total operational time with downtime and recovery time.

The three components of operational time are:

Time the system operates normally (stable state).

Time the system is damaged or unusable (unstable state).

Time to recover from an unusable state back to normal operation.

Availability and stability are positively correlated, but a system cannot remain stable forever. By reframing the formula, we can better analyze the problem.

The goal is to keep the system in a stable working state, avoid negative user impact, and prevent major incidents. The core KPI is system availability.

Improving availability starts with ensuring stability to reduce unstable conditions, then quickly detecting and restoring the system when failures occur.

2. Core Approach to Stability and High Availability

To improve availability, we first identify and define problems. Common unstable situations include:

Function: Application behavior deviates from expectations.

Capacity: Increased request volume leads to exceptions or timeouts.

Security: Unauthorized or malicious requests cause failures.

Fault tolerance: Improper handling of user errors.

These issues typically stem from three root causes:

Human error: Insufficient thinking or careless execution during development.

Hardware failure: Network outages, disk space exhaustion, memory crashes, etc.

Software failure: Thread pool exceptions, JVM crashes, middleware or dependent service errors.

Since failures cannot be eliminated entirely, we establish processes and mechanisms to minimize their occurrence and set up monitoring and alerting to detect issues promptly, enabling rapid recovery.

R&D Standards

Design phase: Document templates, high‑availability design guidelines.

Coding phase: General code standards, project structure standards.

Testing: Unit test pass rate, code coverage.

Logging: Security vulnerability fix guidelines.

Release: Change management procedures.

Capacity Assurance

Capacity assessment: Machine capacity, database capacity, cache capacity.

Load testing and baseline measurement.

Rate‑limiting strategies.

Degradation plans.

Monitoring and Alerting

Log standards.

Monitoring scope: Application basics, gateway, services, business metrics, rate‑limiting.

Alerting standards.

Data verification.

Emergency Response

Daily runbooks: Hardware exception plans, middleware exception plans, business exception plans.

Big‑event (e.g., promotion) plans.

Runbook execution standards.

Conclusion

Ensuring system stability and high availability is a vast topic. This article summarizes a systematic framework covering fault prevention, detection, and recovery, along with R&D norms, capacity planning, monitoring, and emergency response, to help teams build reliable, highly available systems.