Understanding Service Degradation and Its Practical Strategies

What Is Service Degradation

If you have read the previous analysis of service rate limiting, understanding service degradation becomes easy. Imagine a scenic spot that normally allows free entry, but during holidays the visitor flow surges, so the management limits the number of simultaneous entrants – this is rate limiting. Service degradation means cutting less important features when the system is under heavy load to keep core services stable.

In the Internet, similar measures are taken. For example, during a Double‑11 sale, orders may be allowed but returns or modifications are temporarily disabled to preserve service availability. When hardware and software reach their limits, resources are shifted to core business while non‑essential functions are disabled.

Service Level Definition

SLA (Service Level Agreement) is a key metric for judging whether a stress test is abnormal. Monitoring SLA indicators of core services during a test provides a clear view of system health. An SLA typically guarantees a certain uptime, often expressed as "six nines" (99.9999%). This translates to about 31 seconds of downtime per year, indicating extremely high reliability.

Degradation Handling

Fallback Data

Examples include returning a default page when a service fails, setting safe default values (e.g., inventory = 0), providing static data, or using cached data when the live source is unavailable.

Rate‑Limit Degradation

Set a maximum QPS threshold for each request type; requests exceeding the limit are rejected with friendly messages such as "system busy, please try later". Rate limiting is a common stability measure that releases resources for core tasks during traffic spikes.

Timeout Degradation

Define a timeout for remote calls; if a non‑core feature times out, it can be degraded (e.g., hide product recommendations while keeping the main purchase flow functional).

Fault Degradation

When a remote service fails (network, DNS, HTTP error), return default values, fallback data, static pages, or cached responses.

Retry / Automatic Handling

Client‑side high availability can be achieved by providing multiple service endpoints. In micro‑services, mechanisms like Dubbo retry, API retry with a limit and idempotency handling, or a web‑side retry button improve user experience.

Feature Switch Degradation

During incidents, operators can manually toggle switches to disable problematic services. Switches can be stored locally, in databases, Redis, or Zookeeper, and are also useful for gray‑release rollbacks.

Read Degradation

When caches or DBs are unavailable, front‑end caches or fallback data can be used. Strategies include temporarily switching to read‑only caches, static pages, or blocking read access entirely for non‑critical services.

Write Degradation

Under high write pressure, writes can be directed to fast caches (e.g., Redis) and later synchronized to the database, achieving eventual consistency. This approach is common for inventory deduction, flash‑sale orders, or user reviews during peak traffic.

Front‑End Degradation

When backend services are partially or fully unavailable, use local caches or fallback data, and in special scenarios (e.g., flash sales) provide mock data.

JS Degradation

Embed degradation switches in JavaScript to prevent requests when system thresholds are exceeded, allowing graceful feature disabling.

Access‑Layer Degradation

Use Nginx + Lua or HAProxy + Lua to filter invalid requests before they reach services, providing an early degradation point.

Application‑Layer Degradation

Configure feature switches within the application; for example, Spring Cloud’s Hystrix can perform manual or automatic fallback based on timeout thresholds, offering circuit‑breaker functionality to isolate failures.

Fragment Degradation

When loading a page like Taobao’s homepage, if some resources fail, they can be omitted and replaced with alternative data, ensuring the page still renders acceptably.

Pre‑Warming

Static data can be pre‑loaded onto devices before major events (e.g., Double‑11) to reduce network load during the peak.

--- End of Summary ---