Service Rate Limiting, Degradation, and Caching Strategies for High‑Concurrency E‑commerce Interfaces

When a product API experiences a sudden traffic surge—such as a popular item trending on social media and receiving tens of thousands of orders—proper protection mechanisms are required to maintain system stability.

The typical protection stack includes caching , rate limiting , and service degradation .

Service Rate Limiting

Rate limiting controls the request rate either per request or per time window, rejecting, queuing, or degrading traffic once a threshold is reached.

Rate‑Limiting Algorithms

1. Leaky Bucket (漏桶算法) : Requests are placed into a bucket; if the bucket is full, excess requests are dropped. The bucket drains at a fixed rate, ensuring the output rate never exceeds the configured limit.

2. Token Bucket (令牌桶算法) : Tokens are added to a bucket at a steady rate v = time_period / limit . A request succeeds only if a token can be consumed, allowing short bursts when tokens have accumulated.

3. Sliding Window (滑窗算法) : The time window is divided into smaller sub‑windows; each sub‑window records its request count. When the sum of counts exceeds the limit, the request is throttled.

These algorithms are implemented in various layers, such as Nginx (which uses the leaky bucket) and custom Java services.

Local Interface Rate Limiting

Java’s Semaphore from the concurrency library can limit concurrent access to a resource. Example:

private final Semaphore permit = new Semaphore(40, true);

public void process() {
    try {
        permit.acquire();
        // TODO: business logic
    } catch (InterruptedException e) {
        e.printStackTrace();
    } finally {
        permit.release();
    }
}

This limits the service to 40 simultaneous executions.

Distributed Rate Limiting

Message queues (e.g., MQ or Redis List) can act as a buffer, applying the leaky‑bucket principle across multiple nodes. When request volume exceeds a threshold, requests are queued and consumed at the service’s sustainable throughput.

Service Degradation

If traffic continues to rise after risk control, a fallback plan can be triggered to degrade non‑critical services:

Stop edge‑case features (e.g., historical order queries during peak sales).

Reject requests when the threshold is breached.

Reject strategies: random rejection, reject oldest requests, reject non‑core requests.

Recovery involves scaling additional consumer instances and gradually re‑enabling degraded features.

Data Caching Strategies

When a surge is detected, the following steps can be taken:

Apply a distributed lock to serialize access.

Cache hot data in a distributed cache (e.g., Redis).

Allow requests to read/write the cache first.

Send the final results to a message queue for asynchronous persistence.

Potential cache problems, such as inventory over‑selling, can be mitigated by:

Read‑Write Separation : Use Redis Sentinel master‑slave replication; reads dominate writes, and when stock reaches zero, reads fail fast.

Load Balancing : Partition inventory across multiple cache nodes (similar to ConcurrentHashMap.counterCells ) and distribute requests evenly.

Page Cache : Aggregate short‑term writes in memory before flushing to the database, a pattern used in OS page caches and MySQL.

These techniques together help maintain high availability and data consistency under extreme load.

— End of article.