High Concurrency: Challenges, Caching Strategies, Rate Limiting, and Degradation

High concurrency in internet applications results in massive simultaneous requests, demanding systems to achieve the three "high" goals: high performance, high concurrency, and high availability.

Challenges of high concurrency include performance degradation, resource competition, and stability problems, which can lead to system overload or failure.

Definition : High concurrency refers to a system’s ability to process a large number of requests in the same time period without performance loss or response delay.

Typical Scenarios : E‑commerce platforms, social media, and other popular web services where many users perform actions concurrently.

Cache Strategies : Various caching layers—browser cache, client cache, CDN cache, reverse‑proxy cache, local cache, and distributed cache—are introduced with their principles, common technologies, advantages, disadvantages, and applicable scenarios.

Common cache problems such as cache penetration, cache breakdown, cache avalanche, and cache consistency are described, followed by mitigation techniques (Bloom filter, empty‑object caching, delayed double‑check, random expiration, multi‑level cache, etc.).

Rate Limiting is essential to protect systems from overload. The article covers several classic algorithms:

Fixed Window Algorithm – simple counter per fixed time window.

public class FixedWindowRateLimiter {
private static int counter = 0; // request count
private static long lastAcquireTime = 0L;
private static final long windowUnit = 1000L; // 1 second
private static final int threshold = 10; // max requests per window
public synchronized boolean tryAcquire() {
long currentTime = System.currentTimeMillis();
if (currentTime - lastAcquireTime > windowUnit) {
counter = 0;
lastAcquireTime = currentTime;
}
if (counter < threshold) {
counter++;
return true;
}
return false;
}
}

Sliding Window Algorithm – divides a window into smaller sub‑windows for smoother limiting.

import java.util.LinkedList;
import java.util.Queue;
public class SlidingWindowRateLimiter {
private Queue
timestamps;
private int windowSize;
private long windowDuration;
public SlidingWindowRateLimiter(int windowSize, long windowDuration) {
this.windowSize = windowSize;
this.windowDuration = windowDuration;
this.timestamps = new LinkedList<>();
}
public synchronized boolean tryAcquire() {
long currentTime = System.currentTimeMillis();
while (!timestamps.isEmpty() && currentTime - timestamps.peek() > windowDuration) {
timestamps.poll();
}
if (timestamps.size() < windowSize) {
timestamps.offer(currentTime);
return true;
}
return false;
}
}

Leaky Bucket Algorithm – controls outflow rate, discarding excess requests.

public class LeakyBucketRateLimiter {
private long capacity; // max tokens
private long rate; // tokens per second
private long water; // current tokens
private long lastTime; // last request time
public LeakyBucketRateLimiter(long capacity, long rate) {
this.capacity = capacity;
this.rate = rate;
this.water = 0;
this.lastTime = System.currentTimeMillis();
}
public synchronized boolean tryAcquire() {
long now = System.currentTimeMillis();
long elapsed = now - lastTime;
water = Math.max(0, water - elapsed * rate / 1000);
if (water < capacity) {
water++;
lastTime = now;
return true;
}
return false;
}
}

Token Bucket Algorithm – generates tokens at a fixed rate; requests consume tokens.

import java.util.concurrent.ScheduledExecutorService;
import java.util.concurrent.ScheduledThreadPoolExecutor;
import java.util.concurrent.TimeUnit;
public class TokenBucketRateLimiter {
private long capacity;
private long rate;
private long tokens;
private ScheduledExecutorService scheduler;
public TokenBucketRateLimiter(long capacity, long rate) {
this.capacity = capacity;
this.rate = rate;
this.tokens = capacity;
this.scheduler = new ScheduledThreadPoolExecutor(1);
scheduleRefill();
}
private void scheduleRefill() {
scheduler.scheduleAtFixedRate(() -> {
synchronized (this) {
tokens = Math.min(capacity, tokens + rate);
}
}, 1, 1, TimeUnit.SECONDS);
}
public synchronized boolean tryAcquire() {
if (tokens > 0) {
tokens--;
return true;
}
return false;
}
}

Sliding Log Algorithm – records request timestamps in an ordered list for precise rate control.

import java.util.LinkedList;
import java.util.List;
public class SlidingLogRateLimiter {
private int requests;
private List
timestamps;
private long windowDuration;
private int threshold;
public SlidingLogRateLimiter(int threshold, long windowDuration) {
this.requests = 0;
this.timestamps = new LinkedList<>();
this.windowDuration = windowDuration;
this.threshold = threshold;
}
public synchronized boolean tryAcquire() {
long currentTime = System.currentTimeMillis();
while (!timestamps.isEmpty() && currentTime - timestamps.get(0) > windowDuration) {
timestamps.remove(0);
requests--;
}
if (requests < threshold) {
timestamps.add(currentTime);
requests++;
return true;
}
return false;
}
}

Each algorithm’s advantages and disadvantages are discussed, along with typical use cases such as protecting third‑party APIs, smoothing traffic spikes, and ensuring fair request distribution.

Degradation and Circuit Breaker – When a system is overloaded, degradation (fallback) and circuit‑breaker mechanisms protect core functionality. The article explains the state machine (closed, half‑open, open), trigger conditions, and tools such as Guava RateLimiter, Sentinel (single‑node and cluster modes), and Nginx rate‑limit directives.

Finally, the article emphasizes that combining caching, rate limiting, and degradation strategies provides a robust approach to handling high‑concurrency scenarios.