How Datagateway Handles High‑Traffic: Caching, Throttling, and Failover Strategies

1. Origin of the Data Access Layer

Large internet companies typically evolve from low‑traffic, tightly coupled sites to high‑traffic, high‑concurrency, distributed architectures. Rapid traffic growth demands a stable system, prompting the introduction of a dedicated data access layer (Datagateway) to abstract database differences, manage connections, and improve concurrency.

The initial simple architecture performed direct CRUD operations on a single database. As traffic grew, business logic was split across multiple databases, leading to a more complex logical topology.

Datagateway sits between the stateless business interface layer and the underlying databases, providing a unified API and response format (JSON or MsgPack) while shielding the business layer from database heterogeneity.

2. Caching Mechanism

Cache Types

To reduce database load, a cache layer is placed in front of the databases. Two main types are used:

Local cache – suitable for small data sets; fast memory reads/writes, no network overhead, but inconsistent across cluster nodes.

Distributed cache – handles massive data, provides cluster‑wide consistency, but incurs network latency.

Datagateway adopts a distributed cache (e.g., Redis, Memcached, DCache) and uses consistent‑hashing for Memcached to minimize cache invalidation when nodes change.

Cache Update

Two approaches exist: passive (set a TTL, e.g., 5 minutes, and let stale data be served until expiration) and active (update or delete the cache immediately after a write). Datagateway prefers passive updates for most data, offering a nocache parameter for callers needing fresh data.

Cache Miss / Thundering Herd

When many concurrent requests hit an expired key, each may query the database, causing a spike. The solution adds a timestamp‑based lock ( key_lock ) so only one request refreshes the cache while others receive the stale value.

To further reduce latency, an “old‑first” mode spawns a background thread to refresh the cache while the original request returns the stale data.

Hot Cache

Hot keys can overload a single cache node. Mitigations include compressing large values, adding a local cache layer on the application server, and sharding hot keys into multiple sub‑keys to distribute load across nodes.

Cache Disaster Recovery

Datagateway maintains two cache clusters (Redis and DCache). Writes are duplicated asynchronously, and health metrics trigger automatic failover when error rates exceed thresholds. Manual controls allow operators to enable/disable auto‑switching.

3. Degradation and Circuit Breaker

When a database becomes slow (e.g., due to heavy export operations), request threads can saturate, causing connection refusals. Datagateway limits concurrent threads per database; excess requests are rejected, preserving overall system availability.

4. Summary

Datagateway exemplifies a centralized data access system that leverages layered architecture, distributed caching, thundering‑herd protection, hot‑key handling, disaster‑recovery, and degradation‑circuit‑breaker strategies to achieve high stability and performance in high‑traffic environments.