Evolution of Bilibili's Relationship Chain Service: From MySQL to KV Storage, Multi‑Layer Caching, and Hotspot Resilience
The article details how Bilibili's relationship‑chain service scaled from a MySQL‑based design to a distributed KV store, introduced successive caching layers (memcached, Redis hash, Redis KV, bloom filter), and implemented hotspot mitigation techniques to sustain near‑million QPS traffic while ensuring high availability and data consistency.
From a product perspective, the relationship chain represents user follow relationships, supporting features such as follow, blacklist, whisper, mutual follow, and special follow, with query peaks approaching one million QPS.
Initially, a simple MySQL schema with sharded tables handled the workload, but growing data volume (now terabytes) and heavy transactional writes exposed scalability limits, prompting a migration to a custom distributed KV store and an asynchronous write model.
The KV store uses a key format {attr|mid}fid and stores a struct containing the relationship attribute and modification time, eliminating the need for separate count tables and enabling efficient point‑gets and scans.
To reduce KV scan load, a multi‑layer caching strategy was introduced: memcached caches full follow lists, Redis hash caches per‑user relationships for point queries, Redis KV replaces the hash for finer‑grained point‑gets, and a bloom filter filters out empty‑sentinel queries before reaching the KV layer.
Hotspot mitigation evolved from manual key blacklisting to an automated detection tool combined with a local cache, allowing the system to handle sudden traffic spikes on popular creators without manual intervention.
Future directions include exposing the relationship service as a multi‑tenant platform, extending relationship types to other entities, and improving isolation through quota‑based configurations to protect core services from traffic surges.
Architect
Professional architect sharing high‑quality architecture insights. Topics include high‑availability, high‑performance, high‑stability architectures, big data, machine learning, Java, system and distributed architecture, AI, and practical large‑scale architecture case studies. Open to ideas‑driven architects who enjoy sharing and learning.
How this landed with the community
Was this worth your time?
0 Comments
Thoughtful readers leave field notes, pushback, and hard-won operational detail here.