Kuaishou's Big Data Service Platform: Architecture, Key Technologies, and Future Outlook

Kuaishou is a data‑driven company where data engineers are responsible for developing high‑quality structured data tables and stable, reliable data services delivered via APIs. They face two major pain points: a high threshold for developing data services and repetitive development across business lines.

Pain Point 1 – High Development Threshold includes challenges such as how to deliver data (favoring API‑based access over raw tables), how to develop services (requiring microservice knowledge, service discovery, high concurrency), handling permissions and availability, and addressing operational concerns like scaling, migration, decommissioning, and alerting.

Pain Point 2 – Duplicate Development arises because many business lines (payments, live streaming, accounts, etc.) independently build similar data pipelines and microservices, leading to wasted effort and long delivery cycles.

To solve these issues, Kuaishou built a unified data service platform that follows a "configuration‑as‑service" model: data engineers no longer write service code manually but configure service definitions, and the platform automatically generates and deploys the corresponding services.

System Architecture : Raw data is stored in a Data Lake, processed into domain‑organized data assets (typically in a data warehouse), then accelerated to high‑speed storage (Redis, HBase, Druid, etc.) before being exposed through various service interfaces.

Key Technology 1 – Configuration‑as‑Development : The platform defines two roles—service producers and service consumers. Producers configure four items: data source, acceleration target, interface type, and isolated test environment. After configuration, the platform automatically creates, tests, and deploys the service, after which consumers request access permissions.

Key Technology 2 – Multi‑Mode Service Forms : The platform offers three API types: KV API (high‑throughput key‑value lookups with Protobuf responses), SQL API (flexible OLAP/OLTP queries via a fluent interface), and Union API (composite APIs that combine multiple atomic APIs in serial or parallel, reducing latency).

Key Technology 3 – Efficient Data Acceleration : Two acceleration strategies are used—full‑data acceleration (synchronizing raw data from sources like Kafka, MySQL, logs to fast stores such as Redis, HBase, Druid) and multi‑level caching for hot data. The platform also supports data compression (ZSTD, Snappy, GZIP) to reduce storage size dramatically.

Key Technology 4 – High‑Availability Guarantees : Services run in Kuaishou's self‑developed elastic container cloud, register with KESS (service registry), and benefit from automatic unhealthy instance removal. Resource isolation separates workloads by business line and priority, preventing cross‑impact. Full‑link monitoring tracks data synchronization, service stability, and business correctness, providing alerts on latency, QPS, CPU, memory, and data consistency.

Summary and Outlook : Since 2017, the platform has supported diverse online scenarios (live streaming, short video, e‑commerce, advertising) and internal systems, handling up to 10 million QPS with millisecond latency. Future work will focus on tighter alignment with evolving business needs, deeper data‑asset management, and evolving toward a unified OneService framework that supports richer data sources, varied query modes, and a consolidated API gateway with built‑in permission, rate‑limiting, and traffic management.