Design and Implementation of iQIYI Content Platform Data Center (OLTP)

The article discusses the challenges of data management in large‑scale microservice architectures and presents iQIYI's Content Platform Data Center (OLTP) as a solution for unified data collection, storage, and access.

Design Goals – The team identified problems such as data islands, duplicated development, complex integration, tight coupling, and cumbersome development. To address these, the system aims to support hundred‑billion‑level data storage, high QPS reads/writes, unified field‑change notifications, minimal operational cost with UI‑driven schema changes, and generic query/save capabilities.

Architecture – A unified data hub receives writes from all business lines, broadcasts field changes, and allows other services to subscribe to relevant updates. The architecture diagram (Fig. 1) illustrates this flow.

Data Consistency – Data is persisted in both MongoDB and Elasticsearch, introducing consistency concerns. The solution adopts a final‑consistency model with field‑level permission control, MongoDB‑to‑ES field mapping, and a write‑once‑to‑both‑stores approach that overwrites ES documents based on the latest MongoDB data.

High Availability – The system uses distributed file storage, sharded MongoDB clusters deployed across multiple cities for active‑active resilience, real‑time backup with dual‑write configuration, and a switchable database layer to fail over to backup clusters when the primary is unavailable.

Massive Data Storage – Horizontal scaling is achieved via sharded MongoDB for the DB layer and careful index design for Elasticsearch. Indexes are created with appropriate shard counts, and alias‑based routing is used to split data across multiple physical indices while presenting a single logical view to consumers.

Best Practices – A standardized, generic message‑notification SDK allows services to listen for field changes via RabbitMQ, reducing coupling and simplifying integration. The SDK encapsulates RMQ, performs client‑side filtering, and supports fine‑grained (second‑level) field filtering.

Technical Challenges – The team tackled optimistic‑lock granularity by scoping version locks to business‑specific fields, leveraged Elasticsearch Bulk API for high‑throughput writes, employed Search‑After/Scroll APIs for deep pagination, and implemented per‑business‑unit rate limiting to isolate traffic spikes.

Results and Outlook – After deployment, the OLTP hub integrated 26 business lines, runs four isolated clusters, handles >2 K read QPS and >500 write QPS, and stores tables with up to 250 million rows. Ongoing work includes linking the OLTP system with an OLAP data warehouse for real‑time and offline analytics.