NetEase Cloud Music Real-Time Data Warehouse Architecture and Low-Code Platform Practices

The presentation introduces NetEase Cloud Music's real-time data warehouse architecture, divided into four parts: an overall overview, consistency implementation of the stream‑batch model, low‑code platform achievements, and future planning.

Real‑time scenarios include feature generation for recommendation and search, monitoring of resource allocation and AB testing, and data products such as real‑time dashboards that support personalized content delivery.

The architecture follows a dimensional modeling approach with four layers: ODS (log collection from servers and binlog), CDM (data cleaning and intermediate layers for traffic and business details), ADS (application layer using OLAP engines like ClickHouse, Impala, Kudu), and a business decision layer that feeds recommendation, risk control, and precise song promotion.

Technical choices consist of Flink + Kafka for streaming from ODS to CDM, Spark + Hive for offline processing, and ClickHouse/Impala/Kudu for low‑latency queries; the system balances second‑level real‑time needs, minute‑level near‑real‑time analytics, and hour‑level batch processing.

To ensure model consistency, a Lambda architecture is employed alongside a partition flow table that maps partition fields to Kafka topics, supporting static and dynamic partition management via ZooKeeper and metadata services.

The FastX low‑code platform addresses computation consistency by providing model‑centric, component‑based, configurable, and service‑oriented development; it abstracts tasks into views, AB models, and relational models, enabling drag‑and‑drop pipeline creation, automated binlog parsing, ClickHouse sink simplification, and lineage tracking.

Future plans focus on strengthening infrastructure, integrating low‑code with stream‑batch processing, and exploring lake‑warehouse solutions, while leveraging FastX's data model and lineage for asset governance and evaluation.

The session concludes with a Q&A covering metric definition, high availability, binlog usage, data consistency checks, and handling late‑arriving data.