NetEase Data: Practices and Architecture of a Metrics Middle Platform

NetEase Data (网易数帆) shares its journey of developing a comprehensive metrics middle platform, starting from its early adoption of big‑data components in 2006, the launch of products like Mengtang and NetEase YouShu, and the commercial rollout of data‑centered solutions across multiple business lines.

The platform’s product matrix includes low‑level data computation and storage engines (HDFS/S3, Amoro, Yarn/K8s, Spark, Hive, Impala, Flink), a DataOps‑based lifecycle suite, and nine upper‑layer products covering data standards, metadata, data maps, indicator systems, data quality, asset centers, model design, security, and services, culminating in BI, machine learning, and profiling applications.

Key challenges that motivated the creation of the metrics middle platform are inconsistent metric definitions, fragmented consumption entry points, difficulty quantifying metric value, low development efficiency, redundant computations, and poor metric quality, with statistics showing up to 60% of issues stemming from long processing chains.

The solution, EasyMetrics, provides a unified logical semantic model that abstracts physical data sources, a custom metric definition language that simplifies metric creation, composition, and time‑period handling, and a MetricsDSL layer that translates high‑level metric expressions into optimized SQL via Apache Calcite, supporting various aggregation, logical, and arithmetic functions.

Implementation details include plugin‑based data source management (PF4J), reverse engineering of physical tables into logical models, generation of materialized DDL, and decoupled engine adapters that allow seamless integration with scheduling engines (e.g., Apache DolphinScheduler), query engines (Impala, JDBC), and compute engines (Spark, Flink).

Future plans involve deeper metric application scenarios such as dashboards, KPI management, and metric maps, broader BI integration, support for additional data sources like Doris, and the incorporation of AIGC for natural‑language metric queries.