NetEase Game Streaming ETL Architecture and Practices Based on Flink

NetEase Game's data integration relies on a streaming ETL pipeline powered by Flink, transforming heterogeneous game logs—operational, business, and program logs—into structured data for both real‑time and offline warehouses.

The system handles challenges such as schema‑free sources (e.g., MongoDB), deeply nested fields, high log variety, and frequent schema changes, requiring flexible parsing, transformation, and robust error handling.

Three ETL services are offered: a dedicated operational‑log ETL with custom logic, the generic EntryX ETL for all other text logs, and ad‑hoc jobs for special cases. EntryX defines Source, StreamingTable, and Sink modules, automatically generating Flink jobs from user configurations.

Architectural evolution progressed from Hadoop Streaming (Python scripts) to Spark Streaming (POC) and finally to Flink DataStream, preserving Python UDFs via a Jython‑based Runner layer that executes cross‑language functions within the JVM.

Runtime optimizations include hot‑updating lightweight configuration changes, consolidating multiple stream tables into a single Flink job to avoid redundant Kafka reads, and separating real‑time and offline sinks to mitigate HDFS back‑pressure.

Further performance tuning addresses HDFS small‑file explosion by pre‑partitioning streams (keyBy) and limiting parallelism, while SLA metrics are collected via OperatorState‑based utilities supporting static, dynamic, and TTL metrics.

Fault tolerance is achieved using SideOutput for error streams, with downstream recovery strategies involving batch reprocessing or targeted back‑fill jobs that replace corrupted Hive partitions.

Future plans focus on data‑lake support for update/delete workloads, automatic small‑file merging and deduplication, and extending Python support to the full Flink stack via PyFlink.