Ant Group's Real-Time Data Warehouse Architecture, Solutions, and Data Lake Outlook

Ant Group shares its recent two‑to‑three‑year exploration and practice in the real‑time data warehouse field, outlining the overall architecture, data quality guarantees, stream‑batch integration, and future data lake plans.

The real‑time data warehouse consists of six core modules—compute engine, development platform, compute resources, real‑time assets, development tools, and data quality—covering the entire real‑time data development lifecycle while addressing challenges such as asset management, resource control, and platform capability.

Data quality is ensured through both pre‑deployment measures (debugging, stress testing, rate limiting) and in‑process monitoring (task exception metrics, DQC rules, baseline monitoring), providing end‑to‑end assurance of timeliness and accuracy.

To bridge streaming and batch workloads, Ant Group adopts a unified stream‑batch approach using Flink and ODPS, low‑code development, virtual columns for schema alignment, and a mixed meta‑table that enables a single task to compute cumulative metrics across real‑time and offline data.

Looking ahead, the team plans to integrate a data lake using Paimon, consolidating storage and compute across ODS, ADM, and other layers, simplifying the pipeline with three scheduling granularities (daily, hourly, real‑time) and a unified asset model.

The presentation concludes with a summary of these innovations and an invitation for further discussion.