Near‑Real‑Time Data Lake Practices in TikTok E‑commerce Data Warehouse

Speaker: Ma Wenyuan, TikTok E‑commerce Real‑Time Data Warehouse team.

Overview: The talk introduces data‑lake technology characteristics, near‑real‑time architecture, and practical applications in e‑commerce data warehousing.

Data Lake Features: Stores massive raw data with low cost, uses schema‑on‑read, supports Apache Hudi with streaming primitives, upserts, indexing, and both read‑optimized and real‑time query modes.

Near‑Real‑Time Architecture: Describes analysis‑oriented and operation‑oriented scenarios, the need for high efficiency and low storage cost, and how data lake enables reuse of batch results, unified storage, and simplified computation pipelines.

Implementation: Presents a hybrid architecture that combines batch and streaming, leveraging Flink, Spark, Presto, and Hudi to achieve hourly incremental updates while preserving batch richness.

E‑commerce Use Cases: Includes marketing promotion analytics, traffic diagnosis, logistics monitoring, risk governance, and operational monitoring such as data product anomaly detection and real‑time message landing.

Future Challenges: Calls for higher performance, deeper integration with Flink/Spark, and a shift from analytical to product‑level near‑real‑time applications.

Conclusion: Data lake technology proves feasible for near‑real‑time scenarios in TikTok e‑commerce, with ongoing plans to address scalability and reliability.