Batch-Stream Fusion in Education: TAL’s Real-Time Data Platform Practices

The presentation begins with an overview of TAL Education, a leading education technology company, and introduces its comprehensive data platform architecture, which consists of three layers: the data empowerment layer, the global data layer, and the data development layer.

The data empowerment layer provides business intelligence tools, data capabilities, and thematic analysis systems, including event tracking, A/B testing, and dashboard tools.

The global data layer aims to unify data across all business units through ID mapping, creating a large, integrated user pool for cross‑business analytics.

The data development layer hosts the real‑time platform, T‑Streaming, which supports data integration, development, job assurance, resource management, and security.

Key requirements for building the real‑time platform were identified: a unified multi‑tenant cluster, lowered development barriers for real‑time jobs, reusable generic solutions, and comprehensive lifecycle management of jobs.

The platform offers a one‑stop solution for real‑time analytics, supporting database, event, and log integration, Flink SQL and Flink JAR job development, automatic job monitoring, and multi‑tenant resource isolation.

Batch‑stream fusion is discussed on two levels: Flink’s native ability to run the same SQL on batch and streaming data, and TAL’s architectural approach that continuously updates the ODS layer to provide minute‑ and second‑level analytics.

ODS layer real‑timeization is achieved by three jobs: offline data initialization (e.g., MySQL full load), binlog capture to Kafka, and Kafka‑to‑Hive ingestion, all orchestrated through a unified tool.

The Flink SQL development workflow is simplified: users only write the core SQL logic; the platform automatically handles source registration, schema inference, UDF integration, and job submission to Flink clusters.

Job execution is divided into a preparation phase (SQL validation, table creation, script assembly) and an execution phase (StreamTableAPI registration, UDF loading, DataStream conversion, and sink attachment).

Template jobs and JAR‑based custom jobs are provided for common scenarios such as MySQL binlog to Hive/Kudu/Hologres and Alibaba Cloud SLS log ingestion.

A hybrid‑cloud deployment model enables seamless switching of jobs between on‑premise and Alibaba Cloud clusters, ensuring elasticity and resource efficiency during traffic spikes.

A concrete K12 renewal reporting case illustrates challenges (historical data dependence, order chain effects, frequent dimension changes) and the solution using two parallel Flink pipelines (minute‑level via Kudu, second‑level via MySQL) with ETL and update jobs handling dimension changes and order impacts.

Stability measures include active‑active deployment across regions, fault‑tolerant Flink jobs with in‑memory caching, and automated monitoring with alerts and automatic job restarts.

The outlook emphasizes deeper batch‑stream integration, expanded hybrid‑cloud capabilities, and real‑time support for algorithm platforms and data applications.