Advances in Apache Flink AI Ecosystem: ML Pipeline, AI Flow, and Mini‑Batch Streaming Iteration

Flink is a distributed compute engine that supports unified batch‑and‑stream processing, making it suitable for AI workloads such as feature engineering, online learning, and online prediction. To address the maintenance and consistency challenges of traditional Lambda architectures, Flink and Spark were introduced as unified solutions.

The article outlines the background of building AI systems with Flink, then details the Flink ML Pipeline, which abstracts data preprocessing (Transformer) and model training (Estimator), and introduces Alink, a machine‑learning library built on Flink's Table API.

AI Flow is presented as an end‑to‑end API that connects data acquisition, preprocessing, model training, validation, and serving, allowing users to write a single piece of code that runs both offline and online. AI Flow defines components such as Example, Transformer, Trainer, and Model, and uses Translators to convert these definitions into executable jobs for various deployment targets (local, Kubernetes, YARN).

To enable iterative algorithms on streaming data, a new mini‑batch streaming iteration mechanism is proposed. It treats static datasets as a single mini‑batch and splits streaming data into multiple mini‑batches that can be processed in parallel, supporting shared state across mini‑batches and progress tracking via special marker messages.

The article concludes that the mini‑batch iteration design bridges the gap between limited‑data batch iteration and infinite‑stream processing, paving the way for more robust online machine‑learning training and graph‑processing scenarios.