Why Choose Apache Flink for Real‑Time Stream Processing: Features and Lessons Learned

In recent years, big‑data computing engines such as MapReduce and Spark have dominated the development scene, while real‑time engines include Storm, SparkStreaming, and Flink. In the past two years Flink has been at the forefront; Alibaba acquired Flink’s parent company and its Blink branch is gradually merging into the Flink mainline.

First Encounter with Flink

I first learned about Flink two years ago at a conference held by a big‑data security company that uses data analysis for threat warning. They demonstrated three stream‑processing technologies—Storm, SparkStreaming, and Flink. Storm was labeled “past”, SparkStreaming “present”, and Flink “future”. At that time our business did not require real‑time processing, so we knew little about it.

Later, when our business added real‑time processing needs, we revisited the three technologies. We quickly ruled out Storm based on the earlier presentation and evaluated SparkStreaming versus Flink, ultimately selecting Flink.

Why Flink?

1. High throughput, low latency, high performance – Within the community Flink is the only framework that simultaneously supports all three characteristics. SparkStreaming uses micro‑batch processing, which cannot guarantee low latency, and Storm can only provide high performance and low latency but not high throughput.

2. Support for event time, processing time, and ingest time – Flink can process data based on the event’s original timestamp, preventing errors caused by out‑of‑order arrival and preserving the natural order of data despite network or hardware delays. Other systems that rely on processing time or system time may produce incorrect results under similar conditions.

3. Stateful computation – Flink includes built‑in state management, allowing intermediate results to be stored in memory or files and reused when the next batch of events arrives, dramatically improving performance by avoiding recomputation.

4. Flexible window mechanism – In streaming scenarios data is continuous, and Flink offers both data‑driven and time‑driven windows (rolling, tumbling, session, etc.) that can be freely combined to handle various data‑slice requirements such as “one minute” or “100 records”.

5. High fault tolerance – Flink’s fault‑tolerance mechanism can recover from cluster anomalies caused by hardware or network failures, guaranteeing exactly‑once semantics for processed data.

Research Questions

Choosing Flink at that time was arguably unreasonable: mainstream real‑time frameworks like SparkStreaming had far higher market share, and Flink’s documentation and books were scarce, leaving many unknown pitfalls. For a small company, thorough market research is crucial because resources to maintain a dedicated team for new technologies are limited. Ultimately, the most suitable technology is the one that enables rapid iteration, learning, and implementation, even if it means navigating many unknowns.

Recommended Reading: 1 , Distributed Systems Consistency , 2 , … (additional links omitted for brevity).

Welcome to follow the public account 👇

If you liked it, remember to click “Watch”.