Flink 1.13 Release Highlights: Passive Scaling and Performance Analysis Features
Flink 1.13 introduces passive scaling that lets users adjust parallelism to resize jobs, adds visual tools such as load/back‑pressure charts, CPU flame graphs, and state‑backend metrics for deeper performance insight, and includes numerous community optimizations for easier upgrades and operation.
Flink 1.13 has been released, incorporating contributions from over 200 developers and more than 1,000 fixes and optimizations.
A primary goal of this version is to make stream‑processing applications as simple and natural to use as ordinary applications, highlighted by the newly introduced passive scaling feature that allows scaling by merely adjusting the job’s parallelism.
The release also includes a series of important changes that help users better understand job performance, such as load and back‑pressure visualizations for identifying bottleneck nodes, CPU flame graphs for analyzing operator hotspot code, and state‑backend access performance metrics.
Beyond these features, the Flink community has added many other optimizations; the article will discuss some of them and outline important considerations when upgrading to the new version.
1. Important Features
Passive Scaling
Flink’s original aim was to make stream‑processing applications as easy and natural as regular applications, and passive scaling is the latest progress toward that goal.
When deployed on resource managers like Kubernetes or YARN, Flink can actively manage resources (active scaling) based on the parallelism set by the user. In contrast, passive scaling lets the system automatically adjust the parallelism when the number of workers changes, without requiring users to manually manage resources.
With the Application deployment mode, starting a Flink job no longer requires separate steps for cluster startup and job submission; passive scaling completes this vision by keeping worker count in sync with the desired parallelism automatically.
To try passive scaling, users can set scheduler-mode: reactive and launch an application cluster (Standalone or Kubernetes). See the passive scaling documentation for more details.
Performance Analysis
Understanding and analyzing application performance is crucial, especially for data‑intensive, near‑real‑time Flink jobs.
When a Flink job cannot keep up with incoming data or exceeds expected resource usage, the following tools can help diagnose the cause.
Laravel Tech Community
Specializing in Laravel development, we continuously publish fresh content and grow alongside the elegant, stable Laravel framework.
How this landed with the community
Was this worth your time?
0 Comments
Thoughtful readers leave field notes, pushback, and hard-won operational detail here.