Shopee Tech Team
Oct 13, 2022 · Big Data
Improving Flink Unaligned Checkpoint: Problems, Principles, Optimizations, and Production Practices at Shopee
Shopee tackled frequent Flink checkpoint failures caused by back‑pressure by adopting and extending the community’s Unaligned Checkpoint mechanism—adding overdraft buffers, improving legacy sources, introducing an aligned‑checkpoint timeout, enabling output‑buffer switching, merging small HDFS files, and fixing network‑buffer deadlocks—now running hundreds of jobs with stable UC deployment and plans to enable it universally.
Big DataCheckpoint OptimizationFlink
0 likes · 18 min read