Whole-stage code generation

Qunar Tech Salon

Aug 29, 2016 · Big Data

Whole‑Stage Code Generation and Vectorization in Apache Spark’s Tungsten Engine

The article explains how Spark 2.0’s second‑generation Tungsten engine replaces the traditional Volcano iterator model with whole‑stage code generation and vectorization, eliminating virtual calls, keeping temporary data in CPU registers, and using loop unrolling and SIMD to achieve order‑of‑magnitude performance gains on large‑scale data workloads.