Big Data Technology Architecture
Apr 1, 2021 · Big Data
Spark Adaptive Execution: Dynamic Shuffle Partition, Broadcast Join, and Skew Handling
The article explains the limitations of static shuffle partitions, execution‑plan estimation, and data skew in Spark SQL, and describes how Spark Adaptive Execution can automatically adjust shuffle partition numbers, switch join strategies, and mitigate skew through configurable parameters and code examples.
Adaptive ExecutionBroadcast JoinData Skew
0 likes · 11 min read