Spark Performance Optimization: Principles, Memory Model, Resource Tuning, Data Skew and Shuffle Tuning

The article is a comprehensive technical guide for Spark developers and engineers, organized into several sections that explain core optimization concepts and practical tuning methods.

1. Spark Development Principles

It briefly lists the ten Spark development principles, such as avoiding duplicate RDDs, reusing RDDs, persisting frequently used RDDs, and choosing appropriate persistence levels (MEMORY_ONLY, MEMORY_AND_DISK, MEMORY_ONLY_SER, etc.). A table summarizes each persistence level and its meaning.

2. Spark Memory Model

The article contrasts the static memory model used before Spark 1.6 with the unified (dynamic) memory model introduced later. It explains how the executor memory is divided into storage and execution regions, how the spark.memory.fraction and spark.memory.storageFraction parameters control the allocation, and why the unified model allows storage memory to be evicted for execution when needed.

3. Resource Parameter Tuning

Key Spark configuration parameters are described with practical advice:

num-executors : total number of executor processes; typical range 50‑100.

executor-memory : memory per executor; usually 4‑8 GB.

executor-cores : CPU cores per executor; 2‑4 cores are recommended.

driver-memory : memory for the driver; 1 GB is often sufficient unless large collect operations are used.

spark.default.parallelism : default number of tasks per stage; 500‑1000 or 2‑3 × (num‑executors × executor‑cores) is a good rule of thumb.

spark.storage.memoryFraction and spark.shuffle.memoryFraction : fractions of executor memory dedicated to cached RDDs and shuffle aggregation respectively.

Example spark-submit command:

./bin/spark-submit \
  --master yarn-cluster \
  --num-executors 100 \
  --executor-memory 6G \
  --executor-cores 4 \
  --driver-memory 1G \
  --conf spark.default.parallelism=1000 \
  --conf spark.storage.memoryFraction=0.5 \
  --conf spark.shuffle.memoryFraction=0.3 \

4. Data Skew Detection and Mitigation

The article explains how data skew occurs during shuffle when a few keys dominate the data volume, leading to extremely slow tasks or OOM errors. It outlines a systematic way to locate skewed stages via Spark UI and to inspect key distributions using countByKey or SQL queries.

Several mitigation techniques are presented:

Pre‑process data with Hive ETL to reduce shuffle.

Filter out low‑importance skewed keys.

Increase shuffle parallelism (though limited for extreme keys).

Two‑stage aggregation (local + global) for reduce‑by‑key operations.

Convert reduce join to map join by broadcasting the small dataset.

Sample skewed keys, split them into separate RDDs, add random prefixes, and join with an expanded counterpart.

Apply random prefixes and expand the whole RDD when many keys are skewed.

5. Shuffle‑Related Tuning Parameters

Important shuffle configurations are listed with default values and tuning suggestions:

spark.shuffle.file.buffer (default 32 KB): increase to reduce disk writes.

spark.reducer.maxSizeInFlight (default 48 MB): increase to reduce network round‑trips.

spark.shuffle.io.maxRetries (default 3) and spark.shuffle.io.retryWait (default 5 s): raise for higher stability on large shuffles.

spark.shuffle.memoryFraction (default 0.2): allocate more memory to shuffle aggregation when possible.

spark.shuffle.manager (default sort): choose between sort, hash, or tungsten‑sort based on sorting needs.

spark.shuffle.sort.bypassMergeThreshold (default 200): increase to skip sorting when the number of read tasks is low.

6. Conclusion

By combining principle‑based coding practices, appropriate memory model selection, careful resource parameter tuning, targeted data‑skew solutions, and fine‑grained shuffle configuration adjustments, Spark jobs can achieve significant performance gains and stability improvements.