Apache Spark Memory Management: Storage and Execution Memory (Part 2)

This article is a continuation of the previous piece on Apache Spark memory management, focusing on two major aspects: storage memory management and execution memory management.

Storage Memory Management

RDDs are the fundamental immutable data abstraction in Spark. Their persistence is handled by Spark’s Storage module, which decouples RDDs from physical storage. The driver and executor each run a BlockManager (master and slave respectively) that manages Block objects, where each RDD partition corresponds to a unique block identified by an ID such as rdd_RDD-ID_PARTITION-ID .

Spark defines seven storage levels (e.g., MEMORY_ONLY , MEMORY_AND_DISK ) that are composed of five boolean variables. The source code of the StorageLevel class is shown below:

class StorageLevel private(
    private var _useDisk: Boolean, //磁盘
    private var _useMemory: Boolean, //这里其实是指堆内内存
    private var _useOffHeap: Boolean, //堆外内存
    private var _deserialized: Boolean, //是否为非序列化
    private var _replication: Int = 1 //副本个数
)

The dimensions are:

Storage location – disk, on‑heap memory, or off‑heap memory (e.g., MEMORY_AND_DISK stores data in both memory and disk; OFF_HEAP stores only off‑heap).

Storage format – whether the block is stored deserialized (in‑memory objects) or serialized (byte buffer).

Replication – number of copies; values >1 trigger remote replication.

When an RDD is cached, its partitions are first read as Iterator s. Spark then “unrolls” the iterator into a Block . Unrolling requires temporary space from the MemoryManager . If sufficient space is unavailable, unroll fails; otherwise the block is stored either deserialized ( DeserializedMemoryEntry ) or serialized ( SerializedMemoryEntry ) depending on the chosen storage level.

Because executor memory is limited, Spark may evict old blocks using an LRU policy on the LinkedHashMap . Evicted blocks that have a disk component are spilled to disk; otherwise they are simply removed. Eviction rules ensure the evicted block shares the same memory mode, belongs to a different RDD, and is not currently being read.

Execution Memory Management

All tasks running in an executor share a pool of execution memory. Each task can request between 1/2N and 1/N of the total executor memory, where N is the number of concurrent tasks. If a request cannot be satisfied, the task is blocked until enough memory is released.

During Shuffle, execution memory is used for both the write and read phases. In the write phase, Spark may use an ExternalSorter (heap‑based) or a ShuffleExternalSorter (Tungsten‑based). The latter can store data in either on‑heap or off‑heap memory depending on configuration. In the read phase, data is aggregated by an Aggregator and may also be sorted, both consuming on‑heap memory.

Both sorter and aggregator rely on an AppendOnlyMap hash table. When this map grows beyond the memory manager’s allowance, Spark spills its contents to disk (a “spill”), later merging the spilled files.

Tungsten introduces a page‑based memory manager. Memory is divided into 64‑bit logical addresses composed of a 13‑bit page number and a 51‑bit offset. Each page is represented by a MemoryBlock with fields obj (object reference for on‑heap pages or null for off‑heap) and offset (the absolute address). This unified addressing allows Spark to sort pointers without deserialization, greatly improving memory‑access and CPU efficiency.

Conclusion

Spark’s storage memory and execution memory are managed by distinct mechanisms: storage memory uses a LinkedHashMap of blocks derived from RDD partitions, while execution memory relies on AppendOnlyMap and Tungsten’s page‑based system to handle Shuffle data. Understanding these internals helps developers tune Spark applications for optimal performance.