Understanding Linux Kernel Memory Management: Process Allocation, OOM, and Memory Reclamation

This article is reproduced with permission from Luo Daowen’s personal blog.

During an internship the author became interested in Linux kernel memory management after an OOM presentation, and after accumulating knowledge writes this comprehensive blog to share insights.

The article analyzes memory management from a global viewpoint, focusing on four main topics: process memory allocation, OOM behavior, where allocated memory resides, and system memory reclamation.

1. Process Memory Allocation

When a program is started, the terminal calls exec to load the executable, mapping the code, data, BSS, and stack segments into the process address space via mmap . The heap is created on demand. After exec , control is handed to the dynamic linker, which loads required shared libraries.

Memory allocation begins with malloc , which invokes the brk system call. If no VMA exists for the heap, an anonymous mmap creates one and links it into the process's mm_struct red‑black tree. The user‑space allocator (ptmalloc, tcmalloc, jemalloc, etc.) then subdivides the region. Large allocations may use mmap directly, returning virtual memory that is backed by physical pages only upon first access.

When free is called, memory obtained via mmap is released with munmap ; otherwise the allocator returns the pages to the kernel. All memory is reclaimed when the process exits.

2. Out‑of‑Memory (OOM) Handling

When the system runs out of memory, the OOM killer selects a victim process based on factors such as memory consumption, runtime, priority, user ID, child count, and the /proc/<pid>/oom_adj value. The process with the highest oom_score is terminated.

The oom_adj parameter can be set between –16 and 15; a value of –17 protects a process from being killed. The kernel also uses /proc/sys/vm/overcommit_memory to control allocation policies:

0 – heuristic overcommit (default); allocation succeeds unless the request is dramatically larger than physical RAM.

1 – always allow overcommit; OOM occurs only when memory is actually exhausted.

2 – strict limit based on swap + RAM * overcommit_ratio .

3. Where Allocated Memory Resides

Memory can be mapped as shared or private file mappings, or as anonymous mappings. Shared file mappings (code and shared libraries) are cached in the kernel page cache. Private file mappings are also cached, but modifications trigger copy‑on‑write, allocating new private pages.

Anonymous mappings (BSS, heap, stack) are not cached; they consume only the used memory counter. Shared anonymous mappings (used for inter‑process communication) are backed by the page cache.

Tmpfs, procfs, sysfs, and ramfs are memory‑based filesystems. Files stored in tmpfs reside in the page cache and may also use swap; they cannot be freed by drop_caches while referenced.

3.1 Example Commands

Create a 1 GB file:

dd if=/dev/zero of=fileblock bs=M count=1024

Force the kernel to drop clean caches:

echo 1 >> /proc/sys/vm/drop_caches

4. Memory Reclamation

When memory is low, the kernel can reclaim memory manually (e.g., echo 1 >> /proc/sys/vm/drop_caches ) or automatically via the kswapd daemon. kswapd scans LRU lists, evicts inactive pages, and may swap out anonymous pages. The vm.swappiness parameter (0‑100) controls the preference for swapping versus cache reclamation.

File pages are written back if dirty before being freed; clean pages are simply discarded. Anonymous pages are swapped out because they have no backing store.

5. Summary

The article reviews the Linux process address space, explains how memory allocation and deallocation work, describes OOM selection criteria, shows where different types of memory are stored (cache, anonymous, tmpfs), and outlines both manual and automatic memory reclamation mechanisms employed by the kernel.