Understanding Linux Memory Management, Page Reclamation, and OOM Killer

What is the role of virtual memory?

Virtual memory gives each process its own isolated address space by using per‑process page tables, preventing address conflicts and allowing the kernel to enforce read/write permissions and other page attributes for security.

What happens when system memory is tight?

When an application calls malloc , it initially receives virtual memory without immediate physical allocation. Accessing that memory triggers a page‑fault interrupt, causing the kernel to map a physical page if one is free. If no free physical page exists, the kernel starts memory reclamation.

Memory reclamation methods

Background reclamation (kswapd) : An asynchronous kernel thread scans and frees pages when memory pressure rises, without blocking the requesting process.

Direct reclamation : A synchronous path that runs when the background thread cannot keep up, blocking the process while freeing pages.

If both reclamation methods fail to satisfy the request, the Out‑of‑Memory (OOM) killer is invoked.

Which pages can be reclaimed?

Two main page types are reclaimed:

File‑backed pages : Clean pages are freed directly; dirty pages must be written back to disk before being released.

Anonymous pages : Reclaimed via the swap subsystem, which writes infrequently used pages to disk and later reads them back when needed.

Both types use an LRU (Least Recently Used) algorithm that maintains active_list (recently used pages) and inactive_list (rarely used pages). The kernel prefers to reclaim pages from the tail of the inactive list.

Performance impact of memory reclamation

Background reclamation is asynchronous and does not block processes, while direct reclamation is synchronous and can cause noticeable latency and higher CPU utilization. Reclaiming dirty file pages or swapping anonymous pages incurs disk I/O, which can degrade system responsiveness.

Adjusting reclamation preferences

The /proc/sys/vm/swappiness parameter (0‑100) controls the bias toward swapping anonymous pages (higher values) or reclaiming file pages (lower values). A common recommendation is to set it to 0.

# cat /proc/sys/vm/swappiness
0

Triggering kswapd earlier

Metrics such as pgscank/s (pages scanned by kswapd), pgscand/s (pages scanned directly), and pgsteal/s (pages actually reclaimed) can be observed with sar -B 1 . Large pgscand values indicate frequent direct reclamation.

Memory watermarks and kswapd activation

The kernel defines three watermarks: pages_min , pages_low , and pages_high . When free pages fall below pages_low , kswapd starts reclaiming until free pages exceed pages_high . If free pages drop below pages_min , direct reclamation is forced, which blocks the requesting process.

These watermarks are derived from /proc/sys/vm/min_free_kbytes :

pages_min = min_free_kbytes
pages_low = pages_min * 5 / 4
pages_high = pages_min * 3 / 2

NUMA‑aware reclamation

On NUMA systems, memory can be reclaimed locally or from remote nodes. The /proc/sys/vm/zone_reclaim_mode setting controls this behavior. Setting it to 0 (default) allows the kernel to look for free memory on other nodes before reclaiming locally, reducing the chance of direct reclamation under moderate pressure.

Protecting a process from OOM killing

The kernel selects a victim using the oom_badness() function, which scores each process based on its resident page count and the configurable oom_score_adj value (range –1000 to 1000). A higher score increases the likelihood of being killed.

// points = process_pages + oom_score_adj * totalpages / 1000

Setting oom_score_adj to -1000 makes a process virtually immune to OOM killing, which is useful for critical system services like sshd . However, it is not advisable for regular applications, as it may cause other processes to be killed repeatedly.

Summary

When physical memory is insufficient, the kernel reclaims memory via asynchronous background reclamation (kswapd) or synchronous direct reclamation. Reclaimable pages are file‑backed and anonymous, both managed by an LRU algorithm. Frequent reclamation leads to disk I/O and performance degradation.

Typical mitigation steps include:

Adjust /proc/sys/vm/swappiness to favor file‑page reclamation.

Increase /proc/sys/vm/min_free_kbytes to trigger kswapd earlier.

Set /proc/sys/vm/zone_reclaim_mode to 0 on NUMA machines to allow cross‑node reclamation.

Tune oom_score_adj for critical processes to reduce their OOM‑kill probability.

By understanding these mechanisms and tuning the relevant kernel parameters, administrators can mitigate memory‑pressure‑induced latency and avoid unwanted OOM terminations.