Investigation of Slab‑Lock Issues and CPU Power Management on Linux Servers

Background – In Q4 2018 a batch of servers used for storage showed write latencies exceeding 800 ms (normally <300 ms) and sub‑optimal read performance.

Analysis revealed that the monitoring agent’s use of the ss command reads /proc/slabinfo and acquires a large slab lock, causing RocksDB write operations to block for a long time. Two remedies were identified: modify the agent or upgrade the kernel from 2.6 to 3.10/4.18, the latter being applied.

Read latency was linked to BIOS CPU performance‑mode settings; after adjusting these settings, I/O performance improved, though final verification is pending.

Slab Lock – Older kernels (2.6.32) protect slab information with a single coarse lock, leading to long wait times when ss queries TCP statistics. Newer kernels (3.10, 4.18) split this into finer‑grained locks, dramatically reducing contention. The issue can also be mitigated by stripping the network‑monitoring module from the agent, a workaround used on legacy servers.

CPU Power Management – Subsequent read‑performance tests on the upgraded 4.18 kernel showed the Q4 machines performed worst. A custom pread benchmark and a simple compute loop were used to confirm the trend.

echo "i=0; while (( i < 1000000 )); do (( i ++ )); done;" | perf stat sh

The root cause was identified as BIOS power‑saving settings (P‑State, C‑State, MONITOR/MWAIT) enabled on the Q4 servers. Disabling these settings restored compute performance on the 4.18 kernel, while the 2.6 kernel performance degraded.

CPU Power‑State Overview

Modern CPUs balance frequency and power using three main state families:

P‑States – Performance states that adjust core frequency and voltage. The OS moves between P0 (max performance) and lower states (P1, P2…) to save power. Enabling P‑States generally has minimal impact on performance while reducing energy consumption.

C‑States – Power‑saving (sleep) states ranging from C0 (active) to deeper states (C1, C3, C6…). Deeper states shut down more circuitry, increasing wake‑up latency. They can affect latency‑sensitive workloads.

T‑States – Throttling states triggered when the CPU exceeds thermal limits, intermittently pausing execution to cool down.

Tools such as turbostat or i7z can display current C‑State residency. Example output shows an Intel Xeon E5‑2630 v4 with 40 logical cores, 20 physical cores, and two packages.

Package‑level C‑States (PC‑States) require all cores to be in deep sleep, also disabling uncore components like PCIe, which lengthens wake‑up time.

BIOS options like MONITOR/MWAIT allow the OS to place the CPU into deeper idle states without generating extra inter‑processor interrupts, improving efficiency; however, in some cases enabling them can degrade performance.

Further details on how different Linux kernels utilize these instructions and states will be covered in a future article.