Analysis of Slab Lock Issues and CPU Power Management on Linux Servers

Background: In Q4 2018 a batch of servers used for storage exhibited write latencies exceeding 800 ms (normally <300 ms) and sub‑optimal read performance.

Investigation revealed that the monitoring agent’s use of the ss command reads /proc/slabinfo and acquires a large slab lock, causing RocksDB write stalls; the issue can be resolved by either modifying the agent or upgrading the kernel from 2.6 to 3.10/4.18.

Read latency was later linked to the BIOS CPU performance mode; after tuning the BIOS settings, I/O performance improved noticeably.

Slab lock details: the 2.6.32 kernel employs a single coarse‑grained lock for slab information, while kernels 3.10 and 4.18 split this into multiple fine‑grained locks, dramatically reducing lock‑wait time and impact on other processes.

Thus, upgrading the kernel eliminates the write‑timeout problem; alternatively, older servers could patch the monitoring agent to remove the problematic network‑monitoring module.

CPU performance testing with a small pread benchmark showed that Q4 machines running the 4.18 kernel performed worst, and the root cause was identified as BIOS power‑saving settings (P‑State, C‑State, MONITOR/MWAIT). Disabling these settings restored compute performance on the 4.18 kernel machines, while the 2.6 kernel machines saw a performance drop.

Example script used for compute testing: echo "i=0; while (( i < 1000000 )); do (( i ++ )); done;" | perf stat sh

CPU power‑management overview: modern CPUs use three main energy‑saving state families—P‑States (performance levels), C‑States (sleep states), and T‑States (throttling states). P‑States adjust frequency and voltage; C‑States shut down parts of the core when idle; T‑States trigger when temperature exceeds safe limits.

Tools such as turbostat or i7z can display detailed CPU state information. For example, an Intel Xeon E5‑2630 v4 @ 2.20 GHz has 40 logical cores, 20 physical cores, and 2 packages, each with its own CC‑State and PC‑State hierarchy.

Further discussion on how different Linux kernels utilize these instructions and power states will be continued in a future post.