CPU Monitoring and IRQ Affinity Tuning for Performance Testing

When conducting performance testing, proper monitoring is essential. This article describes issues discovered through CPU monitoring and the corresponding solutions.

During a stress test, one CPU core showed very high usage, and the soft‑interrupt (si) accounted for 42% of the load.

First, the NIC multi‑queue situation is examined. Using the command cat /proc/interrupts | grep eth2 it can be seen that the network card provides 24 queues (queues 131‑155 are shown as an example).

Next, the soft‑interrupt affinity is checked. The command for i in {131..155}; do cat /proc/irq/$i/smp_affinity_list; done reveals that all interrupt queues are bound to CPU 0, which explains the high soft‑interrupt usage on that core.

To resolve the issue, each interrupt queue is bound to a distinct CPU core. Example commands:

echo "0" > /proc/irq/131/smp_affinity_list

echo "1" > /proc/irq/132/smp_affinity_list

echo "2" > /proc/irq/133/smp_affinity_list

echo "3" > /proc/irq/134/smp_affinity_list

echo "4" > /proc/irq/135/smp_affinity_list

... and so on for all 24 queues, resulting in a one‑to‑one mapping between the 24 interrupt queues and the 24 CPU cores.

Alternatively, a ready‑made script can be used, e.g., ./set_irq_affinity.sh ethX , which automates the binding process.

After rebinding, the performance test shows that total CPU usage and soft‑interrupt usage are evenly distributed across all 24 cores, demonstrating a clear improvement.

Images in the original article illustrate the monitoring data before and after the changes, as well as the script content.