Master Linux Performance: Essential Monitoring Tools & Commands

Motivated by an interest in Linux and a strong desire for low‑level knowledge, this article compiles essential performance analysis tools and commands for Linux systems, serving both as a knowledge checklist and a practical guide.

Performance Analysis Tools Overview

The following diagram (originally from Brendan Gregg) illustrates the landscape of Linux observability tools; each tool can be queried via

man

for detailed help.

1. vmstat – Virtual Memory Statistics

vmstat monitors virtual memory, processes, CPU, and overall system health. Usage:

vmstat interval times

(e.g.,

vmstat 5 10

). The first line shows averages since boot; subsequent lines show real‑time data at the specified interval.

procs : r = processes waiting for CPU, b = processes in uninterruptible sleep (I/O wait).

memory : swpd = swapped memory, free = idle memory, buff = buffers, cache = cache.

swap : pages swapped in/out per second.

io : bi = blocks received, bo = blocks sent, system = interrupts and context switches per second.

cpu : %us (user), %sy (system), %id (idle), %wa (I/O wait), etc.

Signs of memory shortage include rapid drop in free memory, high swap usage, increased I/O, and many processes waiting for I/O.

2. iostat – CPU and I/O Statistics

iostat reports CPU usage and detailed device I/O statistics. Example to show extended device stats:

<code>iostat -x</code>

Common Linux disk I/O abbreviations: rq (requests), r (reads), w (writes), qu (queue), sz (size), avgrq‑sz (average request size), avgqu‑sz (average queue length), await (average wait), svctm (service time), %util (utilization).

3. dstat – System Monitoring

dstat provides colorful, real‑time output of CPU, disk I/O, network packets, and paging. Example command:

<code>dstat -cdlmnpsy</code>

4. iotop – Real‑time I/O Monitoring

iotop displays per‑process I/O usage, similar to

top

but focused on disk I/O.

<code>iotop -bod interval</code>

5. pidstat – Process Resource Statistics

pidstat monitors CPU, memory, I/O, and thread activity for all or selected processes.

<code>pidstat -d interval</code>

<code>pidstat -u interval</code>

<code>pidstat -r interval</code>

6. top – Interactive Process Viewer

top shows system load, process states, CPU usage breakdown, memory usage, and swap information. The task area lists PID, user, priority, nice value, virtual/physical memory, CPU and memory percentages, and command line.

7. htop – Enhanced top

htop offers an interactive, color‑enhanced interface with mouse support, horizontal/vertical scrolling, and easier process management.

8. mpstat – Multiprocessor Statistics

<code>mpstat -P ALL interval times</code>

9. netstat – Network Statistics

<code>netstat -npl   # show listening ports</code>

<code>netstat -rn   # display routing table</code>

<code>netstat -in   # interface statistics</code>

10. ps – Process Status

<code>ps aux</code>

To kill a specific process:

<code>ps aux | grep mysqld | grep -v grep | awk '{print $2}' | xargs kill -9</code>

To kill zombie processes:

<code>ps -eal | awk '{if ($2 == "Z") {print $4}}' | xargs kill -9</code>

11. strace – System Call Tracing

Trace system calls and signals of a program, e.g.:

<code>strace -e stat64 mysqld -print -defaults &gt; /dev/null</code>

12. uptime – System Uptime and Load

Displays how long the system has been running and the 1‑, 5‑, and 15‑minute load averages.

13. lsof – List Open Files

Examples:

<code>lsof /boot</code>

<code>lsof -i :3306</code>

<code>lsof -u username</code>

<code>lsof -p 4838</code>

<code>lsof -i @192.168.34.128</code>

14. perf – Kernel Performance Analyzer

perf leverages kernel instrumentation to profile functions, cache misses, and hotspots.

Typical workflow: sample with tick‑based interrupts, identify hot functions (e.g.,

foo()

), and focus optimization efforts.

Additional Observability and Benchmarking Tools

perf_events : kernel‑maintained performance diagnostics.

eBPF tools : tracing via BCC, allowing user‑space management of kernel maps.

perf‑tools : collection based on perf_events and ftrace.

bcc (BPF Compiler Collection) : creates efficient kernel tracing programs.

ktap : dynamic kernel tracing similar to DTrace/SystemTap.

Flame Graphs : visualizations generated from perf, SystemTap, or ktap data.

Basic Tools

uptime, top (htop), mpstat, iostat, vmstat, free, ping, nicstat, dstat.

Advanced Commands

sar, netstat, pidstat, strace, tcpdump, blktrace, iotop, slabtop, sysctl, /proc.

By combining these utilities, one can comprehensively monitor and diagnose Linux system performance across CPU, memory, I/O, and network dimensions.