If You Can't Use These Linux Performance Tools, Your Server Is Just a Paperweight

The author organizes a practical reference for Linux performance analysis, beginning with an overview image from Brendan Gregg’s presentation and stating that all tools can be accessed via their manual pages.

vmstat – Virtual Memory Statistics

vmstat reports overall system activity, including virtual memory, processes, and CPU usage. The basic syntax is vmstat interval times, where interval is the sampling period in seconds and times is the number of samples (omitting times runs indefinitely). The first line shows averages since boot; subsequent lines show current snapshots every interval seconds. Columns include:

procs : r – runnable processes; b – processes blocked in uninterruptible sleep (I/O).

memory : swpd – swapped-out pages; other columns show free, buffered, and cached memory.

swap : pages swapped in/out per second.

io : blocks read ( bi) and written ( bo).

system : interrupts ( in) and context switches ( cs) per second.

cpu : percentages of user, system, idle, and I/O wait time.

Typical signs of memory pressure include a rapid drop in free memory, high swpd, frequent page swaps, increased I/O, and rising cs and b counts.

iostat – CPU and I/O Statistics

iostat reports CPU usage and detailed device I/O statistics. The default view mirrors vmstat’s CPU columns; adding device options shows per‑device metrics. Common column abbreviations are rrqm/s, wrqm/s (merged reads/writes), r/s, w/s (requests per second), rsec/s, wsec/s (sectors per second), avgrq‑sz (average request size), avgqu‑sz (average queue length), await (average I/O wait), svctm (service time), and %util (device utilization).

dstat – Combined System Monitoring

dstat displays CPU, disk I/O, network packets, and paging in a colorful, easy‑to‑read format, extending vmstat and iostat. A typical invocation is dstat -cdlmnpsy, which shows CPU, disk, load, memory, network, and system statistics together.

iotop – Real‑Time Disk I/O per Process

iotop works like top but focuses on I/O. It lists processes with their I/O read/write rates. Non‑interactive usage example: iotop -bod interval To view I/O for a specific process, combine with pidstat -d interval.

pidstat – Per‑Process Resource Monitoring

pidstat can report CPU ( -u), memory ( -r), and I/O ( -d) usage for all or selected processes. Example syntax:

pidstat -d interval

pidstat -u interval

pidstat -r interval

top / htop – Interactive Process Viewers

top

summarizes system load, process states, CPU distribution, memory usage, and swap. The task area lists PID, user, priority, nice value, virtual/physical/shared memory, state, CPU% and MEM%, cumulative CPU time, and command line. htop offers a richer UI with color, mouse support, horizontal/vertical scrolling, and the ability to kill processes without typing PIDs. Compared with top, htop starts faster, allows easier navigation, and supports mouse interactions.

mpstat – Multiprocessor Statistics

mpstat reports per‑CPU and overall CPU statistics from /proc/stat. Typical usage:

mpstat -ALL interval times

netstat – Network Connections

netstat displays IP, TCP, UDP, and ICMP statistics and can verify listening ports or print routing tables. Common commands:

netstat -npl   # show listening ports

netstat -rn    # display routing table

netstat -in    # interface statistics

ps – Process Status

Use ps aux for a full snapshot or pipe to grep for filtering. To kill a specific process:

ps aux | grep mysqld | grep -v grep | awk '{print $2}' | xargs kill -9

To remove zombie processes:

ps -eal | awk '{if ($2 == "Z") {print $4}}' | xargs kill -9

strace – System Call Tracing

strace records system calls and signals for a program, useful for diagnosing runtime issues. Example to see which configuration file mysqld loads:

strace -e stat64 mysqld -print -defaults > /dev/null

uptime – System Run Time and Load Averages

Displays how long the system has been up and the 1‑, 5‑, and 15‑minute load averages.

lsof – List Open Files

lsof enumerates open files and can be filtered to show files in a directory, ports in use, files opened by a user, or files opened by a specific PID. Examples:

lsof /boot

lsof -i :3306

lsof -u username

lsof -p 4838

perf – Kernel‑Integrated Performance Analyzer

perf, part of the Linux kernel, samples program execution based on timer ticks. If a function consumes 90 % of runtime, 90 % of samples will fall in its context, allowing developers to pinpoint hotspots and cache‑miss ratios.

sar – System Activity Reporter

sar is a comprehensive performance collector that can report CPU efficiency, memory usage, disk I/O, system calls, and IPC activity. Basic syntax: sar [options] [-A] [-o file] t[n] Here t is the sampling interval, n the number of samples, and -o file stores results in binary format.

Putting It All Together

By selecting the appropriate tool from the list above—basic tools like uptime, top, mpstat, vmstat, free, ping, nicstat, dstat and advanced tools such as sar, netstat, pidstat, strace, tcpdump, blktrace, iotop, slabtop, sysctl, and the /proc filesystem—you can systematically observe, diagnose, and tune Linux server performance.

Images illustrating each tool’s output are retained to aid visual understanding.