Mastering iostat: How to Diagnose Linux I/O Bottlenecks with Real‑World Examples

1. Basic iostat Commands

1.1 Command Introduction

$ iostat -d -k 1 10

-d shows device (disk) usage status.

-k forces output to kilobytes.

1 10 means refresh every 1 second for a total of 10 samples.

1.2 Usage Example

# iostat -x 1 10

avg-cpu: %user %nice %system %iowait %steal %idle

Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm %util

sda 0.00 3.50 0.40 2.50 5.60 48.00 18.48 0.00 0.97 0.97 0.28

sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00

1.3 Parameter Explanation

rrqm/s – reads merged per second.

wrqm/s – writes merged per second.

r/s – reads completed per second.

w/s – writes completed per second.

rsec/s – sectors read per second.

wsec/s – sectors written per second.

rkB/s – kilobytes read per second (half of rsec/s).

wkB/s – kilobytes written per second (half of wsec/s).

avgrq-sz – average size of I/O requests (in sectors).

avgqu-sz – average queue length.

await – average wait time per I/O (ms).

svctm – average service time per I/O (ms).

%util – percentage of time the device was busy.

1.4 Parameter Analysis

If %util approaches 100 %, the disk is saturated and a bottleneck exists.

await < 70 ms usually indicates moderate I/O pressure; higher values suggest queuing delays.

Combine with vmstat to examine bi (blocked processes) and wa (CPU time spent waiting for I/O, >30 % indicates high pressure).

Typical observations:

svctm < await (service time is counted once per request).

When svctm ≈ await, I/O experiences little waiting.

If await » svctm, the queue is long and response time degrades.

2. Understanding I/O Through Real‑World Analogy

2.1 Supermarket Checkout Analogy

r/s + w/s corresponds to the total number of customers.

avgqu-sz is like the average number of people waiting per unit time.

svctm represents the cashier’s processing speed.

await reflects the average waiting time per customer.

avgrq-sz is comparable to the average amount of goods each customer buys.

%util shows the proportion of time the checkout line is occupied.

These parallels help visualize I/O patterns, speed, and response times.

3. Advanced iostat Usage

3.1 What iostat Shows

iostat reports CPU activity, device I/O statistics, and can display interval‑based data. The default output includes terminal, disk, and CPU metrics (tdc). Specifying other options replaces the default set.

3.2 Syntax

Basic syntax: iostat <options> interval count Options let you select devices (‑d, ‑c, ‑t) or request extended statistics (‑x).

interval – seconds between samples.

count – number of samples to collect.

3.3 Example

# iostat -xtc 5 2

extended disk statistics   tty   cpu

sd0 2.6 3.0 20.7 22.7 0.1 0.2 59.2 6 19 0 84 3 85 11 0

3.4 Detailed Parameter List

disk – device name.

r/s – reads per second.

w/s – writes per second.

Kr/s – kilobytes read per second.

Kw/s – kilobytes written per second.

wait – average number of requests waiting (queue length).

actv – average number of active requests.

%w – percentage of time requests spent waiting (queue non‑empty).

%b – percentage of time the device was busy.

3.5 Interpreting Results and Solutions

Key values to watch:

Reads/writes per second (r/s, w/s).

Percentage busy (%b).

Service time (svc_t).

If a disk shows sustained high reads/writes, %b ≫ 5 % and svc_t ≫ 30 ms, consider:

Optimizing the application to use I/O more efficiently (caching, queue tuning).

Distributing the filesystem across multiple disks or using RAID striping.

Increasing system parameters such as inode cache (ufs_ninode).

Upgrading to faster disks or controllers.