Understanding Linux I/O: Types, Schedulers, and Performance Testing with FIO

I/O Classification

File read/write methods differ, leading to many I/O categories. The most common are:

Buffered vs. unbuffered I/O. Buffered I/O uses standard library caches to accelerate file access, while unbuffered I/O accesses files directly via system calls.

Direct vs. indirect I/O. Direct I/O bypasses the OS page cache and interacts directly with the file system; indirect I/O goes through the page cache.

Blocking vs. non‑blocking I/O. Blocking I/O stalls the thread until the operation completes; non‑blocking I/O returns immediately and later notifies the result.

Synchronous vs. asynchronous I/O. Synchronous I/O waits for the entire operation to finish; asynchronous I/O returns immediately and notifies completion via events.

Linux File System I/O

To read or write data on a file system, a process must cooperate with many kernel components. At the application level, syscalls such as

sys_open

,

sys_read

, and

sys_write

are used. The kernel maintains data structures for each opened file and for all open files system‑wide.

Generic Block Layer

The generic block layer is a kernel component that handles requests from all block devices.

Maps disk data into memory; pages are mapped to kernel linear addresses only when the CPU accesses them and unmapped after use.

Uses techniques such as DMA to achieve zero‑copy, placing disk data directly in user‑space addresses.

Manages logical volumes used by LVM and software RAID.

The generic block layer is the core of Linux disk I/O. It provides a standard interface to block devices for file systems and applications, abstracts heterogeneous disk hardware, and performs request reordering and merging to improve efficiency.

I/O Scheduler Layer

Linux kernel supports four I/O scheduling algorithms: NOOP, CFQ, Deadline, and Anticipatory.

NOOP is a simple FIFO queue with minimal request merging, often used for SSDs.

CFQ (Completely Fair Scheduler) maintains a separate queue per process and distributes I/O time slices fairly; it also supports priority scheduling, suitable for desktop and multimedia workloads.

Deadline uses four queues (read/write sorted by sector and deadline queues) to balance throughput and latency, prioritizing requests with approaching deadlines, which benefits mechanical disks but may suffer with many sequential I/O patterns.

Anticipatory builds on the deadline mechanism, scanning queues and waiting up to ~7 ms for a subsequent read request to arrive, favoring mixed read/write workloads but not ideal for random‑read‑heavy databases like MySQL.

Disk I/O Benchmarking

Key metrics include IOPS (operations per second), bandwidth (throughput), and latency. Small block sizes (e.g., 512 B, 4 kB, 8 kB) emphasize IOPS, while large block sizes (e.g., 256 kB, 512 kB, 1 MB) emphasize bandwidth.

1. FIO Overview

FIO is a tool for testing disk performance, capable of measuring IOPS, throughput, and latency, and supports multiple I/O engines.

2. Download FIO

Download from

http://brick.kernel.dk/snaps/

.

3. Install

<code>wget http://brick.kernel.dk/snaps/fio-3.5.tar.gz
# tar -xzvf ./fio-3.5.tar.gz
# cd fio-3.5
# make && make install
# which fio
/usr/local/bin/fio</code>

4. Usage Example

<code>filename=/dev/sdb1    # test device
direct=1              # bypass OS buffers
rw=randwrite          # random write test
rw=randrw             # random read/write test
bs=16k                # block size
bsrange=512-2048      # block size range
size=5g               # test file size
numjobs=30            # number of threads
runtime=1000          # test duration in seconds
ioengine=psync        # I/O engine
rwmixwrite=30         # write proportion in mixed workload
group_reporting       # aggregate results
lockmem=1g            # limit memory usage
zero_buffers          # initialize buffers with zero
nrfiles=8             # number of files per job
</code>

5. Test Scenarios

<code>Mixed test:
fio -filename=/tmp/test -direct=1 -iodepth 1 -thread -rw=randrw -rwmixread=70 -ioengine=psync -bs=512b -size=200m -numjobs=10 -runtime=60 -group_reporting -name=mytest

Sequential read:
fio -filename=/dev/test -direct=1 -iodepth 1 -thread -rw=read -ioengine=psync -bs=16k -size=2G -numjobs=10 -runtime=60 -group_reporting -name=mytest

Random write:
fio -filename=/dev/test -direct=1 -iodepth 1 -thread -rw=randwrite -ioengine=psync -bs=16k -size=2G -numjobs=10 -runtime=60 -group_reporting -name=mytest

Sequential write:
fio -filename=/dev/test -direct=1 -iodepth 1 -thread -rw=write -ioengine=psync -bs=16k -size=2G -numjobs=10 -runtime=60 -group_reporting -name=mytest
</code>

6. Reference Script

<code>https://github.com/sunsharing-note/fio_test.git</code>