IO Performance Evaluation, Monitoring, and Optimization Guide

In production environments, long I/O latency often leads to reduced throughput and slow response times, caused by issues such as switch failures, aging cables, insufficient storage stripe width, cache limits, QoS restrictions, or improper RAID settings.

1. Prerequisite for Evaluating I/O Capability

Understanding the system's I/O model is essential before assessing its I/O capacity.

(1) I/O Model

Different business scenarios exhibit varied I/O characteristics (read/write ratios, I/O sizes, etc.). A model is built for a specific scenario to support capacity planning and problem analysis.

Basic metrics: IOPS, bandwidth, I/O size.

For disk I/O, also consider which disks are involved, read/write ratios, sequential vs. random patterns.

(2) Why Refine an I/O Model?

The maximum IOPS, bandwidth, and response time differ between random small I/O and sequential large I/O tests; therefore, capacity planning and performance tuning must be based on the actual business I/O model.

2. Evaluation Tools

(1) Disk I/O Tools

Tools such as Orion, iometer, dd, xdd, iorate, iozone, and postmark simulate various workloads; Orion can emulate Oracle database I/O patterns.

(2) Network I/O Tools

ping – basic latency test with configurable packet size.

iperf, ttcp – measure maximum TCP/UDP bandwidth, latency, and packet loss.

Windows tools – NTttcp, LANBench, pcattcp, LAN Speed Test, NETIO, NetStress.

3. Key Monitoring Indicators and Common Tools

(1) Disk I/O

On Unix/Linux, use Nmon and iostat for real‑time and post‑analysis data.

IOPS: Nmon DISK_SUMM (IO/Sec), iostat -Dl (tps), per‑disk read/write IOPS.

Bandwidth: Nmon DISK_SUMM (Disk Read/Write KB/s), iostat -Dl (bps), per‑disk read/write bandwidth.

Response Time: iostat -Dl (read‑avg‑serv, write‑avg‑serv).

Other: queue depth, busy degree, etc.

(2) Network I/O

Bandwidth: Nmon NET sheet, topas (BPS, B‑In, B‑Out).

Response Time: ping for basic latency; for precise measurement, capture SYN‑SYNACK timing or use dedicated network probes.

4. Performance Diagnosis and Optimization

(1) Disk I/O Contention

Identify whether contention originates from excessive application I/O or system limits; address application‑level inefficiencies (e.g., enlarge sort buffers, reduce unnecessary logging) before tuning storage.

(2) Storage‑Side Analysis

Examine the entire I/O path (host → network → storage) and pinpoint the bottleneck layer.

Host side: check queue depth, driver limits, HBA configuration.

Network side: verify bandwidth, switch settings, multi‑path routing, cable integrity.

Storage side: assess RAID level, stripe width, cache size, QoS limits, LUN type (thin vs. thick), controller CPU usage, etc.

(3) Low‑Latency Transaction Scenarios

For high‑speed trading, consider SSDs, SSD cache tiers, RAMDISK, appropriate RAID (e.g., RAID10), and high‑performance networking instead of iSCSI.

(4) Network I/O Issue Diagnosis

Use packet capture and analysis to locate latency or loss within specific network segments.

5. Mis‑diagnosed Cases

Examples show that apparent I/O problems may stem from database buffer waits or excessive LPAR sharing causing CPU contention, highlighting the need for holistic analysis.

Author: Yang Jianxu, senior technical manager with extensive experience in performance testing and tuning for banking systems.