Uncover Hidden Performance Bottlenecks with Deep CPU, Memory, Disk & Network Analysis

1. Background

When conducting performance testing, common metrics catch most obvious problems, but subtle performance anomalies often require deeper data analysis. This article records methods and ideas for analyzing such hidden data changes to uncover performance issues.

2. Diagnostic Tools Overview

2.1 CPU

When a high CPU usage alert appears, identify the offending process from monitoring, then log into the Linux server. Use

strace

for system call summaries,

perf

for hotspot functions, or dynamic tracing to observe execution and pinpoint the bottleneck.

2.2 Memory

When a memory shortage alert occurs, find the top memory‑consuming processes from monitoring, examine their historical usage for leaks, then investigate the process’s memory space on the server to understand why it consumes so much memory.

2.3 Disk

If

iostat

shows disk I/O bottlenecks (high utilization, long response time, or sudden queue length increase), use

pidstat

,

vmstat

to locate the source, then analyze filesystem, cache, and process I/O to determine the cause.

2.4 Network

Network performance analysis starts from protocol layers: link layer (throughput, packet loss, errors, soft interrupts), network layer (routing, fragmentation), transport layer (TCP/UDP metrics), and application layer (HTTP/DNS QPS, socket buffers). All metrics originate from kernel interfaces such as

/proc/net

. When a network alert arrives, query these metrics to locate the problematic layer, then use

netstat

,

tcpdump

, or BCC tools on the Linux host to pinpoint the root cause.

3. In‑Depth Data Analysis Cases

3.1 Data Ramp‑up Issue and Analysis

During a performance test of a custom service integrated into a middle platform, TPS exhibited a ramp‑up period after each concurrency burst. Investigation revealed that the

parammodeldetail

API made excessive database calls (product, brandgood, fenshua123) during the ramp, indicating a cache‑miss logic problem. Optimisation: improve cache handling for missing data to reduce DB interactions.

3.2 Stack Data Analysis

A pressure test of a cloud‑map integration showed that beyond 12 concurrent threads, response time spiked and TPS dropped to zero. Stack traces indicated the bottleneck lay in the storage middle‑platform service. CPU usage on storage nodes reached >98%, causing GC pressure, possible memory paging, and overall service slowdown.

3.3 Single‑Interface Latency Analysis and Optimisation

In a Grep service test, detailed timing points were instrumented across four stages: buildParamModels , buildBaseModelInfoMap , buildElements , and buildGrepRemains . Before optimisation, the build stage dominated latency. After enabling session‑persistence, caching model attachments, and caching pre‑append types, build time decreased, yielding a 20‑30% overall response‑time reduction.

4. Summary

Performance testing can easily spot obvious issues like high TPS or CPU usage, but subtle anomalies require deeper data analysis. By systematically examining CPU, memory, disk, and network metrics and applying targeted profiling, hidden bottlenecks can be uncovered and mitigated, helping prevent production incidents.