When CPUs Hide GPU Bottlenecks: How Btune 2.0 Automates Latency Analysis to Uncover Performance Issues

Hidden CPU bottleneck in GPU/XPU workloads

A core challenge in AI infrastructure operations is that expensive GPU compute often cannot be fully utilized because the bottleneck resides in the CPU‑GPU coordination layer. A real migration of a critical inference service from a GPU cluster to a domestically produced XPU cluster showed high request load but QPS far below theoretical limits, with both CPU and XPU utilization fluctuating and overall low.

Root‑cause investigation

Cross‑team troubleshooting revealed that the service performed well when deployed directly with Docker, but degraded when run as a Kubernetes pod on the company’s container platform, suggesting a lower‑level component issue. By disabling agents one by one, the team identified a component named halolet. Concurrently, hardware‑team hotspot analysis with Btune 1.0 highlighted the kernel lock _unlocked_loctl as a hotspot. Combining these clues, the root cause was traced to frequent driver‑interface calls by halolet that held the kernel lock _unlocked_loctl for extended periods, causing CPU‑XPU mutual waiting and performance collapse. Optimizing the halolet call logic to avoid frequent lock acquisition restored inference performance.

Btune 2.0 architectural upgrade

Btune 1.0 relied on the USE (Utilization, Saturation, Errors) and TSA (Thread State Analysis) methods, building five bottleneck analysis trees that addressed most CPU resource issues. However, AI scenarios with complex CPU‑GPU co‑execution and multi‑process interference required more than a resource‑centric view. Btune 2.0 introduces a three‑layer architecture: Load Portrait + Performance Diagnosis Tree + AI Agent.

Full‑dimensional load profiling and diagnosis tree

CPU, memory, disk, network, GPU/XPU, interconnect, parallelism, latency – eight dimensions that capture not only resource occupancy but also where time is spent.

Deep latency analysis modules

Scheduler Latency : time a thread spends waiting in the ready queue.

Interrupt/soft‑interrupt preemption latency: impact of hardware interrupts on normal task execution.

System call latency: cost of user‑kernel mode switches.

Task preemption latency : how high‑priority tasks deprive the current task.

D‑state (uninterruptible wait) latency : precise location of I/O or lock‑induced process blocking.

This fine‑grained analysis lets developers quickly distinguish whether a performance problem stems from compute intensity, I/O blocking, or synchronization primitive contention.

AI Agent for automated decision making

Btune 2.0 integrates an AI Agent that fuses hardware metrics, a knowledge base, and real‑time load portraits. The Agent automatically builds multi‑dimensional models, reasons over the performance diagnosis tree, invokes appropriate toolchains (e.g., lock analysis, stack collection), and generates two reports: a cost analysis pinpointing resource waste and a performance analysis delivering root‑cause and optimization suggestions. The Agent acts like an tireless “chief performance architect” that makes decisions clear and executable.

Case study: digital‑human model training

To validate automation in a complex scenario, Btune 2.0 was applied to a digital‑human training workload where throughput fluctuated and average performance degraded. The Agent performed comprehensive sampling across CPU, GPU/XPU, interconnect, network, and disk I/O. Initial diagnosis ruled out XPU compute and interconnect resource limits, focusing on abnormal kernel‑time data.

The Agent then invoked the deep latency module, dissecting scheduler latency, interrupt preemption, system calls, task preemption, and D‑state latency. It discovered a significant anomaly in D‑state latency. The Agent automatically collected data within a time window, recorded D‑state duration distributions, kernel call stacks, and per‑path timings, and identified the lock object responsible. By correlating the lock’s kernel address, the Agent linked all processes contending for the lock and pinpointed the “culprit process” that blocked the training job.

Compared with traditional single‑process resource views, Btune 2.0’s cross‑process correlation automatically produced an interpretable performance report without manual call‑chain tracing.

Results and conclusions

After developers optimized the identified culprit process, training throughput stability improved markedly. The evolution from manual troubleshooting to automated, latency‑centric diagnostics demonstrates a deeper understanding of AI infrastructure performance. In an era of soaring compute costs and ever‑larger models, each millisecond saved translates to substantial cost savings and efficiency gains. Btune 2.0 is presented not merely as a tool but as a standardized performance governance solution, with open‑source code and concepts that aim to help developers overcome the hidden traps of heterogeneous computing.