How Critical Path Tracing Cuts Latency in Large Distributed Systems

As user experience increasingly depends on low response latency, analyzing service latency has become essential for large distributed systems. This article introduces common latency analysis techniques and focuses on critical path tracing, a method adopted by companies like Google, Meta, and Uber, and successfully applied in Baidu's App recommendation service.

Background

Internet services face growing pressure to reduce response latency, yet traditional analysis methods struggle with the fast iteration cycles and complex call graphs of modern micro‑service architectures.

Common Distributed‑System Latency Analysis Methods

RPC Monitoring

Most RPC frameworks (e.g., BRPC, gRPC, Thrift) embed telemetry that records method names and execution times. External monitoring systems such as Prometheus collect these metrics and display them on dashboards.

While simple and effective for straightforward call graphs, RPC monitoring cannot capture internal component latency and may mislead optimization when parallel sub‑calls have differing costs.

CPU Profiling

CPU profiling samples call stacks and aggregates the most frequent functions, which are considered the main latency contributors. The result is often visualized as a flame graph.

CPU profiling overcomes some RPC limitations but still cannot differentiate parallel sub‑calls, making it costly to optimize large systems.

Distributed Tracing

Distributed tracing records spans for each request across services, reconstructing the end‑to‑end call topology with timestamps. Tools such as Google Dapper and Uber Jaeger provide this capability.

Although powerful, tracing usually lacks fine‑grained component data, and the sheer number of internal nodes can make detailed analysis expensive.

Critical Path Analysis

Introduction

A critical path is the longest‑latency sequence of nodes inside a service. Even if a system contains hundreds of components, the critical path typically includes only dozens, dramatically reducing the optimization scope.

In the example, service A calls B, and the critical path A1→A2→B1→B4→B2→A4 has a total latency of 195 ms, illustrating how focusing on this path can significantly reduce overall response time.

Practical Solution

The data‑collection pipeline gathers per‑service critical‑path information, aggregates it across services, and visualizes the results.

Core Path Production and Reporting

Each service emits its critical‑path data via an operator‑based execution framework. The framework records start and end timestamps for each operator (P1‑P4 in the diagram) and reports them to a collector.

Core Path Aggregation and Computation

Aggregating critical paths across services is performed in time windows. Three aggregation strategies are used:

Node‑level aggregation : concatenate all node sequences and select the most frequent as the overall critical path.

Service‑level aggregation : collapse internal computation nodes into a single inner node, keeping external calls separate, then pick the most frequent service‑level path.

Flat‑node type aggregation : when many nodes appear with similar frequencies, compute a "core‑share" metric and retain nodes whose share exceeds a configurable threshold.

Storage and Visualization

Aggregated results are stored in an OLAP engine for multi‑dimensional queries (by time, user type, traffic source, etc.). The UI presents several metrics for each node:

Core share : probability that the node appears in the critical path.

Core contribution : proportion of the node’s latency to the total path latency when it appears.

Combined contribution : product of core share and contribution, used to rank nodes for optimization.

Mean latency and percentiles (50th, 80th, 90th, etc.).

Application

Baidu’s App recommendation platform has deployed a critical‑path latency analysis platform called Focus . After more than a year of operation, it continuously monitors and optimizes the millisecond‑level response of the feed recommendation API, earning praise from R&D, operations, and algorithm teams.

When a latency anomaly is detected, Focus automatically identifies the offending service (e.g., Service B), drills down to the problematic node (e.g., Node X), and, using flat‑node aggregation, discovers an abnormal increase in a downstream queue (e.g., Queue Y). The issue is then routed to the responsible owner for rapid resolution, all without manual investigation.

Summary

Low‑latency service interfaces are vital for user experience in modern large‑scale distributed systems. Critical‑path analysis provides a cost‑effective way to pinpoint the slowest steps across complex call graphs, enabling targeted optimizations that dramatically improve overall response time. The Baidu case demonstrates a production‑grade, platform‑level implementation, and the technique continues to offer rich opportunities for further research and innovation.