Log Sampling and Cross‑Thread Propagation in High‑Throughput Java Services

System logs are essential for tracing user actions and diagnosing issues, but in high‑throughput services the volume of logs can severely affect performance and disk I/O, especially during peak traffic.

To balance observability with efficiency, the article suggests sampling requests at the entry point, adjusting the sampling rate according to system load, and ensuring that sampled logs still provide useful traceability.

Because modern services often involve asynchronous processing, the sampling identifier must be propagated across the request thread, child threads, and thread‑pool threads; the article explores using Java's ThreadLocal mechanism and the challenges of maintaining consistency in complex async flows.

Three main approaches are discussed: (1) wrapping thread‑pool or task classes (similar to TransmittableThreadLocal) to carry the sampling flag; (2) explicitly passing the flag in code wherever asynchronous execution occurs; and (3) creating a reusable component with an API that abstracts the sampling logic, allowing each business system to adopt it with minimal code changes.

The article recommends choosing the component‑based approach when a unified corporate API is unavailable, and provides guidance on when to apply simple request‑thread sampling versus full async‑thread coordination based on the amount of asynchronous logic in a service.

Additional design considerations include the choice of sampling algorithm (random vs. trace‑ID modulo), scene‑based sampling tied to specific interfaces or parameters, API features that minimize business code modifications, ensuring global trace‑ID consistency (e.g., using existing pfinder solutions), determining the granularity of sampling rates, deciding between a single global probability or per‑level settings, and strategies for handling bursts of error logs such as rate limiting or coupling log volume to disk I/O.

In practice, the author’s team applied these ideas to a promotion‑transaction system by extending an existing JD JSF log component, adding request‑level sampling, and exposing an API for other services; a flow diagram (image) illustrates the implementation, and a code snippet shows how to wrap a thread‑pool task to propagate the sampling flag across threads.

// Before refactor: thread pool task execution
threadPoolExecutor.execute(() -> "your business logic");

// After refactor: wrap task to propagate sampling identifier
threadPoolExecutor.execute(XxxUtils.wrap(() -> "your business logic"));