How Pinpoint Enables Zero‑Impact Distributed Tracing with Bytecode Injection

1. What is Pinpoint

Pinpoint is a full‑stack tracing tool that provides non‑intrusive call‑chain monitoring, method execution details, and application status monitoring, based on the Google Dapper paper.

The core idea is to record and propagate an application‑level identifier across service nodes, which is attached to HTTP headers or other protocols and used by the Pinpoint collector to link requests into a complete call chain.

Key features include:

Distributed transaction tracing across services.

Automatic detection of application topology.

Horizontal scaling for large server clusters.

Code‑level visibility to pinpoint failures and bottlenecks.

Bytecode enhancement to add functionality without modifying source code.

Pinpoint offers plugins for many components, such as JDK 6+, Tomcat, Jetty, Spring Boot, various HTTP clients, Thrift, Dubbo, databases (MySQL, Oracle, PostgreSQL, etc.), caches (Redis, Cassandra), and logging frameworks.

2. Plugin Knowledge and Data Structures

Plugins can intercept methods at the bytecode level to record execution time, parameters, return values, or inject identifiers for RPC calls. They are easy to extend, and Pinpoint provides many ready‑made plugins for common libraries.

2.1 Plugin Structure

A Pinpoint plugin consists of implementations of

TraceMetadataProvider

and

ProfilerPlugin

. The former supplies

ServiceType

and

AnnotationKey

metadata; the latter modifies target classes to collect tracing data.

Plugins are packaged as JAR files. The agent loads them via Java’s

ServiceLoader

mechanism, which requires provider configuration files in

META-INF/services

.

2.1.1 TraceMetadataProvider

Provides

ServiceType

definitions, which identify the library a span belongs to and dictate handling logic.

Each

ServiceType

includes attributes such as a unique integer code, name, and properties. The code is sent as an integer to keep payloads small, and a mapping from code to

ServiceType

is maintained by the provider.

Public

ServiceType

codes must be obtained from the Pinpoint team; private plugins can use codes from a reserved private range.

2.1.2 AnnotationKey

AnnotationKey

represents key‑value pairs attached to spans for detailed data. Built‑in keys exist, and custom keys can be added via

TraceMetadataProvider

.

2.2 ProfilerPlugin

The

ProfilerPlugin

modifies target classes to collect trace data. Its workflow:

Pinpoint Agent starts with the JVM.

Agent loads all plugins from the

plugin

directory.

Agent invokes each plugin’s

setup(ProfilerPluginSetupContext)

method.

During setup, the plugin registers class transformers.

When the application starts, classes are loaded.

For each loaded class, the agent checks for registered transformers.

If a transformer is found, the agent calls its

doInTransform

method.

The transformer modifies the bytecode (e.g., adds interceptors or fields).

The modified class is loaded by the JVM.

When the transformed method executes, the injected interceptor’s

before

and

after

methods run, recording trace data.

The essential steps are selecting methods to trace and injecting interceptors to capture execution details.

3. How Bytecode Injection Works

Pinpoint injects tracing code into target classes at load time using bytecode manipulation, which minimizes runtime overhead and keeps trace data compact.

Interceptors are injected around the target method, invoking

before()

and

after()

to record data.

Below is a simplified example of a custom plugin that intercepts configured methods.

First, define constants for

ServiceType

and

AnnotationKey

codes:

<code>/* Example constants definition (image omitted) */</code>

Define a configuration reader that parses lines like:

Method with parameters

package.Clazz.MethodArgs=arg1,arg2

Method without parameters

package.Clazz.MethodArgs

Each line represents one configuration entry.

Implement

GeneralConfigMetadataProvider

to supply

ServiceType

metadata:

<code>/* Metadata provider implementation (image omitted) */</code>

Create

GeneralConfigInterceptor

extending

SpanEventSimpleAroundInterceptorForPlugin

to record method name, arguments, and return value after execution:

<code>/* Interceptor implementation (image omitted) */</code>

Finally, implement the plugin class that registers the interceptor with the transform template:

<code>/* Plugin registration code (image omitted) */</code>

Add the plugin and metadata provider entries to

META-INF/services

:

<code>com.navercorp.pinpoint.plugin.ProfilerPlugin--com.navercorp.pinpoint.plugin.general.config.GeneralConfigPlugin</code>

<code>com.navercorp.pinpoint.common.trace.TraceMetadataProvider--com.navercorp.pinpoint.plugin.general.config.GeneralConfigMetadataProvider</code>

Deploy the JAR to the agent’s

plugin

directory, configure the target methods in

project.properties

, and start the application. The intercepted methods will appear in Pinpoint’s UI with the recorded trace information.