Deep Dive into pfinder: Architecture, Bytecode Enhancement, and Tracing Mechanisms

In modern software development, performance optimization and fault diagnosis are essential for stable applications. Java, as a widely used language, has many monitoring tools such as SkyWalking and Zipkin; JD uses its own self‑built pfinder for full‑link monitoring.

pfinder Overview

pfinder (Problem Finder) is a new‑generation APM system created by the UMP team. It provides call‑chain tracing, application topology, and multi‑dimensional monitoring without code changes—only two lines of script are needed in the startup file. It supports JD’s internal middleware (jimdb, jmq, jsf) and common open‑source components (Tomcat, HTTP client, MySQL, ES).

Key Features

Multi‑dimensional monitoring (by data center, group, JSF alias, caller, etc.)

Automatic instrumentation for SpringMVC, JSF, MySQL, JMQ, etc.

Application topology visualization

Call‑chain tracing for performance bottleneck analysis

AI‑driven automatic fault analysis

Traffic recording and replay for testing

Cross‑unit traffic monitoring for JSF

APM Component Comparison

Component

Zipkin

Pinpoint

SkyWalking

CAT

pfinder

Contributor

Twitter

Korean company

Huawei

Meituan

JD

Implementation

Request interception, HTTP/MQ

Bytecode injection

Bytecode injection

Proxy pointcuts

Bytecode injection

Integration

Linkerd/Sleuth config

javaagent bytecode

javaagent bytecode

Code intrusion

javaagent bytecode

Protocol

HTTP, MQ

Thrift

gRPC

HTTP/TCP

JMTP

OpenTracing

Supported

Supported

Supported

Granularity

Interface level

Method level

Method level

Code level

Method level

Bytecode Modification Techniques

Several mature bytecode manipulation frameworks are demonstrated to implement the same functionality (printing "start" before method execution and "end" after).

ASM Implementation

@Override
public void visitCode() {
super.visitCode();
mv.visitFieldInsn(GETSTATIC, "java/lang/System", "out", "Ljava/io/PrintStream;");
mv.visitLdcInsn("start");
mv.visitMethodInsn(INVOKEVIRTUAL, "java/io/PrintStream", "println", "(Ljava/lang/String;)V", false);
}
@Override
public void visitInsn(int opcode) {
if ((opcode >= Opcodes.IRETURN && opcode <= Opcodes.RETURN) || opcode == Opcodes.ATHROW) {
mv.visitFieldInsn(GETSTATIC, "java/lang/System", "out", "Ljava/io/PrintStream;");
mv.visitLdcInsn("end");
mv.visitMethodInsn(INVOKEVIRTUAL, "java/io/PrintStream", "println", "(Ljava/lang/String;)V", false);
}
mv.visitInsn(opcode);
}

Javassist Implementation

ClassPool cp = ClassPool.getDefault();
CtClass cc = cp.get("com.ggc.javassist.HelloWord");
CtMethod m = cc.getDeclaredMethod("printHelloWord");
m.insertBefore("{ System.out.println(\"start\"); }");
m.insertAfter("{ System.out.println(\"end\"); }");
Class c = cc.toClass();
cc.writeFile("/path/to/classes");
HelloWord h = (HelloWord) c.newInstance();
h.printHelloWord();

ByteBuddy Implementation

// Dynamically generate a new HelloWord class
Class
dynamicType = new ByteBuddy()
.subclass(HelloWord.class)
.method(ElementMatchers.named("printHelloWord"))
.intercept(MethodDelegation.to(LoggingInterceptor.class))
.make()
.load(HelloWord.class.getClassLoader())
.getLoaded();
LoggingInterceptor dynamicService = (LoggingInterceptor) dynamicType.newInstance();
dynamicService.printHelloWord();

ByteKit Implementation

// Parse interceptor class and related annotations
DefaultInterceptorClassParser parser = new DefaultInterceptorClassParser();
List
processors = parser.parse(HelloWorldInterceptor.class);
ClassNode classNode = AsmUtils.loadClass(HelloWord.class);
for (MethodNode methodNode : classNode.methods) {
MethodProcessor methodProcessor = new MethodProcessor(classNode, methodNode);
for (InterceptorProcessor interceptor : processors) {
interceptor.process(methodProcessor);
}
}

The table below compares performance, usability, and functionality of the four frameworks.

Metric

ASM

Javassist

ByteBuddy

ByteKit

Performance

Highest (direct bytecode ops)

Lower than ASM

Between Javassist and ASM

Between Javassist and ASM

Ease of Use

Requires deep bytecode knowledge

Java‑style API but no debug support

More Java‑friendly, supports debugging

Similar to ByteBuddy, debug‑friendly

Features

Full control, most powerful

Relatively complete

Relatively complete

Relatively complete, prevents duplicate enhancement

Bytecode Injection

Debugging in IDEs like IDEA relies on JVMTI. The JVM specification defines JVMTI (merged from JVMPI and JVMDI) as the native interface for profiling and debugging. Agents are loaded via -agentlib or -javaagent options.

JVMTI Agent Example

Agent_OnLoad(JavaVM *vm, char *options, void *reserved) { /* called when agent is loaded via -agentlib */ }
Agent_OnAttach(JavaVM *vm, char *options, void *reserved) { /* called when agent is attached to a running JVM */ }
Agent_OnUnload(JavaVM *vm) { /* called when agent is unloaded */ }

IDEA’s debug uses the built‑in jdwp agent, which is loaded as libjdwp.dylib on macOS.

Java Instrumentation API

Since JDK 5, developers can write agents in Java using java.lang.instrument.Instrumentation . Key methods include:

void addTransformer(ClassFileTransformer transformer);
Class[] getAllLoadedClasses();
Class[] getInitiatedClasses(ClassLoader loader);
void redefineClasses(ClassDefinition... definitions);
void retransformClasses(Class
... classes);

Instrumented Agent with ByteBuddy

public class GhlAgent {
public static void premain(String args, Instrumentation inst) { boot(inst); }
public static void agentmain(String args, Instrumentation inst) { boot(inst); }
private static void boot(Instrumentation inst) {
new AgentBuilder.Default()
.type(nameStartsWith("com.jd.aviation.performance.service.impl"))
.transform((builder, type, loader, module) ->
builder.method(isMethod().and(isPublic()))
.intercept(MethodDelegation.to(TimingInterceptor.class)))
.installOn(inst);
}
}

Timing Interceptor

public class TimingInterceptor {
public static Object intercept(@SuperCall Callable
callable) throws Exception {
long start = System.currentTimeMillis();
try { return callable.call(); }
finally { long end = System.currentTimeMillis(); log.info("Method call took {} ms", end - start); }
}
}

pfinder Internals

When the pfinder agent starts, it loads META-INF/pfinder/service.addon and META-INF/pfinder/plugin.addon files, registers services, and performs bytecode enhancement via the JMTP protocol.

Service Loading

A SimplePFinderServiceLoader reads the service configuration and creates an iterator of service factories.

Plugin Loading & Enhancement

Plugins are loaded by PluginLoader , which supplies type matchers for classes to be enhanced. The AgentBuilder is configured with a combined matcher chain, exclusion policies, and listeners. During transformation, each plugin’s InterceptPoint array is processed, and the appropriate enhancer (Advice, MethodDelegation, etc.) is applied.

Advice‑Based Enhancement Flow

Advice injects onMethodEnter and onMethodExit hooks into target methods. These hooks delegate to the plugin’s interceptor methods, completing the bytecode enhancement.

Trace‑ID Propagation Across Threads

pfinder stores the trace‑ID in MDC (a ThreadLocal). To avoid loss in asynchronous execution, it wraps user Runnable instances with TracingRunnable , capturing the originating snapshot and restoring it in the child thread.

public class TracingRunnable implements PfinderWrappedRunnable {
private final Runnable origin;
private final TracingSnapshot
snapshot;
public TracingRunnable(Runnable origin, TracingSnapshot
snapshot, ...) {
this.origin = origin; this.snapshot = snapshot; /* other fields */ }
public void run() {
LowLevelAroundTracingContext ctx = SpringAsyncTracingContext.create(...);
ctx.onMethodEnter();
try { origin.run(); } catch (RuntimeException e) { ctx.onException(e); throw e; } finally { ctx.onMethodExit(); }
}
}

Hot Deployment Prototype

Using the JMTP command channel, pfinder can perform class search, decompilation, and hot‑update. The prototype demonstrates UI screens for each operation, though it lacks support for Spring/MyBatis XML and is limited by the JVM’s inability to change class structure without DCEVM.

Open Issues & Future Work

Support for Spring XML and MyBatis XML configuration files.

Instrumentation cannot add fields or change class hierarchy; using DCEVM could overcome this limitation.

The author invites interested readers to collaborate and explore further.