Investigating Off‑Heap Memory Leak in Dble Using Java NIO and BTrace

When using Java NIO with Dble, a client experienced a complete timeout of MySQL heartbeat checks, which was temporarily resolved by restarting the service. The root cause was identified as an off‑heap memory leak.

Phenomenon

All backend MySQL instances reported heartbeat timeouts, and the logs contained repeated warnings about exceeding the DirectByteBufferPool size.

Analysis Process

Log inspection revealed messages such as //心跳超时 and You may need to turn up page size. The maximum size of the DirectByteBufferPool that can be allocated at one time is 2097152, and the size that you would like to allocate is 4194304 . These indicated possible long GC pauses due to insufficient off‑heap memory.

Verification

Monitoring graphs showed a short‑term spike in memory usage and high CPU load on the Dble host, while the free‑buffer metric demonstrated a gradual decline of off‑heap memory after startup, confirming that the leak exhausted the off‑heap pool and forced Dble to allocate on‑heap buffers for network packets.

Off‑Heap Memory Leak Analysis

BTrace was employed to trace allocation and release of off‑heap buffers. The following BTrace script records the addresses of allocated and recycled DirectByteBuffers:

package com.actiontech.dble.btrace.script;

import com.sun.btrace.BTraceUtils;
import com.sun.btrace.annotations.*;
import sun.nio.ch.DirectBuffer;
import java.nio.ByteBuffer;

@BTrace(unsafe = true)
public class BTraceDirectByteBuffer {
    @OnMethod(clazz = "com.actiontech.dble.buffer.DirectByteBufferPool", method = "recycle", location = @Location(Kind.RETURN))
    public static void recycle(@ProbeClassName String pcn, @ProbeMethodName String pmn, ByteBuffer buf) {
        String threadName = BTraceUtils.currentThread().getName();
        if (!threadName.contains("writeTo")) {
            String js = BTraceUtils.jstackStr(15);
            if (!js.contains("heartbeat") && !js.contains("XAAnalysisHandler")) {
                BTraceUtils.println(threadName);
                if (buf.isDirect()) {
                    BTraceUtils.println("r:" + ((DirectBuffer) buf).address());
                }
                BTraceUtils.println(js);
            }
        }
    }

    @OnMethod(clazz = "com.actiontech.dble.buffer.DirectByteBufferPool", method = "allocate", location = @Location(Kind.RETURN))
    public static void allocate(@Return ByteBuffer buf) {
        String threadName = BTraceUtils.currentThread().getName();
        if (!threadName.contains("writeTo")) {
            String js = BTraceUtils.jstackStr(15);
            if (!js.contains("heartbeat") && !js.contains("XAAnalysisHandler")) {
                BTraceUtils.println(threadName);
                if (buf.isDirect()) {
                    BTraceUtils.println("a:" + ((DirectBuffer) buf).address());
                }
                BTraceUtils.println(js);
            }
        }
    }
}

Running the script and filtering the logs:

$ btrace -o /path/to/log -u 11735 /path/to/BTraceDirectByteBuffer.java
$ grep '^a:' /tmp/142-20-dble-btrace.log > allocat.txt
$ sed 's/..//' allocat.txt > allocat_addr.txt
$ grep '^r:' /tmp/142-20-dble-btrace.log > release.txt
$ sed 's/..//' release.txt > release_addr.txt
$ sort allocat_addr.txt release_addr.txt | uniq -u > res.txt

The resulting res.txt lists addresses that were allocated but never released, pointing to the leak.

Code Review

Stack traces from the leaked addresses highlighted the allocation path:

com.actiontech.dble.buffer.DirectByteBufferPool.allocate(DirectByteBufferPool.java:82)
com.actiontech.dble.net.connection.AbstractConnection.allocate(AbstractConnection.java:395)
com.actiontech.dble.backend.mysql.nio.handler.query.impl.OutputHandler.
(OutputHandler.java:51)
...

Reviewing the relevant Dble source revealed that OutputHandler creates a DirectByteBuffer during construction:

public OutputHandler(long id, NonBlockingSession session) {
    super(id, session);
    session.setOutputHandler(this);
    this.lock = new ReentrantLock();
    this.packetId = (byte) session.getPacketId().get();
    this.isBinary = session.isPrepared();
    // allocate off‑heap memory
    this.buffer = session.getSource().allocate();
}

The allocation occurs even when the complex query execution chain is later abandoned (e.g., when routeSingleNode != null causes an early return), leaving the buffer unreleased.

Fix

The resolution is to defer buffer allocation until it is actually needed, removing the eager allocation in the OutputHandler constructor.

Conclusion

By combining log inspection, monitoring, BTrace instrumentation, and targeted code review, the off‑heap memory leak in Dble was traced to premature buffer allocation in OutputHandler , and the fix prevents unnecessary off‑heap consumption.