Resolving Metaspace and Off‑Heap Memory Issues After JDK 11 Upgrade

In April, the Tmall International technical team began upgrading backend services to JDK 11. While non‑core services upgraded smoothly, two typical problems appeared in core applications.

Problem 1 : After a pre‑release machine reboot, the service became unresponsive and CPU usage spiked. Initial suspicion of a memory leak was disproved; the top command showed only GC threads consuming resources. Further investigation revealed a 5% increase in Metaspace utilization after the JDK upgrade.

Metaspace replaces PermGen in Java 8+. Its size is not fixed and grows with the number of loaded classes. Possible reasons for the increase include:

Frequent class loading without unloading.

JVM parameters such as -XX:MetaspaceSize and -XX:MaxMetaspaceSize .

New libraries or features (reflection, dynamic proxies) introduced by JDK 11.

Additional monitoring or debugging tools.

Solution: increase Metaspace to 768 MB, which reduced utilization from 95% to 80% and stopped the continuous Full GC loop.

Problem 2 : After the JDK 11 release, off‑heap memory grew by ~500 MB, eventually triggering memory‑usage alerts.

Investigation showed that Netty’s native direct‑buffer allocation was disabled due to module‑access restrictions introduced in JDK 9+. Netty fell back to java.nio allocation, which does not reuse memory efficiently, leading to growth.

Relevant Netty code (simplified):

if (maxDirectMemory == 0 || !hasUnsafe() || !PlatformDependent0.hasDirectBufferNoCleanerConstructor()) {
    USE_DIRECT_BUFFER_NO_CLEANER = false;
} else {
    USE_DIRECT_BUFFER_NO_CLEANER = true;
    if (maxDirectMemory < 0) {
        maxDirectMemory = MAX_DIRECT_MEMORY;
        DIRECT_MEMORY_COUNTER = (maxDirectMemory <= 0) ? null : new AtomicLong();
    } else {
        DIRECT_MEMORY_COUNTER = new AtomicLong();
    }
}

Solution: add JVM options to restore Netty’s native memory management and open required modules:

# Enable Netty reflection access
SERVICE_OPTS="${SERVICE_OPTS} -Dio.netty.tryReflectionSetAccessible=true"
# Open Unsafe
SERVICE_OPTS="${SERVICE_OPTS} --add-opens=java.base/jdk.internal.misc=ALL-UNNAMED"
# Open java.nio package
SERVICE_OPTS="${SERVICE_OPTS} --add-opens=java.base/java.nio=ALL-UNNAMED"

After rebuilding the image with these options, off‑heap memory usage returned to normal and the alert disappeared.

Conclusion: Upgrading JDK versions can introduce hidden memory‑management issues. Proper JVM tuning and understanding of underlying libraries (e.g., Netty) are essential for a stable migration.