Diagnosing Metaspace OOM in Java Applications: A Step‑by‑Step Analysis

During development, deploying code to the test environment repeatedly caused container restarts due to OOM, while the same deployment to prod ran without problems. The suspicion fell on a recent change of the hot‑deployment base image used only in the test environment.

Viewing JVM Memory Usage

We used Arthas ’s dashboard command and the JDK’s jstat tool to observe memory consumption. The Memory panel of the Arthas dashboard showed a continuously rising metaspace usage that eventually exceeded the value set by -XX:MaxMetaspaceSize , triggering a Metaspace OOM.

Further inspection with jstat confirmed that the last GC was caused by Metaspace memory exceeding the GC threshold, and the Metaspace usage stayed above 90%.

Analyzing the Metaspace OOM Cause

Since JDK 8, metaspace backs the method area and stores class metadata, constants, static variables, and JIT‑compiled code. It uses native memory and has no intrinsic size limit, but can be capped with -XX:MaxMetaspaceSize , which we set to 2 GB.

We compared class loading between test and prod using jstat . The test environment loaded many classes and almost never unloaded them, while prod both loaded and unloaded a large number of classes.

Test environment

Prod environment

The key difference was that the hot‑deployment agent in test held strong references to custom class loaders, preventing their garbage collection.

Quick Remedy

The project used the Aviator expression engine. The original method was:

public static synchronized Object process(Map
eleMap, String expression) {
    AviatorEvaluatorInstance instance = AviatorEvaluator.newInstance();
    Expression compiledExp = instance.compile(expression, true);
    return compiledExp.execute(eleMap);
}

Because each call created a new AviatorEvaluatorInstance (and thus a new AviatorClassLoader ) and also used synchronized , many class‑loader instances accumulated, leading to Metaspace growth.

We removed the unnecessary synchronization and leveraged the thread‑safe cached execution method:

// Delete synchronized
public static Object process(Map
eleMap, String expression) {
    AviatorEvaluator.execute(expression, eleMap, true); // true enables caching
}

Code Analysis

Further digging revealed that each AviatorEvaluatorInstance creates its own AviatorClassLoader , which generates many Script_* classes via ASM. The hot‑deployment agent kept strong references to these class loaders, causing the Metaspace OOM.

Relevant snippet from Aviator’s source:

public Object execute(final String expression, final Map
env, final boolean cached) {
    Expression compiledExpression = compile(expression, expression, cached);
    return compiledExpression.execute(env);
}

private Expression compile(final String cacheKey, final String exp, final String source, final boolean cached) {
    return innerCompile(expression, sourceFile, cached);
}

private Expression innerCompile(final String expression, final String sourceFile, final boolean cached) {
    ExpressionLexer lexer = new ExpressionLexer(this, expression);
    CodeGenerator codeGenerator = newCodeGenerator(sourceFile, cached); // creates new AviatorClassLoader
    return new ExpressionParser(this, lexer, codeGenerator).parse();
}

Because the hot‑deployment agent held strong references to each AviatorClassLoader , the generated Script_* classes could not be unloaded, inflating Metaspace.

Summary

The investigation linked the Metaspace OOM to three main factors: excessive creation of custom class loaders by the Aviator engine, the hot‑deployment agent’s strong references to those loaders, and the lack of class unloading in the test environment. The fix involved using a singleton or cached Aviator evaluator, removing unnecessary synchronization, and planning to replace strong references with weak references in the hot‑deployment agent.

Key JVM tools used during the analysis:

jps – JVM Process Status Tool

jstat – JVM Statistics Monitoring Tool

jinfo – JVM Configuration Info

jmap – Memory Map for Java

jhat – JVM Heap Analysis Tool

jstack – Stack Trace for Java

jcmd – Multi‑purpose diagnostic command‑line tool