Investigation of Excessive Off‑Heap Memory Usage After Migrating a Spring Boot Project to the MDP Framework

After moving an internal project to the MDP framework (built on Spring Boot), the team observed frequent "Swap usage too high" alerts. Although the JVM was started with a 4 GB heap (‑Xmx4g ‑Xms4g and other GC flags), the process’s resident memory grew to about 7 GB, which was unexpected.

The JVM start‑up parameters were:

#include<sys/mman.h>
#include<stdlib.h>
#include<string.h>
#include<stdio.h>
// ... custom malloc / calloc / realloc / free implementation ...

Initial investigation used the -XX:NativeMemoryTracking=detail flag and the jcmd <pid> VM.native_memory detail command, revealing that the memory reported by jcmd (heap, code cache, unsafe allocations) was far smaller than the physical usage shown by top and pmap . This indicated that native code allocations were not accounted for.

Further analysis with pmap displayed many 64 MB anonymous mappings that were absent from the jcmd output, suggesting a large amount of off‑heap memory allocated by native libraries.

System‑level tools were then employed:

gperftools was enabled to monitor malloc activity. The graph showed malloc usage spiking to 3 GB before dropping to a steady 700‑800 MB, implying that most of the growth came from native allocations.

strace was run with -e brk,mmap,munmap to trace system calls. No suspicious allocation calls were observed during normal operation.

gdb was used to dump memory regions identified via /proc/<pid>/smaps . The dumped data resembled extracted JAR contents, pointing to the Spring Boot loader.

Deep diving into the Spring Boot loader revealed that the Meituan Unified Config Center (MCC) uses Reflections to scan all JAR packages. The scanning process invokes ZipInflaterInputStream , which internally creates a Java Inflater that allocates off‑heap memory via native malloc . The wrapper never explicitly frees this memory; it relies on the Inflater ’s finalize method, which is triggered only during garbage collection.

Because the GC only finalizes objects after a collection cycle, the off‑heap memory remains allocated for a long time. Moreover, glibc’s per‑thread memory arenas (64 MB each) retain the memory after it is freed, so the operating system still reports a high resident set size.

To confirm the hypothesis, the team replaced the default memory allocator with a custom one (shown in the code block above) that logs allocations. Tests showed that the application consistently used 700‑800 MB of off‑heap memory, while the OS reported a much larger footprint due to mmap’s page‑size rounding and lazy allocation.

Finally, the root cause was identified as MCC’s default configuration to scan every JAR. By configuring MCC to scan only specific packages, the off‑heap memory spike disappeared. The issue was also fixed in newer Spring Boot versions (2.0.5.RELEASE), where ZipInflaterInputStream now releases its native buffers explicitly.

In summary, the excessive memory consumption was not a JVM heap leak but a combination of aggressive package scanning, reliance on finalizers for native buffer release, and glibc’s memory‑pool behavior. Adjusting scan paths or upgrading Spring Boot eliminates the problem.