JD's JDK Customization and Optimization for HDFS: Experience, Challenges, and Future Directions

The article provides an overview of JD's efforts in the JDK domain, focusing on the background, basic features, and future work directions of the JD JDK project, with technical discussions on JDK features to be detailed in subsequent articles.

HDFS is the distributed storage layer of Hadoop, designed for high fault tolerance on low‑cost hardware, using a master/slave architecture with NameNode and DataNode components.

Since HDFS runs on the JVM, optimizing JVM behavior for NameNode (long‑lived, memory‑intensive) and DataNode (short‑lived) processes is a key research direction for JD.

JD initially used Oracle JDK 1.8 for HDFS, but faced performance optimization limits, uncontrollable GC pauses, low memory utilization, and the upcoming end‑of‑life of JDK 1.8.

OpenJDK 11 was then evaluated; its default G1GC offers smaller memory fragmentation, controllable GC pause times, richer profiling tools (JFR), and higher load capacity for HDFS NameNode.

However, OpenJDK 11 still lacks optimizations for large heaps, efficient JMap scanning, and HDFS‑specific customizations.

JD therefore built a customized JDK on top of OpenJDK 11, adding features such as a re‑implemented javah tool, enlarged G1 region size for large heaps, dynamic heap expansion, scheduled periodic GC, timely return of uncommitted memory (JEP 346), abortable mixed G1 GC (JEP 344), default Class Data Sharing archives, and a parallel, incremental JMap implementation.

Extensive testing showed that the customized JDK reduced TLAB fill frequency by 33%, improved NameNode performance by about 50%, increased file‑count capacity from 400 million to 1 billion, and achieved shorter GC pause times compared to the previous CMS collector.

Further optimizations addressed lock performance, bias‑lock revocation, and various HDFS operations (mkdir, delete, getFileInfo, rename), with detailed benchmark images.

Future directions include dedicated heap regions for specific scenarios, semi‑automatic GC, new GC algorithms tailored for big‑data workloads, Graal‑based AOT support, and active contributions to the OpenJDK community.