Remote Hot Deployment (Mark42/Jarvis): Design, Implementation, and Practice

Background

The remote hot‑deployment component, codenamed Mark42 or Jarvis, is inspired by Meituan Sonic and adapted to Zhaozhuan's business scenarios. It consists of a Java Agent and an IDEA plugin. The full process covers a wide range of knowledge and is split into several topics for sharing.

In a typical front‑backend integration scenario, a minor code change required recompilation, Docker image rebuild, and deployment, causing both front‑end and back‑end developers to waste over ten minutes, dramatically reducing collaboration efficiency. The goal is to achieve a "magic" hot‑update similar to front‑end hot‑reloading for back‑end services.

Project Background

Developers often write code once, then repeatedly compile, package into a Docker container, and deploy for testing. Complex scenarios require remote debugging, code modification, and redeployment, leading to many compile‑deploy cycles via the internal Beetle platform.

The existing workflow is long and inefficient; the aim is to simplify it, reduce compile‑deploy cycles, and make code changes take effect quickly.

Expected Goals

Minimize the number of code submissions, compilations, and deployments in daily development, saving fragmented waiting time and allowing developers to focus on implementation, thereby indirectly improving development efficiency.

Problem Analysis

Hot deployment means updating Java class files at runtime by reloading bytecode in the JVM, reinitializing Spring containers and third‑party frameworks without downtime.

Two ways to obtain the new bytecode:

Compile Java source locally and push the generated .class files to the remote server.

Push the source code to the remote server and let the server compile it.

Solution 1 is low‑cost but suffers from IDE‑Maven compatibility issues, long compile time, and potential JDK version mismatches.

Solution 2 is more complex but preferable: the remote server performs dynamic compilation, keeping the process transparent to the developer.

Key challenges for Solution 2 include:

Spring‑Boot fat‑jar structure prevents classpath reconstruction for dynamic compilation.

Lombok and MapStruct dependencies are lost during runtime compilation.

ClassLoader handling varies across service types (SpringBoot, SCF, ZZJava).

Solution Selection

Option 1 (Dockerfile‑based extraction) is simple but not supported by the current architecture and cannot solve Lombok/MapStruct issues.

Option 2 focuses on solving dynamic compilation for fat‑jar services by reusing Spring‑Boot's URL‑handler mechanism to obtain resources from nested jars.

Spring‑Boot creates a custom URL handler that converts nested jars into URLs, enabling URLClassLoader to load classes. The implementation extracts the URL, opens a JarURLConnection , and retrieves JavaFileObject instances for compilation.

public URL getUrl() throws MalformedURLException { ... }

protected Class
findClass(final String name) throws ClassNotFoundException { ... }

Exploration and Practice

When the Agent starts, it enhances the Spring framework via bytecode manipulation, captures the service's ClassLoader (e.g., LaunchedURLClassLoader for SpringBoot), and stores it globally. During runtime compilation, the Agent uses this ClassLoader to resolve resources and create JavaFileObject instances.

Missing dependencies such as Lombok or MapStruct are manually added to the URL list.

public DynamicCompiler(ClassLoader userClassLoader) { ... }

Dynamic compilation then produces bytecode:

public Map
buildGetByteCodes() { ... }

Special handling is required for MapStruct, which generates implementation classes; the compiler must treat JavaFileObject.Kind.SOURCE correctly to avoid loading source code instead of bytecode.

Adding the -g option is essential to generate full debug information; otherwise, method variable names and line numbers are lost, causing debugging difficulties.

After these steps, arbitrary Java source can be compiled on the fly, producing class files and completing remote hot deployment preparation.

Summary and Outlook

The original purpose of remote hot deployment is not to replace the Beetle release process but to reduce compile‑deploy cycles and save developers' fragmented time, aiming for "one deployment, arbitrary modification".

Current supported features include remote hot deployment, dynamic compilation, remote debugging, log collection, batch deployment, IDEA plugin integration, and extensive framework support (FastJson, Jackson, JDK proxies, Spring, MyBatis, MapStruct, XXL‑JOB, etc.).

Future work includes supporting configuration‑bean injection, XML‑based bean configuration, SCF‑level Agent calls, remote unit‑test support, remote decompilation, and replacing the Spring loader with DCEVM.

Tags

Java, Hot Deployment, Dynamic Compilation, Spring Boot, Java Agent, Backend Development