Understanding Serialization, Deserialization Vulnerabilities and Mitigation in Java

Serialization (Serialization) is the process of converting an object's state into a format that can be stored or transmitted, allowing the object's state to be written to temporary or persistent storage and later reconstructed via deserialization.

In simple terms, serialization converts a data structure or object into a binary string, while deserialization converts that binary string back into the original data structure or object.

Example code that serializes a string to a local file and then deserializes it back:

public static void main(String args[]) throws Exception {
    String obj = "hello world!";
    // Write serialized object to file object.db
    FileOutputStream fos = new FileOutputStream("object.db");
    ObjectOutputStream os = new ObjectOutputStream(fos);
    os.writeObject(obj);
    os.close();
    // Read data from file object.db
    FileInputStream fis = new FileInputStream("object.db");
    ObjectInputStream ois = new ObjectInputStream(fis);
    // Restore object via deserialization
    String obj2 = (String) ois.readObject();
    ois.close();
}

Deserialization vulnerabilities arise when the deserialization mechanism breaks the boundary between data and objects, allowing an attacker to inject malicious serialized data that, when deserialized, creates objects under the attacker’s control and can execute arbitrary code. Java serialization is used in technologies such as RMI, JMX, and JMS.

Case 1 – Apache ActiveMQ 5.12.0 Remote Code Execution (JMS) – Older versions of ActiveMQ allowed unrestricted serialization of classes in JMS ObjectMessage objects, enabling remote attackers to execute arbitrary code.

Affected messaging systems include Apache ActiveMQ 5.12.0 and earlier, HornetQ 2.4.0 and earlier, Oracle OpenMQ 5.1 and earlier, IBM WebSphere MQ 8.0.0.4 and earlier, Oracle WebLogic 12c and earlier, and many other JMS clients.

Case 2 – JBoss Deserialization Vulnerability (JMX) – JBoss uses HTTP to expose JMXInvokerServlet, which relies on Java serialization. An attacker can craft a malicious serialized object and send it to the target, leading to command execution.

Case 3 – Jenkins Remoting (CVE‑2016‑0788) – Jenkins Remoting API lacks authentication and accepts serialized objects, allowing attackers to trigger JRMP, perform deserialization, and achieve remote code execution. A proof‑of‑concept creates a “pwned” file in the Jenkins root directory.

Mitigation Strategies

1. Implement a whitelist for deserialized classes by validating class names in the resolveClass method.

2. Encrypt serialized data during transmission and add authentication to interface calls (this raises the attack cost).

3. Mark sensitive fields as transient to exclude them from serialization when confidentiality is required.

Example of a custom ObjectInputStream that enforces a whitelist:

public class LookAheadObjectInputStream extends ObjectInputStream {
    public LookAheadObjectInputStream(InputStream inputStream) throws IOException {
        super(inputStream);
    }
    @Override
    protected Class
resolveClass(ObjectStreamClass desc) throws IOException, ClassNotFoundException {
        if (!desc.getName().equals(Bicycle.class.getName())) {
            throw new InvalidClassException("Unauthorized deserialization attempt", desc.getName());
        }
        return super.resolveClass(desc);
    }
}