How to Automate Batch Job Retries and Eliminate Midnight Outages

Story Origin

Xiaoming, an operations support manager at a large company, receives batch interruption alerts at 3 am, repeatedly encountering familiar database exceptions that force him to manually restart batch jobs, leading to frustration and a desire for change.

In‑Depth Analysis

Developers aim to raise automation levels so that when a batch interruption occurs, the system can automatically restart the job. Not all interruptions are suitable for automatic retry; for example, code bugs that cause duplicate entries must not be retried. Only transient issues such as environment jitter are appropriate for automatic restart.

Batch jobs rely on external environments and resources such as the batch execution framework, database, file server, and distributed messaging. The following table lists possible exceptions and mitigation measures:

MySQL error codes that can be safely retried are illustrated below:

An example of a CommunicationsException stack trace that may trigger a retry:

<code>com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failure
The last packet successfully received from the server was 3,008 milliseconds ago.
The last packet sent successfully to the server was 3,006 milliseconds ago.
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
    at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
    at com.mysql.jdbc.Util.handleNewInstance(Util.java:425)
    at com.mysql.jdbc.SQLError.createCommunicationsException(SQLError.java:989)
    at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:3556)
    ... 8 more
Caused by: java.net.SocketTimeoutException: Read timed out
    at java.net.SocketInputStream.socketRead0(Native Method)
    at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
    at java.net.SocketInputStream.read(SocketInputStream.java:171)
    ...</code>

Solution

To implement automatic restart, both the batch controller and executor need modifications. The controller must support a new "retry" status, track the number of retries, and enforce a maximum retry count. For transient environment issues, the controller launches a background task that periodically scans jobs in "awaiting retry" state and re‑issues start commands.

The executor, built with Spring, runs each batch job as a Java class implementing a common interface. Using Spring AOP, a post‑process interceptor examines exceptions; if the exception is deemed retryable, it logs the error and returns a retry status to the framework without altering the business code.

Result

After deploying the redesign, batch job exceptions are automatically detected and retried, dramatically reducing manual midnight interventions, improving system stability, and freeing the support manager to focus on higher‑value work.