Handling Duplicate Requests in Java Backend Using Redis and Parameter Deduplication

Duplicate requests can cause serious problems for write operations such as order placement, and they may arise from replay attacks, client-side resubmissions, gateway retries, or other reasons. This article focuses on server‑side strategies to uniformly handle such cases while ignoring client‑side click‑prevention techniques.

Using a Unique Request Identifier for De‑duplication

When each request carries a unique identifier, Redis can be used to store the identifier temporarily; if the key already exists, the request is considered a duplicate.

String KEY = "REQ12343456788"; // request unique ID
long expireTime = 1000; // 1000 ms expiration, duplicates within this window are rejected
long expireAt = System.currentTimeMillis() + expireTime;
String val = "expireAt@" + expireAt;
Boolean firstSet = stringRedisTemplate.execute((RedisCallback
) connection ->
    connection.set(KEY.getBytes(), val.getBytes(), Expiration.milliseconds(expireTime), RedisStringCommands.SetOption.SET_IF_ABSENT));
final boolean isConsiderDup;
if (firstSet != null && firstSet) {
    isConsiderDup = false; // first access
} else {
    isConsiderDup = true; // key already exists, treat as duplicate
}

Business Parameter De‑duplication

When a request does not include a unique ID, the request parameters themselves can be used as a fingerprint. A simple case concatenates user ID, method name, and a single parameter:

String KEY = "dedup:U=" + userId + "M=" + method + "P=" + reqParam;

For JSON payloads, the parameters are sorted by key, concatenated, and an MD5 hash is taken to keep the key short.

String KEY = "dedup:U=" + userId + "M=" + method + "P=" + reqParamMD5;

If the JSON contains time‑related fields that differ between otherwise identical requests, those fields should be excluded before hashing.

// Example requests differing only by requestTime
String req = "{\n" +
    "\"requestTime\" :\"20190101120001\",\n" +
    "\"requestValue\" :\"1000\",\n" +
    "\"requestKey\" :\"key\"\n}";
String req2 = "{\n" +
    "\"requestTime\" :\"20190101120002\",\n" +
    "\"requestValue\" :\"1000\",\n" +
    "\"requestKey\" :\"key\"\n}";

Request De‑duplication Helper Class (Java)

public class ReqDedupHelper {
    /**
     * Compute MD5 digest of request JSON after removing specified keys.
     * @param reqJSON request parameters (usually JSON)
     * @param excludeKeys fields to ignore before hashing
     * @return MD5 digest string
     */
    public String dedupParamMD5(final String reqJSON, String... excludeKeys) {
        String decryptParam = reqJSON;
        TreeMap paramTreeMap = JSON.parseObject(decryptParam, TreeMap.class);
        if (excludeKeys != null) {
            List
dedupExcludeKeys = Arrays.asList(excludeKeys);
            if (!dedupExcludeKeys.isEmpty()) {
                for (String dedupExcludeKey : dedupExcludeKeys) {
                    paramTreeMap.remove(dedupExcludeKey);
                }
            }
        }
        String paramTreeMapJSON = JSON.toJSONString(paramTreeMap);
        String md5deDupParam = jdkMD5(paramTreeMapJSON);
        log.debug("md5deDupParam = {}, excludeKeys = {} {}", md5deDupParam, Arrays.deepToString(excludeKeys), paramTreeMapJSON);
        return md5deDupParam;
    }

    private static String jdkMD5(String src) {
        String res = null;
        try {
            MessageDigest messageDigest = MessageDigest.getInstance("MD5");
            byte[] mdBytes = messageDigest.digest(src.getBytes());
            res = DatatypeConverter.printHexBinary(mdBytes);
        } catch (Exception e) {
            log.error("", e);
        }
        return res;
    }
}

Test logs demonstrate that when the time field is excluded, two otherwise identical requests produce the same MD5, confirming the de‑duplication logic.

req1MD5 = 9E054D36439EBDD0604C5E65EB5C8267 , req2MD5 = A2D20BAC78551C4CA09BEF97FE468A3F
req1MD5 = C2A36FED15128E9E878583CAAAFEFDE9 , req2MD5 = C2A36FED15128E9E878583CAAAFEFDE9

Complete Solution Summary

Combine the parameter MD5 generation (optionally excluding volatile fields) with a Redis SETNX‑plus‑expiration operation to achieve atomic, time‑windowed de‑duplication.

String userId = "12345678"; // user
String method = "pay"; // API name
String dedupMD5 = new ReqDedupHelper().dedupParamMD5(req, "requestTime");
String KEY = "dedup:U=" + userId + "M=" + method + "P=" + dedupMD5;
long expireTime = 1000; // 1 second window
long expireAt = System.currentTimeMillis() + expireTime;
String val = "expireAt@" + expireAt;
Boolean firstSet = stringRedisTemplate.execute((RedisCallback
) connection ->
    connection.set(KEY.getBytes(), val.getBytes(), Expiration.milliseconds(expireTime), RedisStringCommands.SetOption.SET_IF_ABSENT));
final boolean isConsiderDup = (firstSet != null && firstSet) ? false : true;

By using this approach, duplicate write requests within the configured time window are reliably identified and rejected, improving system robustness.