Mastering Idempotent Design: 8 Proven Strategies for Reliable APIs

Preface

Hello, I am Sanyou. Today we will discuss idempotent design.

What is idempotence

Why idempotence is needed

How to handle interface timeout

How to design idempotence

Eight solutions to achieve idempotence

HTTP idempotence

1. What is Idempotence?

Idempotence is a concept in mathematics and computer science.

In mathematics, an idempotent function satisfies f(x) = f(f(x)) . For example, the absolute‑value function is idempotent: abs(x) = abs(abs(x)) .

In computer science, idempotence means that one or multiple requests to a resource should have the same side effect; the impact of multiple requests is identical to a single request.

2. Why Idempotence is Needed

When implementing a transfer function, if the downstream interface times out, should we retry? If we retry, could we transfer money twice?

In modern distributed systems, remote calls can be successful, failed, or timed out. Timeout is an unknown state. If the downstream system implements idempotence, we can safely retry without risking duplicate transfers.

Other common scenarios requiring idempotence include:

Message queue consumers may receive duplicate messages (duplicate consumption).

Rapid repeated clicks on a form submit button may create duplicate records (frontend duplicate submission).

3. How to Handle Interface Timeout?

If a downstream call times out, there are two solutions:

Solution 1: The downstream system provides a query interface. After a timeout, query the record; if it succeeded, proceed with success flow, otherwise treat as failure.

In the transfer example, the channel system queries the transfer record after a timeout to determine success or failure before deciding whether to retry.

Solution 2: The downstream interface itself supports idempotence, so the upstream system can simply retry after a timeout.

For MQ duplicate‑consumption scenarios, Solution 1 is not ideal, so we prefer downstream interfaces that support idempotence.

4. How to Design Idempotence

Idempotence requires a globally unique identifier for each request.

If you use a unique index, the index must be unique.

If you use a primary key, the key must be unique.

If you use a pessimistic lock, the underlying marker is still a globally unique ID.

4.1 Global Unique ID

Common ways to generate a global unique ID:

UUID – large string, random, not ordered.

Snowflake algorithm – 64‑bit IDs generated by Twitter.

Snowflake generates Snowflake IDs , which consist of:

1 bit sign (always 0 for positive IDs)

41 bits timestamp (milliseconds since a custom epoch)

10 bits machine identifier

12 bits sequence number per millisecond

Other generators include Baidu's Uidgenerator and Meituan's Leaf .

4.2 Basic Idempotent Flow

The process is to store each request with its global unique ID, check if a record exists, and if so return the previous result; otherwise process the request.

5. Eight Implementation Schemes

Below are eight common ways to achieve idempotence.

5.1 select + insert + primary/unique key conflict

When a transaction request arrives, first select the business sequence bizSeq from the flow table:

If the record exists, treat it as a duplicate request and return success.

If not, insert the record; if the insert succeeds, return success; if a primary‑key conflict occurs, catch the exception and also return success.

Flow diagram:

Pseudo‑code:

<code>/**
 * Idempotent handling
 */
Rsp idempotent(Request req){
  Object requestRecord = selectByBizSeq(bizSeq);
  if(requestRecord != null){
    log.info("Duplicate request, return success, bizSeq: {}", bizSeq);
    return rsp;
  }
  try{
    insert(req);
  }catch(DuplicateKeyException e){
    log.info("Primary key conflict, duplicate request, return success, bizSeq: {}", bizSeq);
    return rsp;
  }
  // normal processing
  dealRequest(req);
  return rsp;
}
</code>

5.2 Direct insert + primary/unique key conflict

If the probability of duplicate requests is low, we can directly insert and rely on primary‑key or unique‑key conflict to detect duplicates.

Flow diagram:

Pseudo‑code:

<code>/**
 * Idempotent handling
 */
Rsp idempotent(Request req){
  try{
    insert(req);
  }catch(DuplicateKeyException e){
    log.info("Primary key conflict, duplicate request, return success, bizSeq: {}", bizSeq);
    return rsp;
  }
  // normal processing
  dealRequest(req);
  return rsp;
}
</code>

Note: Anti‑repeat (dedup) and idempotence are different. Anti‑repeat only blocks duplicate data, while idempotence also guarantees identical effects for repeated requests.

5.3 State‑machine Idempotence

Business tables often have status fields (e.g., 0‑pending, 1‑processing, 2‑success, 3‑failure). Updating the status can be made idempotent by using a conditional update.

SQL example:

<code>update transfr_flow set status=2 where biz_seq='666' and status=1;
</code>

Flow diagram:

Pseudo‑code:

<code>Rsp idempotentTransfer(Request req){
  String bizSeq = req.getBizSeq();
  int rows = "update transfr_flow set status=2 where biz_seq=#{bizSeq} and status=1;";
  if(rows == 1){
    log.info("Update success, process request");
    return rsp;
  } else if(rows == 0){
    log.info("Update not successful, ignore request");
    return rsp;
  }
  log.warn("Data anomaly");
  return rsp;
}
</code>

First request updates status from 1 to 2, rows = 1, business logic runs.

Second request finds status already 2, rows = 0, no further processing.

5.4 Extract Dedup Table

Instead of relying on the business flow table’s unique bizSeq , create a separate deduplication table with a unique primary key. Inserting into this table will either succeed (process request) or conflict (treat as duplicate).

5.5 Token Scheme

Two‑phase token approach:

Client requests a token; server generates a globally unique token and stores it in Redis with an expiration.

Client includes the token in the actual request; server checks Redis, deletes the token atomically, and proceeds if deletion succeeds.

Flow diagram:

5.6 Pessimistic Lock (select for update)

Pessimistic lock acquires a row lock during a transaction, ensuring only one thread can modify the row at a time.

Example: Update an order only if its status is “processing”.

<code>begin;
select * from order where order_id='666' for update;
if(status != 'processing') return;
-- business logic
update order set status='completed' where order_id='666';
commit;
</code>

Notes:

The lock column must be indexed or a primary key; otherwise a table lock occurs.

Pessimistic locks can degrade performance under high contention.

5.7 Optimistic Lock

Add a version column to the table. Each update checks the current version and increments it atomically.

Process: read version, attempt update with where version = oldVersion . If the update affects zero rows, treat as duplicate.

<code>select order_id, version from order where order_id='666';
update order set version=version+1, status='P' where order_id='666' and version=1;
</code>

Flow diagram:

Version should be monotonically increasing to avoid ABA problems.

5.8 Distributed Lock

Use a distributed lock (e.g., Redis SET EX PX NX) with the business unique identifier as the key. Acquire the lock before processing; if acquisition fails, discard the request.

Redis is lightweight and commonly used; ZooKeeper is also possible.

Set an appropriate expiration time—long enough to cover processing, short enough to avoid stale locks.

6. HTTP Idempotence

HTTP methods and their idempotence:

GET – safe and idempotent (like a database SELECT).

HEAD – idempotent, returns only headers.

OPTIONS – idempotent, queries supported methods.

DELETE – idempotent; deleting the same resource repeatedly has the same effect.

POST – not idempotent; each request creates a new resource.

PUT – idempotent; creating or updating the same URI multiple times yields the same result.

6.1 GET

GET retrieves resources without side effects, thus idempotent.

6.2 HEAD

HEAD is similar to GET but returns only headers; it is also idempotent.

6.3 OPTIONS

OPTIONS queries supported methods and is idempotent.

6.4 DELETE

DELETE removes a resource; repeated deletions have the same effect, making it idempotent.

6.5 POST

POST creates resources; repeated POSTs create multiple resources, so it is not idempotent.

6.6 PUT

PUT creates or updates a resource at a known URI; repeated PUTs produce the same outcome, thus idempotent.

References

Elastic Design Chapter – “Idempotent Design” https://time.geekbang.org/column/article/4050