Ensuring Idempotency in Distributed Systems: Unique ID Generation Strategies

In programming, idempotency means that executing an operation multiple times yields the same effect as executing it once; designing idempotent APIs is crucial for scenarios like user order placement where retries may occur due to timeouts.

To avoid duplicate orders, inventory deductions, or double charging, each order should carry a unique identifier. The downstream service can record the status of that identifier and reject repeated executions, while the upstream service should not be responsible for enforcing idempotency.

Unique IDs can be allocated either by a centralized ID‑generation service or by the upstream service itself, provided it guarantees no collisions. Centralized allocation adds an extra request, whereas client‑side generation can use methods such as UUIDs or Snowflake‑style algorithms.

UUID (Universally Unique Identifier) is a 128‑bit value with an extremely low collision probability, but it consumes more storage, is not human‑readable, and does not provide monotonic ordering.

Snowflake is a Twitter‑originated distributed ID algorithm that produces a 64‑bit long value composed of a 41‑bit timestamp (≈69.7 years), a 10‑bit machine identifier (up to 1024 nodes), and a 12‑bit sequence number (up to 4096 IDs per millisecond). Implementations exist in many languages, and similar concepts are used in Redis or MongoDB global ID generators.

When the idempotency service is distributed, the unique IDs must be stored in shared storage (e.g., MySQL or Redis) so that each instance remains stateless. Using MySQL, an insert into ... values ... on DUPLICATE KEY UPDATE ... statement can detect existing IDs; with Redis, SETEX can be used to atomically set a key only if it does not already exist.

Because most requests are not duplicates, it is inefficient to query the storage for every request. Instead, rely on the storage engine’s atomic insert‑or‑ignore semantics to determine whether a key already exists, reducing unnecessary read overhead.