Idempotency in APIs: Building Operations That Are Safe to Retry

Networks fail. Connections drop. Load balancers time out. Servers restart mid-request. In a distributed system, any request that travels over a network can be sent but not confirmed, leaving the caller uncertain whether the operation completed. This is not a rare edge case — it is a routine condition that well-designed APIs must handle. Idempotency is the mechanism that makes retrying safe when certainty is unavailable.

What Idempotency Means

An operation is idempotent if performing it multiple times produces the same result as performing it once. The outcome is identical whether you call it one time or ten times; subsequent calls do not change anything that the first call already changed.

Some HTTP methods are idempotent by definition. GET, PUT, and DELETE are idempotent in the HTTP specification: reading a resource multiple times returns the same data (or its current state), updating a resource with the same PUT payload results in the same final state regardless of repetition, and deleting a resource that is already deleted is a no-op (or a 404). POST is not idempotent — posting the same order request twice creates two orders, not one.

The design challenge is that most consequential API operations are POST requests: creating a payment, placing an order, sending a notification, provisioning a resource. These are the operations where duplicate execution causes real problems, and they are the operations most vulnerable to “did that actually go through?” uncertainty.

The Idempotency Key Pattern

The standard solution for making non-idempotent operations safe to retry is the idempotency key. The client generates a unique key — typically a UUID — for each logical operation and includes it in the request as a header:

POST /payments HTTP/1.1
Idempotency-Key: f4e8b2d1-6a3c-4e7f-9d2b-1c5a8e3f7b9d
Content-Type: application/json

{
  "amount": 5000,
  "currency": "usd",
  "customer_id": "cus_abc123"
}

The server, upon receiving a request with an idempotency key it has seen before, does not execute the operation again. Instead, it returns the stored result from the first execution. The operation happened once; subsequent requests with the same key are answered from cache.

If the client’s request timed out before it received a response, it retries with the same idempotency key. If the server had already completed the operation, the retry receives the original result. If the server had not yet completed it — the timeout happened during processing — the retry either receives the completed result if it finished, or initiates a fresh attempt if the server did not receive the original request. Either way, the payment is charged exactly once.

The client is responsible for generating the key before sending the request and using the same key on all retries of the same logical operation. A new key must be generated for each distinct logical operation, even if the payload is identical. Two separate payment attempts for the same amount to the same customer are different operations; they should have different keys.

Server-Side Implementation

The server stores idempotency keys and their associated results, typically in a fast persistent store like Redis or a database with appropriate indexing. The storage includes the key itself, the response status code, the response body, and a timestamp for expiration (idempotency keys do not need to be stored forever — 24 hours to 30 days is typical).

The key lookup happens before the operation executes:

  1. Receive request with idempotency key
  2. Check whether this key exists in the store
  3. If yes: return the stored result, do not execute again
  4. If no: execute the operation, store the result under the key, return the result

The implementation must handle concurrent requests with the same key — two retries arriving simultaneously before either has completed and stored a result. The standard approach is to use a distributed lock or an atomic compare-and-set operation on the key in the store, ensuring only one execution proceeds while others wait for the stored result.

Scope the idempotency key to the specific endpoint and API consumer. A key from one account should not affect operations from another account. A key from the payments endpoint should not collide with a key from the orders endpoint.

What to Do with Conflicting Requests

A subtle case: what if two requests arrive with the same idempotency key but different payloads? This should not happen in a correct client implementation — the idempotency key identifies a specific logical operation, so the same key should always accompany the same payload. But it does happen due to bugs.

The correct response is to reject the conflicting request with a 409 (Conflict) or a 422 with an error code indicating the idempotency key is already associated with a different payload. Do not execute the second operation. Do not silently return the result of the first. Surface the conflict explicitly so the client can investigate.

Idempotency in Event-Driven Systems

The same principle applies to webhook processing. Webhooks can be delivered more than once — the provider’s retry logic may re-deliver an event if your endpoint’s 200 response did not arrive, even if the response was actually sent. Your webhook handler must check whether it has already processed an event (by event ID) before processing it.

Store processed event IDs. Before executing any action in a webhook handler, check whether the event ID has been seen. If yes, return 200 immediately without processing. If no, process and store the ID. This is the same pattern as server-side idempotency key handling, applied to the event consumer role.

Designing Operations to Be Idempotent

Some operations can be made intrinsically idempotent through careful schema design. A set_status operation that sets a resource’s status to a specific value is idempotent — calling it twice with the same status produces the same result. A toggle_status operation is not idempotent — calling it twice undoes the first call. Prefer declarative (set to X) over procedural (toggle, increment, append) operations where idempotency matters.

For operations that are inherently additive — sending a message, logging an event — idempotency keys are the only practical mechanism. Design these endpoints to accept and respect idempotency keys from the start; retrofitting idempotency into APIs that were built without it is significantly more difficult.

The operational reality is that distributed systems guarantee message delivery at-least-once, not exactly-once. Building APIs that handle duplicate delivery correctly is not defensive programming — it is accurate programming for the environment in which these systems actually run.

Related