idempotency
repeating a request must not repeat its effect.
what happens when it runs twice
an order with a payment processed twice charges the customer twice. that case has one "good" trait: someone notices. there is a customer complaining. the problem is expensive, but it is visible.
now take a button that toggles a user's access. the request goes out twice. the server enables it, and right after disables it. the system simply ended up in the wrong state, and it will stay that way until someone calls saying they cannot log in (in the best case, but in a critical system it will not be that simple).
the second case is worse than the first. a loud failure you fix on the spot; a silent one you only find out about when the problem is already too big.
and that is why idempotency is not a subject for fintechs only, it is a subject for any application.
it is not the fault of whoever clicked
the natural reaction is to treat this as sloppiness on the front end: disable the button, add a debounce, done. but the repetition may or may not come from there.
when the client sends a POST and the response times out, it is left without
knowing what happened. the server may not have received it. it may have received
it and failed. it may have processed the whole thing and the response got lost
on the way back. all three cases reach the client looking exactly the same: a
timeout.
given that, retrying is the natural decision, and the responsibility for withstanding the repetition belongs to the server. that is what idempotency solves.
and even if the user clicks only once, the browser may send several. look at this code, it is the most common way it happens:
function setupCheckout() {
// we are stacking this listener on the #pay button
document.querySelector('#pay').addEventListener('click', () => {
fetch('/api/payments', {
method: 'POST',
body: JSON.stringify(order),
});
});
}
setupCheckout();
cart.on('change', () => {
renderCart();
setupCheckout(); // the previous listener is still alive
});
every change to the cart registers one more listener on the same button, and
none of the previous ones were removed. after three changes, a single click
fires four POST. the user did everything right.
notice that the bug depends on addEventListener sitting inside something that
runs again: a re-render, a fetch that reloads the screen, a setup function
called more than once.
the definition
according to MDN Web Docs:
An HTTP method is idempotent if the intended effect on the server of making a single request is the same as the effect of making several identical requests.
the simple model
with a front end, an api and a database, nothing beyond that:
- the client generates a key and sends it in the header, something like
Idempotency-Key: 7c1f... - the api checks whether a response is already stored for that key
- if it is, it returns the same response, without processing again
- if it is not, it processes, stores key and response together, and returns
on the first time nobody finds anything and the work happens:
front api db
│ │ │
│ POST /payments │ │
│ Idempotency-Key: 7c1f │ │
├──────────────────────>│ │
│ │ select where key = 7c1f │
│ ├─────────────────────────>│
│ │ empty │
│ │<─────────────────────────┤
│ │ │
│ │ charges the card │
│ │ insert (7c1f, response) │
│ ├─────────────────────────>│
│ 201 created │ │
│<──────────────────────┤ │
on the second time the key is already there and nothing is processed:
front api db
│ │ │
│ POST /payments │ │
│ Idempotency-Key: 7c1f │ (the same as before) │
├──────────────────────>│ │
│ │ select where key = 7c1f │
│ ├─────────────────────────>│
│ │ stored response │
│ │<─────────────────────────┤
│ 201 created │ │
│<──────────────────────┤ no charge at all │
who generates the key? it has to be the client, because only it knows that the second attempt is the same intent and not a new request. the server has no way to tell whether the user wanted to pay again or whether the response got lost and they tried once more.
in other words, the key identifies the intent, not the bytes of the request. a new attempt at the same intent reuses the key. a new intent requires a new key.
the key, on the client side
the browser already generates this without any library:
const idempotencyKey = crypto.randomUUID();
crypto.randomUUID() returns a UUID v4, which is the format the industry
adopted. stripe, for instance, accepts any string up to 255 characters, but UUID
is the de facto standard because it has enough entropy never to collide and
carries no information at all about the order.
it is a good idea to prefix the key with context, because a bare UUID in a log says nothing:
user42-payments-9f3c1b7e2a4d...
the prefix is there to make our investigations easier, not for the system. when
the double charge shows up on a sunday night, user42-payments-... tells you
who and where in a single grep, and the bare UUID tells you that a key exists.
but a prefix is not a security boundary. user{id}-payments-... is predictable,
and in a model that returns a stored response by key, a guessed key becomes
someone else's response (you do not want that). what protects you is the server,
always scoping the lookup by whoever is authenticated:
select * from idempotency_keys
where user_id = :authenticated_user and key = :key
with that, user A's key never matches user B's record, even if they get the
whole string right. the user42- in the prefix goes back to being what it
should be: a help for us developers, or even for support.
what really matters is where the key is born. it belongs to the intent, so it is generated once, when the user decides to pay, and reused on every attempt:
const idempotencyKey = crypto.randomUUID();
async function pay(order, attempt = 0) {
const res = await fetch('/api/payments', {
method: 'POST',
headers: {
'Content-Type': 'application/json',
'Idempotency-Key': idempotencyKey, // the same on every attempt
},
body: JSON.stringify(order),
});
if (!res.ok && attempt < 3) return pay(order, attempt + 1);
return res.json();
}
the mistake that cancels all of it is generating the key inside the call:
const res = await fetch('/api/payments', {
method: 'POST',
headers: {
'Idempotency-Key': crypto.randomUUID(), // wrong
},
body: JSON.stringify(order),
});
here every attempt becomes a new intent in the server's eyes. the header is there, the backend implementation is correct, and the customer is still charged once per retry. it is the kind of bug that passes code review because the line looks right.
same key, different body
what if the same key arrives with a completely different payload?
it happens through a client bug, through a key generated once and reused by accident, or through someone probing your api. the way described above, the server would return the old response silently, and the second request would just vanish without any error.
the solution is to store a hash of the body alongside the key. same key and a different hash means it is not the same intent, and the request is rejected (or not, you can also just warn if that makes sense for your application).
it is computed from the body as it arrived, before anything parses it:
import { createHash } from 'node:crypto';
const bodyHash = createHash('sha256').update(rawBody).digest('hex');
note rawBody, not the parsed object. JSON.stringify is not canonical:
{"a":1,"b":2} and {"b":2,"a":1} are the same payload and produce different
strings, so hashing the parsed object would reject a client that merely ordered
its keys differently. hash the bytes that came off the wire.
with that in hand the check has four outcomes instead of two:
row = select ... where user_id = :user and key = :key
no row -> process, insert, return
row and hash matches -> replay row.response
row and hash does not match -> 422, this key is not yours to reuse
row match but hash does not match -> you decide
what this model does not cover
it has a window, and it shows up when you put the two side by side:
time request A request B
│
1 select 7c1f -> empty
│ ┐
2 │ select 7c1f -> empty
│ │
3 charges the card │ the window: A has not written yet,
│ │ so B does not find anything either
4 │ charges the card
│ │
5 insert (7c1f, ...) ┘
│
6 insert (7c1f, ...)
│
▼ two charges, and the key worked exactly as designed
the interval between checking and writing is where both slip through. writing faster does not help, and checking again before the insert does not help either: however small it gets, the window keeps existing, because these are two separate operations.
this model covers repetition separated in time, which is the timeout and retry
case. it does not cover simultaneous requests, which is the case of the stacked
listener firing four POST at the same instant.
solving that requires a second layer, and it is in idempotency middleware.