Exactly-Once Processing and the Idempotency Problem

Introduction

"We need exactly-once processing." It's one of the most common requirements in payment systems, and it's also, taken literally, impossible. You cannot guarantee that a message is delivered and processed exactly one time across an unreliable network. Anyone who tells you their system does is either wrong or quietly relying on the thing this post is actually about.

The good news: you don't need exactly-once delivery. You need exactly-once effect — the guarantee that no matter how many times a message arrives, the outcome on your system is as if it happened once. A user charged once even if the webhook fires five times. A transfer recorded once even if the client retries on a timeout. That property is achievable, and the tool that gets you there is idempotency.

I learned this the hard way building payment infrastructure, where a duplicated event isn't a cosmetic bug — it's real money credited twice. This post is about why exactly-once delivery is a myth, what you build instead, and the concrete patterns — idempotency keys, idempotency tables, stored responses — that make processing safely repeatable.

Why Exactly-Once Delivery Is Impossible

Consider the simplest possible interaction: a client sends a request, the server processes it, the server replies. Now introduce the one thing that always exists — an unreliable network — and look at where it can fail:

1. The request is lost before reaching the server.
2. The server processes it, but the response is lost on the way back.

From the client's perspective, cases 1 and 2 are indistinguishable: both look like "I got no response." So the client must retry. But in case 2 the work already happened — so the retry causes a second execution. The client cannot know, the server cannot tell the client what it doesn't ask again, and so duplicates are not an edge case. They are a fundamental consequence of distributed communication.

You have exactly two choices for delivery semantics:

• At-most-once: never retry. You'll lose messages. Unacceptable for money.
• At-least-once: always retry until acknowledged. You'll get duplicates.

There is no third option at the delivery layer. So you pick at-least-once and you handle the duplicates one layer up — in processing.

The Real Goal: At-Least-Once Delivery + Idempotent Processing

This is the formula worth memorising:

Exactly-once effect = at-least-once delivery + idempotent processing.

You let the network and your queues be aggressive about redelivery (so nothing is ever lost), and you make your processing logic immune to repetition (so nothing is ever double-counted). The combination behaves like exactly-once even though no single layer provides it. Every robust payment system, message queue consumer, and webhook handler is built on this.

So the engineering problem reduces to one question: how do I make an operation idempotent?

Idempotency Keys: The Foundation

An operation is idempotent if running it twice with the same input produces the same result as running it once. Some operations are naturally idempotent — SET balance = 100 gives the same result no matter how many times you run it. But most money operations are not: balance = balance + 100 doubles every time it repeats.

To make a non-idempotent operation idempotent, you attach an idempotency key: a unique identifier for this specific operation attempt, supplied by the caller and stable across retries.

POST /transfers
Idempotency-Key: 7f3c9a2e-...   ← same key on every retry of THIS transfer
{ "to": "acct_123", "amount": 50000 }

The contract: the first request with a given key does the work; every subsequent request with the same key returns the original result without redoing anything. The key is what lets the server recognise "I've seen this before."

The key must come from the client (or the upstream event), not be generated server-side — because the whole point is that a retry carries the same key. A server-generated key would be different each time and defeat the purpose. For inbound webhooks, the provider's event ID or transaction ID is your idempotency key.

The Idempotency Table Pattern

The mechanism is a dedicated table that records which keys you've already processed, with a uniqueness constraint doing the heavy lifting:

CREATE TABLE idempotency_keys (
  key          TEXT PRIMARY KEY,
  status       TEXT NOT NULL,          -- 'in_progress' | 'completed'
  response     JSONB,                  -- the stored result to replay
  created_at   TIMESTAMPTZ DEFAULT now()
);

The processing flow, all inside one database transaction:

async function processTransfer(key: string, input: TransferInput) {
  return db.$transaction(async (tx) => {
    // Try to claim the key. The UNIQUE constraint is the gatekeeper.
    const claimed = await tx.idempotencyKey
      .create({ data: { key, status: 'in_progress' } })
      .catch(() => null) // duplicate key → already exists

    if (!claimed) {
      const existing = await tx.idempotencyKey.findUnique({ where: { key } })
      if (existing!.status === 'completed') return existing!.response  // replay
      throw new ConflictError('Request already in progress')           // concurrent dup
    }

    // First time: do the real work in the SAME transaction
    const result = await executeTransfer(tx, input)

    await tx.idempotencyKey.update({
      where: { key },
      data: { status: 'completed', response: result },
    })
    return result
  })
}

The critical detail: claiming the key and doing the work happen in the same transaction. If the work fails, the transaction rolls back and the key claim disappears with it — so a retry can legitimately try again. If the work commits, the key is marked completed atomically with the effect, so a retry replays the stored response instead of re-executing. There's no window where the money moved but the key wasn't recorded, or vice versa.

Natural vs Synthetic Idempotency Keys

You don't always need a separate key — sometimes the domain gives you one for free:

• Natural key: the operation already has a unique business identifier. A bank webhook carries a transactionId; use it directly as the idempotency key. Inserting a ledger row with a UNIQUE(bank_transaction_id) constraint is your idempotency check — the second insert simply fails.
• Synthetic key: the caller has no natural unique ID, so they generate a UUID per logical operation and send it as an Idempotency-Key header (Stripe's model).

Prefer natural keys when they exist — they need no extra coordination and tie idempotency directly to the data. Fall back to synthetic keys for client-initiated actions like "create a payment" where the client must decide what counts as "the same attempt."

Storing the Response, Not Just the Key

A subtle but important refinement: it's not enough to record that a key was seen — you should store the original response and replay it. Why? Because the retrying caller still needs an answer, and that answer must be identical to the first one. If the first call created transfer tx_abc, the retry must also return tx_abc, not a fresh ID or a 409 the client doesn't expect. Storing the response (as in the table above) makes the retry transparent: the caller can't tell whether it hit the real execution or a replay.

Concurrency: Two Copies of the Same Request at Once

Retries aren't always sequential. A user double-clicks; a queue redelivers before the first delivery finished. Now two requests with the same key execute simultaneously. This is exactly why the in_progress status exists.

The UNIQUE constraint guarantees only one of the racers can insert the key — the database serialises them for you. The loser sees the key already exists with status in_progress and must back off (return a 409, or wait and retry the read). Without this, both copies could pass an application-level "have I seen this key?" check before either writes, and you'd double-process. Let the database's uniqueness constraint be the lock; never check-then-act in application code.

Pitfalls

• Generating the key server-side. It must be stable across retries, so it has to originate from the client or the source event.
• Recording the key in a different transaction than the effect. They must commit together or you get a gap where one exists without the other.
• Forgetting to store the response. The retry then can't be answered consistently.
• Unbounded key growth. Idempotency tables grow forever; expire old keys (e.g. after 24–72h, longer than any realistic retry window) with a TTL job.
• Assuming your queue gives exactly-once. Even "exactly-once" queues mean exactly-once into the queue; your consumer can still be redelivered. Stay idempotent.

Conclusion

Exactly-once delivery is a comforting fiction; exactly-once effect is an achievable engineering goal. You get there by accepting at-least-once delivery — let everything retry freely, lose nothing — and making your processing idempotent so repetition is harmless. The idempotency table, guarded by a uniqueness constraint and committed in the same transaction as the work, is the concrete machinery.

Once this clicks, your relationship with retries inverts. Instead of fearing duplicate events, you welcome aggressive redelivery, because you've made duplicates a no-op. That's the whole game in reliable money movement: assume everything happens twice, and make sure it only ever counts once. Idempotency is also the property that makes the transactional outbox pattern safe, and it underpins how virtual accounts route transfers without double-crediting.