Design a payment system

What are we actually building?

The analogy: paying online is like mailing a check that might get delivered twice. You can't stop the post office from duplicating the envelope — so instead you write a unique serial number on every check, and the bank cashes each serial number exactly once, no matter how many copies arrive. The idempotency key is the serial number.

Capacity math

The line to say out loud: "The dedupe store is 12 GB — the cheapest insurance in the architecture. The ledger is append-only because money movement is a fact, and facts don't get updated."

The system, end to end

flowchart TB
    CL["Client app"]
    GW["API gateway"]
    IDF["Idempotency filter
Redis: key → stored response"]
    PS["Payment service"]
    PSP["Card processor
(Stripe / Adyen)"]
    LED[("Ledger
Postgres, append-only")]
    WH["Webhook handler"]
    REC["Reconciler
daily PSP settlement check"]

    CL --> GW
    GW --> IDF
    IDF -->|"key unseen"| PS
    IDF -->|"key seen"| CL
    PS --> PSP
    PS --> LED
    PSP -. "async events" .-> WH
    WH --> LED
    REC --> PSP
    REC --> LED

The idempotency filter sits in front of the payment service — a duplicate never reaches the charging logic at all. And note the belt-and-suspenders: a UNIQUE constraint on the key column in Postgres means even if Redis loses a key, the database refuses the second charge.

First charge vs retry

sequenceDiagram
    autonumber
    participant C as Client
    participant F as Idempotency filter
    participant P as Payment service
    participant S as Card processor
    participant L as Ledger
    C->>F: POST /charges {key: k1, $19.99}
    F->>F: k1 unseen
    F->>P: forward
    P->>S: authorize $19.99
    S-->>P: approved (ch_8f2k)
    P->>L: append charge + key k1
    P->>F: store k1 → receipt
    F-->>C: 200 charged (ch_8f2k)
    Note over C,S: network retry — same key k1
    C->>F: POST /charges {key: k1, $19.99}
    F->>F: k1 seen → return stored receipt
    F-->>C: 200 charged (ch_8f2k) — no new charge

The key lifecycle

flowchart TB
    A["Client creates payment intent
generates UUID key"] --> B["Send request with
Idempotency-Key header"]
    B --> C{"key in store?"}
    C -->|"yes"| D["Return stored response
200, no side effects"]
    C -->|"no"| E["Reserve key: processing"]
    E --> F["Charge via processor"]
    F --> G["Append to ledger"]
    G --> H["Store key → full response
TTL 72h"]
    H --> I["Return 200 to client"]
    F -->|"timeout, unknown outcome"| J["Reconcile: query processor
by key before deciding"]
    J --> G

The contract

POST /v1/charges
Idempotency-Key: 7f3a9c2e-…          # required, client-generated
{"amount": 1999, "currency": "usd", "payment_method": "pm_123"}
→ 200 OK {"charge_id": "ch_8f2k", "status": "succeeded", "amount": 1999}

# same key again → identical 200, no new charge
POST /v1/charges/{id}/refunds {"amount": 1999}
GET  /v1/charges/ch_8f2k

Data model: Charge {id, idempotency_key UNIQUE, amount, currency, status, processor_ref, created_at} — the unique constraint is the last line of defense. IdempotencyRecord {key, response_body, status_code, expires_at} in Redis for the fast path. LedgerEntry {id, charge_id, type: charge/refund, amount}, append-only, immutable.

What you give up, on purpose

What breaks, and what saves you

• Processor timeout, unknown outcome → never guess; reconcile by idempotency key before deciding. This is the one that actually causes double-charges in the wild.
• Redis evicts a key early → the Postgres UNIQUE constraint rejects the duplicate write. Defense in depth.
• Client reuses one key for two different carts (client bug) → second request returns the first cart's receipt. Mitigate: keys should embed the intent (cart id); document it loudly.
• Double-click before first response → both requests carry the same key (generated at page load); the "processing" reservation makes the second wait for the first.
• Webhook delivered twice → webhook handler is idempotent too — event IDs deduped the same way.

Opinionated, concrete, shippable

API: Node or Go service, Idempotency-Key header required on all mutating endpoints. Dedupe: Redis with 72h TTL in front, Postgres UNIQUE(idempotency_key) as the backstop. Money: Stripe as the processor — don't build card handling. Ledger: append-only Postgres table; daily reconciliation job against Stripe. Webhooks: outbox pattern, idempotent handlers. Two engineers, one month, and the demo below is the core loop.

How to run the room

1. Say "exactly-once effects, not exactly-once delivery" early. It shows you know the theory and the practice.
2. Draw the idempotency filter before the payment service in your diagram — placement is the insight.
3. Name the DB unique constraint as defense in depth. Interviewers light up at backstops.
4. Have the processor-timeout story ready — it's the hardest question and the most realistic failure.
5. State who generates the key and why (the client owns the intent). It's a classic follow-up.
6. End with the never-event framing: "double-charge isn't a metric we monitor, it's a property we guarantee."

Idempotency demo — double-click all you want

A real idempotency filter running in this page. Pay creates a payment intent with a fresh key. Retry resends the same key — like a network retry or a double-click. Triple-submit fires three concurrent requests with one key. Watch the request log: retries return the stored receipt, and the ledger below records exactly one charge per key.

The server logic is 6 lines: if (store.has(key)) return store.get(key) — check the key before touching money. Everything else in this article is making that check fast, durable, and impossible to skip.