Idempotency

Source: src/lib/api/idempotency.ts (store + lock) and src/lib/api/send-helpers.ts (prepareIdempotency / storeResponseIfIdempotent, the choke point every route goes through).

Header

Idempotency-Key: <any unique string — usually uuid>

Accepted on:

• POST /api/v1/emails
• POST /api/v1/sms
• POST /api/v1/emails/batch
• POST /api/v1/sms/batch
• POST /api/v1/phone-numbers
• POST /api/v1/brands
• POST /api/v1/campaigns
• POST /api/v1/verifications
• POST /api/v1/audiences/{id}/send (and its dashboard twin, POST /api/internal/audiences/{id}/send)

Both SDKs mint a key automatically on exactly these endpoints and reuse it across their retries. They never resend any other POST, PATCH or DELETE through a timeout, network error, 408 or 5xx — only through a 429, which the rate limiter answers before anything runs.

Audience sends

A pollable job_ id was once thought to be enough here, and it is not: a caller whose request times out never sees the id, so its retry scheduled the whole list again — up to 10,000 rows, and a 10k-row blast can run for minutes against a client's 30s timeout. Now the key binds to the job:

• A retry under the same key replays the recorded response, job_id included. Nothing is scheduled again.
• The key is bound to the audience as well as the body. The audience id is in the path, not the body, so the same key and body sent to a different audience is 409 IDEMPOTENCY_MISMATCH — not a replay of the first audience's job, which would answer 200 for a blast that never ran.
• A retry while the first request is still inserting gets 409 IDEMPOTENCY_IN_FLIGHT. That window is the whole blast, not a few hundred milliseconds — retry later under the same key to learn the job_id.
• A refusal (404, 422, 413, a usage cap, …) writes nothing, so the key is released and a corrected retry under it runs.
• Once the send_jobs row exists every outcome is stored — including a failure part-way through the message inserts, answered 500 AUDIENCE_SEND_INCOMPLETE with the job_id in error.job_id. Rows that landed will send unless the job is canceled; a retry under the same key replays that 500 rather than blasting over them. Deal with the job (GET / DELETE /v1/jobs/{id}), then send under a new key.
• That 500 is written first, provisionally, the moment the send_jobs row commits (recordProvisionalResponse), and the real answer overwrites it. On these two routes the real answer is written before the response returns (storeResponseBeforeReturning), not in after(), so a request that finished and returned never leaves the provisional record standing. A request that never records its own answer — killed at maxDuration, or its store failing — still leaves it behind. For the lock's TTL (330s from the record) a retry reads it as 409 IDEMPOTENCY_IN_FLIGHT, since the request may still be running; after that it replays the 500 with the job_id. Its message says the blast may have scheduled every message, and points at GET /v1/jobs/{id} before any cancel: a kill can land after the last insert. What remains: a kill in the milliseconds between the job row and that record, or the record's own write failing (logged as idempotency.provisional_store_failed), leaves only the lock — once it lapses, a retry under the same key schedules the list again.

Verifications

A verification is a billed send, and without replay a provider call that answered after the client's timeout texted the end user a second code — for a different verification id than the one the caller ended up holding. A retry under the same key now replays the recorded 201. Only the 201 is stored: every refusal, and a 502 SEND_FAILED where the provider threw, releases the key so the retry is a real second attempt. POST /v1/verifications/{id}/check takes no key. Use one key per code request, not per user or destination: the replay answers for 24h with the first verification, even after its code has expired, and sends nothing — a per-user key makes every "resend code" a silent no-op.

Batch

Batch is where retries actually happen: it is the request most likely to hit a timeout or a client-side retry wrapper, and the one whose duplicate is most expensive — several hundred duplicate messages, charges, and message.sent webhooks. Ignoring the header there left a caller who received a 5xx with no safe move.

A batch replay returns the recorded per-item outcome array verbatim. It is not a re-attempt: items that failed on the first call stay failed in the replay. The alternative — retrying just the failures — would mean re-sending the successes alongside them, which is the duplicate the key exists to prevent. To retry only the failed items, send them as a new batch under a new key.

Behavior

1. The raw request body is read once and hashed (sha256) before parsing. The header value is then namespaced (see Scoping) — that scoped value is what gets stored, locked on, and replayed.
2. checkIdempotency looks up idempotency_keys where key matches, org_id matches, and expires_at > now().
   • Hit + same body hash → the cached response body and status code replay with an added header:

     Idempotent-Replayed: true
     

   • Hit + different body hash → 409 IDEMPOTENCY_MISMATCH, and nothing is sent. Bodies are compared byte-for-byte via the hash, so re-serializing the same object with different key order counts as a different body.
   • Miss → continue.
3. A Redis lock (sendoka:idem:lock:{orgId}:{scopedKey}, SET NX) is claimed for the duration of the send. A second request that arrives while the first is still running gets 409 IDEMPOTENCY_IN_FLIGHT (type: "rate_limit_error") rather than a duplicate send.
4. After the response is produced, storeResponseIfIdempotent runs in after(): it writes the cache row, then releases the lock. (The two audience send routes await the write before responding and leave only the release to after(); see Audience sends.) Storing before releasing means a retry landing in between still hits the cache branch and replays instead of 409-ing. If the write fails, the lock is kept until its TTL runs out: released, it would free the key with nothing recorded under it, and the next retry would run the request again — a second message, or a second blast. Retries inside the TTL get 409 IDEMPOTENCY_IN_FLIGHT instead. Past the TTL the key is free again with nothing recorded — except on an audience send, where the provisional record written with the job row replays instead (see Audience sends).
5. storeIdempotency upserts with ON CONFLICT DO UPDATE on key, refreshing status, body and expires_at. (It used to be DO NOTHING, which meant a key legitimately reused after its TTL — the row lingers until the daily cleanup cron — never stored its new response and lost replay protection for up to a day.) The body hash + in-flight lock are what keep the refresh from clobbering a live entry.

When Redis is unavailable

claimInFlight fails closed in production — no lock, no send — so two concurrent retries during an Upstash outage can't both race past the gate. The refusal is the same 409 IDEMPOTENCY_IN_FLIGHT a held lock gives, so during an outage it means "not sent", not "still running". It reaches every keyed request: the SDKs key the nine endpoints above automatically, and the dashboard's audience drawer keys every Send. Outside production (or with ALLOW_IDEMPOTENCY_WITHOUT_REDIS=true) it fails open so npm run dev works without extra infrastructure. A Redis call that throws is treated the same as Redis being absent.

Scoping

Keys are namespaced by scopeIdempotencyKey as:

{orgId}:{tenantId ?? ""}:{environment}:{key}

so Idempotency-Key: ship-1001 from three different callers never collides. All three segments matter:

• org — the table's primary key is key alone, so without the org prefix a second org's store would collide with the first's row and its later retry would re-send.
• tenant — in platform mode one org fans out to many tenants (one per shop, via X-Sendoka-Tenant-Ref) and keys like ship-<orderno> are only unique per shop. Without this segment, shop B would replay shop A's response or hit IDEMPOTENCY_MISMATCH and never send. A non-platform send has a null tenant, so the segment is empty (org_x::live:ship-1001), not the literal null.
• environment — live / test is not part of the request body, so a deterministic key reused across environments would replay the test response for the live call, silently skipping a real send.

org_…, ten_… and live/test are all colon-free, so distinct tuples always map to distinct strings even when the raw key contains colons.

POST /v1/verifications and the audience sends (the v1 route and its dashboard twin) add a route segment to the environment: {orgId}:{tenantId}:live/verifications:{key} and …:live/audience-send:{key}. They started honouring the header long after the send routes did, so a caller that already sent a stable key there (one per user, or one reused for an email and the code that follows it) would otherwise replay the other route's record or get a non-transient IDEMPOTENCY_MISMATCH, and its users would stop receiving codes. On these routes the same key means the same request only within the route. The other routes keep the plain form, so records stored before this still replay.

Lookups still filter on org_id in SQL as well, so a key from another org is never readable.

TTL

• Cache row: IDEMPOTENCY_TTL_HOURS, default 24, env-tunable up to 168 (7 days); a value that is unparseable, ≤ 0, or > 168 silently falls back to 24. After expires_at the key is reusable and the next store refreshes the row. Expired rows are deleted nightly by /api/cron/cleanup-idempotency.
• In-flight lock: IN_FLIGHT_TTL_SECONDS = 330. It must outlast the longest a single send can take before it stores its cache row, which is the route's own maxDuration (300s on the v1 send routes) — not any caller's timeout. A client abort does not stop the server-side send, so bounding the lock by the MCP self-call's 60s UPSTREAM_TIMEOUT_MS was wrong: the lock could expire mid-send and let a retry re-claim and send again. On the happy path the lock is released as soon as the cache row lands, so the real 409 window is a few hundred milliseconds, not 330 seconds — except on the audience send, where the request itself can run for minutes and the lock is held for all of it. The verification and audience-send routes pin maxDuration = 300 like the send routes, and idempotency-lock-ttl.test.ts holds every lock-claiming route under the TTL.

Schema

response_body carries an internal __body_hash key alongside the response fields. It is what checkIdempotency compares the incoming body against, and it is stripped before the response is replayed — clients never see it. A provisional record (see Audience sends) also carries __provisional_until, epoch ms: before it, the row answers 409 IDEMPOTENCY_IN_FLIGHT; after it, the row replays like any other. The body hash is still compared first, so a different body under the key is a mismatch either way.

Errors

All three are detailed in errors.md.