Runbook — repair domains.region after the create-path fix

One-time deploy step. Required on any deployment whose AWS_REGION is not us-east-1 and that added domains before the fix landed. A us-east-1 deployment is unaffected — the wrong value and the right value are the same string there.

This deployment runs AWS_REGION=ap-southeast-2, so it needs it.

Why

createDomainIdentity has always created the SES identity in AWS_REGION. Until the fix, both add-domain routes then inserted a domains row with no region, so Postgres filled in the schema column default "us-east-1" (src/lib/db/schema/domains.ts). Every row written before the fix therefore names a region its identity was never in, and every later operation keys off the row, not off the env:

  • domain-verify-poll calls checkDomainStatus(d.domain, d.region), gets NotFoundException from us-east-1, and substitutes verified: false. The "identity missing" early return is guarded on !wasVerified, so a row that is already verified skips it, falls through to justUnverified, flips to pending and fans out a domain.unverified webhook at the customer — for a domain that is verified and sending.
  • Deleting a domain (internal/domains) calls deleteDomainIdentity(record.domain, record.region), so the delete goes to us-east-1, finds nothing, and the real ap-southeast-2 identity survives and keeps sending. The row is gone by then, so nothing in the product mentions that identity again.

The fix stops new rows being written wrong. It does not touch the rows that already are — nothing self-heals.

Run it dry first

scripts/repair-domain-regions.ts is dry run by default: it prints every row it would change and writes nothing.

npx tsx --env-file=.env scripts/repair-domain-regions.ts

It does not guess the target region. For each domains row it issues GetEmailIdentity in every plausible region (ALLOWED_REGIONS, plus AWS_REGION and the row's own value when those sit outside that list) and records which regions answer. Read-only at AWS in both modes--apply changes what happens in Postgres and nothing else.

Read the output, then apply:

npx tsx --env-file=.env scripts/repair-domain-regions.ts --apply

⚠️ Dev and prod share one Neon database. There is no staging copy to rehearse against — whatever DATABASE_URL resolves to is production. Run the dry pass and read it before --apply.

Idempotent and resumable: a second run finds the rows it corrected already correct and writes nothing. Exit code is 0 when every row resolved to exactly one region, 1 when any row needs a human.

What it will not write, and what to do about it

Only a row whose identity is found in exactly one region is corrected automatically. Three outcomes are reported and left alone:

Reported What it means What to do
found in no region The identity was deleted at AWS, or it lives in a region this deployment has never named. Re-add the domain (mints a fresh identity) or delete the row. Do not hand-write a region to make it look fixed — that just moves the NotFoundException somewhere harder to spot.
found in several regions The domain is a real identity in more than one region — created by hand, or left behind by a region move. Decide which one is the live sender (check DNS and which identity is verified — the script prints verified and DKIM status per region), set region by hand, and delete the spare at AWS. Picking wrong re-creates the delete bug on purpose.
probe incomplete At least one region did not answer — throttle, timeout, AccessDenied. Re-run. A non-answer is not an absence, so the row is refused rather than written on partial evidence. If every row skips with the same regions erroring, the IAM key is missing ses:GetEmailIdentity there (see scripts/sendoka-app-runtime-policy.json).

After

Nothing else to do — no migration, no redeploy. The schema default stays "us-east-1"; it is simply no longer reachable from either create path, which src/app/api/domain-create-region.test.ts asserts.