Cron Jobs
Registered via vercel.ts. Each route is a standard Next.js GET handler under /api/cron/* that authorizes via the CRON_SECRET env.
Thirteen jobs — one row below per entry in the crons array. Adding a route without a vercel.ts entry means it never fires; adding an entry without a row here means nobody knows it exists. Keep all three in lockstep.
| Schedule (UTC) | Path | Purpose |
|---|---|---|
* * * * * |
/api/cron/send-scheduled |
Claim due scheduled messages (FOR UPDATE SKIP LOCKED) and call the provider |
*/5 * * * * |
/api/cron/retry-webhooks |
Re-attempt failed webhook deliveries with exponential backoff, drained round-robin per org so one org's backlog can't starve the rest |
*/5 * * * * |
/api/cron/probe-health |
Probe Postgres + Redis liveness, schema drift (migrations applied vs. the _journal.json bundled with the deploy) and the sandbox parent SES identity; write rows to health_probes for the public status page. Only liveness decides the stored status — the two operator checks ride in checks.schema / checks.sandbox_identity, in the response's schema_drift / sandbox_identity_unverified, and in cron.probe_health.schema_drift / cron.probe_health.sandbox_identity warn lines, because neither is a customer-visible incident and status is what /status renders and divides its 30-day uptime by. The identity check reads one sample row per distinct region from domains where platform_managed = true and asks SES whether any parent of that name is verified in the row's own region — the one dependency nothing in the repo creates (aws-ses.md). It fails open (no credentials, throttle, timeout, column not migrated yet) and is skipped entirely with no AWS_ACCESS_KEY_ID |
*/5 * * * * |
/api/cron/domain-verify-poll |
Poll SES for two cohorts — pending domains, and verified domains past their recheck interval. Emits domain.verified / domain.unverified (DKIM revoked, DNS pulled, or the SES identity deleted — dkim_status: IDENTITY_NOT_FOUND, counted as identity_missing in the response). Also the only caller that attaches SES_CONFIGURATION_SET to identities predating the attach in createDomainIdentity, because it is the only one holding a row to count attempts on: capped at MAX_CONFIG_SET_ATTACH_ATTEMPTS (12) consecutive failures, then retired into domains.config_set_attach_state = 'attempt_cap' with one cron.domain_config_set.gave_up. Skipped for an identity SES no longer has |
*/5 * * * * |
/api/cron/sms-verify-poll |
Poll AWS End User Messaging for brand / campaign / phone-number registration — pending rows plus verified-row rechecks |
*/10 * * * * |
/api/cron/run-exports |
Drain org_exports queue — upload to Vercel Blob when configured, fall back to in-row JSON. Also reclaims jobs stuck running after an OOM / wall-clock kill |
*/15 * * * * |
/api/cron/reconcile-registrations |
Retry brand/campaign create when the AWS provider call failed at POST time (rows with providerBrandId/providerCampaignId = NULL). Rows older than 7 days flip to failed with reconcile_state = 'age_cap' |
*/15 * * * * |
/api/cron/ip-pool-warmup |
Poll SES for dedicated-IP allocation + warmup percentage; advance pools provisioning → warming → active and invalidate the pool cache |
*/30 * * * * |
/api/cron/dlt-status-poll |
Poll the DLT provider for status flips on submitted entities / headers / templates. Approved templates are what makes an Indian SMS send eligible |
0 2 * * * |
/api/cron/rollup-stats |
Daily roll-up of messages into daily_stats (org × day × channel × status), over a day-aligned window that excludes today; idempotent via upsert |
0 3 * * * |
/api/cron/cleanup-idempotency |
Batched drain of idempotency_keys rows past expires_at |
0 4 * * * |
/api/cron/retention |
Nightly prune: health_probes >30d, cron_runs >30d (the run ledger withCronRun writes — nothing else reaps it, and at thirteen jobs with one every minute it grows ~2.3k rows a day), daily_stats >180d, delivered webhook_deliveries >30d, phone_verifications >7d, email_verifications >7d, verifications >2d, message_events >90d (MESSAGE_EVENTS_TTL_DAYS — the open/click trail, which holds the recipient's raw IP, user agent and clicked URL), inbound_messages + inbound_sms >90d (INBOUND_TTL_DAYS, one window for both; suppressions is never touched, so sweeping the row that carried a STOP does not forget the opt-out), OAuth auth-codes/refresh-tokens/revoked keys, plus message-content redaction, where the effective window per org is min(plan logRetentionDays — Free 7d, Pro 90d, no promise on PAYG/Enterprise — and MESSAGE_CONTENT_TTL_DAYS when set). Comped owners and any plan_status this build does not recognise use the env window only: an unknown tier must never inherit Free's 7 days. audit_logs >365d are NOT pruned by default — see below. Also retries signup notifications that were claimed, failed to send and released their claim — the mailed code is the last step of every signup, so nothing else triggers them again. Finally sweeps stranded signups (see below) |
0 6 1-5 * * |
/api/cron/report-overage |
Days 1-5: report the closed month's Pro overage to Stripe meter events (ledger-guarded, idempotent, priced by the plan in effect during the closed month) |
Audit-log retention — keep is the default
The audit_logs sweep is the one table in the retention job that is not
pruned on a default run. shouldDeleteAudit in
src/app/api/cron/retention/route.ts
is true only when one of two things holds:
ARCHIVE_AUDIT_LOGS=trueand the Blob upload actually succeeded. The sweep selects up to 5000 rows past the 365d cutoff, oldest first, uploads them to Vercel Blob as a privateaudit-archive/YYYY-MM-DD.json, and then deletes exactly the ids it archived — never the whole< cutoffset. Repeated runs drain the backlog batch by batch. The narrower delete is the point: the unbounded form used to drop the remainder beyond the batch that had just been archived, which is silent, permanent loss.AUDIT_LOG_HARD_DELETE=true, which deletes aged rows with no archive at all. This is the only setting that destroys audit history outright.
Neither set — or ARCHIVE_AUDIT_LOGS=true with BLOB_READ_WRITE_TOKEN
missing, or with the upload throwing — and aged audit rows are simply
kept. That is deliberate: the audit log is the tamper-evidence trail, and
the sweep refuses to delete what it could not archive. The no-token case logs
cron.retention.audit_archive_skipped_no_blob; an upload failure logs
cron.retention.audit_archive_upload. Both leave the rows in place, so a
deployment that turned archiving on but never provisioned a Blob store grows
audit_logs unbounded rather than losing it — check for those two events before
assuming the archive is running.
Stranded-signup sweep
The retention job ends with the only sweep in the tree that deletes an account rather than pruning a log or a dead code: self-serve signups that never proved their address, are older than 30 days, never held a session, and whose orgs are solo, key-less and message-less. It is what unsticks an address a squatter registered and abandoned when no OAuth provider can claim it — see authentication and issue #91.
It carries two gates the other sweeps do not:
STRANDED_SIGNUP_REAP=1— an explicit operator opt-in. Anything else runs the identical query and logsauth.signup_reap.dry_runwith the ids it would have deleted, deleting nothing. Run it that way for a cycle and read the log before turning it on.cronSecretPresented, notcronAuthorized. The latter is deliberately permissive whenCRON_SECRETis unset outside production so a localcurlworks — and dev and prod share one Neon database, so thatcurlwould be deleting production accounts. The destructive path requires a real bearer in every environment; without one the sweep degrades to its dry run.
Reported in the run summary as
stranded_signups: { candidates, deleted, skipped, orphaned, dryRun }.
skipped and orphaned are separate counters on purpose. A candidate one of
the guards turned away cost nothing and the next run re-evaluates it — that is
skipped. orphaned is an incident, not a statistic: it counts candidates
that ended the run with at least one organization irreversibly cascaded and the
account still standing, i.e. an owner left with no org, no membership and no way
back. Nothing can undo that — deleteOrgCascade is a hard delete and the Neon
HTTP driver has no transaction spanning it and the users delete that follows —
so a non-zero orphaned means a Neon point-in-time restore of specific rows.
The run logs auth.signup_reap.cascaded_then_abandoned (a guard refused a
second org after the first was gone) or auth.signup_reap.cascaded_then_aborted
(the guarded account delete matched nothing after a cascade) at error level,
each carrying orgIds — the only record of which orgs were destroyed. Grep
cascaded_then_ for both, and auth.signup_reap.failed for the third door into
the same state: a throw after the first cascade, which carries the same orgIds
and counts against the same orphaned.
Authentication
Vercel sends Authorization: Bearer ${CRON_SECRET} on scheduled invocations. Every handler gates on the shared cronAuthorized and 401s when it returns false:
export function cronAuthorized(req: NextRequest): boolean {
const header = req.headers.get("authorization");
const secret = process.env.CRON_SECRET;
if (!secret) {
return process.env.NODE_ENV !== "production";
}
if (!header) return false;
const expected = Buffer.from(`Bearer ${secret}`);
const actual = Buffer.from(header);
if (actual.length !== expected.length) return false;
return timingSafeEqual(actual, expected);
}
CRON_SECRET is required in production — the check fails closed. An unset secret rejects every caller, Vercel Cron included; it does not leave the endpoints open. All thirteen jobs then 401 on every tick with no alert, because a 401 is an ordinary response and the platform records the invocation as having happened: scheduled messages never leave scheduled, webhook retries never fire, overage is never metered, retention never prunes. Set it on the production deploy before the first tick.
Outside production (NODE_ENV !== "production") an unset secret is permissive, so curl localhost:3000/api/cron/* works with no setup. Set the var locally and it is enforced locally too.
Ten of the thirteen jobs refuse to ride that permissiveness, because dev and prod share one Neon database. Every cron that spends money, calls a provider about a real customer's resources, or mutates customer-visible state additionally gates on cronSecretPresented — a real presented bearer, in every environment — and degrades to a logged no-op without one (never a 401; cronAuthorized above still owns that answer):
| Cron | What a bare local curl used to reach |
|---|---|
send-scheduled |
Claims real customers' due rows and fires or terminally fails them |
retry-webhooks |
POSTs to customers' live endpoints, burning each delivery's 8-attempt budget |
reconcile-registrations |
Retries billed, non-idempotent AWS CreateRegistration calls |
report-overage |
Emits Stripe meter events and flips overage_reports ledger rows |
| stranded-signup sweep | Deletes abandoned accounts (also behind STRANDED_SIGNUP_REAP) |
retention |
DELETEs audit logs, stats, deliveries, inbound messages, OAuth tokens |
run-exports |
Claims export jobs and writes orgs' full PII dumps to Vercel Blob |
domain-verify-poll |
Flips live domains out of verified (stops that org's email) + webhooks |
sms-verify-poll |
Flips live 10DLC registrations out of verified (stops that org's SMS) + webhooks |
dlt-status-poll |
Flips DLT templates in and out of approved (India send-eligibility) + webhooks |
ip-pool-warmup |
Promotes pools to active, routing mail over IPs AWS has not finished warming |
The no-op response carries skipped: "cron_secret_not_presented" alongside the route's own zeroed counters, and the skip logs as cron.<route>.skipped_no_secret. The exception is ip-pool-warmup, whose summary already has a numeric skipped counter — there the marker rides on skipped_reason and skipped stays the honest 0. To exercise any of them locally, set CRON_SECRET and send it as the bearer.
Only cleanup-idempotency, rollup-stats and probe-health stop at cronAuthorized. They prune already-expired idempotency keys, aggregate yesterday's counters into daily_stats, and record a health_probes row — none of it spends money, reaches a customer's endpoint, or changes a resource a customer can see, so a stray local run is noise rather than damage. Add the cronSecretPresented gate to any new cron that does more than that.
Each gate carries a secret-gate.test.ts beside its route asserting all three arms — 401 when unauthorized, a no-op that reaches neither the database nor the provider when authorized-but-unpresented, and a normal run when the bearer is sent.
The comparison is timingSafeEqual over the whole Bearer <secret> header, with a length pre-check because timingSafeEqual throws on unequal-length buffers. Don't "simplify" it back to === — that hands the secret to anyone who can time the endpoint, one byte at a time.
send-scheduled
Requires a presented
CRON_SECRETin every environment. After the usualcronAuthorized401 check, the handler gates oncronSecretPresented— the stranded-signup-sweep convention — because the permissive dev path (secret unset outside production) let a bare localcurlclaim real customers' due scheduled rows on the shared Neon database and either fire them from the laptop or burn theirattemptsbudget into a terminalfailed. Without a real bearer the run is a logged no-op (cron.scheduled.skipped_no_secret; the response carriesskipped: "cron_secret_not_presented"alongside the zeroed counters) that touches nothing — no reaper, no claim, no provider call, no job settlement. SetCRON_SECRETlocally and send the bearer to run it in dev.Atomic claim:
UPDATE messages SET status='sending', updated_at=now() WHERE id IN (SELECT id FROM messages WHERE (status='scheduled' AND scheduled_at <= now()) OR (status='sending' AND updated_at < now() - interval '5 minutes') ORDER BY scheduled_at LIMIT 500 FOR UPDATE SKIP LOCKED) RETURNING *. Two cron runners can't see the same row — Postgres hands each a disjoint set.Suspended orgs are filtered inside the claim, not canceled after it:
AND org_id NOT IN (SELECT id FROM organizations WHERE status = 'suspended'). Their due rows stayscheduledand resume when the org is unsuspended, and because the filter sits in the innerSELECTthey cannot fill theLIMIT 500batch and starve everyone else's due rows. So "an org's sends stopped without an error" has a second cause besides the ones below — checkorganizations.statusand see _org-suspension.md.Stale-claim recovery: rows stuck in
sendingfor > 5 min (runner crashed mid-provider-call) become eligible again on the next tick. Bounded double-send window matches the recovery threshold; SES/SNS p99 is well under 8s so it shouldn't fire normally.Pre-publish: recipient-suppression recheck — STOP'd between schedule and send → flip to
canceled.Pre-publish: sender recheck — the domain or number the row was authorized against at enqueue must still be the org's (verified, matching tenant, not a simulated test row) or the message flips to
canceledwith the reason inerror_message.resolveDomainis routing, not a gate: it selects region/pool columns with no status or tenant filter, so a domain that lost verification, was rebound to another tenant, or was re-registered by a different org still resolves — and SES identities are account-wide, so the mail would be DKIM-signed as that identity. An alphanumeric SMS sender ID has no provider check behind it at all. A row from the org's sandbox sender is also canceled when its fullto+cc+bccreaches anyone who is not a verified member of the org, or when it carries ajob_id(an audience blast, which a sandbox sender never carries) — rows queued before that rule shipped included — with theSANDBOX_RECIPIENT_NOT_ALLOWEDmessage aserror_message; these cancels log under the samecron.scheduled.sender_unverifiedevent as the rest of this gate. The daily cap is not re-applied here: the row was counted the day it was accepted. Batched per(org, tenant)per claim, so a 500-row ramp costs one lookup rather than 500. Two exemptions: test rows (never reach a provider) and rows already carrying aprovider_message_id(that mail is out; the pass only finishes bookkeeping).allowed_domain_idsis deliberately not re-applied — that is key state, not sender state.Pre-publish: spend-cap recheck, answered by DEFERRAL. The free-plan and per-tenant caps cancel a row they cannot admit; the org spend cap (
organizations.spend_cap_cents) does not — it is a ceiling the owner can raise, so the row waits. Per batch the handler seeds one spend budget per live org off the samecheckUsageLimitread the free cap uses plus onecheckSpendCaplookup, then reserves synchronously per row by re-costing a projected copy of the period counters with the realevaluateSpendCap(both channels together — it bounds the bill, not a counter). A row that would cross goes back toscheduledwithnext_retry_atanderror_messagecleared, under the samestatus='sending'guard, and the run logscron.scheduled.spend_cap_deferredper row. Nomessage.failed, noattemptsbump — the provider was never called. cap=0 on a metered tier defers everything metered, an hour at a time, until the cap moves or the job is canceled; sends inside a plan allowance cost nothing and are never deferred. Exemptions match the other gates: test rows and rows already carrying aprovider_message_id. Four details are load-bearing:- The push is measured from the row's own
scheduled_at, not fromnow()— advanced by whole 60-minute intervals until it lands strictly in the future. A flatnow() + 60 minstacks a whole ramped cohort on one instant, voiding the per-recipient spacing computed at enqueue and handing the next tick exactly the burst the ramp exists to prevent. - Quiet hours are re-evaluated for SMS rows on the
broadcaststream, for orgs that have not setsms_quiet_hourstooff.quietHoursDeferralis otherwise an enqueue-only gate (its only other callers are the two audience-send routes), so without this a blast row placed at 19:30 destination-local would drift to 20:30, 21:30 … and fire inside the window the moment the cap moved. Transactional SMS is never re-timed — it is never deferred for quiet hours in the first place. - The
scheduledstatus event is written once per row, viarecordStatusEventOnce. The deferral repeats for as long as the cap binds, nothing prunesmessage_status_events, andGET /v1/{emails,sms}/{id}returns the wholestatus_historyarray — an entry per pass would grow the customer's response by one identical line an hour, forever. - An inline send (
scheduled_at IS NULL) is canceled, not deferred. The stale-claim arm can hand over a/v1/emailsor/v1/smsrow wedged insending;scheduledwith noscheduled_atmatches no claim arm, so deferring it would strand it, and a transactional send is never deferred. It joins the absolute caps and cancels withOrganization spend cap reached before scheduled send.
The first deferral of a batch also fires the 100%
spend_capusage alert (deduped through theusage_alertsunique constraint, and only when the cap is genuinely binding rather than overshot by one large row). Without it a ramp accepted before the cap started binding is silent in both directions: the enqueue counters never crossed, soenforceUsageLimit's refusal alert never fired either, and the org's whole queue could sit frozen with nobody told. The deferral UPDATE carries its owntry/catch— a Neon blip landing in the outer catch would writestatus='failed'and could fan outmessage.failed, which is exactly what a deferral must never do; instead the row is left insendingfor the 5-minute stale-claim arm.- The push is measured from the row's own
Provider call: SES (email) or SNS (SMS). Test env: skip provider, set
providerMessageId = "test_<id>".provider_message_idis stamped as soon as the provider accepts, in its ownstatus='sending'-guarded UPDATE, before the terminalstatus='sent'write. This is what makes the recording-only path reachable: a runner killed between the send and the terminal write leaves a row carrying the id, so the next stale re-claim finishes the bookkeeping instead of sending the message a second (and third) time and metering each delivery.Update predicate includes
AND status = 'sending'so a late update from a crashed runner can't stomp a row already re-claimed and completed.Wall-clock skip releases the claim (back to
scheduled, whenscheduled_atis set) rather than leaving the row insending. Left claimed, a batch abandoned at the deadline burned anattemptsincrement per stale re-claim without ever calling the provider, and after three of those the reaper wrote it permanentlyfailed— a message reported as failed that was never attempted. A batch is also only claimed with ≥30s of budget left.Reaper: rows wedged in
sendingpast the attempt cap flip tofailedand emit amessage.failedwebhook. Without that fanout the customer got no notification at all — the SES/SNS handlers never fire for a send that died before the provider answered, so the failure showed up only in the dashboard. The permanent "missing required fields" failure fans out too; transient failures do not, since those rows are still retriable.Fan-out: increment usage + emit
message.sentto webhooks viaafter().
reconcile-registrations
- Requires a presented
CRON_SECRETin every environment — the send-scheduled convention. Every retry is a billed, non-idempotent AWSCreateRegistration, so the permissive dev path let a bare localcurlreplay real orgs' pending registrations on the shared Neon database — paying for each — and burnreconcile_attemptstoward the attempt-cap give-up. Without a real bearer the run is a logged no-op (cron.reconcile.skipped_no_secret; the response carriesskipped: "cron_secret_not_presented"alongside zeroed brand/campaign counters). - Picks up orphan brand/campaign rows where
status='pending'ANDproviderBrandId/providerCampaignId IS NULL(POST-time AWS failure). - Retries
CreateRegistrationper row. UPDATE predicate keeps theIS NULLguard so overlapping ticks can't double-stamp a row that was just fixed. - Campaign retries only fire once the bound brand finally has a
providerBrandId. - The batch is
ORDER BY created_atand capped at 20, so the oldest live orphan is always picked first. Age-expired rows are retired in one bulk statement before the batch is selected — retiring them inside the loop let dead rows occupy slots and starve the fresh ones a retry would actually fix. - Rows older than
MAX_RETRY_AGE_DAYS = 7are bumped tofailedwithreconcile_state = 'age_cap'; rows pastMAX_RECONCILE_ATTEMPTS = 24withreconcile_state = 'attempt_cap'. Neither sweep touches a row whose claim is live, because attempts are counted on claim — retiring one would raise a gave-up alarm for a brand whose final attempt is still in flight. - Scheduler state never touches
provider_status. That column is mirrored verbatim to customers byGET /v1/brands, so the claim lives inreconcile_claimed_at(stale after 5 min) and the give-up reason inreconcile_state.reconcile_claimed_atis also what staleness is measured from — it used to beupdated_at, which every other writer bumps, so "this claim is stale" and "somebody touched this row" were the same test.
retry-webhooks
- Requires a presented
CRON_SECRETin every environment — the send-scheduled convention. The permissive dev path let a bare localcurlclaim real orgs' due deliveries on the shared Neon database and POST them at customers' live endpoints from a laptop, burning each row's 8-attempt budget. Without a real bearer the run is a logged no-op (cron.retry_webhooks.skipped_no_secret; the response carriesskipped: "cron_secret_not_presented"alongside zeroedprocessed/total_due). - Claims up to 100
webhook_deliveriesrows wherestatus = 'pending'andnext_attempt_atis past the run's frozen due-horizon — round-robin per org, see below. - For each: re-run
attemptDelivery()which POSTs to the endpoint and updates attempt count/status. - If the endpoint row is missing or
enabled = false, the delivery is markedfailedwith reasonendpoint disabled or removed.
Fairness — the claim is round-robin per org, not a global FIFO. Due rows are ranked within each org by due-ness (row_number() OVER (PARTITION BY org_id ORDER BY next_attempt_at, id)) and the batch is filled in rank order: every org's most-overdue delivery first, then every org's second, and so on, ties inside a rank going to the most overdue row. Under the old global ORDER BY next_attempt_at, one org recovering a 10k-row backlog owned every 100-row batch for the whole run — every other org's deliveries sat behind it, and could burn through MAX_ATTEMPTS purely from queue latency rather than receiver failure. The interleave caps a backlogged org at its share of each batch, so a healthy org's single due delivery goes out in the first one. The claim is still one atomic statement: Postgres refuses FOR UPDATE in a query level that computes window functions, so the ranking runs unlocked and the locking level re-applies the due predicate — a row a competing runner claimed in the gap has been leased forward and drops out rather than being delivered twice.
Backoff: nextBackoffMs(attempt) returns min(2^attempt * 60_000, 6h) + random(0..30s) ms. Capped at 8 attempts (MAX_ATTEMPTS in webhook-fanout.ts). Minute-scale because this cron only runs every 5 minutes — a second-scale backoff was entirely absorbed by the tick interval, burning every attempt within ~25 minutes of a receiver going down. The run drains in batches of 100 until the backlog clears or it hits its wall-clock budget, and leases each claimed row 10 minutes forward so a slow batch can't redeliver its own rows.
The drain claims rows due as of the run's start instant, not now(). The 10-minute lease only covers the window before attemptDelivery writes the real next_attempt_at, and the minimum backoff it writes (2 min) is shorter than the drain's wall-clock budget (4 min) — so against now() a row that failed early in a run came due again inside that same run and was delivered twice. Freezing the horizon makes that impossible; rows that fall due mid-run are picked up by the next tick.
cleanup-idempotency
DELETE FROM idempotency_keys WHERE expires_at < now(), in DELETE_BATCH = 20_000 chunks until the backlog clears or the run hits its wall-clock budget. Not one unbounded statement: every idempotent send writes a row, so against a real backlog the single DELETE outran the Neon HTTP timeout and the sweep failed every night while the table only grew. The batched form drains across ticks. Response reports deleted / remaining / complete; a batch that fails answers 500 with an error field rather than 200 with complete: false, because the latter reads exactly like "there is more to drain" and left the cron_runs row green for a sweep that had stopped working.
Query budgets
Every Neon query is bounded — fetch has no default timeout, and a connection that opens and then stalls otherwise holds the invocation to maxDuration. The request path and the per-minute crons get 20s (sql / db from @/lib/db); a query that has not answered in twenty seconds there is gone, not slow.
The three batch sweeps — cleanup-idempotency, retention and rollup-stats — import sweepSql / sweepDb instead, which are the same database on a 120s budget. Their statements are legitimately slow (20 000-row DELETEs, a 30-day GROUP BY over messages), and the request-path cap would abort a batch that is working. Anything new on a request path or on a per-minute schedule stays on the plain clients.
report-overage
Requires a presented CRON_SECRET in every environment — the send-scheduled convention. This cron moves real money: it emits Stripe meter events against real customers and flips overage_reports ledger rows, where reported is terminal and a row stranded pending by an interrupted laptop run blocks the period until a human reconciles it against the Stripe meter. Without a real bearer the run is a logged no-op (cron.overage.skipped_no_secret; the response carries skipped: "cron_secret_not_presented" alongside zeroed evaluated / billed / results / failed).
For every org with a Stripe customer — there is deliberately no plan_status filter, see "Plan history" below:
- Read
usage_recordsfor the closed period (previousPeriodKey()), live env. Orgs are keyset-paginated 200 at a time, one usage query per page, plus oneorg_plan_eventsquery per page. - Skip the org outright if its owner is comped. A comp is an uncapped entitlement no
plan_statusrecords, so there is nothing to meter — and it must not reachplanForPeriod(), which would price the month off the column and flag the volume as ahard_capped_freecap-enforcement bug. Skipped orgs that actually sent are listed ascomped_skipped(and logged ascron.overage.comped_orgs_skipped) so the volume the platform absorbed stays visible. - Decide which plan applied to the closed period via
planForPeriod()(src/lib/api/plan-history.ts). Skip the org if it isn't billable for that period (see below). - Compute overage vs. the period's plan limits —
PLANS[decision.plan].limits, which isPLANS.profor a metered period:email_overage = max(0, emails - 10_000) sms_overage = max(0, sms - 1_000) - Claim an
overage_reportsrow for(org, period, channel), then emit the Stripe meter event:event_name: STRIPE_EMAIL_OVERAGE_METER_EVENTevent_name: STRIPE_SMS_OVERAGE_METER_EVENT- Payload:
{ stripe_customer_id, value: <overage> }. identifier: overage:{org}:{period}:{channel}.
- Flip the row to
reportedon success, orfailedon a Stripe error so the next day's run retries it.
Plan history — which plan the closed month is billed at
organizations.plan_status says what an org is on right now, which is not what it was on during the month being billed. Selecting or pricing on it was wrong in both directions:
- An org that was Pro all of July and signed an enterprise contract before the Aug 1 run was filtered out by
plan_status <> 'enterprise'. Nooverage_reportsrow was ever claimed, so it couldn't appear instuck/skipped_*/unresolved_prioreither — those only re-surface rows that already exist — and the run returned 200. A whole month of real overage vanished with zero signal. - Enterprise is uncapped (
usage-limit.tsreturnsInfinityand skips the usage read) butusage_recordsare still written for it. Flip a contract org off enterprise before the run and its entire enterprise-era volume got measured againstPLANS.proand metered to Stripe as a genuine charge.
org_plan_events is the fix: an append-only row per plan transition (org_id, from_plan, to_plan, effective_at, source). Writers are applyPlan in the Stripe webhook and the self-heal in /api/internal/billing, both via recordPlanChange(), which flips plan_status and appends the event in one statement so the two can never disagree. No-op transitions aren't recorded, so the table stays a list of real changes rather than a webhook journal.
planForPeriod(period, currentPlan, events) then decides, and the rules are:
| Situation | Decision |
|---|---|
| Plan changed inside the period | Bill at the highest entitlement held during it. usage_records is one monthly aggregate with no per-send timestamps, so the volume cannot be split between eras — and pricing an uncapped era at Pro rates is a real, refundable overcharge. Consequence: an org that converts to enterprise mid-month is not billed overage for that month. |
| Plan changed after the period closed | Irrelevant to the bill. The event's from_plan names what applied during the period, so a July-Pro org that went enterprise on Aug 1 is still billed for July. |
| Enterprise held at any point in the period | billable: false, reason uncapped_enterprise. Never priced at Pro rates. |
| Free for the whole period | billable: false, reason hard_capped_free. An overage here means the Free hard cap leaked — logged as an anomaly. |
| Live plan is a higher tier than history explains | billable: false, reason unrecorded_plan_change. Somebody moved the org up a tier without appending an event, so the move can't be dated and the period may have been uncapped for any part of it. Logged via logError; fix by recording the flip (below) and the next run bills it correctly. An unrecorded downgrade does not block billing — the last recorded tier is still what applied. |
| No history at all (every org predating the table) | Assume the org's current plan_status held for the whole period. Degrades exactly to the pre-history behaviour; counted as assumed_plan_from_current so the number can be watched decaying toward zero. |
Out-of-band plan flips must be recorded. applyPlan only knows pro/free and refuses to touch an enterprise org, so enterprise conversions are applied by hand — and a hand-applied flip with no event row is the unrecorded_plan_change case above. Whenever you change plan_status directly, append the transition in the same statement:
WITH prev AS (
SELECT id, plan_status FROM organizations WHERE id = 'org_...' FOR UPDATE
), upd AS (
UPDATE organizations o SET plan_status = 'enterprise', updated_at = now()
FROM prev p WHERE o.id = p.id
)
INSERT INTO org_plan_events (id, org_id, from_plan, to_plan, effective_at, source, created_at)
SELECT 'ope_<nanoid24>', p.id, p.plan_status, 'enterprise', now(), 'manual', now() FROM prev p;
Set effective_at to when the contract actually started if it isn't now — it is the column the period decision reads, and backdating it correctly is what puts the change in the right month.
Runs 0 6 1-5 * * — daily for the first five days rather than once, because Vercel Cron never retries and a single failed or wall-clock-truncated firing would strand the closed month unbilled. Re-runs are safe because of the overage_reports ledger, not because of the meter identifier: Stripe's identifier de-dup only spans a rolling ~24h, so day 1 and day 3 would otherwise both bill. previousPeriodKey() resolves to the same closed month on any of those days.
Ledger epoch (OVERAGE_LEDGER_FIRST_PERIOD). The ledger only speaks for periods it existed for. It shipped empty, so on any period predating it a missing row means "we have no idea" — not "not yet billed" — and claiming one would re-fire a meter event the pre-ledger cron already sent. The cron therefore refuses to meter any period earlier than this env var, and refuses everything when it is unset: an unbilled month is a reconciliation, a double-billed month is a refund to every paying customer. Set it to the first period whose day-1 run happens after the ledger deploy (deploying during 2026-08 → 2026-08, first metered 2026-09-01), and reconcile anything older by hand against the Stripe meter.
Response fields: evaluated (orgs that had prior-month overage under a billable plan — what the run looked at) and billed (how many of those money actually moved for on this run; each results row carries the metered channels behind it, so a healthy day-2 run legitimately reports evaluated: N, billed: 0 next to already_reported: N), already_reported (expected to climb to the full billable count on days 2-5 — the ledger doing its job), stuck (rows left pending by a run that died mid-Stripe-call; never re-fired automatically, reconcile against the Stripe meter by hand), skipped_pre_ledger (overage left unmetered because the period predates the epoch, or the epoch is unset), skipped_no_meter, comped_skipped (comped orgs with volume, skipped before pricing — never a failure), complete, failed. Returns 200 only when failed, skipped_pre_ledger, skipped_no_meter and skipped_unbillable are all empty, complete is true and ledger_epoch_unset is false; otherwise 500. The epoch is tested on its own rather than through skipped_pre_ledger, because with no epoch nothing is ever claimed: in a month where no billable org exceeded its allotment that list is empty, and a verdict read off it alone answered 200 complete: true with metering switched off deployment-wide. skipped_no_overage_price and overage_prices_missing are deliberately not in that list — see below.
A meter event is not an invoice. It is addressed to the customer, and Stripe accepts it whatever they are subscribed to; it only turns into money through a price on their subscription that reads the channel's meter. Checkout attaches those prices, so every subscription minted before that line carries the flat price alone — this cron metered them, markReported closed the period, and the invoice came out $0, with the ledger then refusing to re-meter a period it believes it settled. ensureOveragePrices (src/lib/billing/overage.ts) now settles the question before any ledger row is claimed, and settles it on the price's recurring.meter, never on the price id: a configured price already sitting on the subscription is still refused if it names no meter, and a different price reading the channel's meter counts as billable and is left alone — which is what makes a Stripe rate change (prices are immutable, so a new id is minted and the env var repointed) and an operator's hand-repair in the dashboard safe rather than double-billed. Four more response fields come out of it:
skipped_unbillable—[{ orgId, channel, overage, reason }]. Real overage that was not metered because the customer's subscription should carry a metered price for that channel and does not.reasonisno_subscription(a churned org that overflowed the month it cancelled; nothing to attach to — normally already invoiced by final settlement at cancel time, which closes the period in this same ledger, so one still showing up here needs a hand-written invoice),ambiguous_subscription(more than one live subscription and none of them names a configured tier — attaching to the wrong one bills the same usage twice, so it refuses and asks for a human),price_not_metered(STRIPE_*_OVERAGE_PRICE_IDnames a price with norecurring.meter: a licensed price, a pre-meters legacy metered one, or the flat tier price by copy-paste) orstripe_error. Nothing is claimed and nothing is markedreported, so the period stays re-tryable on the next day's run the moment a human fixes the reason. Logged throughlogErrorascron.overage.unbillable_subscriptionswith aby_reasonhistogram, plus onecron.overage.unbillable_subscriptionper (org, channel). This list fails the run.skipped_no_overage_price—[{ orgId, channel, overage }]. Overage nobody is trying to bill, because that channel has noSTRIPE_*_OVERAGE_PRICE_IDat all. That is the supported flat-rate deployment (see environment), not a fault:log.warn("cron.overage.no_overage_price", …), and it does not fail the run — every org answers this way on every run forever, and a cron that is red forever is how the alarm above gets muted. Unclaimed likeskipped_unbillable, so setting the env var mid-window lets a later day-1-to-5 run bill the same period.overage_prices_missing—string[]of the unset env var names (STRIPE_EMAIL_OVERAGE_PRICE_ID/STRIPE_SMS_OVERAGE_PRICE_ID), present on every run so a green one still states this deployment's stance. Mirrorsmeters_missing.backfilled_prices— count of metered subscription items this run created on subscriptions Checkout never gave them to. Counted per (org, channel) item, so one repaired pre-Checkout subscription contributes 2. Never a failure;log.warn("cron.overage.prices_backfilled", …). An item already reading the same meter is not counted here — that reportsalready_attachedand nothing was created. Expected to drain to zero; a non-zero count long after the backfill has drained means subscriptions are still being minted without the prices.
Telling skipped_no_overage_price from skipped_unbillable is the whole point of the split: "this deployment never configured overage pricing" and "this subscription should have billed and could not" used to be the same silent success.
Plan-history fields — no org is ever dropped silently:
excluded_by_plan— every org that had prior-month volume above its own period plan's allotment and was deliberately not billed, each with itsreason, theplans_heldduring the period, and thenotional_*_overagethat allotment implies. The yardstick is the plan the period resolved to, not Pro's: Pay as you go has an allowance of 0 by design, so pricing it at Pro's 10,000 scored a full month of billable PAYG volume as zero and dropped the org out of this list entirely. Plans with an unlimited allowance (enterprise) have no arithmetic overage and keep the Pro yardstick, since the point there is only to record that real volume went unbilled by policy. Logged atwarnascron.overage.excluded_by_plan_history.mid_period_changes— orgs that were billed, but whose amount came from the highest-entitlement rule rather than a measurement. Traceable when an invoice is disputed.assumed_plan_from_current— count of orgs priced off the no-history fallback.comped_skipped— orgs skipped before any pricing because their owner is comped, listed with the volume the platform absorbed. Logged atinfoascron.overage.comped_orgs_skipped. The comp is read off the org's current owner, so transferring an org to a comped owner carries the skip backwards over a closed month — which is why the absorbed volume is listed per-org rather than counted away. Not aPeriodPlanReason: the skip happens beforeplanForPeriod()is reached, which is precisely what keeps a comped org's volume from reading as a leaked Free cap. A comp does not erase a ledger row an earlier run already claimed for the period (possible when an org is comped mid-window, days 1-5): apendingrow still raisescron.overage.stuck_pendingand fails the run, because Stripe may already have metered the customer. Afailedrow is deliberately not retried — a comped org is not billed — and re-surfaces inunresolved_prioronce the period rolls.
These do not make the run non-200: an enterprise-era month genuinely has no overage, and 500-ing on it every month would train everyone to ignore this cron. The two reasons that are anomalies rather than policy — hard_capped_free and unrecorded_plan_change — additionally raise cron.overage.plan_history_anomaly through logError, which is what to alert on. hard_capped_free is held to a tolerance of 100 units first: checkBatchUsage reserves non-atomically, so a batch admitted just under the Free cap can land a whole batch past it, and an alarm that fires on that routine overshoot is one people learn to close.
Env required:
STRIPE_EMAIL_OVERAGE_METER_EVENTSTRIPE_SMS_OVERAGE_METER_EVENTOVERAGE_LEDGER_FIRST_PERIOD
Set the two meter names in Stripe dashboard → Billing → Meters → Event name. A missing one does not short-circuit the run — the totals are still computed so ops can see the magnitude going unbilled — but it raises cron.overage.meters_misconfigured, is named in meters_missing, and puts every overage on that channel into skipped_no_meter, which fails the run.
Also read here, and optional:
STRIPE_EMAIL_OVERAGE_PRICE_IDSTRIPE_SMS_OVERAGE_PRICE_ID
These are the metered prices the meter events actually bill through, and the cron reads them as well as Checkout does: it attaches a missing one to the customer's live subscription before metering (backfilled_prices), and refuses to meter at all when it cannot (skipped_unbillable). Leaving them unset is a supported flat-rate deployment, not a misconfiguration — the overage is reported under skipped_no_overage_price, the env var names come back in overage_prices_missing, and the run stays green.