Billing reconciliation & the unsettled-task sweep

Every generation holds tokens up front — the reservation is a real deduction, not a soft hold — and reconciles that hold against actual usage when the task goes terminal (see the token ledger & money path). That final step is settlement: charge the shortfall, refund the surplus, or refund the whole hold on failure. The happy path settles inline, while the client’s live stream is still open. This page is about what happens when it doesn’t — the disconnects, worker deaths, and stuck queue rows that would otherwise strand a user’s tokens or freeze a chat message at “planning…” forever.

Settlement is idempotent at every layer, so the design is “at-least-once, from several independent places.” Three things can settle a task, and one watchdog forces stuck tasks terminal so they become settleable.

The three settlers

1. In-request settle (first crack)On terminal, the live-stream handler settles exactly once — core-mcp toolcall for agent calls, the gateway generation servicer for the consumer path.

2. core-mcp reconciler (Go)A 60s ticker that sweeps ListUnsettledTasks and settles/refunds any terminal-but-unsettled task the agent path missed.

3. gateway reconciler (Python)A 60s sweep that both settles (ListUnsettledTasks) and reaps (ListStaleTasks) — plus projects the chat message terminal.

The in-request path is the fast path. Both the core-mcp toolcall handler (internal/toolcall/handler.go) and the gateway generation servicer (app/grpc_server/servicers/generation.py) settle the moment the task they are streaming reaches completed/failed, then call MarkTaskSettled. The two reconcilers exist only to catch the tasks whose stream was never opened or was dropped before terminal.

Both reconcilers hit the same global list — and that's fine

core-mcp and the gateway sweep the same global ListUnsettledTasks, so in a full deploy two settlers race over the same rows. That is safe by construction: MarkTaskSettled runs WHERE settled_at IS NULL, every token mutation is keyed by a derived idempotency key, and DeductTokens/RefundTokens move the balance exactly once per key. Retrying a settle a hundred times — across replicas, across reconcilers — lands the same result.

The grace window

The reconciler sweep, ListUnsettledTasks, deliberately ignores freshly-finished tasks. Its SQL (in generation/task/v1/task.proto) returns only terminal tasks that are still unsettled and older than a two-minute grace window:

task.proto — ListUnsettledTasks

SELECT ... FROM generation.tasks
WHERE status IN ('completed', 'failed')
  AND reserved_token_amount > 0
  AND settled_at IS NULL
  AND completed_at < NOW() - INTERVAL '2 minutes'
ORDER BY completed_at ASC
LIMIT COALESCE(NULLIF($1, 0), 100)

That grace window is what keeps the backstop from racing the in-request path: a task that just completed is almost always about to be settled inline by the stream that is watching it. Only after two minutes with no settlement does a reconciler treat it as orphaned and step in. The window is a hardcoded SQL literal, not an env knob — only the sweep cadence is configurable.

What a settle actually does

Both reconcilers and the in-request path funnel terminal settlement through one shared routine — settle_terminal_task in the gateway (app/enforcement/pipeline.py), mirrored by settleOne in core-mcp/internal/reconciler/reconciler.go:

1. Branch on terminal status. completed → re-fetch pricing with FetchResolvedPricing, compute the actual cost from the worker’s reported_usage, and settle (charge the shortfall or refund the surplus vs. what was reserved). failed → refund_all (return the entire hold).

2. Mark it settled. MarkTaskSettled sets settled_at = NOW() WHERE settled_at IS NULL. The predicate makes it a no-op on the second (and hundredth) call, so a task can leave the unsettled list only once even if several settlers reach it.

The idempotency-key contract

The idempotency seed is the task’s own billing_idempotency_key (falling back to recon:<task-id> if the task never got one). Every money mutation derives its key by appending a fixed suffix: reserve uses the bare key, a settlement shortfall uses :settle-extra, a surplus refund uses :settle-refund, and a failure refund uses :refund-all. Go and Python append byte-identical suffixes, which is exactly why the two reconcilers can share one global work-list. The suffix catalog lives on the money-path page.

The stale-task reaper (the watchdog)

Settlement only runs on terminal tasks. A task that is stuck non-terminal — a queued row nobody claimed, a processing zombie whose worker died — never reaches the unsettled list, never refunds, and leaves its chat message frozen at RUNNING. The reaper fixes that by forcing such rows terminal so the settlement sweep can then refund them.

It runs only in the gateway (_reap_stale in app/grpc_server/reconciler.py) — by design there is a single reaper owner. core-mcp has a settlement reconciler but no reaper. Each 60-second gateway sweep does both: _sweep (settle) then _reap_stale.

The ListStaleTasks RPC returns three populations, each mirroring what ClaimNextTask can no longer reclaim:

Population	Predicate	Why it’s dead
Processing zombie	status = 'processing' AND retry_count >= max_retries AND lease expired past a grace (claim_expires_at < NOW() - 120s)	A worker died mid-task and no retry budget remains — no worker will reclaim it.
Unclaimable queued	status IN ('queued','dispatched') AND retry_count >= max_retries	Retries exhausted (e.g. a legacy max_retries = 0 row) — ClaimNextTask skips it forever.
Abandoned queued	status IN ('queued','dispatched') AND started_at IS NULL AND created_at < NOW() - 900s	Never claimed past a generous cap — no worker exists for that tool.

The 120-second dead-lease grace and the 900-second abandonment cap are the SQL defaults (the gateway calls the RPC with page_limit only, so the defaults apply). The cap is deliberately generous: a brief worker outage should self-heal via ClaimNextTask before a row is reaped.

For each stale row the reaper calls FailTask(increment_retry=False), which forces the row to terminal failed, then projects the now-failed state onto the chat message so the loader resolves. It does not refund directly.

The reaper never refunds — the next sweep does

_reap_stale only fails the row. The refund is deliberately left to the settlement sweep: once the task is terminal-and-unsettled it becomes eligible for ListUnsettledTasks and is refunded on the following pass (idempotent on the billing key). This split is what prevents a double-refund — the reaper never touches money, so a failed reap can’t leak or double-credit tokens. Cost: a reaped task’s refund lands one sweep later (≈60 s).

The reaper needs core-database to serve ListStaleTasks

ListStaleTasks shipped alongside the reaper. If the gateway is deployed against an older core-database that doesn’t serve it, _reap_stale gets UNIMPLEMENTED every sweep — the settlement path still works, but stuck-non-terminal messages never resolve. Deploy core-database first. Settlement (ListUnsettledTasks) is long-standing infra; the stale-task reaper is the newer half.

Zombies vs. reaping: the lease is the dividing line

A reaper is a last resort, not the first line of defence against a dead worker. The queue’s own lease reclaims most zombies without any human or reconciler: ClaimNextTask/HeartbeatTask stamp claim_expires_at = NOW() + INTERVAL '2 minutes', and ClaimNextTask will re-grab any processing row whose lease has lapsed as long as retry_count < max_retries. The reaper only sees a processing zombie once that retry budget is spent — i.e. the row is genuinely unrecoverable. See the task queue for the claim/lease/heartbeat mechanics.

Operator knobs

Only the sweep cadence is configurable; the grace/cap windows are baked into the proto SQL. Names only — never put a value in a doc.

Env var (NAME only)	Service	Default	Controls
RECONCILE_INTERVAL	core-mcp	60s	Cadence of the Go settlement reconciler (internal/config/config.go)
RECONCILE_INTERVAL_SECONDS	core-gateway-consumer	60	Cadence of the Python sweep + reap (app/core/config.py)

Fixed in task.proto (change the contract, not an env var): the 2-minute settlement grace, the 120-second dead-lease grace, and the 900-second abandonment cap.

Watching it work

Every sweep is its own OpenTelemetry span, so this background money-path work is never an orphan trace:

• Spans: reconciler.sweep / reconciler.Sweep (per tick), reconciler.settle_task, and reconciler.reap_task, carrying reconciler.settled_count and reconciler.reaped_count attributes. A persistently-failing settle lands on the span with ERROR status, not just in logs.
• Metrics: both services export the settlement_failure_total and refund_failure_total counters (dimensioned by tool_id/status, never user_id). A non-zero, non-decreasing refund_failure_total means a hold leaked and can’t be auto-retried — alert on it.

Query these in SigNoz — see observability in practice. A steady stream of reconciler.settled_count > 0 on every tick is a smell: the in-request path should be settling almost everything, so the backstop should usually settle zero. Sustained backstop settlements point at live streams that never open or drop early.

                          ┌──────────────────────────┐
   task goes terminal ───▶│  in-request settle        │  (first crack, exactly once)
   (stream still open)     └──────────────────────────┘
                                      │ missed?
                                      ▼
                          ┌──────────────────────────┐
   +2 min, still unsettled│  ListUnsettledTasks sweep │  core-mcp AND gateway, 60s
                          └──────────────────────────┘
   stuck non-terminal ───▶┌──────────────────────────┐
   (zombie / abandoned)   │  ListStaleTasks reaper    │  gateway only → FailTask
                          └──────────────────────────┘
                                      │ now terminal
                                      ▼
                                (next sweep refunds)

Related

• The token ledger & the money path — estimate → reserve → settle, and the full idempotency-suffix catalog
• Task queue on Postgres — SKIP LOCKED claiming, leases, heartbeats, zombie reclaim
• The agent + MCP path — the second (Go) billing pipeline whose orphans core-mcp’s reconciler catches
• core-mcp · core-gateway-consumer — the two services that run reconcilers
• Observability in practice — query the reconciler spans and money-path counters in SigNoz
• Proto contract reference — where the ListUnsettledTasks / ListStaleTasks SQL lives