Request lifecycle: life of a generation

Deployed This is the human-in-the-browser consumer path: one prompt in, one generated master out. Between those two moments about a dozen services authenticate the caller, charge tokens, queue a job, run a model, stream live progress, store the result, and reconcile the bill. Read this once and the rest of the Architecture section slots into place.

The agent (LLM) path is a second pipeline that re-runs the same bill-and-queue logic inside core-mcp — covered on its own page. Everything below is the browser-driven path through core-gateway-consumer.

The sequence

Follow the autonumbered arrows in the diagram; the prose below is grouped by the same numbers.

sequenceDiagram
  autonumber
  actor U as User (browser)
  participant FE as saas-2.0 BFF
  participant GW as core-gateway-consumer
  participant ID as core-identity
  participant DB as core-database
  participant R as Redis
  participant W as Tool worker
  participant ST as core-storage
  U->>FE: send prompt (POST /api/…)
  FE->>GW: CreateGeneration (Bearer token)
  GW->>ID: POST /v1/auth/validate
  ID-->>GW: CallerIdentity (server-derived user_id)
  GW->>GW: gate license · rate-limit · price · reserve
  GW->>DB: QueueTask (status = queued)
  GW->>R: publish {env}:common:task:wake
  GW-->>FE: task_id
  FE->>GW: StreamGeneration (SSE)
  W->>DB: ClaimNextTask (FOR UPDATE SKIP LOCKED)
  W->>R: XADD per-task event stream (progress)
  R-->>GW: read stream
  GW-->>FE: SSE StreamGenerationResponse (relayed)
  W->>ST: UploadFile (final master)
  ST-->>W: db_blob_hash
  W->>DB: CompleteTask (output + reported_usage)
  W->>R: XADD terminal event
  GW->>DB: settle (shortfall / refund) + MarkTaskSettled
  GW-->>FE: terminal event + output

Walk the sequence

1 — Browser to BFF

The user submits a prompt in the Next.js app soundverse-saas-2.0. The request hits that app’s own server-side BFF (Backend-For-Frontend) — Next.js route handlers under /api — not the gateway directly. The BFF holds the user’s session, attaches the Logto access token, and speaks gRPC to the backend on the browser’s behalf. The browser never receives a raw gateway credential.

2 — BFF to gateway

The BFF calls CreateGeneration on core-gateway-consumer over the consumer/v1 contract (consumer.proto), carrying the user’s token as a Bearer credential. The consumer gateway is the trust boundary — the only backend service the outside world is allowed to reach.

3–4 — Validate identity (never trust the client)

The gateway hands the token to core-identity, whose POST /v1/auth/validate verifies it against Logto’s JWKS and resolves it to our internal user_id. If the token is valid but the user is new, identity provisions them just-in-time and grants a starting token balance. See Identity & auth for the JWKS + JIT details.

Identity is server-derived, always

The caller’s user_id comes only from the validated token, never from a field in the request body. A client cannot ask to act as another user — the gateway discards any client-supplied identity and uses what identity returns. Workspace, project, and file ownership are checked the same way.

5 — Gate, rate-limit, price, reserve

Still inside the gateway, the enforcement pipeline (app/enforcement/pipeline.py) runs, in order:

1. Resolve entitlements + gate the license — the requested license is checked against the user’s subscription plan (fail-closed to FREE) before any hold is taken.
2. Rate-limit — resolved per-tool/per-model limits, scaled by the plan multiplier.
3. Price — FetchResolvedPricing produces an estimate of the token cost. A missing pricing row surfaces as ToolNotPriced here, mirroring core-mcp; an explicit zero-cost price is a real row and resolves normally.
4. Reserve — the estimate is held (deducted) from the user’s balance up front, keyed by a billing idempotency key (the per-call call_id) so a retried request can’t double-charge. If the balance is short, the call fails with InsufficientFunds.

FREE must be an explicit price row

Pricing is a lookup, not a default. A tool with no pricing row makes step 3 return NotFound and every call fails — declare cost_base = 0 for FREE. See Add pricing to a tool and the tool pricing model.

6–8 — Queue and acknowledge

The gateway calls QueueTask, inserting a row into the generation.tasks Postgres table with status = 'queued' — the queue is a database table, not a separate broker (see Task queue on Postgres). The QueueTask request carries the durable billing context (reserved_token_amount, reservation_ledger_entry_id, billing_idempotency_key) plus a w3c_traceparent so the worker’s task.run span parents to this request and the whole generation is one connected trace in SigNoz.

Immediately after the row commits, the gateway publishes a tiny message on the env-namespaced Redis channel {env}:common:task:wake (constant WAKE_CHANNEL_SUFFIX = "task:wake") to nudge idle workers awake. It then returns a task_id to the BFF — nothing blocks on the actual generation.

The wake is best-effort; polling is the correctness floor

task:wake is a lossy pub/sub push published after QueueTask commits. Workers also poll ClaimNextTask on a jittered interval, so a dropped wake just falls back to the next poll. This is what makes the wake deploy-order-independent. See Task queue on Postgres.

9 — Open the stream

The BFF re-connects with StreamGeneration, and the gateway relays everything as Server-Sent Events (SSE) to the browser. The long-running work and the live updates are decoupled: the work happens on a worker; the gateway just tails an event stream. On connect the gateway synthesizes the current DB-authoritative status so a mid-flight or already-done task still shows correctly.

10 — Worker claims the task

A tool worker — built on the soundverse-py WorkerFleet — wakes and calls ClaimNextTask (task.proto). The claim is an UPDATE … WHERE id = (SELECT … FOR UPDATE SKIP LOCKED): FOR UPDATE locks the chosen row, SKIP LOCKED lets other workers grab different rows concurrently, so a hundred workers never fight over one task. The same statement sets claim_expires_at = NOW() + INTERVAL '2 minutes' — a lease the worker must renew with HeartbeatTask or the task becomes re-claimable by a zombie sweep.

11–13 — Run and stream progress

The worker calls its upstream provider (an LLM via litellm, or a model service like Sansaarm) and reports progress by XADD-ing events onto a per-task Redis stream. Its TaskContext exposes the friendly helpers — ctx.emit_progress(pct, msg), ctx.set_streaming_url(url), ctx.emit_partial(data), ctx.log(msg). The gateway reads that stream and forwards each event as the SSE the browser is already listening to.

14–17 — Upload and complete

When the master is ready the worker streams it to core-storage via UploadFile, which content-dedupes the bytes into Azure Blob Storage and returns a scoped blob hash — db_blob_hash = sha256("{raw_file_hash}:{container_name}"), so identical bytes in different containers stay distinct (see Storage & media plane). The worker then calls CompleteTask with the output and its reported_usage (the real cost), and writes one terminal event onto the Redis stream.

18–19 — Settle the bill

On seeing the terminal event, the gateway runs settle_terminal_task: it re-prices the reported_usage, compares it to the reserved estimate, and either deducts the shortfall or refunds the surplus, then marks the task settled with MarkTaskSettled. Settlement is idempotent — the token ledger has a unique idempotency index, so running it twice is a no-op. The user sees the final output and a corrected balance.

Two reconcilers back this up

If the browser closes before the terminal event arrives, who settles? Background reconcilers in both the gateway and core-mcp periodically sweep terminal-but-unsettled tasks (behind a grace window so the live path gets first crack). This makes settlement at-least-once; the ledger’s idempotency guard makes it effectively exactly-once. The trade-off — two un-coordinated sweepers plus two copies of the billing pipeline — is tracked as known debt. See Billing reconciliation and the token ledger & money path.

Why it’s shaped this way

• One door to the data. Every QueueTask / ClaimNextTask / CompleteTask / MarkTaskSettled goes through core-database — the sole Postgres-credentialed service. No gateway or worker holds a DB password.
• Money before work. Reserve is a real up-front deduct; settle only true-s-up the difference. A user can never run inflight jobs they can’t pay for.
• Long work is decoupled from the stream. The gateway never runs a model — it queues, tails a Redis stream, and relays SSE. Workers can crash and the lease + reaper recover the task; the browser can disconnect and a reconciler still settles.

Related

• System overview — the full hero architecture this path lives in
• The agent + MCP path — the second billing pipeline
• The token ledger & money path — reserve / settle in detail
• Task queue on Postgres — SKIP LOCKED, leases, the task:wake nudge
• Identity & auth — Logto validation + JIT provisioning
• Billing reconciliation — the unsettled-task sweep
• Glossary — BFF, reserve/settle, lease, SAS, and the rest