The contract: proto as source of truth

This is the load-bearing idea of Soundverse 2.0 Deployed. There is exactly one source of truth for every wire shape and for every SQL-backed data service: the soundverse-proto repository. You write .proto files, annotate the data-plane RPCs with SQL, and a custom toolchain generates the per-language SDKs plus the entire Go Connect service that talks to Postgres. Nobody hand-writes a data service. Get this page and the rest of the codebase reads itself.

This is the explanation — the “why” and the shape. For the step-by-step recipes see the how-to pages linked throughout and in Related.

Why a single contract

The backend is a contract-first polyrepo: ~20 submodules in Go, Python, TypeScript, Swift and Kotlin, all speaking the same wire types. A shared shape that lives in more than one place drifts, so it lives in exactly one — the proto repo, mounted as the soundverse-proto/ submodule (it tracks the staging branch). Two very different things flow out of it:

The wire types

Every request/response message, enum and service, generated once and published to a per-language repo. A field renamed in the proto is a compile error in every consumer.

The data plane itself

Annotate a data RPC with the SQL it runs, and the toolchain emits the whole Go Connect handler — pool selection, $N binding, row scanning, Redis caching, error mapping. See How the DB codegen plugin works.

Generated code is published, never committed

Everything the toolchain emits lands under gen/ and is .gitignored. It is pushed to the downstream language repos on release, so this repo only ever contains hand-written .proto, schema.sql, and the two codegen programs. buf breaking is intentionally disabled during the startup phase — the contract may reshape freely until GA.

The proto layout

Every package is named soundverse_proto.<domain>.<...>.v1, one directory per domain under proto/soundverse_proto/. Two directories are special:

• Directoryproto/soundverse_proto/

  • Directorycommon/v1/ shared RequestContext (embedded in every request) + ErrorDetail

    • …
  • Directorydb/v1/ db_options.proto — the custom extensions that drive codegen (the keystone)

    • …
  • Directoryidentity/v1/ users, links, identity data service

    • …
  • Directorygeneration/

    • Directorytask/v1/ the async generation queue — the canonical annotated example

      • …
    • Directorytool/v1/ tool registry, credentials, rate limits, pricing

      • …
  • Directorybilling/tokens/v1/ token balances + the ledger

    • …
  • Directorystorage/v1/ · chat/v1/ · consumer/v1/ files/blobs · chat · consumer reads

    • …
  • … 17 domain packages in total (studio, collabstudio, migration, compat/*, tests/*, …)

The full package/table catalog lives in the Proto contract reference.

SQL-in-proto: the keystone

All of codegen is driven by a handful of custom proto extensions declared in db/v1/db_options.proto. Method options describe an RPC; field options map a message field to a SQL column. A data RPC is fully specified by its annotations — the plugin needs nothing else.

Annotation	Extends	Option #	Role
target_db	method	50001	Pool key (e.g. "prod") selecting which pgxpool.Pool runs the query
sql_query	method	50002	The literal SQL the handler executes (explicit columns, no SELECT *; mutations need a WHERE)
cache_ttl_seconds	method	50003	>0 turns on read-through Redis caching of the marshalled response
db_column	field	50004	Maps a message field to a SQL result column for scanning
require_internal_auth	method	50005	Reject any bearer token ≠ the internal secret with CodeUnauthenticated
cache_key	method	50006	Key template like tools:active:{environment}; {name} binds to request fields
invalidates_keys	method	50007	Repeated key templates the handler DELs after a successful write
db_enum_text	field	50008	Enum request field targeting a Postgres ENUM: send the lowercase label, not the int

From those, cmd/protoc-gen-soundverse-db emits a complete Connect handler per data service. It fails the build on a dangerous or inconsistent annotation (no SELECT *, no WHERE-less mutation, every projected column must have a matching db_column). The full handler walkthrough is in How the DB codegen plugin works.

The $N placeholder rule is the fleet’s sharpest footgun

The plugin binds request fields to $N 1-based over the scalar/enum fields only — the leading common.v1.RequestContext (and any message-typed field) consumes no placeholder. A sql_query that assumes context is $1 is off-by-one and fails at runtime, not at build time. This once silently broke tool registration fleet-wide. Same rule: never nest a filter/page message in a List* request — put the cursor as flat scalars. Details in Add a field or RPC to a data service.

schema.sql files sit next to the protos they back (e.g. generation/task/v1/schema.sql). They are hand-maintained reference DDL — not generated, not auto-applied, there is no migration runner. Four things must always agree: the sql_query column list, the db_column field annotations, the schema.sql table, and the request fields bound to $N.

The four generation steps

Generation runs through four buf templates, in this exact order (it mirrors ci.yml). The first three are buf generate runs; the fourth is a plain Go program.

#	Template / command	Emits
1	buf.gen.yaml	Standard SDKs — Go (protocolbuffers/go + grpc/go), Python (+ mypy stubs), Swift, Kotlin
2	buf.gen.connect.yaml	Connect Go handlers (connectrpc/go) + Connect-ES TypeScript clients (bufbuild/es, protobuf-es v2) into soundverse-proto-web
3	buf.gen.soundverse-db.yaml	The custom DB plugin ./bin/protoc-gen-soundverse-db → one *.soundverse_db.go per data service
4	cmd/gen-db-registry (go run .)	Scans the generated files → registry/db_registry.go with one RegisterAllDBServices(...) mount function

Build the DB plugin before step 3

buf.gen.soundverse-db.yaml references a local binary at ./bin/protoc-gen-soundverse-db. If you skip building it, step 3 fails. The exact ordered commands are in Run proto codegen locally.

The cascade: one edit fans out

A single .proto edit propagates across the fleet along two lanes with very different ergonomics — automatic for Python, manual for Go.

flowchart TD
  P["soundverse-proto<br/>.proto + DB annotations + schema.sql"]

  P -->|"buf.gen.yaml"| SDK["soundverse-proto-go / -python<br/>-swift / -kotlin"]
  P -->|"buf.gen.connect.yaml"| WEB["soundverse-proto-web<br/>Connect-ES TS clients"]
  P -->|"protoc-gen-soundverse-db<br/>+ gen-db-registry"| DBH["*.soundverse_db.go + db_registry.go<br/>(in soundverse-proto-go)"]

  SDK -->|"proto_updated dispatch<br/>uv.lock bump PR"| PY["Python consumers<br/>gateways · identity · storage · soundverse-py"]
  PY -->|"soundverse_updated dispatch<br/>2nd hop"| WK["tool workers<br/>core-tool-*"]
  WEB --> FE["soundverse-saas-2.0"]
  DBH -.->|"MANUAL: go get @staging + go mod tidy"| GO["core-database · core-mcp"]

  classDef gateway  fill:#0ea5e9,color:#fff,stroke:#0369a1
  classDef data     fill:#8b5cf6,color:#fff,stroke:#6d28d9
  classDef worker   fill:#10b981,color:#fff,stroke:#047857
  classDef frontend fill:#6366f1,color:#fff,stroke:#4338ca

  class P data
  class GO data
  class PY gateway
  class WK worker
  class FE frontend

Release lanes. Pushing to the proto repo’s staging branch publishes rolling dev builds to each language repo’s staging branch; pushing to prod cuts immutable versioned releases.

Python — two hops, automatic. The staging publish also fires a proto_updated repository_dispatch to the core* repos and to soundverse-py, each of which opens an update-private-deps PR that re-locks soundverse-proto in uv.lock. Merging the soundverse-py bump moves its main, which fires a second soundverse_updated dispatch to the tool workers (core-tool-* + template-core-tool) so they re-lock soundverse. So a contract change reaches a worker in two hops: proto → soundverse-py → worker.

Go — fully manual. The two Go consumers, core-database and core-mcp, are outside the auto-bump system. They pin a frozen go.mod pseudo-version and bump it by hand via each repo’s update-proto.sh (go get github.com/soundversegit/soundverse-proto-go@staging && go mod tidy).

The highest-fan-in service can run against a stale contract

Because Go is manual and buf breaking is off, core-database can lag the contract while the Python gateways have already moved ahead. Worse, the prod cascade is currently dead on both ends: the prod release fires proto_updated_prod, but every consumer listens only for proto_updated and the prod-deploy dispatch triggers are commented out — a prod contract change propagates to nothing until a human bumps each consumer. Treat prod proto rollouts as an ordered, manual runbook. Full mechanics: Drive the proto → deploy cascade.

Related

Add a field or RPC to a data serviceThe proto + schema.sql edit, the $N rule, and the four-things-must-agree.

Run proto codegen locallyThe exact ordered commands for the four buf templates.

Drive the proto → deploy cascadeTwo-hop Python auto-bump vs fully-manual Go, and the prod caveats.

Refresh proto stubsPull the regenerated SDKs into each consumer.

How the DB codegen plugin worksInside protoc-gen-soundverse-db and the generated handler.

Proto contract referencePackage dirs, all 8 annotations, and the DB-error → Connect-code map.