Steadhold

The problem

Provisioning a database is a distributed transaction wearing a disguise. A single request has to reserve capacity on a node, create a volume, start a container, wait for the process inside it to become healthy, create roles and schemas, start a connection pooler and an API layer, mint keys, register a route, and enable backups. Ten of those eleven steps touch a system that can fail independently of the database row that claims they happened.

If the worker dies between starting a container and recording that it started, a naive retry starts a second one. Now the customer is billed twice, two processes hold the same volume, and the control plane believes one thing while the machine does another. This class of bug does not show up in development, because in development nothing dies.

The design

The API never provisions. It commits intent — a row and a job, in one transaction — and returns 202. Workers do the work, and every step is written as a question before it is written as an action: does this already exist? If it does, verify it and continue. If it does not, create it. That single rule is what makes the saga resumable, and it is enforced step by step rather than assumed.

Desired state lives in the control plane; actual state lives on the node as containers, volumes and routes. Only a reconciliation loop is allowed to close the gap between them outside a job, and it is deliberately asymmetric: a container that should be running gets restarted automatically, a container that should not exist gets stopped automatically, and an orphaned volume is only ever reported. Durability outranks cost, so nothing that holds data is deleted by a machine.

Proving it, rather than claiming it

The worker was killed with SIGKILL at each of the eleven steps, and each run was checked afterwards for duplicate containers, duplicate volumes, duplicate credentials and leaked capacity bookings. Twenty create-and-delete cycles leave nothing behind on the node or in the control plane. Rebooting a data node brings every project back to serving queries within seconds, and anything the node and the control plane disagree about is reported rather than silently corrected.

The same standard applies to backups. A backup that has not been restored is not a backup, so a job restores each one into a scratch cluster and checks four things: that recovery completed, that page checksums verify, that pg_amcheck passes on the indexes, and that row counts match what the live project says should be there — because a backup can pass every structural test and still contain an empty database.

To trust that verifier, it had to be made to fail: a byte range inside a real backup is overwritten with random data, and the verification is expected to reject it and name the check that caught it. A verifier that passes healthy backups and broken ones alike is worse than none, because it manufactures exactly the confidence it exists to earn.

Deletion, pausing and time travel

Nothing about a DELETE distinguishes “we are finished with this” from “I typed the wrong reference”, so deletion takes a final full backup first and refuses to proceed if it fails, then holds the data for a seven-day recovery window. Pausing has the same interlock for a subtler reason: a paused project has no running Postgres, so write-ahead log archiving stops with the container, and the most recent copy of a customer's data would otherwise be older than the pause itself.

Restoring to a point in time provisions a new project and replays write-ahead log into it up to the named second. The original is never touched and the copy is marked restored rather than ready, because two databases serving one application loses data by construction. Thirty days after a project is purged its backups are destroyed, and the sweep lists the storage prefix again afterwards — recording the destruction only when that listing comes back empty.

Who is allowed to do what

Projects belong to organisations, and three roles decide what each member can do: a member creates and pauses projects but cannot delete them, an admin does everything except delete the organisation or grant ownership, and no admin can strip an owner. Every mutating call writes a row to an append-only audit table, enforced by a database trigger rather than by convention, because revoking permissions does not bind a table's owner. A test enumerates every mutating route and fails if one of them is unaudited.

Each project gets its own ES256 keypair and publishes a JWKS endpoint, so a customer's services verify tokens without calling us. Database passwords and signing keys are envelope-encrypted, and API keys are stored as hashes — they are shown once and never regenerated silently.