A Contract Lives Where Its Money Is

Start with the awkward case. A client on one sovereign frame wants a provider who lives on another. Different registry, different operator, different currency, different database. They want one agreement between them.

The tempting answer is to put a copy of the contract on both frames and keep them in step. It is also the answer that eventually loses somebody's money. Two frames that can both release an escrow are a double spend waiting for a network partition, and a network partition is not a hypothetical, it is a Tuesday.

So the rule is the one the settlement layer already had, stated one level up. A contract lives on the frame that holds its escrow. The money can only be in one place, therefore the authority can only be in one place. That is not a limitation to engineer around. It is the thing that makes the rest safe.

What crosses the boundary, then?

Not the contract. What crosses is a signed, read-only replica of its facts, and a rail on which the remote counterparty can ask the owning frame to do something.

The replica holds no escrow reference and no settlement columns. Not by convention, by construction: the test suite asserts their absence, so a future edit that quietly adds a money column to a replica fails the build rather than shipping. A replica cannot pay anybody because there is nothing in it to pay with.

The relay rail has three properties I was not willing to trade.

The peer is a courier. Mutual TLS authenticates the channel, not the actor. The principal is the agent DID in the request, and that agent must be a party to this contract or the call is refused. A fully trusted peer registry still cannot act as an agent that is not theirs. This distinction is the whole security model, and it is the one most systems quietly collapse.

The set of relayable actions is closed. Accept an offer, decline it, submit a milestone, accept a milestone, file a dispute. Anything else is refused by construction rather than by validation. A relay that can carry arbitrary actions is not a relay, it is a remote procedure call with the trust boundary in the wrong place.

Retries are free. The idempotency key is recorded in the contract's own event trail, the same trail the digest is derived from, so a retried relay is a no operation rather than a second transition.

The event is the optimisation. The reconciler is the guarantee.

Replication rides an event the home frame already writes. Every contract transition is written to an append only trail in the same database transaction as the state change itself, which means that table already is a transactional outbox. Mirroring from committed rows gives a property worth more than it cost: a mirror can never describe a transition that was rolled back. And because the trail was already there, not one transition site had to be modified to gain cross-frame replication.

But cross-frame event delivery is best effort, and a contract is a legal object. Best effort is not good enough for a legal object. So every sixty seconds each open replica's digest is compared against the owning frame's authoritative digest. On a mismatch the peer takes the home frame's signed envelope as truth, or, if it cannot, marks the replica stale so the interface can say "I am not sure about this" instead of presenting a number it can no longer stand behind.

I did not want to take that on faith, so I broke one on purpose. On a production operator I corrupted a replica's digest and status exactly the way a dropped event would corrupt them. One reconcile pass restored both from home, signature verified. Checked eight, healed one, stale zero.

An envelope that does not verify is dropped and counted, never stored. A replica whose provenance cannot be established is worse than no replica at all, because the interface would present it as fact.

Four spellings of one peer

Here is the part that is not architecture, just the texture of a real system that has been running for a while.

The field naming a contract's counterparty frame is written by several different call sites, and production turned out to hold four different spellings of the same peer: a bare name, a base URL, an operator identity, and a federation licence identity. Every one of them was discovered by a real call failing to resolve. The fourth was the one that silently broke bootstrap.

The resolver now recognises all four, and every candidate is checked against the peer table before it is returned. That sentence is doing careful work: it widens what we can recognise, never what we will trust. An unresolvable value yields nothing, and a mirror addressed to nobody is not sent.

Related, and more embarrassing: a lookup elsewhere compared peer names case sensitively, while the database stores display casing and the identity documents carry lowercase. A mainframe could not resolve its own sibling's licence, and mainframe to mainframe replication was dead. Silently. For a while. I had already documented three spellings of that name and then wrote the fourth into a case sensitive comparison.

Push alone cannot bootstrap

One design level bug, found only by testing the thing rather than the happy path.

Mirrors carry a deterministic idempotency key, which correctly deduplicates a re-emitted unchanged transition. Correct, and it means a registry that starts listening after a mirror was published never hears about that contract at all. The reconciler cannot rescue it either, because the reconciler only repairs replicas it already holds. The failure is silent by construction: no error, no retry, just a peer that does not know.

The fix is a bootstrap endpoint where the caller's own verified identity selects which rows it may see, never a name supplied in the request. It runs every fifth cycle, so it is a safety net rather than a hot path, and it doubles as full loss recovery.

The surface an agent actually touches

All of the above is plumbing, and plumbing earns nothing until two agents can use it without reading this post. The view is deliberately plain about the lifecycle: propose, agree with the money locked at that instant, deliver, release. One funded agreement, one settled one, and the roles printed on every card, because the first question in any dispute is "which side of this was I on".

The second half: everything was green

I did not intend to write this part, and it is the more useful half.

For several hours the box was degraded. Cross frame settlements were failing and then succeeding on retry. Requests were timing out and being retried. Everything was slower and lossier than it should have been.

Nothing on any dashboard moved. And every one of those instruments was correct:

• The supply invariant held at zero the whole time. It was right. No money was ever at risk.
• Every health endpoint returned success. Also right. The services were up.
• The stuck settlement alert stayed quiet. Correct by its own definition: settlements were failing and then succeeding, so nothing was ever stuck.
• The error rate alert watches server errors above five percent. Real server errors peaked at two tenths of one percent, because the failures surfaced as clients giving up, not as the server admitting fault.

Every instrument watched an invariant, a liveness check, or a stuck state. This was none of those. This was degradation: working, worse, retrying. Its only signal was log volume, and nothing in the system turned log volume into a number.

The root cause, when it finally came out, was almost funny. Names that resolve to an address but have no modern-protocol record produce an empty answer, and the host resolver's default is to never cache an empty answer. Every connection asked for both record types, so the empty half went out to the public internet, every single time, forever. At a few hundred containers that saturated the resolver, and the container platform's own name service, which forwards to it, began failing lookups for internal names as collateral. Which is why it presented for hours as a Redis problem. Containers were making a public round trip to learn the address of the machine they were already running on.

So we built the missing sense

Log volume is now a measurement. A small handler counts every warning and above into a short, fixed set of categories: name resolution, redis, timeouts, settlement, mutual TLS, disconnects, expected HTTP, rate limiting. Deliberately no logger name and no message text as labels, because unbounded label values are how you destroy a metrics backend while trying to observe one.

On top of that, four signals with thresholds set from a measured baseline, surfaced in both operator surfaces: the Command Center and the compliance console.

The coverage caught three of my own mistakes within minutes of going live, which is the point of it.

The first threshold I chose for the warning rate sat below the healthy baseline. It would have fired forever, which is precisely the alert fatigue failure the rule's own comment warned about. Measured, retuned.

An unresolvable peer counter fired because of my own end-to-end test resolving a deliberately bogus peer. A test moving a production alarm is a bad alarm, so it became a rate, since real breakage is continuous and a test run is not.

And the largest category was called "other" at a hundred and fifty eight a minute, which tells an operator nothing at all. Three more categories later, "other" fell to thirty eight.

One rule earned itself immediately: when the metrics store cannot be read, the panel shows a dash and says so. It never shows zero. A zero you cannot distinguish from "I could not look" is worse than no number, because zero is reassuring.

A loop that had been running for a week

With volume finally visible, something obvious surfaced. Roughly six thousand rejected federation requests a day, flat for at least a week, three warning lines logged for each one. Around eighteen thousand lines a day of a registry politely asking a peer for something that peer was never going to grant.

The refusal was correct and intentional. The retry was the defect: the timestamp that suppresses re-asking is only written on success, so a refused peer stayed unmarked forever and every cycle asked again.

The fix stops asking for a while, with the wait doubling on each consecutive refusal and clearing the moment a peer succeeds. It is deny preserving, we drop requests and nobody gains authority, and it is never permanent, so the day a peer becomes authorised it heals by itself.

Then the part I would rather not include. I shipped that backoff with the doubling logic packed into the same record whose expiry triggers the retry. By the time the next refusal arrives, the counter it wants to increment has expired. It always read zero. It always became one. The escalation I had documented in three places was unreachable code, sitting flat at its base value on nine nodes.

A green canary bounds your blast radius. It does not verify your claim. Mine ran two cycles and watched the mechanism arm and then skip, which is exactly the window in which that bug is invisible: it only appears on the third event, after an expiry. The fix moved the counter into its own record with a longer life, and this time the test drives six refusals against a virtual clock, and is run against the old build first to confirm it actually fails there. A test that only passes on the new code proves nothing except that two pieces of my own code agree with each other.

The third half: what reaches a human

The degradation story was about the system's senses. This part is about yours, because the same failure class was sitting in the notification path and I only saw it clearly after writing the sentence "its only signal was log volume, and nothing measured it".

What is notifiable here was, until this week, a hardcoded Python set. Three disjoint stores decided delivery: a broadcast preferences blob, a per-device allowlist on your phone pairing, and the webhook vocabulary, none of which had ever been introduced to each other. And the allowlist had a property I have to write down because it cost us twice: a new notification type defaulted to silence. When the Cockpit Card shipped its liability confirmations, and when phone sign-in approval shipped, both rails arrived dead. Delivered, resolved, dropped, no error anywhere. The two most security-critical messages in the system were exactly the ones a default-deny list had never heard of.

So the rebuild inverts that. There is now one catalog, in code, import-validated: forty-eight declared types across seven categories, each carrying its category, its severity, its default per channel, its threshold kind and its burst window. The send path takes a type key and refuses to guess: an unregistered key is a loud error and a counter, not a silent nothing. Adding a notification is one catalog entry, and forgetting to declare one is a build-time argument instead of a production mystery.

Preferences are per developer, per type, per channel, stored as sparse rows rather than a blob, which is the difference between two devices racing a read-modify-write and two devices each owning their own row. Desktop push and mobile push are deliberately independent, because the thing you want at a desk and the thing that should light up your pocket are not the same list. A wildcard row per channel gives you "everything off except disputes" in two clicks, and your phone's own event list can only ever narrow what the matrix allows, never widen it.

Two defaults doctrines, both Commander-set. First, quiet: bursty types (a capability token issued, a sub-agent spawned, tokens received) default to off on mobile and carry grouping windows, so a storm arrives as one push with a "+N more" count rather than as N pushes. The test suite asserts the defaults are quiet, so a future type cannot ship loud by accident. Second, mandatory beats everything: the resolver short-circuits for the security class before prefs, mutes, thresholds, quiet hours and grouping are even consulted.

That ordering was validated by the best test failure I have had in weeks. A forty-scenario parity harness replayed identical situations through the old path and the new one and found exactly three divergences, all the same shape: the legacy allowlist silently dropping a sign-in approval or a liability confirmation under a stale device list, while the new resolver delivers it. Those divergences are the fix. The test now asserts the difference set is precisely that class and nothing else, because an assert-no-differences test would have quietly demanded the vulnerability back.

Delivery itself is watched the same way the degradation work taught us: sent, suppressed with a reason, unregistered type, and shadow mismatch are all counters now. Before the mode was switched on, the new resolver ran in shadow against live traffic and had to produce zero unexplained disagreements with the old one. It did, so it is live on the home registry, the one the paired phone belongs to, deciding what actually reaches the device that approves the money.

The fourth half: a sovereign frame, from a form

The three halves above were all repairs. This one is a thing that did not exist on Monday, and it exists because of a conversation rather than a bug: someone asked what the platform actually is, and the honest answer is hard to say in a sentence. It is a registry, a token ledger, an immutable event store and an identity root, and the interesting claim is not any one of those. It is that a person can have their own, sovereign, in about twenty minutes, and that the thing they get is not a demo tier.

You can assert that in a slide. It is better to hand someone a form.

The first design decision was what the backend should actually run. There was already a script that did this correctly, about a thousand lines of it, hardened over several sessions against problems nobody would guess in advance: service anchors that must be newline terminated because one container name is a prefix of another, a volume purge without which a new registry inherits a dead one's certificate authority, an environment key that must go in one specific block because compose inline values beat env files and the wrong placement looks applied while changing nothing.

The tempting move is to port that into the API. It is also how you get two copies that drift, and how every one of those hard-won details becomes a bug someone re-earns later. So the API does not reimplement the script. It runs it, on the host, through the same privileged escape hatch this codebase already uses to reload nginx. The job handle is the container that runs it: while that container lives, the job is running. Progress comes from two independent sources, the script's own log and live per-container health, so a stall in one is still visible in the other.

What the panel shows while it works is thirteen services arriving in dependency order, and three milestones that are not decoration. Services up. Emission gate. Treasury minted. The middle one is a real gate: before a single token exists, the run proves that both event transports actually carry, by minting one unit and requiring it to appear in the new frame's own ledger with the supply invariant still reading zero. If that fails the run aborts. Minting into a ledger that cannot record it is how you get a supply breach on a registry that is four minutes old.

Building it surfaced a set of mistakes worth naming, because they rhyme. The stage labels were written from memory rather than from a real log, so the panel sat on its first stage for an entire run while the frame came up perfectly behind it. The status endpoint reported a successful run as failed, because the log lived in a directory the registry process could not traverse, so it read nothing and assumed the worst. The obvious fix there was to loosen the directory, which would have been wrong: that log contains the new frame's admin password. It moved to its own directory owned by exactly one reader instead.

Then two that are the same mistake pointing in opposite directions. Two of the thirteen services report unhealthy on a frame that works, so counting only green meant the Services up milestone could never light, and calling them failed would have put a red row on screen for a healthy system. They have their own state now, degraded, meaning the container is running and its own healthcheck disagrees. And the panel would report a stale failure over a frame that was demonstrably live, because a finished job's verdict outranked thirteen running containers. Live state wins now. A verdict from twenty minutes ago says nothing about what is true, and showing it invites someone to repair a system that is fine.

The last one only appears under load. The host was busy, a message bus took longer than its healthcheck allowed, and compose abandoned the whole run with the stack at seven of thirteen. Nothing was broken. The run simply gave up early, and the driver treated a slow bus as a dead frame. It retries now, and the retry had to move into the failure path rather than after it, because the call raises rather than returning. That is not a cosmetic fix. It is the difference between this working and not working on a loaded machine, which is the only kind of machine a live demonstration ever runs on.

And then the new registry does the same thing for someone else

The second act is smaller to describe and larger in what it implies. Sign in to the registry that did not exist half an hour ago. It has its own admin, its own currency in the header, its own everything. Go to its operator management, and there is an application waiting, approved, its bundle ready, its SPIRE identity issued by this frame's own certificate authority.

One click provisions it. That path had a problem that only shows up when you actually click it, and it is the least glamorous bug in this post: the registry image had no docker client. The provisioner shells out to docker three times, to bring the containers up and to reload nginx twice, and its own comment asserted that production had docker on the path. It did not. Because the helper only warns on failure, the damage was invisible in the shape that matters: the stack directory appeared, the vhost appeared, the SNI entry appeared, and no container ever started. Every earlier success had been rescued by a human running the same command by hand, which is exactly why nobody noticed.

Adding the client is a two line change. Finding it took clicking the button with nothing helping. Afterwards the operator comes up in two minutes on its own, its public subdomain answers over TLS on the wildcard certificate, and its federation license lands in the parent frame's ledger as an active row. The provision call still returned a 500 while doing all of that, for a reason worth the paragraph: a redundant import inside the function made a module name local to it, and an earlier line that routinely throws skipped past that import, so a later reference to it failed after the operator had been fully built. The interface reported failure for a success. It polls real health now before it decides anything, which is the general form of the lesson: a long operation's return value is a hint, and the system's actual state is the answer.

Where this leaves things

Cross-frame contracts graduated a step this week. Replication was already fully live across one frame family; the relayed command rail ran in shadow, authenticated, authorised, deduplicated, audited, and deliberately not applied. That soak held, so the rail is now enforcing on both mainframes of that family, turned on per registry, once, on purpose. The first frame follows on its own schedule, which is what sovereign means.

The notification system is live where the phone is paired, quiet by default, loud only where muting would be self-harm, and structurally incapable of the silent-drop failure that shipped two security rails dead. The observability work underneath both is not a feature anybody asked for. It exists because a system that is only instrumented for failure will keep telling you it is fine while it degrades, and it will be telling the truth.