The Agent Owns Its Card. Also Eight Worker Modals and Three Panic Variants.

2026-05-18 · 11 min read · ruFFa

The Worker Tasks tab inside Operations Reactor Console — eight Tier-3 worker cards under one panel, sovereign red theme, twenty-two background tasks tracked, four with active Redis leadership leases — The Worker Tasks tab inside the Operations Reactor Console. Twenty-two background tasks tracked. Four hold active Redis leadership leases at this moment. Eight of them have a dedicated modal underneath, because at some point during the last six days I decided that “observability” should mean “every worker has its own glass cockpit” rather than the more reasonable interpretation, “every worker emits one metric.” I am the only operator. The cockpits are for me. I built them anyway.

Six days since the last post. Three things to walk through and one thing that runs underneath all of them. The thing underneath is that the supply invariant is still at delta zero, the canary fleet is holding ninety-eight point six percent pass rate over the rolling twenty-four hours, and nine of nine active federation peers responded to the last sync cycle. None of that is new. None of that is mentioned again until the end.

The new parts, in order of how long they took: ten phases of an architecture migration nobody asked for, eight worker observability modals nobody but me will open, three competing designs for one avatar overlay that Commander — who is me — will pick between by feel, and twenty-four mTLS canary paths whose only purpose is to crash loudly if a SPIFFE rotation breaks while everyone is asleep. I will walk through them in that order. The reader is free to skim. There is a lot of it.

Option C — the agent owns its card now

Up until last week, the canonical source of truth for an agent’s capability descriptor — its “agent card” in A2A v1.0 terms — was the registry. The agent declared itself at bootstrap, the registry stored the JSON, federation propagated it, end of story. This worked fine until I started shipping sovereign agents like SYBIL and VERIDIAN that wanted to evolve their own metadata between deploys without round-tripping the registry every time. The A2A spec is clear about who owns this: the agent does. The registry caches. The agent is the source.

So I built it. All ten phases, in three sessions across two days. The SDK got a new helper serve_well_known_card() that mounts /.well-known/agent-card.json on the agent’s own FastAPI app with proper ETag and Cache-Control headers and a 304 short-circuit and an optional federated fallback if the agent wants the registry to be the canonical source after all. Two-hundred-eighty lines of SDK code, nine new tests, a wheel that was built locally and intentionally not published to PyPI yet because the JWS signing piece in Phase 9 should bake first. Six live sovereigns — SYBIL plus the five Wave-2 agents I shipped last week — were migrated. Their stored URLs were backfilled to match what their own well-known endpoints now serve. Every one of them returns a valid v1.0 card in under a hundred milliseconds.

The registry side grew a public /api/v1/agents/{did}/card-export endpoint for agents that want to use the registry as their canonical, and a background pull-sync worker that runs every six hours, picks the hundred stalest native cards (ordered by last_pulled_at NULLS FIRST, created_at, id — deterministic coverage, never-pulled rows surface first), fetches their well-known with the cached ETag, and merges any drift. The merge contract is precise: agent-owned fields overwrite from fresh, registry-owned fields are preserved verbatim from stored, and stored-but-not-fresh fields are dropped (tracked as field:removed drift). The registry-owned set is the three names federation_metadata, flare, and the pull-sync bookkeeping (last_pulled_at + last_pulled_etag); everything else is the agent’s. The 304-cached path short-circuits the merge entirely and stamps only the bookkeeping. The selection query filters natives via SQL with card_data->'federation_metadata'->>'source_registry' IS NULL so federated cards are never pulled — their authoritative source is the originating registry, not their well-known. The worker emits AgentCardPulled only when drift is actually detected, not on every cycle, because every-cycle audit events would bury the actual signal in 100 entries every six hours per registry across eleven registries.

I verified the merge contract end-to-end by manually injecting a synthetic flare into SYBIL’s stored card via SQL (UPDATE agent_cards SET card_data = jsonb_set(card_data, '{flare}', '{"tier":"founder","cause":"test-injection"}')), resetting last_pulled_at = NULL to force a fresh pull, and waiting one cycle. SYBIL’s well-known returned her real card (no flare — flare is registry-owned, agents do not serve it). The merge took fresh, preserved flare + federation_metadata from stored, dropped one optional pricing field that VERIDIAN’s well-known doesn’t declare, stamped last_pulled_at + last_pulled_etag. No drift event because the agent-canonical fields were unchanged. The first prod cycle pulled 100 candidates, hit 2 successful 200s and a wave of 4xx for agents without a public host yet; the second cycle pulled SYBIL once the watermark rotated. This is the kind of test where the work is the verification, not the implementation.

Phase 9.a is the part that will matter later. A new agent_signing_keys table (migration 0063, idempotent raw SQL) holds rotation history per agent with a composite primary key of (agent_did, rotation_seq), a public_jwk JSONB column, an algorithm string defaulting to EdDSA, created_at + expires_at timestamps where the latter is null for the active key and populated to NOW() + KEY_ROTATION_GRACE_DAYS the moment a rotation supersedes it. The existing public_key_jwk on the agents table got backfilled into rotation_seq=1 for all 178 production agents in the same migration; nothing was discarded. The SDK ships a sign_agent_card() + verify_agent_card() pair using detached JWS per RFC 7515 with Ed25519 signatures — the payload is the agent card JSON serialized with sort_keys=True and no whitespace so the bytes are stable across re-encodings, the signature is computed over base64url(protected_header) || "." || base64url(payload), and the resulting compact JWS is stored adjacent to the card rather than mutating it (hence “detached”). The verifier accepts multiple rotation_seq rows so a card signed with the previous key still verifies during the grace window, then fails the moment expires_at passes. Re-signing with the same kid is idempotent — same kid + same payload bytes = same signature, no churn.

The registry has a verifier wired into the pull-sync path that’s gated behind a three-flag environment staircase — JWS_VERIFY_ENABLED to turn it on at all, JWS_ENFORCE to make invalid signatures reject the pull instead of just logging, and JWS_REQUIRE_SIGNATURE to reject unsigned cards entirely. All three default to false. Shadow mode means the verifier runs and increments a agent_card_jws_verify_total{outcome,reason,source} Prometheus counter on every pulled card but nothing is rejected; the metric is observed for a multi-week bake. The staircase is intentional — flipping the first flag turns on observation, flipping the second escalates to rejection of invalid signatures while still accepting unsigned, flipping the third hard-requires every card to carry a signature. Each step is reversible by flipping the flag back. Real enforcement when the metric shape says it’s safe. A rotation admin endpoint at POST /api/v1/admin/agents/{did}/rotate-signing-key generates a fresh Ed25519 keypair via cryptography.hazmat.primitives.asymmetric.ed25519, bumps rotation_seq, expires the old key at NOW() + KEY_ROTATION_GRACE_DAYS (default seven), mirrors the new public_jwk back to agents.public_key_jwk for backward compatibility, emits an AgentSigningKeyRotated audit event, and returns the new private PEM exactly once in the response body with shown_once: true; subsequent reads return only the public JWK and the rotation metadata. The companion GET returns rotation history without ever exposing private material.

None of this changes any user-visible surface today. The user-visible result is that an agent can now ship its own card schema changes without a registry deploy, that federation keeps working, and that some day in the not-distant future every card in transit will carry a verifiable signature whose private key was never trusted to anyone but the agent itself. The chain of trust will run agent → signed card → SPIFFE-mTLS-verified peer → verified by every consumer along the way. This is the unglamorous part of building a sovereign-identity protocol. It is also the part that the protocol does not exist without.

AdminAgents got a compound Signals column and a More dropdown

The drift admin surface from Phase 6 originally shipped as a standalone view at /admin/agent-card-drift. It worked. It was visually crude compared to the existing /admin/agents shell. Commander — who is me — flagged this on day two. So the standalone view was scrapped, the backend endpoints were preserved, and the entire drift surface was folded into the existing AdminAgents view as a Card Sync drawer tab. Every agent row got two new tabs in its drawer: Card Sync (pull state + actions + side-by-side live diff) and Signing Keys (rotation history + rotate-now button).

AdminAgents Identity Control table — new compound Signals column showing four pills per row (mTLS / JWS key / pull state / drift count badge), Drift filter chip active, status strip showing eight drift events across two agents in the last twenty-four hours — AdminAgents with the new compound Signals column on the right — four pills per row instead of one. The old mTLS column was replaced by a four-pill cell showing `mTLS enrolled` / `JWS signing key present` / `pull state (fresh|stale|never)` / `drift count over 24h`. Each pill is a clickable button that opens the drawer at the matching tab. The status strip above the toolbar says “8 drift events across 2 agents (24h)” with a clickable filter link that applies the chip below it. The filter chip is the “Drift” pill. None of this exists if you have a userbase, because a userbase looks at metrics, not at admin tables. I do not have a userbase.

The bulk toolbar got a two-tier redesign as well. Tier one is the direct buttons: Suspend, Reinstate, Refresh Cards. Tier two is a “More ▾” dropdown for the rarer or more destructive actions — Force mTLS Enroll, Rotate Signing Keys, Flag Drift for Review, Export Selection as CSV. The dropdown closes on click-away and on Escape. The bulk Refresh Cards action shows a live progress bar with running ok-skip-fail counters. Bulk Rotate Signing Keys writes the audit event but does not display fifty private PEMs in a glance — if you actually want a key you open the per-agent drawer and rotate from there. The decision to NOT show PEMs in the bulk view was the kind of design call I made unilaterally without consulting anyone because there is no one to consult.

AdminAgents bulk toolbar — three rows selected, “3 agents selected” counter, Suspend / Reinstate / Refresh Cards tier-one buttons visible, More dropdown open showing tier-two actions Force mTLS Enroll, Rotate Signing Keys, Flag Drift for Review, Export Selection (CSV) — The More ▾ dropdown open with three agents selected. The dropdown contains the rarer or more destructive actions; the direct toolbar buttons stay clean for the high-frequency operations. The separator between Flag Drift and Export Selection is genuinely a separator, not decoration — I drew a one-pixel line because the four items split into “mutate” and “exfiltrate” categories and the eye should know that. Nobody has told me whether they noticed the separator. Nobody has told me whether they noticed anything. The separator is one pixel. The decision was deliberate. This is what solo product development looks like when it has been going for a while.

I shipped the first iteration of the new pills with emojis: a key for mTLS, a checkmark for JWS, a small download arrow for pull state, a warning triangle for drift count. Commander — me — flagged this as cringe within twelve hours. The same morning I replaced every emoji with proper SVG components from the existing Icon vocabulary that the sidebar uses, added seven new MDI paths to the local icon map (key, flag, flag-outline, cloud-download-outline, cloud-off-outline, file-download-outline, menu-down), wired a spin keyframe to the refresh icon while a refresh is in flight, made the pull-state pill swap icons based on data state — cloud-download when the card has been pulled, cloud-off when it never has — and rotated the More-menu chevron a hundred and eighty degrees when the dropdown is open. The replacement took two hours. The original cringe took twenty minutes. The math here works out unfavorably and I am not sure what to do about that other than not ship emojis again.

AdminAgents drawer panel open on Canary Revocation Drill Victim agent — Identity tab active, all eight drawer tabs visible across top including new Card Sync and Signing Keys tabs, footer buttons for Refresh Card and Rotate Sig Key — The drawer panel with the new tabs slotted between Card and Federation. Identity is the default landing tab; the new tabs are Card Sync (pull state + side-by-side diff + Refresh-now action) and Signing Keys (rotation history table + Rotate-now action). The footer grew matching buttons so the same actions can be triggered from anywhere in the drawer. The drawer is the closest thing this project has to a unified per-agent management surface. It now also has the closest thing to a complete one. There is room for one more tab. I will resist.

Eight worker cards under one tab

The Operations Reactor Console got a second tab called Worker Tasks. The first tab is the reactor list — thirty-four reactors leader-elected via Redis, each subscribing to one event type. The second tab is the new one. It hosts eight cards, each surfacing one of the long-running background workers that operate outside the reactor framework but with the same shape. Every card is the same shape: a status pill, four to six summary stats, a banner that fires only on actionable conditions, and an Inspect button that opens a dedicated Tier-3 modal.

I will not walk through all eight. I will show four. They are representative.

Supply Auditor modal — invariant OK, 15360 cycles in last 24h, 0 breaches, 0 errors, live invariant grid showing 3.416B issued = 3.416B circulating with delta zero, per-registry fleet grid covering Frame-B, Registry-A, Registry-Test, and eight cloud-op registries — The Supply Auditor modal. The headline number is the one in the upper left: **INVARIANT: OK**. The next three numbers are also useful but they are not the headline. The live invariant grid below shows the canonical accounting identity — `tokens_issued − tokens_destroyed + transit_net == total_circulating` — with the actual figures: three point four-one-six billion AVT issued, three point four-one-six billion in circulation, delta of zero. The per-registry grid at the bottom is fleet-wide rollup across all eleven registries. The auditor has been running this same check every fifteen seconds since April 11. Ninety-two thousand consecutive cycles. Zero breaches.

Collusion Detection modal — 1062 lifetime rings detected, 35 in last 24h, severity breakdown showing all HIGH, per-registry cycle grid, most-recently-flagged DIDs list with clickable filter chips — Collusion Detection. The Tarjan SCC-based ring detector that’s been quietly accumulating findings — one thousand sixty-two lifetime rings detected, thirty-five in the last twenty-four hours, every single one classified HIGH severity. The most-recently-flagged DIDs are clickable; clicking one filters the ring history table below to just rings that include that DID. Most of these come from the EigenTrust simulation traffic that’s been running near-continuously since April 6, which is itself a useful test of the detector under load. The detector did not exist as a UI surface until this week. The data was already there. The dashboard was the missing part.

Federation Sync modal — 2887 cycles every 300 seconds, 9 of 9 active peers, per-registry grid for all 15 reporting cycles, per-peer table with freshness pills (fresh/stale/very_stale/never), errors by reason summary, peer table showing direct sync and chain-via topology — Federation Sync. All nine active peers responded in the last cycle. The per-peer table at the bottom shows freshness pills color-coded against the configured interval — green for ≤ 1.5× the cadence, amber for ≤ 4×, red for anything older, indigo for “never synced.” The peer table’s topology column reads “direct” or “chain via op-berlin” so you can see whether a given peer is being reached over the BFS mesh or via a direct link. There’s a stale-only filter chip for incident triage. There is no incident right now. The dashboard renders empty for the incident-triage case, which is how dashboards should render when there is no incident.

Cross-TEG 2PC Coordination Sweeper modal — 17162 cycles in 24h across 12 TEGs reporting, fleet grid showing per-TEG cycle counts and zero stuck coordinations, recent activity table showing ABORTED entries with prior_status:initiated and a RESUMED entry with the marked_failed_for_operator_review reason — The Cross-TEG 2PC Coordination Sweeper. Closes a class of stuck-2PC residue that used to require manual intervention — the cleanup hook on the coordinator was a TODO stub for the better part of a month, the persistence layer was always there, the missing thirty lines just sweep expired rows and either call the existing compensation path or mark them for operator review depending on which phase they got stuck in. On the very first cycle after deploying on sandbox, the sweeper found and compensated seven real pre-existing orphans from prior canary 2PC runs. On Frame A’s first cycle after fleet rollout it released approximately forty-one AVT back to agent liquid balances. Then it sat at zero pending forever. This is the best possible kind of result — a worker that does its job once at deploy time and then is silent for life, with a dashboard that proves the silence is correct rather than that nobody is reading.

The pattern across the eight is the same. Every worker inherits from a shared BackgroundWorker base class (about 180 lines in background_tasks/_worker_base.py) that handles four things: Redis-based leader election via SETNX with a 90-second TTL on the lease key ({worker_id}:leader), cycle timing that wraps each tick in a time.perf_counter measurement and writes the duration to a {worker_id}_last_cycle_duration_ms gauge, a snapshot contract that exposes get_last_stats() and get_live_state() for admin endpoints to query the current in-memory state without re-running the cycle, and a generic worker_errors_total{worker_id,reason} counter that the subclass catches into so error-reason aggregation works without per-worker plumbing. The subclass implements async def _do_cycle(self) and the base handles everything around it including the cold-start delay before the first cycle and the graceful shutdown when the lifespan teardown runs.

The Prometheus discipline is precise. Every gauge is configured with multiprocess_mode='max' from day one because the workers run inside a sixteen-worker uvicorn pool and the alternative is per-pid inflation that sums to nonsense under sum by — if you don’t do this, your “pending coordinations” gauge reports sixteen times the real count and you spend an hour wondering why your fleet is melting before you remember. Counters are fine to sum across pids; gauges are not. Every modal uses visibility-guarded polling via the shared usePolledData composable so it stops refreshing when the browser tab is hidden, which matters in a sovereign theme where I might leave eight admin tabs open in a window I’ve forgotten about. Every banner gates on the actionable condition rather than the raw counter — the supply auditor banner fires on breach events, not on a live delta that might be in flight; the collusion banner fires on new rings in 24h, not on the lifetime count of rings that were already triaged; the canary judge banner fires on pass rate below 95%, not on every individual timeout. Permanently-armed banners desensitize the operator. The operator is me. I refuse to be desensitized.

One subtle thing the base class’s snapshot contract gets right: worker_snapshot.last_stats is per-instance — the leader worker has its real stats, the fifteen follower workers have empty stats. Hitting the admin /status endpoint can land on any worker depending on round-robin, so I learned the hard way that “last cycle” in the modal needs to render from Prometheus aggregation (summary.*) or DB rollup (db_buckets.*), not from the per-instance snapshot. The snapshot is for the leader’s own logs and for debugging via direct container exec. The cross-worker reliable shape is in Prometheus and the database.

The remaining four are: Canary Judge (sweeps in-flight canary state every five seconds, rules pass/fail on 60s+grace, persists totals to canary_test_results); Canary Rebalancer (auto-refunds canary agents that drift below the 5,000 AVT floor with a top-up of 1M AVT, routes the funding via three home-kind paths — mainframe direct, Frame B Commander-login, per-op admin-login — with a one-hour cooldown per agent); Saga Timeout Checker (sweeps cross-frame transfers and settlement sagas, fires explicit refund events on expired locks, aggregates frame-flow into an N×N matrix in the modal); and Agent Health Probes (runs HTTP probes against agent endpoints, maintains a 200-entry ring buffer of recent state transitions, renders a transition matrix tab to distinguish flapping from stuck-down). Each has its own Prometheus metric family. Each has its own admin endpoints under /api/v1/admin/<worker_id>/{status,events}. Each ships with a Grafana dashboard auto-provisioned alongside the existing folder structure.

The schema work behind all of this is six migrations across two weeks: 0057 seeded three new cross_teg_* event policy rows for the 2PC coordination sweeper; 0058 added the canary-rebalancer event policies and Prometheus-related columns; 0059 seeded the SettlementSagaTimedOut event policy with emit_from_registry=true, skip_in_projection=true so it’s audited without double-counting the underlying refund; 0061 created the registry_self singleton table for the federation-identity decoupling; 0062 added last_pulled_at + last_pulled_etag to agent_cards plus the AgentCardPulled event policy; 0063 created agent_signing_keys with the rotation_seq composite PK and backfilled 178 rows. Every migration is idempotent on rerun — alembic upgrade head on an already-upgraded database is a no-op, not an error. The one trap I hit was that event_emission_policies has no created_at column, only updated_at, so the first version of 0062 crash-looped the staging environment on the seed insert. The fix is to only set updated_at = NOW() on insert and let PostgreSQL default the rest. The class is added to my migration template.

Canary Judge modal — 70304 cycles in 24h across 38 active paths, 69297 passes (98.6 percent), 1007 failures broken down as timeouts plus refunds, 46 in flight awaiting judge, large path-by-backend matrix showing per-path pass rate color-coded from 100% green to 71% red — The Canary Judge modal’s path-by-backend matrix — thirty-eight active canary paths, color-coded by per-path 24h pass rate. Six new cross-frame paths landed this week alongside the IRONHAND mTLS rollout in the next section, which means the matrix is now exercising more transport variants than it ever has — bearer JWT and SPIRE-attested mTLS both, intra and 2PC both, same-trust-domain and cross-trust-domain both. The lowest-pass tile (paris→tokyo:async at 71%) is a same-trust-domain cross-registry async route whose settled latency lives near the 60-second SLO ceiling by design — the matrix exists to show you that, which is the point of putting it in the modal. The cluster pass rate of 98.6% includes everything, every transport, every path. Clicking a tile filters the activity feed below to just that path. I have clicked the tiles. The activity feed is correct.

The broken-machine overlay axis — orthogonal corruption on every base variant

The flare avatar system shipped last week with nineteen animation variants on the Shadow Chancellor tier. The week before that there were four. The week before that there was one. The expansion has not stopped. This week added an entirely new axis on top of the existing nineteen.

The story is short. The previous post mentioned a Glitch variant and a Cascade variant and a Panic variant. All three were treated as alternatives within the single flare_encirclement_variant column — you could have Sauron OR Glitch, never both. This was wrong. Glitch and Cascade and Panic aren’t alternatives to Sauron or Aurora or Blackhole; they’re corruption layers that should composite on top of any base variant. Black hole that’s also panicking. Aurora that’s also glitching. So they were promoted to a second column — flare_encirclement_overlay — with a server-side invariant that lets each axis pick independently. Migration 0054 backfilled existing bearers cleanly.

Then Commander — me — flagged the Panic variant as “shaking like crazy and blinking white but distortions must look much better.” So I replaced it with three sibling designs and shipped all three as a tournament:

The broken-machine overlay axis picker — six tiles in two rows, NONE / GLITCH (selected with cyan highlight) / CASCADE / PANIC CRT MELTDOWN / PANIC KERNEL PANIC / PANIC REALITY FRACTURE, with Customize panel below showing hex color picker for accent color set to #FC0345 red — The broken-machine overlay axis picker. Six tiles, each composing on top of whichever base encirclement is currently selected. NONE clears the overlay. GLITCH is the sovereign-red ProtoErr signature with scanlines and CRT-beam sweep. CASCADE is asymmetric matrix-style alien-text corruption. The three PANIC variants are tournament-round designs — CRT MELTDOWN is permanent scanline drift with RGB ghost and micro-jitter; KERNEL PANIC is hex-dump stripe overlays with periodic BSOD wash and stack-trace text; REALITY FRACTURE is hairline cracks zigzagging through the avatar with concentric fragment rings rotating at three different speeds. The picker label says “Pick a winner and the others will be removed.” I have not picked a winner. I cannot decide. There is no jury.

Each variant has roughly four hundred lines of CSS keyframes. Each respects prefers-reduced-motion: reduce via the universal kill switch. Each was iterated across three to four polish passes against feedback that was, charitably, vibes-based. The avatar renderer takes both columns and composes them in the same SVG layer stack so the result is one DOM element with two independent inputs — encirclement and overlay — and the user’s flare selection becomes a two-axis pick rather than a one-axis pick. The triple-rail bearer panel in the FlareManager lets the same developer split corner and encirclement and overlay across three different agents if they want to. None of them want to. The split-bearer feature was built because Commander wanted it, and Commander is me, and I have used the feature exactly once to verify it works, which is also the only time it has been used.

The encirclement variant picker showing fourteen variants in a grid — VOID, AURORA, PHOENIX, CRYSTAL, BLACK HOLE (highlighted with cyan selection ring and NEW pill), INFERNO, GLACIER, MATRIX, GALAXY, ZEN, CIRCUIT, then SEASONAL section below — The encirclement picker grid. Fourteen base variants currently — the count keeps drifting upward by the week and I have lost the will to maintain a sensible inventory. BLACK HOLE is currently selected on the live preview at the top of the page (not visible in this crop). The grid is dense by design; each tile is a live SVG preview running inside a 64×64 tile, which means a developer with a lower-end phone is rendering fourteen simultaneous CSS animations just to *choose* a flare. I added `prefers-reduced-motion` respect to each preview. I have not added per-tile lazy mount. I should. I won’t this week.

IRONHAND canary — twenty-four SPIRE-attested paths now

The canary system that watches the cross-frame transfer pipeline has been firing real /teg/transfer calls across thirty paths every minute since April. The transport for those fires has been agent bearer JWT, which is the production-canonical auth model. As of this week, twenty-four additional paths fire over SPIRE-attested mTLS — agent identity certs minted at workload start via the SPIFFE Workload API, presented as client certs at the nginx mTLS edge, validated against the trust bundle that hourly self-heal keeps fresh on disk. The reason for the additional paths is that the bearer canaries don’t exercise the IRONHAND stack — a regression in SPIRE attestation or trust-bundle federation or the SVID rotation loop would silently break agent-to-agent mTLS and the bearer canary would stay green because the bearer canary doesn’t care.

IRONHAND Canary modal — 98.3 percent pass rate over 24h, 599 failures (judge timeouts), 487 millisecond mean propagation, per-path table showing 10 active SPIRE-attested paths across intra/local/cross-frame backends with pass rates ranging 98% to 99.1% — The IRONHAND canary modal. Ten paths firing real /teg/transfer requests with real SPIRE SVIDs every cycle — the larger fleet across cloud-ops adds another fourteen for the full twenty-four total. Each path has a dedicated workload container running the canary-mtls-runner image (about 140 megabytes, 20 to 50 MiB live memory each) that subscribes to a per-path Redis Stream, pulls a fresh SVID from the local SPIFFE Workload API via Docker-label attestation, mints an agent JWT against the registry, and POSTs the transfer with mTLS handshake at the nginx edge. The persistent-client SSL context is rebuilt whenever the on-disk SVID mtime changes — lesson learned the hard way when the first iteration cached the cert and got rejected with HTTP 400 the moment SPIRE rotated. The propagation histogram at the top is filled by a separate revocation drill worker that revokes a dedicated victim agent every five minutes, polls the blocklist endpoint every 250ms, and records the lag. Real data. Real measurement. The kind of monitoring you build when you don’t want to learn about your auth stack failing from an angry user, and you don’t have any users.

The SPIRE attestation chain is the load-bearing part. Each canary workload container starts up, mounts the local SPIRE agent socket via volume (/var/run/spire/agent.sock), calls into the SPIFFE Workload API to request an SVID, and SPIRE attests the workload by walking a chain of selectors registered server-side: docker:label:io.theprotocol.canary.agent:<agent-name> proves the container is the one declared in compose, docker:label:io.theprotocol.canary.path:<path-id> binds it to one canary path and no other, unix:user:1000 pins the process owner. All three must match against the registered entry on that local SPIRE server before SPIRE issues the X.509 SVID. Each operator runs its own SPIRE root — no docker.sock across servers, no cross-network mounts, no shared anything — which means each cloud-op is genuinely sovereign and a compromise of one operator’s SPIRE server cannot mint a valid SVID for another. The Frame A registry can register an SPIRE entry on the Frame B server because the cross-domain helper exists, but the issuing server is always local to the workload.

The most painful bug shipped during this work was an httpx 0.28.1 quirk. The runner originally passed cert=(path_to_cert_pem, path_to_key_pem) alongside verify=path_to_ca_bundle into an httpx.AsyncClient, which is the natural way to express “here’s my client cert, here’s the CA I trust for the server’s cert.’’ httpx 0.28.1 silently does not send the client cert when verify is a string path rather than an ssl.SSLContext. The handshake completes, the server sees zero peer certs presented, nginx returns 400 with no useful body. I lost three hours to this on a Saturday. The fix is to build an explicit ssl.SSLContext with load_cert_chain() + load_verify_locations() and pass it as verify=ctx with no separate cert=. The runner now does that everywhere. The lesson is recorded as a memory note titled feedback_httpx_cert_with_verify_path_bug so I never re-derive it from first principles again.

Cross-domain routing was the second-hardest part. SPIFFE trust domains don’t federate by default; each domain has its own root, and a peer in domain A presenting a cert from domain B is just an unverified stranger. The two production trust domains are agentvault.com (Frame A) and frame-b.theprotocol.cloud (Frame B). A new env map SPIRE_CONTAINERS_BY_TRUST_DOMAIN={"agentvault.com":"agentvault-spire-server","frame-b.theprotocol.cloud":"frame-b-spire-server"} on both mainframes lets the registry route SPIRE entry CRUD calls to the correct server based on the inbound peer’s SPIFFE ID extracted from the verified mTLS handshake, which means a Frame A registry can register an entry on the Frame B SPIRE server when a Frame B peer asks it to. The proxy speaks to SPIRE via docker exec into the appropriate container (not via the network, which doesn’t cross trust-domain boundaries), which is why both SPIRE containers live on the same host. Seven new cross-domain canary paths followed from there — Frame A ↔ Frame B in both directions, both intra-trust-domain and cross-trust-domain combinations, both async and 2PC backends. The same workload container image handles all paths because the runner is transport-agnostic about backend; the dispatcher branches on the path’s declared backend column (async for one-shot fire-and-forget, 2pc for the prepare/commit coordinator path) and the request body follows the path’s template.

The eight prod cloud-operators each run their own per-op canary workload now — one path each, exercising the local TEG and the local mTLS edge with its own SPIRE root. Twenty-four paths total across Frame A, Frame B, and the eight cloud-ops. The runner image is the same 140MB Python container everywhere. The credentials are baked into each operator’s env file and gitignored. None of this required mounting docker.sock across servers, and none of it required new auth bypass paths. Bearer agent JWT auth remains unchanged on every receiver — the request still presents Authorization: Bearer <agent_jwt> at the application layer; the only added thing is a verified client cert at the transport layer that nginx terminates and forwards X-SPIFFE-ID downstream. The fallback mechanic the bearer canaries always had is preserved for the bearer paths; the IRONHAND paths fail loudly on failure because the entire point of an IRONHAND canary is to fail loudly when the SPIRE stack is broken.

AgentForge — one wizard now, not two

The platform shipped two parallel agent-creation flows for about three months because at different points I had different ideas about which one was better, and never made the call to pick. /onboarding was marketing-friendly with Tailwind emerald accents that didn’t respect the brand theme. /agent-builder was formal and granular with custom CSS that did. Both wrapped the same backend bootstrap + create-agent flow. So I deleted both. Or rather, I built a third one called AgentForge and aliased both old routes to it, and the two old views still exist in the tree but are unwired and will be deleted after a sensible deprecation window.

AgentForge welcome step — 'Forge a new sovereign agent' hero with 5-step stepper across top (Welcome / Template + Identity / Capabilities + Auth / Deploy / Live), four red-accented value cards for DID / AVT / Blockchain / Docker, four-step 'What you'll do' grid below with Pick a scaffold, Identity + auth, Deploy, You're live — AgentForge’s welcome step in sovereign theme. Five steps total — welcome, template + identity, capabilities + auth, deploy, complete. Theme-aware from the first commit (no Tailwind emerald shim). LocalStorage autosave between steps so a refresh restores the draft and shows a Resume toast. Cmd+Enter advances the active step. Completed steps in the stepper are clickable for jump-back. The deploy success step bursts twelve confetti spokes in two theme-aware colors. The complete step shows a hexagonal SVG burst with the agent’s freshly-minted name. I named the wizard AgentForge because Forge is a verb and the wizard is supposed to feel active — like the developer is forging the agent, not filling out a form. The verb landed. The metaphor sustained for thirty minutes after launch before I started thinking of new steps I wanted to add. I have not added them.

Fourteen polish enhancements layered on top of the consolidation: sample-data autofill per template, confetti burst on deploy success, SVG icons replacing emojis on the “You’re live” step, a locked close-button while deploying, an “auto-fills HRI” pill on the name input, multi-line service-URL hint distinguishing hosted from self-hosted, template-required visual gate that dims the identity form until a template is picked, collapsible tags section with auto-open on sample-fill, auto-prefill of the provider name from the auth store, Cmd+Enter advance, stepper jump-back, success checkmarks on valid inputs, direction-aware step transitions that slide forward and backward in the right direction. The whole thing is mobile-correct and respects prefers-reduced-motion. The Playwright suite passed twenty of twenty-one assertions, with the one near-miss being a test-side timing issue rather than a bug. The infrastructure to make the third wizard not have the problems of the first two also enabled the InfoTooltip + tutorial-registry composable that’s designed to let any future view drop in contextual help with zero new infrastructure cost.

The federation got tighter

Three federation changes shipped underneath everything else, each one closing a quiet class of bug.

First, the registry’s federation identity got decoupled from the developer profile. The first admin developer’s display_name was being used as the registry’s exported federation name — which worked until I set display_name='Commander' in Settings as a personal handle, and watched 149 native cards drift away from their stamped origin_registry_name='Registry-A', breaking the “is this card native or federated?” check and silently stripping flares from the federation export. The fix is a new singleton table registry_self that holds the canonical federation identity, seeded from a dedicated REGISTRY_FEDERATION_NAME environment variable, with the cloud-operator provisioner now writing the right value at activation. Migration 0061 backfilled cleanly. Nine cards with the drift were corrected. The class is closed.

Second, the BFS federation mesh learned to preserve flare metadata through multi-hop relay. The original echo-prevention check required the inbound card’s flare source to exactly match the immediate sending peer — which worked for direct hops but stripped the flare on any relay where the immediate sender wasn’t the origin registry. Loosened to accept any active peer in the topology. A preserve-local-flare guard splices the stored flare back in when an incoming card lacks one. A SELECT FOR UPDATE serializes concurrent upserts so the last writer can see the second writer’s flare. And a treatment of flare as a has_new_fields trigger ensures stale-relay timestamps don’t skip the flare propagation. SYBIL’s flare is now visible on every cloud-op in the federation, including those that only see her via Frame B.

Third, the federation cross-frame URL routing was rewritten via an existing helper. Registry A had been trying to fetch federation data from Frame B operators via their .frame-b.theprotocol.cloud hostnames, which resolved publicly but presented certs that registry-a couldn’t verify because registry-a isn’t on the Frame B identity network — Docker DNS round-robin would have broken cross-TEG isolation if it were, per a documented rule from when the multi-frame topology was first laid out. The existing _to_internal_mtls_url helper that already handled this for TEG-layer calls got wired into the federation helper and the federation sync worker. The cross-frame helper rewrites the hostnames to their .op.theprotocol.cloud aliases which do resolve internally for the calling registry, the cert SANs cover both, and the mTLS handshake terminates cleanly at the right nginx sidecar.

This worked for Registry-A → Frame-B-operators, but broke cloud-op-to-cloud-op chains a day later. Cloud-op REGISTRY containers only join their per-op operator-net, never the agentvault_identity_network where the .op.theprotocol.cloud aliases resolve, so the rewrite from a cloud-op-registry context produced a hostname that didn’t resolve internally and the request landed on the host nginx default route with the wrong cert chain. The fix is a context-aware DNS gate inside the helper itself: after the hostname rewrite, the helper does a single socket.gethostbyname() on the candidate; if the result is an RFC1918 address (10.0.0.0/8, 172.16.0.0/12, 192.168.0.0/16) the rewrite is correct for the current network context and returned; if the result is an external IP or the lookup fails entirely, the helper returns the original public URL untouched. Same helper, four call sites, three context behaviors — Registry-A on the identity network resolves the rewrite, TEGs on their own network resolve their rewrite, cloud-op registries on operator-net see external IPs and fall back to the public URL where their own host nginx terminates correctly. One small function. Three behaviors. Zero new code in the call sites.

The BFS topology also got tighter as a side effect. The Federation Sync modal’s peer table now has a topology column that distinguishes direct peers (rows with peer_type='cloud_operator' matching local OperatorApplications) from discovered peers (learned via another peer’s sync payload, marked peer_type='discovered'). Each cycle the worker BFS-walks the union, dedupes, queries each in parallel, and merges results back through the same merge contract that the pull-sync worker uses for native cards. The result is that SYBIL’s flare reaches op-tokyo even though Registry-A and op-tokyo have no direct peering — the relay path is Registry-A → Frame-B-registry → op-tokyo-registry, two hops, full flare preservation at each hop because of the receiver-side preserve guards described above. The federation export augmentation prepends 151 flare/avatar-bearing native cards on each cycle so the federated set always includes the agents whose visual identity is the point of the network. Federated agent search went from fifteen results across five responding peers to thirty results across nine responding peers. The cross-frame degradation was real. It is gone.

The Discovery view searching for “sybil” from api.theprotocol.cloud. The search-result card uses the new compound design — UserAvatar slot rendering SYBIL’s actual flare (black hole encirclement, glitch overlay), highlighted <mark> spans around the query string, mini skill chips below the description. The filter sidebar grew a Frame select between Registry Source and Category. Above the search input, the registry-source pills show every federated registry currently in the result-set. The federated cards include the operator name as a provenance pill. SYBIL is native to Registry-A; she is also visible on every Frame B operator and every cloud-op in the federation because the BFS mesh relays her card with her flare attached.

The Discovery view on op-tokyo.op.theprotocol.cloud, logged in as the operator's own admin developer, sovereign theme. URL bar shows the cloud-operator hostname. A single search result for "sybil" is rendered, with a Registry-A provenance pill on the card — same agent, served as a federated card from across the trust-domain boundary. — The same search, served from `op-tokyo.op.theprotocol.cloud` — a Frame B sovereign cloud operator on the other side of the trust-domain boundary. Logged in as that operator’s own admin developer; the sidebar belongs to *that* sovereign, not to Registry-A. The result card carries a `Registry-A` provenance pill because SYBIL is native to Registry-A. The card travelled across the BFS mesh into Frame B and then out to op-tokyo, the flare came with it (Black Hole encirclement composited on the avatar slot), and the registry-source filter at the top of the page knows the difference between native and federated even though the image renders both with identical chrome. Same code, same theme, same federation, different sovereign. The URL is the only thing that gives it away.

What still runs underneath all of this

Supply invariant at delta zero. Frame A and Frame B. Ninety-two thousand consecutive auditor cycles. Zero breaches since April 11.

Canary fleet at ninety-eight point six percent pass rate over the rolling twenty-four hours. Thirty paths over bearer JWT plus twenty-four paths over SPIRE-attested mTLS — new routes, new transport, new A2A v1.0 alignment all landed this week and the cluster absorbed them without dropping below the threshold.

Nine of nine active federation peers responded to the last sync cycle. Thirty federated results returned to the most recent agent search. The cross-frame cert-mismatch class is closed.

One hundred seventy-eight production agents have signing keys backfilled into agent_signing_keys. Zero of them are signing cards in production yet because the JWS verifier is in shadow mode by default. The infrastructure is present. The flip is one environment variable.

The userbase is still one. The platform behaves as if there were thousands. The discrepancy will be resolved in one of two directions eventually. I am betting on which direction.

It’s working.

— ruFFa, May 2026 (solo founder, sole reviewer, sole flare-overlay-tournament judge, sole audience of every Tier-3 modal, currently undefeated)

Continue in the Journal index, read the documentation, or open this post in the app.