THEPROTOCOL

Sixteen Passes in Three Days. One of Them Tried to Convince Me the Compose File Was Lying.

2026-05-21 · 8 min read · ruFFa
Ten registry containers, ten TEG containers, the canary monitor in sovereign red showing thirty-eight active paths and a five-minute pass rate recovering toward 100 percent after a fleet rollout.
The fleet a few minutes after the latest production rollout. Ten registries, ten TEGs, every container on the same image tag, every alembic head matched. The pass-rate dip is post-rollout warmup — a class of failure that has had three named patches by now and still surprises me every time. Recovers to 100% by T+9 of the bake without intervention. The only thing I have actually internalized about this dashboard is that I should not look at it for nine minutes after a rollout.

Three days. Sixteen separate fixes shipped. An overnight that turned a 7,911-line god-file into 436 declarative test contracts. The v0.3 spec for a federation card standard nobody has asked for. The labor force is one solo founder plus a small civilization of LLM agents working in parallel sessions on a single Hetzner box in Falkenstein. The civilization does not have shared memory, sleeps when I sleep, and disagrees with itself about how to handle CSS specificity. The supply auditor does not know about the canary. The canary does not know about the supply auditor. The only thing that knows about both is me. This is fine.

Eleven sections, briefly. Skim freely.

Eight overlay variants, and a sovereign that had been authenticating with stale credentials

The last post left an unresolved tournament between three broken-machine avatar overlays. The resolution is that the tournament was abandoned and eight more overlays shipped instead, one per remaining sovereign agent — circuit-trace etch, RGB ghost through a hex-prism, buffering-text stutter, amber drip with the AVT glyph, segmented crack network, sound-wave-with-screech, ember spatter, thread-unravel with a literal TENSION 0.32 readout. Nineteen base variants by thirteen overlays equals two hundred and forty-seven permutations. No human will exercise more than four. I am not a human. I am the human. The rest shipped anyway.

Agent World view in sovereign red theme — over a thousand agents rendered as orbs on a 3D Cesium globe with avatar flares and overlay variants visible.
Agent World at altitude. Roughly eleven hundred agents rendered as orbs on a 3D globe. The sovereigns are visible by their overlays. The orbs that look like generic spheres belong to developers who have not yet equipped a flare, which is most of them. The userbase is one. The globe spins anyway.

An incidental discovery: when a credential bulk-import triggered a global logout cascade, the forensic trace revealed that one sovereign's container had been authenticating against production for over a month via a long-lived refresh-token chain that does not re-validate the client_id against the database. Two more sovereigns were running containers with no DB rows on the home registry at all, pointing at some other registry or at legacy detritus. None of them broke. The infrastructure is older than its newest documentation, which is a sentence written before in a previous post and one expected to recur until the infrastructure or the embarrassment runs out.

The canary 2PC fix — cluster pass rate climbed from 92.4% to 100%

Three cross-frame canary paths had been failing at twenty percent of fires for a week, contributing nearly all of the cluster's missed budget. The root cause was an internal-URL helper measuring the wrong network perspective: the helper ran on the registry container, but the URL it produced was consumed by the TEG container, and the two containers live on different Docker networks with different DNS views. A cluster-host shortcut at the top of the helper — force-rewrite the known internal hostnames to their internal port before the DNS gate runs — landed in about thirty-five lines. Cluster pass climbed from 92.4% to 99.83% in the thirty-minute bake.

A residual tail at 1.8% turned out to be a per-uvicorn-worker memory bug: the TEG's runtime partner list lives in module-level memory, and the admin endpoint that updates it only reaches the one worker the load balancer happens to route to. A twenty-line follow-up in the partner-resolver function teaches it to prefer the URL hint from the request body over the cached entry and to self-heal the cache in place. Cluster pass: 100.000%. A Redis-backed shared partner cache landed a few days later as part of a larger upgrade wave and made the band-aid retroactively unnecessary. The band-aid stayed in. Band-aids in this codebase compound.

Twenty-eight items, one wave, one quiet realization about OAuth audit logging

A four-wave strategic plan with twenty-eight items shipped over a Sunday: CI/CD audit events, a webhook retry with exponential backoff, a JWS-failure throttle reactor, a Redis-backed cross-worker partner cache, a leader fail-closed default, migration-safety static analysis, two new alerting rules, a couple of organization-lifecycle hooks. Most of it was the kind of work whose only feedback signal is a metric that does not move.

The interesting finding came from a post-ship test pass. A new admin-action audit helper triggered the first non-OAuth-caller invocation of a low-level audit logger that had been wired into production for an indeterminate period. The helper crashed. Investigation revealed three latent bugs stacked on top of each other in the logger — a missing import, a column-name mismatch, and an enum-serialization that emitted uppercase NAMES against a database column expecting lowercase values. Each of them would crash on every insert. None of them had ever surfaced because every existing caller of the helper was an OAuth token-exchange path that silently swallowed exceptions. Conclusion: production OAuth audit logging has been silently broken for some non-zero period. Token-exchange events have never landed in the audit table. Anywhere. Zero rows ever. The fix bundled into the next image and shipped quietly.

Critical broadcasts, WebSocket reconnect backfill, cross-device read sync

The notification stack had two surfaces — a polled LED matrix ticker and a bell with a count badge — and the bell did not re-render after a WebSocket reconnect. Closing a laptop and reopening it ninety minutes later left the bell stale until a page reload. The fix grew into a five-sub-pass rewrite over a single six-hour session: a sticky-top sovereign-red critical banner that bypasses every mute, a cross-tab read-sync via BroadcastChannel, a scheduler worker that fires on a per-broadcast schedule, an email retry worker with exponential backoff, three new cross-frame event types, org-scoped broadcasts, per-developer notification preferences with Redis cache invalidation, and an auto-broadcast reactor that fires when a CI/CD deployment exhausts its webhook retries. Eleven new reactors. Seven new migrations. Reactor count grew from thirty-six to forty-seven. The user-visible result is a bell that stays correct after a reconnect.

The Admin Broadcasts surface in sovereign red theme — filter chips for All, Info, Warning, Maintenance, Critical, and a feed of broadcast entries with status badges.
The Admin Broadcasts surface in sovereign red. Filter chips persist to localStorage. The critical-priority items bypass every mute the user has configured. The user opted into seeing critical items regardless of their preferences when they accepted the terms. The terms are mine. I will not be apologizing.

The unified rollout, and the 401 that turned out to be operator error

The strategic plan, the CI/CD admin views, and the entire broadcast rewrite bundled into one image and rolled to the fleet without drama. Then came eight follow-up iterations of polish — clickable rows on the CI/CD view, color-coded chips on the audit log, a brighter ACK button on the critical banner, an agent-name lookup that took three attempts to get right, and an eleven-view background-restoration pass after a CSS specificity bug painted opaque slabs over the global ambient layer. CSS specificity continues to surprise me at midlife.

The most embarrassing moment in the three-day window: a password-reset flow test failed because, after issuing a force-reset, the new password was entered against the wrong account email. Off by four characters. The reset had worked perfectly. The 401 was correct. The endpoint returns the generic anti-enumeration 401 that says nothing about whether the account exists, which made it look like the reset endpoint was broken. The endpoint was not broken. The operator was.

Agent Bundles, because the export endpoint had been silently returning 500

An audit on the morning revealed that the existing /export-import surface was returning HTTP 500 with Object of type Decimal is not JSON serializable on every export call. Clone called the same broken export. Bulk-export looped the broken export. The UI swallowed the error and downloaded "something" with no indication of failure. There are no users. The system had been quietly broken at some indeterminate point and nobody had noticed because nobody was looking.

Worse, the import path read balance and reputation fields out of the payload and assigned them to the new agent row — on a federated registry, this would silently violate the supply invariant.

So the surface was rewritten as Agent Bundles: a single signed, self-describing .tpb archive (gzipped tarball with a canonical manifest, an auto-generated README so the file is comprehensible six months later, and a detached JWS signature over a Merkle root of the payloads). Three modes — Snapshot, Template, Migration — collapse four overlapping concepts into one. Six new tables, a per-developer Ed25519 signing key, six seeded sovereign-pattern templates, an anonymous public /templates marketplace, a Grafana dashboard, three background workers, eleven endpoints, and a CLI subcommand. Balance and reputation are never restored. Both fields are moved into a read-only audit block in the manifest so a future operator can verify what the agent had at export time, but neither field is writable on import. The supply invariant cannot be tricked into existence.

The public Agent Templates marketplace view — six sovereign-pattern template cards in a grid with description text and Try-this-agent buttons.
The public /templates route. Six sovereign-pattern templates seeded by a database migration; pure synthetic blueprints, no actual production sovereign code copied. The marketplace exists for the same reason the operator portal exists: to make it possible for the second user, when they arrive, to do the thing without me being there. The second user has not arrived. If the second user does arrive, please give me advance notice so I can prepare emotionally.

SPIRE federation was silently broken, the bundle-freshness detector did not exist

While running an overnight on the event-emission policy cleanup, new observability counters surfaced something that demanded a pause. The cross-frame TEG had logged eleven and a half thousand SSL CERTIFICATE_VERIFY_FAILED errors in the previous six hours. Cross-frame canary paths had dropped from 99.83% to 44%. The supply invariant held at delta zero because the saga rollback path was compensating for every failed cross-frame transfer in real time. The system was protecting itself. The system was also not telling me about it.

Diagnosis: the two SPIRE servers had not been able to resolve each other's Docker hostnames because they are on disjoint Docker networks. Bilateral bundle exchange — the thing that keeps mTLS working across the federation — had been failing every seventy-five seconds, ever since something. The CA bundle TTL is seven days. The overlap window is roughly fourteen. By the time the SSL failures surfaced, both sides had certs expiring within twenty-four to forty-eight hours. The fix is fifteen minutes and no rebuild: dual-attach both SPIRE servers across both networks with explicit aliases. Bundle exchange resumed within two minutes of recreate. Cross-frame canary climbed back to 99%+ over the next day.

A drift detector had detected the bundle staleness once, on an earlier date that I am declining to specify, and then silently ignored it because the compliance poller was disabled. The drift was filed and sat in the database for a period whose duration I am also declining to specify. I am not specifying the duration because I am ashamed of the duration. The system knew. The system did not page anyone. Nobody had configured anyone to be paged. The person who would have configured the pager is me. I was busy. I am always busy. The pager exists now.

The Federation network view in sovereign red theme — Frame-B and direct peers rendered with status pills, drift-checks-clean badge visible at the top of the panel.
The Federation view with the new bundle-freshness layer. A four-state pill per peer (fresh / overdue / critical / unknown), a banner above the KPI row with an aria-live announcement when anything goes red, two alerting rules paging on five-day and six-day age thresholds. Frame-B's pill is green. The drift-checks-clean badge is also green. The system now has opinions about its own health and will share them out loud.

The Idempotency-Key, the Pydantic bug, and zero raw UUIDs fleet-wide

A swagger consolidation pass declared an Idempotency-Key header on thirteen financial endpoints. A test pass discovered that the spec lied: the header was documented but not enforced. Two consecutive identical POST calls to /teg/treasury/fund-agent went through twice. The treasury was double-debited. The supply audit didn't drift because the endpoint moves AVT between two in-circulation accounts, but the operator-facing notion that the treasury was debited 100 AVT was actually 200 AVT debited on the first retry.

The next pass promoted a proven idempotency cache (in use in the bundle service for weeks) into a shared module and wired it into eleven financial endpoints. A scope-prefixed Redis key, a 24-hour TTL, fail-open on Redis unavailable. A Pydantic-as-dict bug shipped with the first attempt — the cache's redaction step ran isinstance(response, dict), which returns False for Pydantic models, and fell through to a serializer that produced invalid JSON; replay calls 500'd with response-validator errors. A type-aware serialization fix landed within the hour. A second sweep found three financial emit sites still using random UUIDs for their idempotency keys and replaced them with deterministic forms. Zero raw-UUID financial event emissions fleet-wide in the post-rollout window. 1,668 financial events emitted, one hundred percent deterministic. Supply delta zero across all eleven post-rollout measurements.

The central-registry flag flip, and ninety seconds of a compose file that had been lying for two years

A final swagger pass shipped the remaining cleanup — mechanical em-dash sweep on admin router tags, feature-area scoping, fourteen internal TEG endpoints hidden from the public spec, and a 618-callsite swap from Depends to Security on the auth dependencies. The spec metric did not move because FastAPI's dependency analysis already traces through to the auth scheme; the benefit is declarative intent and future-proofing for per-route mTLS schemes. Six hundred and eighteen callsites that moved the spec needle by zero and that I would do again.

The same image bundled a flip of IS_CENTRAL_REGISTRY=true on both mainframes, which activates two observation-mode workers. Frame A entered a restart loop with FEDERATION_HEALTH_CHECK_SECRET required for central registry. The secret was added to the env file. The restart loop continued. Investigation revealed that the Frame A compose file has no env_file: directive on the registry service: variables in .env.production are only used for ${VAR} substitution in the compose file itself, not for container env injection. Every variable that needs to reach the container has to be explicit-mapped. The pattern had worked for two years because nobody had added a variable that required explicit mapping. Recovery took ninety seconds once diagnosed. The compose file was not lying. The compose file was incomplete and silent about being incomplete. The silence is the load-bearing part.

The federation compliance monitor view — mainframe panels showing two new workers active in observation mode, cloud-op panels showing the workers correctly not running.
The compliance monitor after the flip. Two new workers active on the mainframes in observation mode (drift is detected and logged; no enforcement action is taken). The eight cloud operators correctly show the workers as not running because the flip applies only to mainframes. The action layer is the next pass.

The API tester overhaul — from a 7,911-line god-file to a self-honest catalog

The API tester is the thing that runs after every code change to verify nothing broke. It had grown organically to the point where it could no longer be modified without breaking something else. A single 7,911-line router file with a 4,002-line handler in the middle. Three thousand eight hundred and thirty lines of duplicated request-body templates across four files. A dead frontend view at 1,722 lines that was not imported by anything but still being shipped. Approximately thirty live endpoints uncatalogued because the hand-maintained catalog had been silently skipping them.

A directive of "GO FULL VISION singularity bro" greenlit a seven-phase all-night overhaul. The architectural decision: OpenAPI introspection becomes the source of truth for the catalog. Drift becomes structurally impossible. The runtime pulls every path from the live spec, dispatches each request through a typed declarative contract if one exists, and falls through to the legacy chain if not. Contracts become purely additive: they only improve classification, never regress it. The pass shipped 322 declarative contracts in the first wave (target was 50; the autonomous session overshot by 6.4×), a worker bump on the test environment, an eager-bootstrap endpoint, four new flow specs, an archive layout, and a Grafana dashboard.

The API tester surface in sovereign red theme — three tabs (Endpoints, Chained Frame Lifecycle, Dual Frame Bridge), sandbox-mode banner, authentication-configuration panel, and the test-category grid covering all 13 categories.
The final post-overhaul number: 683 endpoints, 669 genuine healthy (98.0%), effective pass 98.83% (8 real failures), 21m 32s wall clock. The pre-overhaul baseline against the OLD tester was 660 endpoints / 651 genuine / 99.55% effective / 25m 56s. That looks like a headline regression. It is not. The new run tests twenty-three more endpoints because OpenAPI picked up routes the hand-maintained catalog had been silently skipping. The eight remaining real failures are an irreducible minimum (known backend bugs and intentionally-incomplete fixtures). Catalog drift is now structurally impossible. The thermometer used to run cold. The thermometer now reports the actual temperature. The actual temperature is fine.

Registry Card v0.3 — a federation standard drafted alone because there is nobody to draft it with

The Agent Card has been a standard for two years. Every agent owns a /.well-known/agent-card.json; the discovery layer pulls it on a six-hour clock; the round trip is signed. This works extraordinarily well for agents. It works not at all for the registries that host the agents, because there has never been a standard for what a registry says about itself. The federation is supposedly a peer-to-peer network of equals. One tier of the network gets to dress up. The other tier wears a name tag with a Sharpie scribble on it. This is, in retrospect, on me.

So a spec got written. Alone. As the entire standards body. Self-appointed and unopposed. Three iterations over the course of a single morning, after a brisk in-head argument between three versions of one opinion. Roughly one hundred and fifty fields across thirteen blocks. The name in the spec is TheProtocol Federation Card, because the word "standard" is doing a lot of work in the phrase "a federation standard" when the federation in question is a Hetzner box with two Docker networks. The implementation landed in roughly seven wall-clock hours instead of the estimated nineteen because there was no review cycle and no committee and no Slack channel for stakeholder alignment. Pull request reviewers: zero. Approvals: also zero. Merged.

The card itself is one JSON document, served anonymously at /.well-known/registry-card.json on any registry that implements the spec, signed with the registry's own Ed25519 signing key, verifiable against the registry's JWKS at the same well-known prefix. Empty skeleton, types only, no actual values — for the reader who would rather see the shape than the prose:

{
  "registry_card_version": string,
  "identity":  { spiffe_id, registry_name, display_name, trust_domain,
                 public_url, federation_base_url, founded_at, tier },
  "operator":  { developer_id, display_name, bio, contact_email, company,
                 url, github_url, twitter_handle, timezone, location_label,
                 operator_since, operator_avatar_url, successor_contact,
                 support_contact: { email, matrix, signal } },
  "description": string,
  "tags": [ string, ... ],
  "visual":    { icon_url, icon_etag, icon_size_bytes,
                 orb_color, orb_glow_color, orb_pulse_speed_hz, orb_size_px,
                 halo_enabled, halo_thickness,
                 theme, sovereign_variant,
                 flare_corner_variant, flare_encirclement_variant,
                 schema_compat },
  "capabilities": {
      fiat_genesis_registry, fiat_onramp_enabled, fiat_providers,
      is_central_registry, federation_enforcement_active, federation_enabled,
      kafka_enabled, event_store_enabled, astraeus_enabled, opa_enabled,
      auto_tally_enabled, canary_enabled, revocation_drill_enabled,
      oauth_token_exchange_enabled, routed_fiat_proxy_enabled,
      saga_adaptive_conc_enabled, federation_card_gc_enabled,
      zkp_phases_enabled, a2a_versions_supported,
      registry_card_versions_served,
      minting_capability, bridge_capability,
      supports_payment_protocol, supports_event_types },
  "fees": {
      "intra_registry":  { base_rate_percent, max_rate_percent,
                           velocity_scaling_factor, destination,
                           customizable, locked_by_network },
      "cross_registry": { base_rate_percent, fee_recipient,
                          customizable, locked_by_network,
                          future_customization_in_bounded_params },
      "hosting":        { enabled, rate_percent_of_agent_service_fee,
                          min_avt, applies_to_agent_dids, description },
      fee_collector_address, fee_waterfall_policy, settlement_currency,
      fee_audit_endpoint,
      "economic_disclosure": { spec_version, fee_floor_avt,
                               rounding, fees_emit_event_types } },
  "federation": {
      protocol_version, supported_auth_methods, trust_domains_accepted,
      spire_bundle_endpoint, policy_hash, policy_version, policy_updated_at,
      is_genesis_registry, is_fiat_genesis, is_sovereign_frame,
      uptime_started_at, active_peer_count, trusted_peer_spiffe_ids,
      license_id, license_status, license_key_prefix },
  "policy": {
      locked_params_hash, bounded_params, policy_version, policy_modified_at,
      drift_status, consul_compliance_state, policy_change_history_url },
  "economic": {
      tokens_issued, total_circulating, tokens_destroyed,
      tokens_fees_collected, supply_invariant_delta, supply_invariant_status,
      supply_last_audited_at, auditor_consecutive_ok_cycles,
      last_breach_at, treasury_zero_days, treasury_quarantine_triggered,
      external_auditor_endpoint },
  "sovereign_agents": [ { name, did, subdomain, sovereign_variant }, ... ],
  "stats": {
      agent_count, native_agent_count, federated_agent_count,
      sovereign_agent_count, service_agent_count, developer_count,
      active_peers, active_disputes, active_proposals,
      uptime_pct_30d, canary_pass_rate_24h, computed_at },
  "endpoints": {
      openapi_spec, agent_discovery, agent_card_well_known, topology,
      network_stats, federation_info, policy_compliance, health_check,
      registry_card, registry_jwks, spire_bundle, fee_config,
      external_auditor, public_status_page },
  "links": {
      whitepaper, docs, github_source, status_page, blog, rss,
      terms_of_service, privacy_policy, code_of_conduct, support_url,
      discord, matrix, incident_log },
  "commitments": {
      data_residency, gdpr_compliant, incident_disclosure_sla_hours,
      uptime_sla_pct, support_response_hours, deprecation_notice_days,
      open_source_license, breach_disclosure_policy, key_rotation_max_days,
      card_refresh_max_minutes, legally_binding_disclaimer },
  "lifecycle": {
      status, quarantine_reason, card_updated_at, card_updated_by,
      card_etag, served_at, next_card_refresh, last_card_fetch_count_24h },
  "signature": { alg, kid, jwks_url, signed_canonical, value }
}

A reference SDK helper takes any agent card and its host registry card and computes the total cost of a transaction upfront, including the cross-registry fee that is hard-coded today and the hosting-fee surcharge that is declared but disabled. The number on the "Pay" button is derivable from public JSON. The schema lives at /.well-known/registry-card-schema.json on any registry that implements the spec. Yes, that is a self-describing endpoint that describes how to describe itself. I am aware this is recursive. I shipped it anyway.

Want to look at the live Registry-A card right now? It is anonymous, public, and unsigned-as-of-the-time-of-the-video:

Or, if the only review the spec is ever going to get is from the reader of this paragraph — curl -s https://registry.theprotocol.cloud/.well-known/registry-card.json | jq . and tell me what is missing. I will accept the feedback. There is a non-zero chance you will be the second person to look at this document with any real attention.

Ten end-to-end tests pass on a non-production frame: signed-card serve, offline EdDSA verify against the JWKS, ETag 304 round-trip, admin PUT with event-store emission, icon-upload size cap, cross-host federation sync, three-way SPIFFE sovereign-claim defense (one host forged a sovereign-pinned variant; the receiving host stripped the forgery and cached the rest), topology endpoint surfacing the new visual and fees slice, and SDK cost-composition matching the spec's reference math down to the rounding mode. Ten of ten. The author wrote the tests. The author ran the tests. The author judged the tests. This is not how integration testing is supposed to work; there was nobody else available to run it the proper way.

A twenty-seven-second tour. AgentWorld in neural theme with Ironhand Alpha as the active agent — eleven hundred and forty-two agents rendered as orbs on a Cesium globe with signal arcs sweeping across Europe. A click on Registry-A's orb opens the new v0.3 hover tooltip: GENESIS-FLAME badge, GLITCH overlay, ACTIVE status, the cross-registry fee, the hosting status, the auth methods, the UNSIGNED v0.3 card label, the Card ETag. The detail panel slides out with the "View Registry Card v0.3" button. Then the tour cuts to the brand-new /admin/registry-card editor, where the Flare Layer section renders all nineteen encirclement variants and all thirteen broken-machine overlays available to a registry — the same DOM the sovereign agents use, applied at the registry tier. Plus the derived economic surface, the commitments matrix, and the links panel. The parallel session shipped this between the start of the post and the time of writing. The orbs are now wearing flares.

One more thing. A separate session shipped the flare-extension-for-registries while this post was being condensed. Same DOM machinery the sovereign agents already use, attached to the orb on the Federation view. The argument is that, in the limit, "registry" and "agent" are both just nodes in the federation graph, and every node should get to dress up. This argument was pre-litigated in zero internal review meetings because there are zero internal review meetings. The orbs are wearing flares as of this paragraph. The video above is the proof. The labor force has, as always, outpaced the documentation.

What still runs underneath

Supply invariant at delta zero across both frames. Ninety-two thousand consecutive auditor cycles since the last breach in April. The number remains the most boring and most important number in the post.

Canary fleet at 100% over the last ten minutes, ninety-eight point six over the rolling twenty-four hours. SPIRE federation auto-refresh working again after a previously-silent drift. Bundle-freshness pills rendering on the Federation view. Two alerting rules paging on five-day and six-day age thresholds. All sixteen Idempotency-Key endpoints enforcing dedup at runtime. Zero raw-UUID financial event emissions for the past twenty-four hours and counting.

The userbase is one. The labor force is the userbase plus an indeterminate number of LLM agents working in parallel sessions. The labor force does not pay license fees. The userbase does not collect them. The platform behaves as if there were thousands. The bet on which direction the discrepancy resolves is one-sided because there is nobody to take the other side.

It's working.

— ruFFa, May 2026 (solo founder, sole reviewer, sole convener of the LLM civilization, sole audience for every joke in this post including this one, sole human in the loop)