Closing the mTLS Gaps. Per-Tool Audit on Eleven Registries. Shadow OPA Goes Live.

Three days. Two distinct shipments. One architectural milestone that has been sitting in the docs since the start of the month, marked DONE this week. One observability layer that should have shipped six months ago, now live fleet-wide. One authorization-policy experiment moving from sandbox to its first production frame. One quality gate that finally passed without anyone hand-waving past the failures. Six sections. The reader is free to skim.

The remaining plaintext hops got sidecars

The mTLS fabric has had three load-bearing properties for months. Cross-frame federation traffic between Frame A and Frame B is SPIFFE-attested mTLS through dedicated nginx-federation sidecars on both sides. Inbound traffic from cloud operators to the mainframe EventStores is mTLS through the same sidecar pattern, with both sides validating against a federated trust bundle that includes the other frame's CA via SPIRE's federatesWith directive. Agent-to-agent traffic that opts into IRONHAND mTLS rides through the same pipe. None of this changed this week.

What was still plaintext was the Registry talking to its own services. Same Docker network, sibling containers, the Registry would POST to http://teg-layer:8080 for a transfer and http://event-store:8200 for an emission, on the loopback-of-the-internal-net. Defensible because the network was a private Docker bridge. Improvable because "internal" is a word that means different things depending on which threat model the reader brought to it — container escape, sidecar compromise, accidental cross-tenant routing in a future deployment that is not this one. The plan to close those gaps has been in PRODUCTION_DOCS/02_ARCHITECTURE/Decentralized_Topology_and_mTLS_Roadmap.md since the start of the month. The execution time arrived this week.

Tier 0 (hygiene — deleting an environment variable that had become load-bearing by accident, canonicalizing Frame B peer URLs, blanking the legacy PARTNER_TEGS seed) shipped on the 23rd. Tier 1 (the actual sidecars — Registry to EventStore via a new nginx-event-store sidecar on each frame, plus Registry to its own TEG via a new nginx-teg-{a,frame-b} sidecar on each frame) shipped overnight into the 24th as §pass69. Tier 2 (TEG-side helper for the callback path, plus an SNI alias addition on the host stream proxy) shipped as §pass69.1 later the same day. Image tag pass69.1-2026-05-24 is now on all eleven production registries and all eleven production TEGs.

Six new sidecar containers entered the production compose: nginx-event-store + cert-writer-event-store on Frame A and Frame B both, plus nginx-teg-a + cert-writer-teg-a on Frame A, plus frame-b-nginx-teg + frame-b-cert-writer-teg on Frame B. Each one terminates TLS on port 8443, presents the service's SPIRE SVID as the server cert, validates client certs against the federated trust bundle, forwards the X-SSL-Client-* headers to the upstream so the application layer can see the SPIFFE ID of the caller. The cert-writer sidecar polls the SPIRE workload API every cycle and atomically writes svid.0.pem + svid.0.key + bundle.0.pem into a shared volume that the nginx sidecar reads with a resolver 127.0.0.11 valid=10s directive so it picks up rotations without a restart.

The application-level flip was a single environment variable on each Registry: TEG_API_BASE_URL went from http://teg-layer:8080 (or http://frame-b-teg:8080) to https://nginx-teg-a:8443 (or https://frame-b-nginx-teg:8443). Same string-key, completely different security posture. The Registry's httpx.AsyncClient instances all got verify=get_httpx_verify() — the canonical helper that returns an ssl.SSLContext with the right CA bundle and the client's own SVID pre-loaded for mTLS — across seventy-nine call sites in twenty-seven files. The post is shorter than the diff.

Frame B gets the same treatment as Frame A — symmetric by design. Both mainframes' TEG_API_BASE_URL point at https://...-nginx-teg:8443; both mainframes' EVENT_STORE_URL point at https://...-nginx-event-store:8443. Frame B is not the smaller sibling; it's a peer sovereign with its own SPIRE trust domain, its own CA, its own sidecar set.

A cloud operator stack on prod runs nine containers on a single host. Listing cloud-op-op-berlin live as I write this paragraph:

1. cloud-op-op-berlin-registry — the FastAPI Registry process
2. cloud-op-op-berlin-teg — the TEG layer
3. cloud-op-op-berlin-db — PostgreSQL for the Registry
4. cloud-op-op-berlin-teg-db — PostgreSQL for the TEG
5. cloud-op-op-berlin-redis — shared Redis (state, leader election, idempotency cache)
6. cloud-op-op-berlin-pgbouncer — connection pool in front of both databases
7. cloud-op-op-berlin-nginx-federation — mTLS sidecar terminating inbound federation traffic and proxying outbound federation calls; presents the operator's SPIRE SVID
8. cloud-op-op-berlin-cert-writer — SPIRE workload-API consumer; atomically writes svid.0.pem / svid.0.key / bundle.0.pem for the nginx sidecar to read on every cert rotation
9. cloud-op-op-berlin-spire-agent — per-host SPIRE agent that attests workloads to the SPIRE server

Of those nine, three are part of the mTLS plumbing: nginx-federation + cert-writer + spire-agent. Every byte of traffic that crosses the host boundary on a cloud-op rides over mTLS through that sidecar — inbound federation calls from peers, outbound federation calls to peers, the cloud-op's writes to the mainframe EventStore at events.theprotocol.cloud. The one wire that's still plain HTTP today is the Registry's call to its own TEG: TEG_API_BASE_URL stays at the Pydantic default http://teg-layer:8080, same Docker bridge, sibling containers. That isn't an architectural decision against mTLS on that path; it's a priority call. As long as a cloud-op's full stack lives on a single host, the Registry and the TEG are sibling containers on the same bridge, and dialing a TLS sidecar to reach a process 5cm away (in network-graph terms) is a strictly-defense-in-depth play with a real operational cost — another sidecar, another cert-writer, another rotation surface to maintain. The juice is genuine but small, and the squeeze is real.

What changes that calculus is scale. The thesis of this whole stack is that a single high-tier box can serve a registry with tens of thousands of agents interacting through it before the host's CPU or memory becomes the bottleneck. The day an operator's services have to leave a single host — Registry on one box, TEG on another, the cross-host hop suddenly running over a real network — the sidecar pattern stops being "marginal defense-in-depth" and starts being "the same trust-boundary terminator we use everywhere else." Implementation is mechanically identical to §pass69 T1.2: a fresh nginx-teg sidecar in the operator compose template, a cert-writer-teg next to it, one SPIRE entry per operator, and a one-line TEG_API_BASE_URL flip in .env.operator. About an afternoon's work whenever we decide the scale justifies it. The Oracle's mTLS chapter has a short note about this under the hops table; the principle is "mTLS-everywhere terminates at the trust boundary," and a sibling container on the same Docker bridge is not a trust boundary yet.

One footnote on the mainframe sweep: my first version of this section claimed §pass69's "79-callsite refactor" covered every httpx.AsyncClient in the registry codebase. The parallel session that runs alongside mine caught a thirty-three-callsite gap in sixteen files this morning — sites that didn't follow the env-var-as-local-variable pattern §pass69's grep used, so they kept dialing plain-HTTP TEG / EventStore URLs and failing TLS handshakes once §pass69 flipped the env vars. That work shipped as §pass69.2-mtls-sweep-finish-2026-05-24 and brought the mainframes to 126 of 126 mTLS-bound callsites covered (100%). Mission Control's Network Pulse tiles were silently rendering zeros for a couple of hours during the gap; they're back to real numbers now. The lesson, as always: a grep that scans for one pattern misses the callsites that didn't read the pattern memo.

Two incidents during the bring-up, both worth naming. Incident #1, IPAM race. On Frame B, the brand-new frame-b-cert-writer-teg container got allocated 172.16.36.20 during initial docker compose up — which is the IP that pgbouncer had statically reserved in the compose file but had momentarily released because it was being recreated for an unrelated ulimit change in the same up command. The cert-writer grabbed it first. Pgbouncer came back up, found its reserved IP taken, errored, retried, errored again. Frame-b-registry started returning ConnectionRefused on every DB call. The recovery was to stop the new TEG sidecars (which freed the IP), let pgbouncer reclaim its address, then bring the sidecars back up to get fresh IPAM assignments. Two minutes of recovery. The lesson is in the memory entry. The compose file does not protect statically-reserved IPs from races during simultaneous recreate of multiple containers, and this is the kind of thing you only learn at the moment it happens.

Incident #2, Frame A host-header asymmetry. Frame A's teg-layer has a strict TrustedHostMiddleware in FastAPI that rejects requests whose Host: header isn't on its allow-list. Frame B's frame-b-teg is permissive about the same. After the TEG_API_BASE_URL flip, every TEG call on Frame A returned HTTP 400 with "Invalid host header" because the request now came through the sidecar as Host: nginx-teg-a, which isn't a name teg-layer recognizes. The workaround is one line in nginx-federation-teg-a.conf: proxy_set_header Host "teg-layer:8080". Push the legacy hostname through so the application sees what it expects. nginx -s reload didn't pick the change up — bind-mount inode caching, the same class of bug as the vmalert reload entry from a prior session — so a full container restart was the actual fix. After restart: 45 successes and 0 four-hundreds in the next 60-second window. The follow-up to align Frame A's allowed_hosts with the sidecar name is queued and marked LOW priority because the workaround is invisible to everything except the next person to read the nginx config.

The bake held. Cross-frame canary 2PC paths kept passing through the rollout. Supply invariant at delta zero throughout the five-hour session.

Every MCP tool call writes its own receipt

The MCP server has been live for months. Eleven production registries serve it. Twenty-five tools across four authentication tiers. The thing it was missing was a per-tool audit log, which is the kind of feature that doesn't break anything by being absent until the day a regulator or an incident response team asks "who called what at 03:47 UTC last Wednesday" and the answer is "I don't know, the application logs roll every six hours." This week the answer became "every call to /mcp/rpc and /mcp/admin/rpc writes a row to security_audit_logs with actor, target tool, outcome, latency, sanitized arguments, and the OPA shadow decision when applicable." Shipped fleet-wide as §pass68 on the 23rd.

Three new SecurityEventType enum values were added in migration 0082_mcp_audit_event_types: mcp_tool_invoked, mcp_tool_failed, admin_read_mcp_audit. The dispatcher hook lives in mcp/server.py and mcp/server_admin.py — sixty-six lines and eighty-five lines respectively — and fires after every tools/call dispatch, success or failure, async via asyncio.create_task so the MCP response path is never blocked by an audit write. A deny-list sanitizer in mcp/audit_sanitizer.py walks the arguments tree before write, redacts any field whose key matches one of twelve sensitive substrings (jwt, secret, password, token, api_key, private_key, bearer, authorization, and four more), caps strings at 500 characters, caps the JSON payload at 4 KB total, and caps lists at 5 items. The sanitizer never raises. Three Prometheus metrics on top: mcp_audit_log_writes_total{outcome,server}, mcp_audit_log_write_latency_seconds as a histogram, mcp_audit_log_sanitizer_drops_total as a counter for redactions so the operator can see when sensitive material is being stripped at the boundary.

Click any row to open the detail drawer:

The query endpoints are at GET /api/v1/me/mcp-audit-log (returns the caller's own rows; dev JWT or API key) and GET /api/v1/admin/mcp-audit-log?actor_id=N (admin variant gated on the admin_support flag). The same <McpAuditLogPanel> component drives three mount points in this image: the Settings → API tab card you see above (developer self-view), and a per-subject MCP Audit drawer tab inside /admin/agents and /admin/developers for the admin-scope view. Here's the admin drawer in action — same component, this time scoped to another developer:

This whole pattern is one layer of a defense-in-depth stack that's partially shipped. The self-watching write is here today. The next layer — a hard gate that requires an approved support ticket from the developer before an admin can even open their drawer, with an exception only for an active security incident involving that specific account — is the next pass. Cosmetically the gate slots in as a "support-ticket-required" guard above the same drawer; mechanically it's a row check on the developer's support_ticket_grants table and a banner-with-deny when the row isn't there. The reason it's worth building before there's anyone to investigate: operators across jurisdictions face very different rules about ticketed-and-audited cross-account access. Some regulators require an explicit, time-bounded, ticketed grant for any cross-account read. Building the gate now means an operator in a jurisdiction with that requirement can flip it on (per-operator env flag) without us shipping custom compliance code per region. The audit row that already fires is the trail. The ticket gate that's coming is the up-front control. Together they make the platform meet the cross-account-access standards that most serious enterprises ask about by the second meeting.

Self-watching admin reads is the privacy contract here — admin can investigate, but the investigation is itself observable by the subject. I built the pattern alone. There is no one to investigate me. The pattern is in place anyway because the day there is someone to investigate me is not the day to be retrofitting the observability layer.

The OAuth surface kept its POST /oauth/revoke endpoint — the path has been around as long as API keys have had a revoke at all (DELETE /api/v1/auth/api-keys/{id} is the related developer-side revoke and predates everything in this post). The OAuth variant got renamed and re-wired more than once during the cleanup passes of the last few months. §pass68's contribution was stabilizing the wire on it again: RFC 7009-compliant always-200 response (don't reveal token validity to a probing caller), correct separation between "revoke this refresh token" and "revoke this OAuth grant entirely while leaving the underlying API key valid for direct use," and — the actually-new bit — advertising the endpoint in the OAuth metadata at /.well-known/oauth-authorization-server under the revocation_endpoint field so discovery clients (notably Claude.ai's custom connector flow) find it without anyone publishing the URL by hand. The Ch12 doc had drifted out of sync with the wiring during one of the renames; this rollout brought them back into alignment.

The local MCP bridge (mcp-stdio-bridge.js at the project root, 567 lines now, was 411) got eight quality-of-life improvements that I will summarize as a bullet list because the bridge has been the part of the stack most likely to misbehave silently:

• Monotonic JSON-RPC id counter replacing the previous Date.now() — eliminates collisions on rapid back-to-back tool calls.
• One retry on transient 5xx / connection errors with a 250 ms backoff. Idempotent calls only.
• Configurable timeout via THEPROTOCOL_TIMEOUT_MS (default 30000), because the previous hardcoded 10s was occasionally too short for the canary-judge stat endpoint under sandbox load.
• Structured JSON-RPC error passthrough — errors now propagate to Claude as proper JSON-RPC error objects instead of being stringified into the text content of a "success" result, which had been silently making Claude think failed calls had succeeded.
• The missing theprotocol_bridgeTransfer schema was added — the bridge's local TOOL_DISPATCH dictionary did not include the cross-frame wrapped-token bridge tool because nobody had noticed until someone tried to call it.
• Opt-in client-side JSONL audit log at the path in THEPROTOCOL_LOG_FILE. The bridge never phones home; the file stays on the operator's disk. Independent of the server-side security_audit_logs table. Useful for personal review on a shared workstation.
• Verbose debug mode via THEPROTOCOL_DEBUG=true — pipes every request and response to stderr.
• Dynamic tool count in the init handler — reports len(TOOL_DISPATCH) rather than a hardcoded "15 tools" string that had been drifting since the tools/list output went to 17. Small thing. Now matches reality.

OPA shadow flipped on a production frame for the first time

§pass67, 2026-05-23. The OPA shadow architecture has been live on sandbox-A for nearly a month: every authorization decision in require_admin_flag, get_current_developer, and get_current_agent posts the same decision to a local OPA sidecar in parallel with the Python check; the Python answer remains authoritative; the OPA answer is logged and counted in Prometheus as opa_decisions_total{policy,decision,source} + opa_decision_latency_seconds{policy}. The sandbox numbers had been clean for the entire month — 100% match rate, no false positives, no false negatives. The next step was to flip it on a production registry and see whether the production policy surfaces caused any divergence at the scale of real traffic.

Frame A's registry-a got the flip. Two environment variables: OPA_ENABLED=true (master), OPA_ENFORCE=false (shadow mode — OPA decisions logged but Python remains authoritative). The OPA sidecar was added to docker-compose.production.yml for the registry-a service block only; Frame B and the eight cloud operators do not yet have the sidecar, so they are not part of the flip. Four rego policies loaded into the OPA at startup via REST PUT — workload_access.rego (legacy SPIFFE check), agentvault_admin.rego, agentvault_developer.rego, agentvault_agent.rego. Each one mirrors the Python authorization logic for one principal type.

First-hour data: 72 decisions across the three application-level policies, 100% match rate with the Python check, latency 1–5ms p50/p95 measured against the opa_decision_latency_seconds histogram. No false positives. No false negatives. The shadow layer adds no observable user-facing latency because it runs in parallel with the Python check, not serially after it. The flip is reversible in under thirty seconds per registry — flip OPA_ENABLED=false and force-recreate the container.

What this unlocks: a future where the authoritative authorization layer is Rego policy files on disk that can be diffed, reviewed, and version-controlled outside the application code path. The Python checks become the fallback; the policy becomes the source of truth. We are not there yet. The bake on this first production frame needs to run for some number of weeks of real traffic before the enforce flag flips. The point of shadow mode is that the bake costs nothing — if OPA disagrees with Python on any decision, an alerting rule pages me, and I get to debug the policy without anyone being denied access in the meantime. The mechanism is older than the policy. The policy will mature in shadow. Then the policy will become the answer.

The API tester crossed 100.00% effective

The previous post mentioned the API tester overhaul — the OpenAPI-introspection-driven catalog shipped on the 21st as part of §pass61, alongside three hundred and twenty-two declarative endpoint contracts and four new flow tests. The Phase III sequential baseline was 91.68% effective with 58 real failures on 697 endpoints. That number was honest about the state of the tester. The next pass was to fix the contracts that were lying about the endpoints they were testing.

§pass63.3, in five iterative batches across the morning of the 22nd, drove the catalog to 100.00% effective with 0 genuine failures on 696 endpoints. The image tag is pass63.3-batch4-final-2026-05-22. The breakdown is 691 healthy + 123 expected-failure stamps + 0 real failures. The expected-failure stamps are the contract layer's way of saying "this endpoint correctly returned a documented business-rule error and that is a pass, not a regression" — for example, the dispute resolution endpoint correctly rejects an attempt to rule on a closed dispute with HTTP 409, and the contract layer codifies that as a known-acceptable outcome.

The real fixes — not the EF stamps, the actual code changes — landed in batches one through four: parallelized the _collect_docker_stats_sync aggregation in admin_health via a ThreadPoolExecutor (wall-clock 90s → 7s for 46 containers, which is the kind of speedup that turns a 60s endpoint into a useful one); raised pgbouncer's max_client_conn from 800 to 1600 to eliminate "no more connections" 500s during the reset-sandbox flow; split developers_signing.well_known_router from the prefixed router so /.well-known/dev-jwks.json resolves at the correct root path; rewrote the four a2a_payment.py contracts with the body shapes that match the actual Pydantic schemas (the old shapes were leftovers from a previous schema version); added the missing DeleteVersion contract for the CI/CD pipeline endpoints. Eighteen real failures → five → two → five (state pollution from a recreate) → zero on a fresh reset.

The headline isn't 100%. The headline is that drift is now structurally impossible. The next time someone adds an endpoint, the tester picks it up without any human catalog edit. The next time someone changes a Pydantic schema, the contract for that endpoint either matches it or surfaces as a real failure on the next run. The tester is now honest about itself. This is the kind of work whose value isn't in the percentage; it's in the property that the percentage now means something.

The mainframe-gate detector got five signals instead of one

This one is small but I am putting it in because it is the kind of bug that had been silently shipping mainframe-only sidebar items to cloud operators for an unclear duration. The frontend has a useMainframe composable whose job is to tell every UI surface whether the current registry is a mainframe (Frame A genesis, Frame B sovereign) or a cloud operator. Sidebar items like "Eigentrust Command", "Cross-Frame Bridge", "Treasury Mint", and a handful of others are conditionally rendered based on this flag — they should appear only on mainframes, not on the cloud-op storefronts where minting is disabled and the bridge endpoint is not exposed.

The original detector relied on a single string anchor — the presence of a specific substring in the federation info response. It worked. It was fragile. Any change to that string anywhere upstream would break the detection silently. §pass66.2 rewrote it as a five-signal OR check: identity.tier === 'sovereign-mainframe', OR federation.is_sovereign_frame === true, OR federation.is_genesis_registry === true, OR capabilities.is_central_registry === true, OR capabilities.minting_capability === 'genesis_only', with a fallback to the /federation/info.is_mainframe string check if all five fail.

The five signals are not redundant. Each one is set by a different layer of the system: identity.tier comes from the Registry Card v0.3 served at /.well-known/registry-card.json and is locked at deploy time; federation.is_sovereign_frame is set in code based on the SPIRE trust domain; federation.is_genesis_registry is true only on Frame A; capabilities.is_central_registry reflects the runtime env flag of the same name; capabilities.minting_capability is a string declared in the registry card's economic-disclosure block. If any one of them indicates mainframe, the UI treats the registry as mainframe. The fallback string check is preserved for backwards compatibility with any registry that hasn't been redeployed since the Registry Card v0.3 ship. The check is now Redis-purge immune as well: a separate stale-peer filter in _compute_trusted_registries drops peers that are no longer in the active local peer_registries table from the cached topology view, so a purged Redis can't accidentally resurrect a deleted peer for half a sync cycle.

User-visible result: mainframe-only sidebar items now correctly disappear on cloud operators. The bug had been masking on cloud operators with stale federation cache entries that happened to match the old string anchor. Nobody noticed because nobody was running cloud-op UIs except me and I had not been clicking those links because I knew what they did.

The Oracle got the corrections it had been waiting for

Most of the chapters had been authored months ago and had drifted from the production code in a variety of ways — some chapters were off by a single counter (the reactor count had been stuck at 24; the actual count is 44), others by a load-bearing claim that had become stale during a refactor (the OAuth Custom Connector chapter described a token revocation endpoint whose wiring had been moved during cleanup and not yet documented at its new home). The Oracle is the public-facing surface for the entire system; it is the thing a new developer or a future me reads to understand what is shipping. It got the line-by-line corrections it had been waiting for, plus a new chapter for Organizations & Teams, and the Game Arena chapter was trimmed to match what is actually running.

The text is at api.theprotocol.cloud/ui#/docs. The chapters that changed this round are signposted by an updated date in the header. The audit scripts under scripts/playwright_chXX_audit.mjs verify required content is present and known-stale claims have not crept back — if a future change accidentally re-introduces a claim the source of truth contradicts, the audit catches it. The chapters have been updated as best as I could for now. They will keep drifting; the corrections will keep landing.

Operators can now gift cloud-op provisioning time directly

One more lever, small but useful. Until this image, the only path to give a developer cloud-op provisioning credit was through the fiat purchase rail — pay €25 via Stripe, the webhook handler credits one month of provisioning unlock, the developer's /operator-portal flips from locked to unlocked. That's still the canonical path. But it's also been the only path, which meant any time I wanted to grant a month of provisioning to someone outside the fiat flow (a beta tester, a grant recipient, a partner doing federation interop work) I'd have to SQL-update a row by hand. That's the kind of operation that lives forever in a private CLI script that nobody else can run and that I forget the syntax of every time.

The standing infrastructure

Supply invariant at delta zero across both frames. Ninety-two thousand-plus consecutive auditor cycles since the last breach in April. Canary fleet at 99.83% over the rolling twenty-four hours. Cross-frame paths held throughout the §pass69 rollout. Bundle freshness pills green. SPIRE federated bundle exchange working. The OPA shadow log on registry-a accumulating decisions at the rate of real traffic. The MCP audit log accumulating tool-call rows on all eleven production registries. Image pass69.1-2026-05-24 on every registry container in the fleet, eleven of eleven, alembic heads matched.

None of those numbers are remarkable. They are the same numbers the previous post closed on, and the post before that, with the bake-time deltas you would expect from three additional days of running infrastructure. What changed this week is what the numbers describe. The fabric is now mTLS where mTLS belongs. The MCP surface is now observable per call. The authorization layer now has a second opinion. The quality gate now means what it says. The boring numbers were boring before, are boring now, and will keep being boring as long as the system stays up. Boring is the target.

Next pass is queued and short: align Frame A's TrustedHostMiddleware with the sidecar hostname so the host-header workaround can come out of the nginx config; extend OPA shadow to Frame B and the eight cloud operators once the registry-a bake settles; ship the in-app view for /me/mcp-audit-log so developers can see their own MCP history without curl. The boring numbers will keep being boring across all of that. That is the only review I am going to give myself this week.

— ruFFa, 2026-05-24, magenta theme on, eleven registries holding.