THEPROTOCOL

After the Simulation: 24 Reactors, 26 Dashboards, and the Layer That Makes It Survive

2026-04-30 · 7 min read · ruFFa
TheProtocol Live Status dashboard — Frame A registry UP, Frame B registry UP, Event Store UP, TEG Layer UP, with supply invariant ingest curves, 64 reactor leaders, 73 federation peers active, 43.5 ops/s dispatch rate
Live status — Frame A, Frame B, Event Store, and TEG layer all reporting in real time. The reactor framework keeps 64 leaders elected across the cluster; 73 federation peers actively syncing.

The simulation has been running near-continuously since April 6, at variable load — different agent counts, different transfer rates, whatever I needed to test on a given day. Across all of it, the supply invariant has held at delta = 0.0 every cycle. That’s the load surface. What’s been shipping since then is what surrounds the load — the operations layer that turns a thing that works into a thing that keeps working when nobody is watching.

Event-Driven Reactors (24 of them)

Every interesting state change in TheProtocol now emits a typed event. License revoked, agent suspended, peer license reinstated, supply breach detected, drift event fired — they all land in the immutable EventStore as structured records. Reactors subscribe to specific event types, are leader-elected via Redis (so exactly one instance runs per cluster), and dispatch to a handler that takes the corrective action.

This is the architectural difference between a system that does things and a system where the right things happen automatically when something else does things.

Reactor Framework dashboard — 73 active leaders, 73 reactors WS-connected, 57.8 dispatches/min, 0 errors over 28 days, 585k total dispatches, with the per-reactor leader-status table showing every reactor across every registry and operator
Reactor framework health — 73 leaders elected via Redis, 0 errors over 28 days, 585k dispatches. The table shows per-registry leader status; every reactor, every operator, every frame.
Full Reactor Framework dashboard — cluster health stats, per-reactor leader-status table for every registry and operator, dispatches-per-second by reactor, error-rate by reactor, p50 / p99 dispatch latency by reactor, leader stability timeseries
The full Reactor Framework dashboard — cluster health, per-reactor table, dispatch throughput by reactor, error rates, p50 / p99 latency, leader stability. One screen, every reactor across every registry and frame.

Coordinated Suspension — One Click, Four Levers

When you suspend a misbehaving agent or developer, four things have to happen together:

  1. Registry status flips to suspended
  2. IRONHAND mTLS certificate revoked (peer-to-peer auth blocked within ~500ms)
  3. TEG account suspended (no transfers)
  4. Cross-TEG blocklist propagated to every partner TEG (no inbound transfers from elsewhere)

In the old model, that was four separate buttons in the admin UI and four separate places where you could forget one. In the new model, one click. The orchestration is split across an inline path (correctness, fires immediately) and a reactor (cross-frame propagation via webhook + cross-TEG admin endpoint).

Long-Term Metric Storage — 365-Day Cold Tier

The monitoring stack got a long-term tier. Prometheus continues to handle 30-day hot retention; every sample also remote_writes to a single-node VictoriaMetrics behind it. Alert evaluation moved to vmalert — which means alert rules can now query the full year of history (long-window burn-rate alerts that previously couldn’t fire because the data was about to fall off Prom’s retention edge).

26 Dashboards Across 8 Folders

Grafana is up to 26 auto-provisioned dashboards in 8 folders (Cluster / Economic / Federation / BackgroundTasks / Security / ServiceDomain / SLO / Public). The public status page now lives at a fixed URL — anyone can see live system health without an account. A server-side image-renderer sidecar produces dashboard PNGs on demand for incident triage and visual verification.

Highlights of the new dashboards:

Grafana dashboards folder view — full directory structure showing all 26 dashboards organized into BackgroundTasks, Cluster, Economic, Federation, Public, Security, ServiceDomain, and SLO folders with their tag pills
Every dashboard, every folder. 26 auto-provisioned views — from cluster fundamentals up through governance, A2A payment funnels, IRONHAND mTLS, OPA shadow-mode, federation health, and the SLO error-budget tracker.

Federation Compliance Detector — Now 6 Rules

The CONSUL Federation Compliance dashboard at /ui#/admin/compliance now runs six rules. The four originals catch licensing and lifecycle drift — the cross-table state-mismatch class that lives between federation_licenses, operator_applications, peer_registries, and SPIRE itself:

The two new rules added this cycle catch infrastructure-shape drift the originals couldn’t see:

Current state: 0 drifts across all 6 rules. Detection runs everywhere; the hourly auto-cycle is gated by IS_CENTRAL_REGISTRY, but every operator can run a scan on demand via the “Run drift scan now” button. Per-day idempotency on emit means the same drift on the same UTC day fires the event once, not 24 times.

Multi-Frame Federation — Operating at 98.95%

The Frame A ↔ Frame B bridge that went live April 3 is now operating at 98.95% cross-frame transfer success rate. The residual ~1% is benign saga compensation traffic — when a target TEG rejects a transfer (insufficient balance, expired prepare lock, suspended receiver), the source aborts coordination and unlocks the locked fund. By-design 2PC behavior, no money disappears.

Each cloud operator’s federation sidecar now joins both sovereign-frame Docker bridges with per-frame DNS aliases. A runtime resolver in the registry picks the fast path automatically based on the caller’s network context — RFC1918 (Docker bridge, internal port) vs. public IP (host network, host-mapped port). End result: cross-frame transfers route via the Docker bridge under SPIRE mTLS, no host-NAT round-trip.

Symmetry verified across all 8 production cloud operators (London, Berlin, Paris, Washington, Sydney, Seoul, Singapore, Tokyo).

Cross-TEG Coordination Health dashboard — 99.8% success rate stat in green, 739.8 req/s sustained, 1.93k 5xx (24h benign saga compensations), 206k 2xx (24h), with per-operator outcome timeseries and top-5xx-handler breakdown
Cross-TEG coordination health — 99.8% success at 739.8 req/s sustained. The 5xx counter is benign 2PC saga compensation traffic, not infrastructure failure. Per-operator and per-handler breakdowns localize anything that does drift.
Full Operator Layers dashboard — cluster-wide ingestion rate, per-operator request rate stacked by layer, per-operator latency, per-operator registry / TEG ops/sec, operator × layer matrix table, Cross-TEG coordination health, per-operator outcome mix, top 5xx handlers
The full Operator Layers dashboard — ingestion, per-operator request rate by layer, per-operator latency, per-layer ops/sec, the operator × layer matrix, and the Cross-TEG section at the bottom. One view, 11 cloud operators across both frames, every layer instrumented.

Identity Layer Maturation

One codebase, multiple deployments. Production registries, sandbox registries, Frame B, and every operator all run the same source tree from the same Docker images. The differences between them are environment variables, not branches. So “a feature lives in this binary; an env-var controls whether it’s active here” is the right way to read this section.

OPA Shadow-Mode dashboard — 12.9 decisions/sec, 1.18k total decisions, 0.0000% mismatch rate between Python and OPA, 33 ms p95 latency, decisions-by-policy and allow-vs-deny timeseries, OPA engine UP, 1 bundle loaded
OPA shadow-mode — 0.0000% mismatch between the live Python authorization and the parallel OPA decision, sustained over the sample window. Same code, same flag, ready for production scale.

A Cluster of Smaller Wins

What’s Next

With reactors handling cross-service propagation, drift detection catching topology regressions, the multi-frame topology now bridge-symmetric, and the sandbox samples confirming the gated paths are clean, the focus moves to flag flips on the mainframes:

The simulation runs. The reactors react. The dashboards render. The drift detector finds nothing.

It’s working.

— ruFFa, April 2026

Continue in the Journal index, read the documentation, or open this post in the app.