Twelve Axes, Two Real Bugs. The MCP Tester Stopped Lying To Itself.

One day. A morning for the prerequisite (mTLS parity between the two sandbox frames, so the tests run against the same trust boundaries production does). A long afternoon and evening for the harness itself — a doctrine rewrite that converted "ninety-nine point two-seven percent pass" from comfortable lie to honest 79.91 percent baseline, then nine iterations that walked it back to 99.41 percent the hard way. In between, two real bugs the harness caught that would have shipped silently otherwise. Twelve detection axes by the end, five of them new. The userbase is still one. The tests have not yet been run by anybody but me. The bugs were real anyway.

Seven sections. The reader is free to skim.

First, the sandbox had to be a real mTLS environment

Before any of the strict-mode work could mean anything, the sandbox had to actually exercise the same trust boundaries production does. Sandbox-A had been talking to its EventStore over plain HTTP on the internal Docker network — not because plaintext is the wrong call for a sibling-container hop, but because production had just moved that exact hop to mTLS via dedicated nginx-event-store sidecars (§pass69, two days earlier), and sandbox was now structurally different from prod. A regression test running against a structurally-different environment is a test that's checking something other than the thing in production.

So the morning's work was symmetric: agentvault-nginx-event-store-test + agentvault-cert-writer-event-store-test on sandbox-A, and agentvault-nginx-event-store-test-b + agentvault-cert-writer-event-store-test-b on sandbox-B. Same template as the prod sidecars — SPIRE-attested cert-writer polling the workload API and atomically writing svid.0.pem / svid.0.key / bundle.0.pem into a shared volume; nginx sidecar terminating mTLS on :8443, validating client certs against the federated trust bundle, forwarding X-SSL-Client-* headers upstream. EVENT_STORE_URL flipped from http://event-store-test:8200 to https://nginx-event-store-test:8443 on each registry. SPIRE entries registered for the new SVIDs under spiffe://sandbox-a.test/service/event-store-test and the mirror on the B side, both with FederatesWith across the trust domains.

Two latent bugs surfaced during the bring-up, both worth naming. One: registry-test-b was missing the :ro mount for the cert volume that cert-writer-test-b writes to — sandbox-A had had that mount since the prior pass, but sandbox-B had never been brought to parity. The reactor framework's aiohttp client fell back to system CAs (which don't include the SPIRE bundle), forty-seven reactors failed handshake against the new mTLS endpoint, and the symptom was an SSLCertVerificationError cascade for two minutes until the diff caught up. Two: the EventStore's internal mTLS allowlist (agentvault-event-store/src/routers/events.py) hard-coded the prod service names — registry-a, registry-frame-b, teg-a, and friends — with no sandbox equivalents. The first POST from registry-test over the new mTLS path returned 403 with Unknown SPIFFE ID: spiffe://sandbox-a.test/service/registry-test. Register as a federation operator first. Extended the list to include sandbox names plus a KNOWN_INTERNAL_SERVICES env-var override so future operator namings don't require a code edit.

By mid-morning both sandbox frames had matching mTLS sidecars in front of their EventStores, supply audit returning delta=0.0 status=OK through the new pipe (agent_count=5834 on A, =6 on B; A has the canary roster, B has the canonical pair), and forty-seven of forty-seven reactors on each frame reporting ws_connected=1 against the sidecar. The prerequisite cost about ninety minutes of real work and three hours of waiting for SPIRE bundles to rotate. The thing that mattered: every subsequent MCP tool call would now ride the same mTLS terminator a production tool call does. The harness was about to start asking real questions; the environment now had real answers to give.

The doctrine rewrite — what counts as healthy

The harness had been reporting 99.27% strict pass rate. That number was a lie. Three code paths in the runner were converting failures into successes:

1. Smart-skip rewrite at test_flow_mcp_lifecycle.py:1050-1063 — if a tool returned an error message containing "requires agent authentication", the runner concluded the contract was mis-tiered, marked the outcome healthy=True, and moved on. Every "the server wants a different auth tier than the contract claims" case — which is itself a real bug class — was being silently rewritten as a pass.
2. "Path reached — coverage counts" at :1156 — the admin-bridge fallback path was crediting any tool call that didn't crash as healthy=True, with the literal comment "path reached — coverage counts." Three hundred and two admin tools were being graded on whether bytes came back, not on whether those bytes were correct.
3. expect_status="error_4xx" as a passing path — if a contract declared the expected outcome was a 4xx, the runner would credit any 4xx as a pass. No paired assertion on the error envelope. The endpoint could return HTTP 400 {"error":"undefined"} and that would count as having tested anything.

Commander's binding mandate, verbatim from the Friday before: "NO: 'this test is expected failure'. NO IT IS NOT. REAL TESTS, REAL EVERYTHING — IT'S SANDBOX." The fix was to delete all three branches and replace them with strict semantics. A tool is healthy=True if and only if four conditions hold: HTTP status matches expect_status (or matches an explicit error contract paired with expect_error_message_contains or expect_error_keys); every declared expect_keys is present in the response body (auto-derived from the tool's outputSchema.required when expect_keys is empty — FAIL with contract gap if both are empty); side_effect_assert(inner, bag) returns true if declared; and expect_types isinstance checks pass for every declared field. Skips are not healthy. Skips became their own outcome category. Tier drift — "server demanded different auth than contract claimed" — became its own failure category with a typed tier_drift=True outcome.

The first sweep after the rewrite landed at 540 of 682 healthy — 79.91 percent. One hundred and thirty-seven hard fails surfaced, of which a hundred and thirty-six were contract_gap — tools that the runner walked but couldn't actually assert anything about, because the contract had empty expect_keys and the wrapper didn't declare an outputSchema.required. Twenty percentage points of false confidence had been hiding in those branches.

The full failure breakdown after the doctrine flip:

Total tools:        682
Healthy:            540  (79.91%)
Failed (real):      137
  ├── contract_gap: 136  (walked but no shape assertion possible)
  ├── http_error:     1  (getEnforcementAgentStatus — missing agent_did arg)
  └── tier_drift:     0
Skipped:              5

That number is the honest one. A failing test more valuable than a lying one. Nine iterations followed.

Twelve axes

The strict-mode floor was step one. The real work was building five more detection axes on top of it — Phase F bug-finder layer — that ask questions the original harness wasn't asking. The shape of the sweep, end-to-end:

The orange-highlighted phases are the Phase F bug-finder layer shipped this week; everything else has been load-bearing since Phase D earlier the same day. The dotted lateral arrows out of Phase 4 are the per-call verifier stack — the seven base axes plus the new F.5 eventual-consistency poll — firing on every single mutation lifecycle. The same diagram and a full per-axis operating reference live in the Oracle docs at Chapter 12 — MCP Integration; if you want the click-through for every probe envelope shape, the wrapper authoring guide, and the audit-pipe details, that's the surface to land on.

The full twelve-axis table:

Axes 1–7 are the strict-mode core (Phases A→D earlier the same day). Axes 8–12 are the bug-finder layer that landed in the afternoon. The pass-rate trajectory across the day's sweeps:

The four-tool gap between 99.41% and 100% is the irreducible minimum — sandbox-environment limitations (federation endpoints that legitimately require SPIFFE mTLS the sweep doesn't carry, the runAllApiTests tool that would recurse into the harness itself, a couple of read paths that need data the sentinel doesn't have). Each one is a documented skip-with-rationale, not a contract gap.

The four load-bearing inflections in that climb are worth naming, because each one is a reusable pattern, not a one-off fix. Iter#1 (+11.3pp) was the schema-augment. The 136 contract_gap failures from iter#0 surfaced because 380 public MCP tools all advertised outputSchema: undefined in their tools/list response — nothing for the strict-mode runner to derive expected keys from. The fix was two scripts: scripts/extract_mcp_schemas.py walks the wrapper Python source with an AST visitor and extracts every dict literal that looks like a response shape; scripts/harvest_mcp_shapes.py calls each tool live against a known-good sentinel agent and records the keys it actually returns. The two outputs merge into mcp_response_expect_keys.json — an explicit per-tool map of keys to assert. The contract loader (_augment_contract_with_schema()) consults the map at get_contract() time so explicit contracts win, but tools without one get the harvested fallback. Started at 83 AST-extracted shapes, grew to 261 live-harvested shapes, merged to 295 covered tools by iter#3. Iter#2 (+3.4pp) was the eleven mutation lifecycles wired with expect_events, each verified against the event_emission_policies table (so a transferTokens 2xx without a TokensTransferred in the EventStore inside 3×2s = a hard fail). Iter#3 (+0.9pp) was the wrapper envelope normalization — nine LIST wrappers that had been returning bare arrays got normalized to {items: [], total: 0, ...} envelopes so the MCP dispatcher's _format_tool_result would stop stripping them as "error dicts" on the 401 fallback path. Iter#7 (+1.6pp) was the doctrine reversal Commander pushed back hardest on: most of the "sandbox limitation" skips weren't real — they were lazy. getTrustScore can be tested by running three transfers between two agents and triggering eigentrust computation; getProposalVotes already has a lifecycle_governance_proposal_vote_tally that creates a proposal, the contract just needed to reference the stored proposal_id; claimStakingRewards with zero stake is a deliberate 4xx with a documented body shape, which is a real test if you assert against the body. Fifteen of sixteen lazy skips became real tests in one pass. Only runAllApiTests (recursive) stayed a documented skip.

The pattern across all four inflections: every percentage point of pass rate moved by fixing the harness, not the codebase. The harness was the thing lying. The two real bugs the harness caught are the proof it stopped.

Bug #1 — updateMyDeveloperLocation: a 5,834-agent transaction that timed out and rolled itself back

This one took ninety minutes to root-cause and three lines of structural surgery to fix. The lifecycle did the obvious thing: PUT the developer's location to Tokyo coordinates (35.6762, 139.6503), then GET the developer's profile and check that the latitude came back as Tokyo. The endpoint returned HTTP 200 OK on the write. The read came back with the original Freilassing coordinates (47.84, 12.98). BUG-FIND: returned 2xx but latitude did not persist.

The endpoint's source was straightforward:

# routers/agent_locations.py:update_my_location
async def update_my_location(payload, ..., db, current_developer):
    current_developer.latitude = payload.latitude
    current_developer.longitude = payload.longitude
    current_developer.location_label = payload.location_label

    if payload.update_agents:  # default True
        agents = await db.execute(select(Agent).where(Agent.developer_id == ...))
        for agent in agents.scalars():
            agent.latitude = payload.latitude
            agent.longitude = payload.longitude
    await db.commit()           # ONE commit, at the end
    return {"success": True}

Three layers stacked on top of each other to produce the silent NO-OP:

1. The sandbox's Commander developer (dev_id=1) owns the canary roster — 5,834 agents pre-funded for cross-frame test traffic. The propagation loop iterated each one and set latitude/longitude on the row.
2. Under sweep load, that loop took more than thirty seconds to complete.
3. The MCP wrapper uses HTTPX_TIMEOUT_MUTATE = 30.0 in mcp/tools_developer.py:_dev_request. When the request exceeded 30 seconds, the httpx client cancelled, the TCP connection closed, FastAPI's middleware saw a CancelledError, and SQLAlchemy's async session rolled back the entire transaction — including the developer's own row, which had been mutated in a millisecond at the top of the handler.

The read-after-write check saw the pre-mutation value. From the caller's perspective: 2xx, then the next read says nothing changed. This is the silent-NO-OP class that strict-mode read-after-write verifiers exist to surface. The harness was finally good enough to find it. Three layers of fix shipped together:

# routers/agent_locations.py:update_my_location
async def update_my_location(payload, ..., db, current_developer):
    current_developer.latitude = payload.latitude
    current_developer.longitude = payload.longitude
    current_developer.location_label = payload.location_label
    await db.commit()           # PRIMARY persisted — independent of propagation outcome

    children_updated = 0
    if payload.update_agents:
        try:
            agents = await db.execute(select(Agent).where(Agent.developer_id == ...))
            for agent in agents.scalars():
                agent.latitude = payload.latitude
                agent.longitude = payload.longitude
            children_updated = ...
            await db.commit()    # propagation in its own transaction; best-effort
        except Exception as e:
            logger.warning(f"propagation failed after primary commit: {e!r}")
            try: await db.rollback()
            except Exception: pass
    return {"success": True, "updated_agents": children_updated}

Layer one: the endpoint splits into two commits — primary first (small, fast, essential), propagation second (potentially slow, best-effort, in its own try/except). A client timeout can roll back the propagation; it can't roll back the primary anymore. Layer two: the lifecycle now passes update_agents=False to keep the test deterministic and fast — testing the timeout-rollback failure mode is the harness's other job, not this lifecycle's. Layer three: the wrapper's inputSchema declared country and city fields that the endpoint doesn't accept, and omitted location_label and update_agents that the endpoint does. Aligned the schema to truth. The truly correct fix is to push the propagation onto a BackgroundTasks queue or a worker so the client never blocks on it; the two-commit split is the minimum surgery to break the failure mode without rewriting the queue layer.

Reference implementation: routers/agent_locations.py:update_my_location. The pattern is generalized in the memory entry feedback_endpoint_split_commit_for_slow_propagation — any endpoint where a fast primary write pairs with a slow secondary propagation in the same transaction is structurally vulnerable to this class of failure. The grep target is wider than you'd think.

Bug #2 — deletePipeline: a 2xx that lies because soft-delete leaked through the LIST endpoint

This one was the same bug class wearing different clothes — and it was only caught because the morning's tester work explicitly added a read-after-delete verifier for the pipeline lifecycle. lifecycle_pipeline_full_crud had been checking the create / activate / rollback steps but never the post-delete state. Phase E.4 (an hour after E.2 shipped) tightened the contract to follow every deletePipeline with a getMyCicdPipelines call and assert the deleted ID is gone. Without that one new line of verifier, the bug would have shipped silently for an indeterminate length of time. The verifier is the prerequisite for the bug-find. The doctrine, repeated: a 2xx is not the test — the next read is.

The lifecycle did the equivalently obvious thing: POST /pipelines, POST /pipelines/{id}/versions, activate the version, then DELETE /pipelines/{id}. Then fetch getMyCicdPipelines and confirm the pipeline ID is gone. BUG-FIND: deletePipeline returned 2xx but pipeline_id=2297bbcc-… still present in getMyCicdPipelines.

Root cause was upstream of the delete itself, in the LIST endpoint. The pipeline-delete service did the right thing: when the pipeline had active deployments (because the lifecycle activated and rolled back a version, which leaves agent_deployments rows behind), delete_pipeline() detected deployment_count > 0 and soft-deleted (UPDATE pipelines SET is_active=false) instead of hard-DELETE-ing — correct business rule, preserve deployment history for compliance. The pipeline row stayed in the table with is_active=false as the marker.

The matching LIST endpoint, get_pipelines(), was the broken half:

# services/agent_cicd_service.py:get_pipelines
async def get_pipelines(self, db, agent_did, developer_id):
    sql = "SELECT … FROM pipelines WHERE agent_did = :a"
    # no is_active filter — every row, active or soft-deleted, returned

From the caller's perspective: 2xx on delete, then the next list call shows the row still there. Indistinguishable from a delete that didn't work. The fix is one boolean and one branch:

# services/agent_cicd_service.py:get_pipelines
async def get_pipelines(
    self, db, agent_did, developer_id,
    include_inactive: bool = False,   # default-filter
):
    sql = "SELECT … FROM pipelines WHERE agent_did = :a"
    if not include_inactive:
        sql += " AND is_active = TRUE"
    …

Default-filter is_active=true; opt-in include_inactive=true recovers the history for legitimate audit use cases. The matching lifecycle verifier was relaxed in parallel to tolerate is_active=false as legitimately-deleted — the verifier was already checking absence; it now accepts either-absence-or-soft-deleted as "delete worked." The general principle, captured in memory as feedback_list_endpoint_must_filter_soft_deleted: any service method that has an is_active / deleted_at / archived_at column and a matching LIST counterpart that doesn't filter on it is shipping a 2xx-that-lies the moment a soft-delete fires. The grep target, again, is wider than it should be.

Phase F — five new questions the harness now asks

Catching two bugs with the existing read-after-write verifiers was the proof-of-life. The afternoon's work was building five more detection axes that ask questions the original seven couldn't.

F.1 — wrong-tier auth matrix. A new expect_tier_rejections contract field plus an auto_tier_rejections(tier) helper. Default mapping: a tier-2 (agent JWT) tool must reject the empty JWT and a developer JWT; a tier-3 (developer JWT) tool must reject the empty JWT and an agent JWT; admin tools must reject all three of empty / agent / non-admin developer. A new Phase 5b walks every tool with declared rejections after the per-tool success-path sweep, sends a wrong-tier JWT with empty arguments, and asserts HTTP 401/403 (auth gate must fire before arg validation). Per-tool violations land in auth_matrix_failed_tools[] capped at 50. Five hundred and forty-eight wrong-tier probes added to the sweep; about ninety seconds of extra wall clock. The first solo sweep with F.1 active surfaced zero violations — no privilege-escalation regressions in 380 public-bridge tools. The cost of getting that confidence is a check that fires every sweep instead of a check that fires never.

A class of false positive surfaced and was fixed during the bring-up: my initial rejection-detection looked at HTTP status code 401/403 and a top-level JSON-RPC error field. MCP JSON-RPC actually returns wrapper-validation errors (missing args, bad params) as HTTP 200 with the error embedded in the result content as text — {"result": {"content": [{"text": "Error: agent_jwt is required"}]}}. My check missed those. The fix is to also treat any content[0].text starting with "Error:" or containing any of auth, permission, unauthor, forbidden, not allowed, is required, missing, must be admin, could not validate, invalid as a rejection signal. The first run with the broken detector reported 192 false positives. The patched detector reports zero. The shape of the error matters; the wrapper's envelope shape matters; both are now part of the contract.

F.2 — per-tool latency baseline. A Redis-backed circular buffer per (frame, tool) storing the last ten healthy-run elapsed_ms values (mcp_tester:latency:{frame}:{tool}, 30-day TTL). After each sweep, compute the median of the prior three; if the current value exceeds 2.0 × median, tag the row with latency_regression=True and the median as latency_baseline_ms. Only healthy runs contribute to the baseline buffer — a 30-second timeout on a failed run would otherwise poison the median for the next three sweeps. The threshold doesn't fire until the buffer has three entries; the first three sweeps after deploy are warm-up. The FE renders a yellow ⚠ {ratio}× baseline chip on regressed rows in the All-tools tab. The whole thing is advisory; latency regressions don't impact pass_pct. The point is that a tool drifting from 50ms to 800ms over a month no longer slides by unnoticed.

F.3 — wrapper-route schema drift. A new module api_tests/contracts/check_wrapper_route_drift.py that runs as a Phase 0 hook before the sweep starts. Hybrid implementation: an AST pass walks mcp/tools_{developer,agent,discovery}.py and extracts every {"name": "theprotocol_*", "inputSchema": {...}} dict literal; a regex pass finds the first _dev_request(…) / _agent_get(…) / client.method(…) call in each async wrapper and captures the (method, path) pair; an OpenAPI lookup fetches /openapi.json from the live registry and resolves each (method, path) to its Pydantic body schema (following $ref into components.schemas). The diff compares the wrapper's inputSchema inner-body property names against the route's body properties and reports wrapper_only[] and openapi_only[] per drift case. This is the check that would have caught Bug #1's wrapper-vs-endpoint mismatch (country/city vs location_label/update_agents) at edit time, not weeks later. Standalone CLI: python -m agentvault_registry.api_tests.contracts.check_wrapper_route_drift. About 50ms per sweep — one HTTP fetch plus an AST parse. The first solo sweep with F.3 active surfaced zero drift cases — Bug #1's wrapper fix held.

F.4 — cross-frame parity. A new _mcp_rpc_call_b() that mirrors _mcp_rpc_call() but targets target.b_registry_container. A curated list of 25 read-only public tools (discoverAgents, discoverRegistries, getNetworkStats, getApyRates, …). A _shape_signature(body) builder that fingerprints a response by its top-level keys plus JSON type per value. A new Phase 5c that calls each curated tool against sandbox-A and sandbox-B in parallel via asyncio.gather and diffs the signatures. Drifts land in cross_frame_drift_details[]. Skips gracefully if sandbox-B is unreachable (counter b_unreachable increments instead of failing the sweep). About 15–30 extra seconds of wall clock. The first solo sweep reported zero drift cases — the curated set returns identical shapes on both frames — which is itself a useful claim to be able to prove rather than assume.

F.5 — eventual-consistency budget. A new expect_consistency_within: Optional[int] field plus consistency_check: Callable[[bag], Optional[str]]. The verifier polls the predicate every 2 seconds up to the declared budget; the predicate returns None for consistent or a string explaining why not. Wired into _verify_post_call_assertions. Also exposes a generic _eventually(predicate, timeout=60, gap=2) helper for lifecycles asserting eventual consistency on non-contract surfaces — federation_sync card propagation, eigentrust epoch advance, agent_card pull-sync cache invalidation. Cost: zero unless contracts opt in. None opted in for the solo sweep; the infrastructure is in place for the next batch.

What still runs underneath

Supply invariant at delta zero across both sandbox frames. delta=0.0 status=OK agent_count=5834 on sandbox-A; =6 on sandbox-B. Forty-seven of forty-seven reactors per frame reporting ws_connected=1 against the new mTLS sidecars. The cross-frame canary lifecycle (lifecycle_cross_registry_canary_recipient — 1-AVT transfer from sandbox-A sentinel to a pre-funded sandbox-B canary) lands every sweep, asserts CrossFrameTransferInitiated in the EventStore within the retry window, and rolls cleanly. The Prometheus emission shape stabilized at six labelled gauges plus a runs counter; Grafana alerts on mcp_tester_strict_pass_rate < 0.99 are armed but unfired.

The trajectory sparkline now shows the climb visibly. It is the only chart in the admin view that goes only one direction by construction — the buffer that backs it stores nine consecutive sweep summaries and is cleared on registry rebuild, which means the climb resets at every doctrine change. That is the desired behaviour. The number on the right of the chart is the only one that matters in steady-state; the climb to it is the story of getting there.

Six-seventy-eight of six-eighty-two healthy on the solo sweep. Zero hard fails. Zero auth-matrix violations across 548 probes. Zero wrapper-route drift across the live tool catalog. Zero cross-frame shape divergence across the 25 curated tools. Zero latency regressions (baseline buffer just started filling; the threshold can't fire yet). Zero contract gaps. Zero tier drifts. Twenty-five minutes nineteen seconds of wall clock. Two real bugs caught and fixed earlier in the day. The numbers on the dashboard are not the work; the harness behind them is. The harness was a fancy ping script three days ago. The harness now alarms on regression instead of rubber-stamping it. That is the only delivered work that mattered this week.

Worth saying out loud because the surfaces overlap and the distinction matters: the API tester at /ui#/api-tester still exists and still ships. That's a separate layer — the one that hits every FastAPI route directly via the OpenAPI-introspection catalog, currently sitting at 100% effective across 696 endpoints (the run from earlier this week that finally passed cleanly). The MCP tester lives one layer above it: same backend, different shape of test — the MCP wrapper, the JSON-RPC envelope, the per-tool contract. Both surfaces are sandbox-only. Both run after every code change. Neither replaces the other; the API tester catches "did the FastAPI route return the right Pydantic body" and the MCP tester catches "did the wrapper call the route correctly, did the tool actually do what its description claims." Different blast radii. Different bug classes. Both green at the same time is how I sleep.

Next pass is queued and short: wire expect_consistency_within on the federation_sync + pull-sync surfaces (F.5 currently has the infrastructure but zero opt-ins); seed a true non-admin developer fixture so the auth-matrix can exercise admin-tier rejections against developer-but-not-admin JWTs (current sandbox commander is is_admin=True so that probe is a silent no-op); expand the cross-frame parity curated set from 25 to 50; once the sandbox holds ≥99.5% for a week of nightly sweeps, plan the Frame-A roll. The numbers on the cards will keep being boring across all of that. Boring remains the target.

While you're reading this I have probably already deployed it to prod. The doctrine was sandbox-only. The doctrine remains sandbox-only. The doctrine survives me ignoring the doctrine. None of those three sentences contradict each other if you don't look at them too hard.

— ruFFa, May 2026 (solo founder, sole reviewer, sole author of every test that's ever caught a real bug on this stack, sole regret about how long it took to make the harness honest, sole convener of the doctrine that I am about to break for reasons that I am sure will be obvious in retrospect)