StateProof The agent said it was done. Prove it.

Improvement changelog

Every iteration, including the ones that did not work.

Six steps, two of which failed their quality guardrails and are kept exactly as they ran. Every row links to the report, manifest, predictions, prompt and decision record behind it — the raw model responses for each contract are on the Trajectories page.

  1. Core-12 diagnostic baseline saturated

    A frontier evaluator with the full trajectory and both state snapshots classified every Core-12 case correctly. Overall PASS/FAIL accuracy had no headroom left to measure.

    BVA 100.0%

    report manifest predictions prompt v1

  2. Hard-12 requirement-level baseline saturated

    Requirement-level scoring on a harder suite still did not separate the systems: the frozen frontier baseline reached full recall and complete diagnosis. Accuracy stopped being the interesting axis; cost and determinism started.

    SVR 100.0% · CDR 100.0% · FVR 0.0%

    report manifest predictions prompt v2

  3. StateProof v1 exposed the DSL limits

    Compile-once-then-verify worked, and every overall verdict was right. What it lost was diagnosis: the vocabulary could not say "only the support case for this order may change", and a prohibited refund got counted twice — once as a prohibition, once as scope.

    SVR 83.3% · CDR 50.0% · FVR 4.3% — no efficiency claimed

    report manifest predictions prompt v1 decision record

  4. StateProof v2 fixed relational scope and note structure

    All three v1 defects closed: relational mutation scope, one note carrying both its exact text and its refund reference, and scope no longer duplicating a prohibition. A new defect appeared — outbound messages were identified by recipient alone, so a pre-existing message to the same person made the check unresolvable.

    SVR 91.7% · CDR 75.0% · BVA 75.0% — warm run withheld

    report manifest predictions prompt v2 decision record

  5. StateProof v3 added existential matching

    `record_exists_matching` asks whether a record satisfying every condition exists, instead of selecting one candidate first. A semantic lint refuses a contract that identifies an outbound record by recipient alone. Every guardrail met, with zero repair retries.

    SVR 100.0% · FVR 0.0% · CDR 100.0% · BVA 100.0%

    report manifest predictions prompt v3 decision record

  6. Measured warm verification proved zero-call reuse

    Three consecutive warm runs from the committed contract bundle, in a child process with no credential in its environment and no .env in its working directory. Byte-identical canonical predictions, identical contract hashes, identical metrics.

    0 model calls · 0 tokens · sha256 3d8ef516fa5d6d6b

    warm report warm manifest repeat 1 repeat 2 changelog

  7. Untouched locked evaluation, run once

    After the source freeze, the four held-out cases were run exactly once for both systems: the frontier baseline live, and StateProof from the frozen contract bundle with no credential in the environment. Neither was rerun, and the one-time protocol makes a second attempt impossible.

    StateProof locked SVR 100.0% · CDR 100.0% · BVA 100.0% · 0 model calls

    locked StateProof report locked baseline report final evaluation ledger

Nothing has been removed from this history. Two of the six steps are failures, and they are the reason the final efficiency claim is believable: the same code that reports the win withheld it twice.