Three tiers, only one of which costs anything

Probes & judging Preview

Nine deterministic probes over every run, rubric judging over a sample, and SQL-precompressed corpus mining across the campaign.

Tier 1 — nine probes, zero tokens

completion, run_records, session_outcome — did the run actually finish and record itself?
dispatch_hygiene, fan_out — sane dispatch counts, capped at 100.
verifier_coverage — did the verification actually run?
artifact_substance — missing, unreadable, degenerate (under 200 bytes), or written outside its confine.
event_delivery — did the event log receive what it should have?
stall — more than 15 minutes between consecutive dispatches in one session.

What each tier costs to run

Tier 1 — probes100% of runs coveredzero tokens, reads the event store
Tier 2 — rubric judging20% of runs coveredone repetition in five, plus every flagged run
Tier 3 — corpus mining100% of runs coveredone dispatch over a SQL aggregate

The sampling rate for Tier 2 is a campaign setting; the shape is what matters. Everything free runs over everything, and the one expensive tier is the one that reads a pre-compressed aggregate rather than the corpus itself.

A campaign report

Screenshot pendingA teamtester campaign report: one row per subject run with a pass or fail mark under each of the nine Tier-1 probes, the rubric scores for the sampled runs beside them, and the Tier-3 systemic findings listed underneath with links back to the runs that evidence each one.

The probe columns are deterministic — they come from reading each run's own event store, so this half of the report costs nothing and says the same thing every time it is regenerated.

They read the database, not the transcript

Every probe reads the subject run's own event-store SQLite rather than parsing its prose. That is what makes them deterministic and free: there is no model in the loop, so a Tier-1 verdict costs nothing and cannot itself be wrong in an interesting way.

Tier 2 — the judgment seat never rotates

A sampled slice of runs is scored 0–10 per criterion with quoted evidence. Calibration runs and any Tier-1-flagged run are always judged. Critically, the judge is always the same high-reasoning Claude model regardless of which backend produced the artifact under test — you cannot ask a model to grade its own output and call the result a comparison.

Tier 3 — the pre-compression is SQL

Systemic defects are mined across the whole campaign by aggregating probe failures, run states and per-criterion score means with plain GROUP BY queries, then handing only that aggregate to a single tools-disabled dispatch that returns schema-constrained findings from a fixed enum — friction-point, silent-verifier, swallowed-error, pathological-fan-out, serial-opportunity, cache-opportunity, cost-outlier, portability-defect. Each links back to the exact runs that evidence it.