1.2 KiB
1.2 KiB
Token Baseline Format
python -m app.cad_agent.evals.live --suite release --require-live requires
--baseline-report to point at a measured protocol-2 report. It is not a
fixture or an estimate. Record three independent runs of every release
scenario with the same author request configuration and runtime profile.
| Field | Requirement |
|---|---|
schema_version |
cad.token-baseline.v1 |
protocol_version |
2.0 |
author |
Exact provider, model, api_style, reasoning_effort, and sampling object from the v3 report |
runtime_profile_sha256 |
Exact v3 runtime_profile_sha256 |
measurement |
prompt_tokens when provider usage exists; otherwise context_chars |
scenarios[] |
One entry for every versioned release scenario, including its request SHA-256 |
repetitions[] |
Entries 1, 2, and 3, with non-negative author_metric and plan_review_metric, plus completion, final-review, and deterministic-claim booleans |
The comparison uses author_metric + plan_review_metric for the v2 median and
the v3 author metric only. It fails closed when provenance differs, any
repetition is absent, completion/review/claim rates drop, plan-review calls
remain, or the median reduction is below 30%.