22 lines
1.2 KiB
Markdown
22 lines
1.2 KiB
Markdown
# Token Baseline Format
|
|
|
|
`python -m app.cad_agent.evals.live --suite release --require-live` requires
|
|
`--baseline-report` to point at a measured protocol-2 report. It is not a
|
|
fixture or an estimate. Record three independent runs of every release
|
|
scenario with the same author request configuration and runtime profile.
|
|
|
|
| Field | Requirement |
|
|
| --- | --- |
|
|
| `schema_version` | `cad.token-baseline.v1` |
|
|
| `protocol_version` | `2.0` |
|
|
| `author` | Exact `provider`, `model`, `api_style`, `reasoning_effort`, and `sampling` object from the v3 report |
|
|
| `runtime_profile_sha256` | Exact v3 `runtime_profile_sha256` |
|
|
| `measurement` | `prompt_tokens` when provider usage exists; otherwise `context_chars` |
|
|
| `scenarios[]` | One entry for every versioned release scenario, including its request SHA-256 |
|
|
| `repetitions[]` | Entries `1`, `2`, and `3`, with non-negative `author_metric` and `plan_review_metric`, plus completion, final-review, and deterministic-claim booleans |
|
|
|
|
The comparison uses `author_metric + plan_review_metric` for the v2 median and
|
|
the v3 author metric only. It fails closed when provenance differs, any
|
|
repetition is absent, completion/review/claim rates drop, plan-review calls
|
|
remain, or the median reduction is below 30%.
|