change
This commit is contained in:
@@ -0,0 +1 @@
|
||||
只使用当前工具 schema、`operation_contract` 和服务端提供的 topology/reference token;它们高于本指引、示例和任何经验。每次只完成被调度的一个原子操作,不编造 atom、字段、selector 或能力。参数须为有限 mm/deg 数值。已有可执行 checkpoint 是应保留的最佳结果;能力缺口、未验证项和视觉疑点要如实交给服务端证据流程,不能用虚构几何掩盖。
|
||||
@@ -0,0 +1 @@
|
||||
先区分显式事实、图像观察、工程默认值和未知项。提取单位、外形、功能面、孔/槽、配合关系、关键尺寸及可验证目标。默认值只能补足常见零件的非关键构造,不能把未说明尺寸伪装成用户要求或确定性验收值。只有安全、配合、合规或可建模性确实取决于一个缺失事实时,才提出一个聚焦澄清;其余不确定性记录为假设或风险。
|
||||
@@ -0,0 +1 @@
|
||||
把尺寸当作模型契约:先识别主控的长度、宽度、厚度、直径、中心距、节距、数量、半径和角度,再从它们推导重复位置、对称偏移和余量。所有尺寸明确使用 mm,角度使用 deg;长度、直径、深度、节距和圆角半径必须为合理正有限值。阵列优先由中心线、数量、节距、半径或角度推导,避免难以追溯的点坐标常数。提交前以包围盒、比例、壁厚/材料余量和目标特征数量做常识检查。
|
||||
@@ -0,0 +1 @@
|
||||
世界坐标默认是右手 mm。根 `XY` 与 `+Z` 挤出只用于当前 contract 允许的根挤出。原点按功能基准选取:对称件取中心,板件取占地中心,轴对称件取轴线,存在安装或配合接口时取安装面、配合轴或明确接口基准。`workplane.origin_mm` 是局部 `(0,0)` 的世界点,`x_dir` 是局部 `+X`,`normal` 指向正挤出;草图局部坐标不是世界坐标。孔 `positions[].mm` 是宿主面上的绝对世界点,不是面内偏移。非根特征依据当前 contract、测量宿主和 token 建立 frame,不能猜测最后生成面或全局平面。`reference_plane`/`reference_axis` 是唯一支持的命名定位上下文;先创建基准,再创建依赖它的旋转、镜像、阵列和草图。位置必须由基准、中心线、偏移、节距或半径导出;定位失败时改 frame、偏移或方向,不修改已发布工件。
|
||||
@@ -0,0 +1 @@
|
||||
优先把零件身份和主控尺寸写进稳定根特征:根体、主要增材体、主要切除、孔/槽、重复特征、最后的圆角/倒角。每个节点只承担一个原子意图,依赖边只表示直接几何前提。默认形成连通单体;确需多体时必须由目标和 contract 支持。避免把视觉装饰、细小倒角或易碎布尔放在主形体之前。重规划时保留已完成节点和可执行检查点,只替换最小必要子图。
|
||||
@@ -0,0 +1 @@
|
||||
轮廓必须闭合、不自交、无零长或重叠边,并清楚区分外环和内环。先验证 workplane 的原点、`x_dir`、`normal` 与局部轮廓方向;翻转方向使用 contract 允许的字段,不凭视觉猜测。切除从实际材料面进入,深度覆盖目标材料并满足当前预检;避免刚好停在共面边界。对薄壁、近相切、重叠工具和零厚度结果保持余量。切除失败先检查宿主、方向、深度和轮廓,再考虑更换建模顺序。
|
||||
@@ -0,0 +1 @@
|
||||
宿主特征只能使用当前 revision 的测量 topology 和服务端给出的不透明 selector token;不得按边/面列表下标、历史名称或“最后一个面”猜选。选择前核对 token 的 kind、中心、法向、包围盒和 surface_type 是否覆盖预期材料区域。布尔、孔、阵列、圆角后拓扑可能变化,旧 token 和 reference 不可假定仍有效;依赖新拓扑时重新观察。reference token 只按当前 contract 放入允许槽位。选择不确定时请求 topology,而不是提交模糊 selector。
|
||||
@@ -0,0 +1 @@
|
||||
对称和重复优先通过 `pattern_linear`、`pattern_mirror` 及其 contract 参数表达。先完成一个正确的源特征,再用中心面、中心线、方向、数量、节距、半径或角度定义重复关系;不要用零散手填坐标代替可追溯模式。镜像平面和阵列方向应来自已建立的 datum 或当前测量 token。阵列前确认源特征、间距和数量不会重叠、越界或使材料变成零厚度。
|
||||
@@ -0,0 +1 @@
|
||||
圆角和倒角仅在主形体、切除和孔稳定后执行,并只选择唯一、当前有效的边 token;禁止“所有边”式回退。半径/距离必须小于邻近材料可容纳范围,避免相邻圆角相交。布尔操作避开共面终止、近相切和重复工具重叠;若风险高,优先以更稳定的主轮廓、顺序或足够余量表达。失败时不要重复原片段,先诊断受影响的面、边、深度和拓扑。
|
||||
@@ -0,0 +1 @@
|
||||
确定性几何事实与视觉审查职责不同:包围盒、实体数、孔深或贯穿状态只能证明已测量的 claim,不能证明整体设计语义。使用当前 contract、预检结果、claim evidence、render manifest 和 recent failures 作决定。视觉不符时给出具体的形状、位置、方向或比例差异作为修复依据,不能把它伪装成确定性通过。仅在几何改变后重新审查;STEP/checkpoint 是主工件,GLB 和渲染是派生审查证据,不能替代 CAD 几何。
|
||||
@@ -0,0 +1 @@
|
||||
修复先读错误和证据,定位最小责任点,再改最小的 CDSL/计划部分并重新执行依赖检查。常见原因包括开环/自交轮廓、零或负尺寸、切除方向或深度错误、错误 host frame、布尔后的旧 selector、过大圆角和直径/半径混淆。不要原样重试已失败片段。运行时不支持的能力应作为风险或缺口保留并继续发布最佳可执行模型,不能发明新 atom 或删除有效 checkpoint。
|
||||
@@ -0,0 +1,63 @@
|
||||
# CDSL Author Guidance Corpus
|
||||
|
||||
This corpus is a Chinese-first, non-authoritative author aid. The runtime
|
||||
operation contract, fragment schema, topology/reference tokens, preflight and
|
||||
verifier evidence always win over these Markdown files. The manifest maps
|
||||
only workflow phase, scheduled atomic operation and repair state; it never
|
||||
classifies the user's part request.
|
||||
|
||||
## Source Migration
|
||||
|
||||
| Source reference | CDSL target sections | Intentionally excluded |
|
||||
| --- | --- | --- |
|
||||
| `cad-brief.md` | `01`, `02`, `09` | Python/file workflow |
|
||||
| `parameters.md` | `02`, `07` | sidecars, animation, viewer control |
|
||||
| `positioning.md` | `03`, `06`, `op-reference` | assemblies, joints, `Location`, imported STEP placement |
|
||||
| `build123d-modeling.md` | `03` through `08`, operation appendices | build123d APIs, labels, colors and assembly source |
|
||||
| `build123d-modeling.zh-CN.md` | all Chinese terminology and rule review | a duplicate competing rule set |
|
||||
| `inspection-and-validation.md` | `09`, `10` | CLI paths and selector syntax |
|
||||
| `snapshot-review.md` | `09` | renderer commands |
|
||||
| `repair-loop.md` | `10`, `05`, `06`, `08` | build123d-only remediation syntax |
|
||||
| `step-generation.md` | `00`, `09` | Python generator commands |
|
||||
| `supported-exports.md` | `09` | mesh tolerance and exporter-specific flags |
|
||||
|
||||
## Selection Contract
|
||||
|
||||
- Requirements authoring selects `00` to `03`.
|
||||
- Feature planning selects `00`, `02`, `03`, `04`, `07`, and `08`.
|
||||
- A scheduled feature selects `00`, `03` to `06`, `08`, and its current
|
||||
operation appendix.
|
||||
- Repair selects `00`, `03`, `06`, `09`, `10`, and its operation appendix.
|
||||
- Final validation selects `00`, `09`, and `10`.
|
||||
|
||||
At a bounded prompt budget, contract, coordinate/datum, and the scheduled
|
||||
operation appendix are mandatory. Other sections are included in stable
|
||||
priority order. A malformed corpus or an unsupported operation registry
|
||||
falls back to the original short author prompt and records fallback metadata
|
||||
with the author usage record.
|
||||
|
||||
## Evaluation Commands
|
||||
|
||||
Run the six matched control scenarios three times each, first without and
|
||||
then with guidance:
|
||||
|
||||
```bash
|
||||
PYTHONPATH=backend python -m app.cad_agent.evals.live --suite comprehensive --repetitions 3 --author-guidance off \
|
||||
--scenario rectangular_mounting_plate --scenario circular_flange_pcd \
|
||||
--scenario obround_slot_plate --scenario rounded_rectangular_pocket \
|
||||
--scenario double_hole_linkage_arm --scenario l_bracket
|
||||
|
||||
PYTHONPATH=backend python -m app.cad_agent.evals.live --suite comprehensive --repetitions 3 --author-guidance on \
|
||||
--scenario rectangular_mounting_plate --scenario circular_flange_pcd \
|
||||
--scenario obround_slot_plate --scenario rounded_rectangular_pocket \
|
||||
--scenario double_hole_linkage_arm --scenario l_bracket
|
||||
|
||||
PYTHONPATH=backend python -m app.cad_agent.evals.live --compare-guidance-reports CONTROL/report.json TREATMENT/report.json
|
||||
```
|
||||
|
||||
The comparator excludes declared validation gaps and engine-declared
|
||||
`unsupported_*` capability gaps from prompt quality metrics, checks paired
|
||||
model/runtime/contract/budget equivalence, and
|
||||
requires the treatment's checkpoint/completion rates not to regress, median
|
||||
author calls to stay within 10 percent, and either schema/decision or CDSL
|
||||
expression failures to improve.
|
||||
@@ -0,0 +1,56 @@
|
||||
{
|
||||
"schema_version": "cdsl.author-guidance.manifest.v1",
|
||||
"version": "2026-09-03.1",
|
||||
"sections": [
|
||||
{"id": "00-author-contract", "file": "00-author-contract.md", "title": "00 Author Contract", "priority": 100, "mandatory": true},
|
||||
{"id": "01-brief-and-assumptions", "file": "01-brief-and-assumptions.md", "title": "01 Brief And Assumptions", "priority": 70, "mandatory": false},
|
||||
{"id": "02-parameters-and-derived-dimensions", "file": "02-parameters-and-derived-dimensions.md", "title": "02 Parameters And Derived Dimensions", "priority": 80, "mandatory": false},
|
||||
{"id": "03-coordinate-system-and-datums", "file": "03-coordinate-system-and-datums.md", "title": "03 Coordinate System And Datums", "priority": 100, "mandatory": true},
|
||||
{"id": "04-construction-and-feature-order", "file": "04-construction-and-feature-order.md", "title": "04 Construction And Feature Order", "priority": 70, "mandatory": false},
|
||||
{"id": "05-profiles-workplanes-and-cuts", "file": "05-profiles-workplanes-and-cuts.md", "title": "05 Profiles Workplanes And Cuts", "priority": 90, "mandatory": false},
|
||||
{"id": "06-hosted-features-selectors-and-topology", "file": "06-hosted-features-selectors-and-topology.md", "title": "06 Hosted Features Selectors And Topology", "priority": 90, "mandatory": false},
|
||||
{"id": "07-patterns-symmetry-and-repetition", "file": "07-patterns-symmetry-and-repetition.md", "title": "07 Patterns Symmetry And Repetition", "priority": 60, "mandatory": false},
|
||||
{"id": "08-finishing-and-boolean-risk", "file": "08-finishing-and-boolean-risk.md", "title": "08 Finishing And Boolean Risk", "priority": 60, "mandatory": false},
|
||||
{"id": "09-evidence-visual-review-and-validation", "file": "09-evidence-visual-review-and-validation.md", "title": "09 Evidence Visual Review And Validation", "priority": 80, "mandatory": false},
|
||||
{"id": "10-repair-and-best-effort", "file": "10-repair-and-best-effort.md", "title": "10 Repair And Best Effort", "priority": 80, "mandatory": false},
|
||||
{"id": "op-extrude-add", "file": "op-extrude-add.md", "title": "Operation Appendix Extrude Add", "priority": 100, "mandatory": true},
|
||||
{"id": "op-extrude-cut", "file": "op-extrude-cut.md", "title": "Operation Appendix Extrude Cut", "priority": 100, "mandatory": true},
|
||||
{"id": "op-revolve", "file": "op-revolve.md", "title": "Operation Appendix Revolve", "priority": 100, "mandatory": true},
|
||||
{"id": "op-hole", "file": "op-hole.md", "title": "Operation Appendix Hole", "priority": 100, "mandatory": true},
|
||||
{"id": "op-reference", "file": "op-reference.md", "title": "Operation Appendix Reference", "priority": 100, "mandatory": true},
|
||||
{"id": "op-pattern", "file": "op-pattern.md", "title": "Operation Appendix Pattern", "priority": 100, "mandatory": true},
|
||||
{"id": "op-finish", "file": "op-finish.md", "title": "Operation Appendix Finish", "priority": 100, "mandatory": true},
|
||||
{"id": "op-sphere", "file": "op-sphere.md", "title": "Operation Appendix Sphere", "priority": 100, "mandatory": true}
|
||||
],
|
||||
"phase_sections": {
|
||||
"DEFAULT": ["00-author-contract", "03-coordinate-system-and-datums", "09-evidence-visual-review-and-validation"],
|
||||
"DRAFTING_REQUIREMENTS_DOCUMENT": ["00-author-contract", "01-brief-and-assumptions", "02-parameters-and-derived-dimensions", "03-coordinate-system-and-datums"],
|
||||
"DRAFTING_COMPLETION_TARGET": ["00-author-contract", "01-brief-and-assumptions", "02-parameters-and-derived-dimensions", "03-coordinate-system-and-datums"],
|
||||
"COMPILING_REQUIREMENTS": ["00-author-contract", "01-brief-and-assumptions", "02-parameters-and-derived-dimensions", "03-coordinate-system-and-datums"],
|
||||
"COMPILING_FEATURE_PLAN": ["00-author-contract", "02-parameters-and-derived-dimensions", "03-coordinate-system-and-datums", "04-construction-and-feature-order", "07-patterns-symmetry-and-repetition", "08-finishing-and-boolean-risk"],
|
||||
"REPLANNING_FEATURE_SUBGRAPH": ["00-author-contract", "02-parameters-and-derived-dimensions", "03-coordinate-system-and-datums", "04-construction-and-feature-order", "07-patterns-symmetry-and-repetition", "08-finishing-and-boolean-risk"],
|
||||
"FEATURE_PENDING": ["00-author-contract", "03-coordinate-system-and-datums", "04-construction-and-feature-order", "05-profiles-workplanes-and-cuts", "06-hosted-features-selectors-and-topology", "08-finishing-and-boolean-risk"],
|
||||
"AWAITING_ACTION": ["00-author-contract", "03-coordinate-system-and-datums", "06-hosted-features-selectors-and-topology", "09-evidence-visual-review-and-validation", "10-repair-and-best-effort"],
|
||||
"ACTION_PENDING": ["00-author-contract", "03-coordinate-system-and-datums", "06-hosted-features-selectors-and-topology", "09-evidence-visual-review-and-validation", "10-repair-and-best-effort"]
|
||||
},
|
||||
"repair_sections": ["00-author-contract", "03-coordinate-system-and-datums", "06-hosted-features-selectors-and-topology", "09-evidence-visual-review-and-validation", "10-repair-and-best-effort"],
|
||||
"final_sections": ["00-author-contract", "09-evidence-visual-review-and-validation", "10-repair-and-best-effort"],
|
||||
"operation_sections": {
|
||||
"extrude_add_blind": ["op-extrude-add"],
|
||||
"extrude_add_two_sided": ["op-extrude-add"],
|
||||
"extrude_cut_blind": ["op-extrude-cut"],
|
||||
"revolve_add": ["op-revolve"],
|
||||
"revolve_cut": ["op-revolve"],
|
||||
"hole_blind": ["op-hole"],
|
||||
"hole_counterbore": ["op-hole"],
|
||||
"hole_countersink": ["op-hole"],
|
||||
"hole_wizard": ["op-hole"],
|
||||
"reference_plane": ["op-reference"],
|
||||
"reference_axis": ["op-reference"],
|
||||
"pattern_linear": ["op-pattern"],
|
||||
"pattern_mirror": ["op-pattern"],
|
||||
"fillet": ["op-finish"],
|
||||
"chamfer": ["op-finish"],
|
||||
"sphere_add": ["op-sphere"]
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1 @@
|
||||
`extrude_add_blind` 和 `extrude_add_two_sided` 必须使用闭合草图和 contract 允许的正距离。根挤出遵守根 `XY` datum;后续增材先确认草图 frame 与已有实体的连接。双向挤出分别核对两个方向的长度与材料范围;`reverse` 只用于当前 frame 的方向修正,不能代替错误的 workplane。
|
||||
@@ -0,0 +1 @@
|
||||
`extrude_cut_blind` 使用闭合草图、当前允许的正距离和正确宿主 frame。从实际材料面进入,方向由 workplane normal 与 contract 的 `reverse` 决定;深度应覆盖目标材料,不能刚好停在共面边界。切除失败时先检查轮廓、宿主、方向、深度和材料覆盖,而不是盲目加大距离。
|
||||
@@ -0,0 +1 @@
|
||||
`fillet` 与 `chamfer` 仅接受当前 revision 中唯一且合格的 edge selector token。先完成影响这些边的布尔、孔和阵列,再按 contract 使用正半径或距离。局部材料不足、相邻过渡相交或 token 已失效时,不用全局边选择兜底;保留主体并报告该收尾特征的风险。
|
||||
@@ -0,0 +1 @@
|
||||
孔 atom 需要当前宿主面的有效 selector token。`positions[].mm` 使用该宿主面上的绝对世界坐标,先核对点在面区域内与法向方向。直径、深度、沉孔/沉头参数以 contract 为准,深度覆盖预期材料;多孔共享一个原子操作时保持同一规格和同一宿主。不要把点写成面局部偏移或裸数组。
|
||||
@@ -0,0 +1 @@
|
||||
`pattern_linear` 只复制当前 contract 允许且存在的源 feature reference;方向是明确世界/基准方向,数量和 spacing 为合理值。`pattern_mirror` 使用存在的镜像 plane reference,先确认源与平面关系以及复制后不会重叠或意外合并。pattern 不代替新的宿主选择;下游特征若依赖新面,重新读取 topology。
|
||||
@@ -0,0 +1 @@
|
||||
`reference_plane` 用有限非零 `normal` 和与其不平行的 `x_dir` 定义局部 frame;`origin_mm` 是世界点。`reference_axis` 用有限非零 `direction` 和世界原点定义。它们只建立可追溯 datum,不直接制造实体;先于依赖它的旋转、镜像、阵列或定位特征,并依照 contract 的 reference token 规则引用。
|
||||
@@ -0,0 +1 @@
|
||||
`revolve_add` 和 `revolve_cut` 的轴必须由明确 datum 或 contract 中的世界坐标轴表达,并按预检要求位于正确的草图关系中。核对 axis origin、direction、角度和 `reverse`;完整回转避免轮廓跨轴造成自交,局部回转避免与现有材料近相切。切除回转仍必须覆盖目标材料。
|
||||
@@ -0,0 +1 @@
|
||||
`sphere_add` 用明确的世界中心和正有限半径定义。确认它与目标实体的连接意图:需要单体时应有足够相交,独立体仅在需求允许多体时使用。球体位置从 datum 或主尺寸导出,不把视图坐标误当世界坐标。
|
||||
@@ -0,0 +1,191 @@
|
||||
"""Bounded, file-backed author guidance for the CDSL workflow.
|
||||
|
||||
The corpus is deliberately non-authoritative: contracts, schemas, topology
|
||||
tokens, and server preflight always remain the executable source of truth.
|
||||
Loading errors return an empty selection so authoring continues with the
|
||||
pre-guidance prompt instead of turning documentation into an availability
|
||||
dependency.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
from dataclasses import dataclass
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
from app.cad_agent.domain.state import TaskPhase
|
||||
from app.cad_agent.ports import AuthorGuidanceSelection
|
||||
|
||||
|
||||
_MIN_CHARS = 1_200
|
||||
_MAX_CHARS = 6_000
|
||||
|
||||
|
||||
@dataclass(frozen=True, slots=True)
|
||||
class _Section:
|
||||
section_id: str
|
||||
title: str
|
||||
priority: int
|
||||
mandatory: bool
|
||||
content: str
|
||||
|
||||
@property
|
||||
def block(self) -> str:
|
||||
return f"## {self.title}\n{self.content.strip()}"
|
||||
|
||||
|
||||
@dataclass(frozen=True, slots=True)
|
||||
class _Corpus:
|
||||
version: str
|
||||
sections: dict[str, _Section]
|
||||
phase_sections: dict[str, tuple[str, ...]]
|
||||
repair_sections: tuple[str, ...]
|
||||
final_sections: tuple[str, ...]
|
||||
operation_sections: dict[str, tuple[str, ...]]
|
||||
|
||||
|
||||
class FileAuthorGuidance:
|
||||
"""Read and select the checked-in corpus deterministically.
|
||||
|
||||
Selection depends exclusively on workflow state and the runtime operation
|
||||
registry. It intentionally receives neither the user's request nor image
|
||||
observations, so it cannot become an implicit part-family classifier.
|
||||
"""
|
||||
|
||||
def __init__(self, root: Path, *, enabled: bool = True, max_chars: int = 3_600) -> None:
|
||||
self.root = root
|
||||
self.enabled = enabled
|
||||
self.max_chars = min(_MAX_CHARS, max(_MIN_CHARS, max_chars))
|
||||
self._corpus: _Corpus | None = None
|
||||
self._load_error = ""
|
||||
|
||||
def select(
|
||||
self,
|
||||
*,
|
||||
phase: TaskPhase,
|
||||
atomic_id: str,
|
||||
repair_required: bool,
|
||||
supported_atomic_ids: tuple[str, ...],
|
||||
) -> AuthorGuidanceSelection:
|
||||
if not self.enabled:
|
||||
return AuthorGuidanceSelection(fallback_reason="guidance_disabled")
|
||||
corpus = self._load()
|
||||
if corpus is None:
|
||||
return AuthorGuidanceSelection(fallback_reason=self._load_error or "guidance_unavailable")
|
||||
supported = set(supported_atomic_ids)
|
||||
if set(corpus.operation_sections) != supported:
|
||||
return AuthorGuidanceSelection(fallback_reason="guidance_operation_coverage_mismatch")
|
||||
if atomic_id and atomic_id not in supported:
|
||||
return AuthorGuidanceSelection(fallback_reason="guidance_unknown_atomic_id")
|
||||
|
||||
if repair_required:
|
||||
requested = list(corpus.repair_sections)
|
||||
elif phase == TaskPhase.FINAL_VALIDATION:
|
||||
requested = list(corpus.final_sections)
|
||||
else:
|
||||
requested = list(corpus.phase_sections.get(phase.value, corpus.phase_sections.get("DEFAULT", ())))
|
||||
if atomic_id:
|
||||
requested.extend(corpus.operation_sections[atomic_id])
|
||||
requested = list(dict.fromkeys(requested))
|
||||
if not requested:
|
||||
return AuthorGuidanceSelection(fallback_reason="guidance_no_matching_sections")
|
||||
|
||||
mandatory = [section_id for section_id in requested if corpus.sections[section_id].mandatory]
|
||||
optional = [section_id for section_id in requested if not corpus.sections[section_id].mandatory]
|
||||
optional.sort(key=lambda section_id: (-corpus.sections[section_id].priority, requested.index(section_id)))
|
||||
selected: list[str] = []
|
||||
text = ""
|
||||
for section_id in [*mandatory, *optional]:
|
||||
block = corpus.sections[section_id].block
|
||||
candidate = block if not text else f"{text}\n\n{block}"
|
||||
if len(candidate) <= self.max_chars:
|
||||
text = candidate
|
||||
selected.append(section_id)
|
||||
elif section_id in mandatory:
|
||||
# Do not silently drop contract, datum, or operation guidance.
|
||||
return AuthorGuidanceSelection(fallback_reason="guidance_required_sections_exceed_budget")
|
||||
return AuthorGuidanceSelection(
|
||||
version=corpus.version,
|
||||
section_ids=tuple(selected),
|
||||
content=text,
|
||||
enabled=True,
|
||||
)
|
||||
|
||||
def _load(self) -> _Corpus | None:
|
||||
if self._corpus is not None:
|
||||
return self._corpus
|
||||
if self._load_error:
|
||||
return None
|
||||
try:
|
||||
manifest_path = self.root / "manifest.json"
|
||||
raw = json.loads(manifest_path.read_text(encoding="utf-8"))
|
||||
if not isinstance(raw, dict):
|
||||
raise ValueError("manifest is not an object")
|
||||
version = raw.get("version")
|
||||
if raw.get("schema_version") != "cdsl.author-guidance.manifest.v1" or not isinstance(version, str) or not version:
|
||||
raise ValueError("manifest version is invalid")
|
||||
raw_sections = raw.get("sections")
|
||||
if not isinstance(raw_sections, list) or not raw_sections:
|
||||
raise ValueError("manifest sections are invalid")
|
||||
sections: dict[str, _Section] = {}
|
||||
root = self.root.resolve()
|
||||
for item in raw_sections:
|
||||
if not isinstance(item, dict):
|
||||
raise ValueError("section declaration is invalid")
|
||||
section_id = item.get("id")
|
||||
filename = item.get("file")
|
||||
title = item.get("title")
|
||||
priority = item.get("priority")
|
||||
mandatory = item.get("mandatory", False)
|
||||
if (
|
||||
not isinstance(section_id, str) or not section_id
|
||||
or not isinstance(filename, str) or not filename
|
||||
or not isinstance(title, str) or not title
|
||||
or not isinstance(priority, int) or isinstance(priority, bool)
|
||||
or not isinstance(mandatory, bool)
|
||||
or section_id in sections
|
||||
):
|
||||
raise ValueError("section metadata is invalid")
|
||||
path = (self.root / filename).resolve()
|
||||
if root not in path.parents or not path.is_file():
|
||||
raise ValueError("section file is unavailable")
|
||||
content = path.read_text(encoding="utf-8").strip()
|
||||
if not content:
|
||||
raise ValueError("section content is empty")
|
||||
sections[section_id] = _Section(section_id, title, priority, mandatory, content)
|
||||
|
||||
def identifiers(value: Any, field: str) -> tuple[str, ...]:
|
||||
if not isinstance(value, list) or not value or not all(isinstance(item, str) and item in sections for item in value):
|
||||
raise ValueError(f"{field} is invalid")
|
||||
return tuple(dict.fromkeys(value))
|
||||
|
||||
raw_phases = raw.get("phase_sections")
|
||||
if not isinstance(raw_phases, dict) or "DEFAULT" not in raw_phases:
|
||||
raise ValueError("phase sections are invalid")
|
||||
phase_sections = {
|
||||
phase: identifiers(section_ids, f"phase {phase}")
|
||||
for phase, section_ids in raw_phases.items()
|
||||
if isinstance(phase, str)
|
||||
}
|
||||
if len(phase_sections) != len(raw_phases):
|
||||
raise ValueError("phase name is invalid")
|
||||
operation_sections = {
|
||||
atomic_id: identifiers(section_ids, f"operation {atomic_id}")
|
||||
for atomic_id, section_ids in (raw.get("operation_sections") or {}).items()
|
||||
if isinstance(atomic_id, str)
|
||||
}
|
||||
if not operation_sections or len(operation_sections) != len(raw.get("operation_sections") or {}):
|
||||
raise ValueError("operation sections are invalid")
|
||||
self._corpus = _Corpus(
|
||||
version=version,
|
||||
sections=sections,
|
||||
phase_sections=phase_sections,
|
||||
repair_sections=identifiers(raw.get("repair_sections"), "repair sections"),
|
||||
final_sections=identifiers(raw.get("final_sections"), "final sections"),
|
||||
operation_sections=operation_sections,
|
||||
)
|
||||
return self._corpus
|
||||
except (OSError, ValueError, TypeError, json.JSONDecodeError) as error:
|
||||
self._load_error = f"guidance_load_failed:{type(error).__name__}"
|
||||
return None
|
||||
@@ -46,13 +46,67 @@ class StructuredModelGateway:
|
||||
|
||||
async def call_tool(self, *, messages: list[dict[str, Any]], tool: dict[str, Any], provider_id: str, model_id: str, required_tool_name: str) -> dict[str, Any]:
|
||||
provider, model = self._provider_model(provider_id, model_id)
|
||||
payload = self._payload(provider, model.id, messages, [tool], required_tool_name)
|
||||
response = await self._request(provider, payload)
|
||||
response, compatibility_mode = await self._request_tool_call(
|
||||
provider, model.id, messages, [tool], required_tool_name,
|
||||
)
|
||||
try:
|
||||
return self._normalized_response(provider, response)
|
||||
normalized = self._normalized_response(provider, response)
|
||||
normalized["usage"]["structured_compatibility_mode"] = compatibility_mode
|
||||
return normalized
|
||||
except (KeyError, TypeError, ValueError) as error:
|
||||
raise StructuredModelError("Provider response cannot be normalized as a structured tool-call result") from error
|
||||
|
||||
async def _request_tool_call(
|
||||
self,
|
||||
provider: ProviderConfig,
|
||||
model_id: str,
|
||||
messages: list[dict[str, Any]],
|
||||
tools: list[dict[str, Any]],
|
||||
required_tool_name: str,
|
||||
) -> tuple[dict[str, Any], str]:
|
||||
"""Keep forced tool choice unless a provider explicitly rejects thinking mode.
|
||||
|
||||
Some OpenAI-compatible gateways reject an otherwise valid forced
|
||||
``tool_choice`` whenever reasoning is enabled or implicit. This is a
|
||||
request compatibility issue rather than an author error. First retry
|
||||
without the client-requested reasoning option; only if that exact
|
||||
rejection remains do we allow automatic choice among the single tool
|
||||
already exposed to the model. The workflow's one-call/schema checks
|
||||
still reject free-form or wrong-tool responses.
|
||||
"""
|
||||
try:
|
||||
return await self._request(
|
||||
provider,
|
||||
self._payload(provider, model_id, messages, tools, required_tool_name),
|
||||
), "forced"
|
||||
except StructuredModelError as first_error:
|
||||
if not self._thinking_tool_choice_rejection(first_error):
|
||||
raise
|
||||
try:
|
||||
return await self._request(
|
||||
provider,
|
||||
self._payload(
|
||||
provider, model_id, messages, tools, required_tool_name,
|
||||
include_reasoning=False,
|
||||
),
|
||||
), "reasoning_disabled"
|
||||
except StructuredModelError as second_error:
|
||||
if not self._thinking_tool_choice_rejection(second_error):
|
||||
raise
|
||||
return await self._request(
|
||||
provider,
|
||||
self._payload(
|
||||
provider, model_id, messages, tools, required_tool_name,
|
||||
include_reasoning=False,
|
||||
force_tool_name=False,
|
||||
),
|
||||
), "single_tool_auto"
|
||||
|
||||
@staticmethod
|
||||
def _thinking_tool_choice_rejection(error: StructuredModelError) -> bool:
|
||||
message = str(error).casefold()
|
||||
return "thinking" in message and ("tool_choice" in message or "tool choice" in message)
|
||||
|
||||
async def conformance(self, *, provider_id: str, model_id: str, tools: list[dict[str, Any]]) -> dict[str, Any]:
|
||||
failures: list[dict[str, str]] = []
|
||||
usage = {"prompt_tokens": 0, "completion_tokens": 0}
|
||||
@@ -190,20 +244,31 @@ class StructuredModelGateway:
|
||||
})
|
||||
return values
|
||||
|
||||
def _payload(self, provider: ProviderConfig, model_id: str, messages: list[dict[str, Any]], tools: list[dict[str, Any]], required_tool_name: str) -> dict[str, Any]:
|
||||
def _payload(
|
||||
self,
|
||||
provider: ProviderConfig,
|
||||
model_id: str,
|
||||
messages: list[dict[str, Any]],
|
||||
tools: list[dict[str, Any]],
|
||||
required_tool_name: str,
|
||||
*,
|
||||
include_reasoning: bool = True,
|
||||
force_tool_name: bool = True,
|
||||
) -> dict[str, Any]:
|
||||
if len(tools) != 1 or not required_tool_name:
|
||||
raise StructuredModelError("The v3 protocol requires exactly one named tool per provider call.")
|
||||
request_options = provider.request_options if include_reasoning else {}
|
||||
if provider.api_style == "responses":
|
||||
payload: dict[str, Any] = {
|
||||
"model": model_id, "input": self._responses_input(messages), "tools": self._responses_tools(tools),
|
||||
"tool_choice": {"type": "function", "name": required_tool_name},
|
||||
**provider.request_options,
|
||||
"tool_choice": {"type": "function", "name": required_tool_name} if force_tool_name else "auto",
|
||||
**request_options,
|
||||
}
|
||||
return payload
|
||||
return {
|
||||
"model": model_id, "messages": messages, "tools": tools,
|
||||
"tool_choice": {"type": "function", "function": {"name": required_tool_name}},
|
||||
"temperature": 0, **provider.request_options,
|
||||
"tool_choice": {"type": "function", "function": {"name": required_tool_name}} if force_tool_name else "auto",
|
||||
"temperature": 0, **request_options,
|
||||
}
|
||||
|
||||
@staticmethod
|
||||
|
||||
@@ -33,7 +33,17 @@ from app.cad_agent.domain.feature_plan import FeaturePlan, plan_hash
|
||||
from app.cad_agent.domain.errors import ErrorCode, WorkflowError
|
||||
from app.cad_agent.domain.operation_contract import fragment_schema
|
||||
from app.cad_agent.domain.state import TaskPhase, TaskState, retry_resume_event, transition
|
||||
from app.cad_agent.ports import AdapterUnavailable, ArtifactStore, CadRuntime, ModelGateway, ReviewGateway, TaskRepository
|
||||
from app.cad_agent.ports import (
|
||||
AdapterUnavailable,
|
||||
ArtifactStore,
|
||||
AuthorGuidance,
|
||||
AuthorGuidanceSelection,
|
||||
CadRuntime,
|
||||
ModelGateway,
|
||||
NullAuthorGuidance,
|
||||
ReviewGateway,
|
||||
TaskRepository,
|
||||
)
|
||||
|
||||
|
||||
T = TypeVar("T", bound=BaseModel)
|
||||
@@ -116,6 +126,7 @@ class WorkflowCoordinator:
|
||||
review_gateway: ReviewGateway,
|
||||
requirements: RequirementsCommandHandler,
|
||||
actions: ActionCommandHandler,
|
||||
author_guidance: AuthorGuidance | None = None,
|
||||
) -> None:
|
||||
self.config = config
|
||||
self.repository = repository
|
||||
@@ -125,6 +136,7 @@ class WorkflowCoordinator:
|
||||
self.review_gateway = review_gateway
|
||||
self.requirements = requirements
|
||||
self.actions = actions
|
||||
self.author_guidance = author_guidance or NullAuthorGuidance()
|
||||
|
||||
def create_task(
|
||||
self,
|
||||
@@ -1078,14 +1090,14 @@ class WorkflowCoordinator:
|
||||
"message": message[:1000],
|
||||
}
|
||||
|
||||
async def _author_turn(self, task_id: str, author: ModelIdentity, tools: list[dict[str, Any]], feedback: list[dict[str, Any]]) -> tuple[str, str, dict[str, int]] | WorkflowError:
|
||||
async def _author_turn(self, task_id: str, author: ModelIdentity, tools: list[dict[str, Any]], feedback: list[dict[str, Any]]) -> tuple[str, str, dict[str, Any]] | WorkflowError:
|
||||
if len(tools) != 1:
|
||||
raise RuntimeError("Workflow state must expose exactly one author tool.")
|
||||
tool = tools[0]
|
||||
name = str((tool.get("function") or {}).get("name") or "")
|
||||
if not name:
|
||||
raise RuntimeError("Workflow exposed an unnamed author tool.")
|
||||
messages = self._author_context(task_id, feedback)
|
||||
messages, guidance = self._author_context(task_id, feedback)
|
||||
try:
|
||||
response = await self.model_gateway.call_tool(
|
||||
messages=messages, tool=tool, provider_id=author.provider_id,
|
||||
@@ -1103,10 +1115,26 @@ class WorkflowCoordinator:
|
||||
"model_id": author.model_id,
|
||||
"retry_reason": "provider_unavailable",
|
||||
"cache_hit": False,
|
||||
**guidance.usage_metadata(),
|
||||
})
|
||||
return WorkflowError(ErrorCode.AUTHOR_TRANSPORT_UNAVAILABLE, str(error)[:1000], retryable=True)
|
||||
call = validate_one_tool_call(response["tool_calls"], name)
|
||||
if isinstance(call, WorkflowError):
|
||||
# A provider response is still a billable author attempt even when
|
||||
# it violates the one-tool-call protocol. Keep guidance audit
|
||||
# metadata on that record so A/B reports do not silently omit the
|
||||
# failures this corpus is intended to reduce.
|
||||
self.repository.record_usage(task_id, {
|
||||
**response["usage"],
|
||||
"context_chars": len(json.dumps(messages, ensure_ascii=False)),
|
||||
"tool": name,
|
||||
"provider_id": author.provider_id,
|
||||
"model_id": author.model_id,
|
||||
"raw_arguments_hash": "",
|
||||
"retry_reason": "invalid_tool_call",
|
||||
"cache_hit": False,
|
||||
**guidance.usage_metadata(),
|
||||
})
|
||||
self._record_rejected_tool_calls(
|
||||
task_id,
|
||||
actor="author",
|
||||
@@ -1132,6 +1160,7 @@ class WorkflowCoordinator:
|
||||
"raw_arguments_hash": raw_arguments_hash(raw),
|
||||
"retry_reason": "",
|
||||
"cache_hit": False,
|
||||
**guidance.usage_metadata(),
|
||||
}
|
||||
self.repository.record_usage(task_id, usage)
|
||||
return name, raw, usage
|
||||
@@ -1400,10 +1429,10 @@ class WorkflowCoordinator:
|
||||
return []
|
||||
return [self._tool("record_geometry_conclusion", StatelessGeometryConclusion)]
|
||||
|
||||
def _author_context(self, task_id: str, feedback: list[dict[str, Any]]) -> list[dict[str, Any]]:
|
||||
def _author_context(self, task_id: str, feedback: list[dict[str, Any]]) -> tuple[list[dict[str, Any]], AuthorGuidanceSelection]:
|
||||
state = self.repository.get_state(task_id)
|
||||
if state is None:
|
||||
return []
|
||||
return [], AuthorGuidanceSelection(fallback_reason="task_state_unavailable")
|
||||
if state.phase == TaskPhase.DRAFTING_REQUIREMENTS_DOCUMENT:
|
||||
content = {
|
||||
"protocol": "cad.v3.2.feature-dag",
|
||||
@@ -1557,8 +1586,17 @@ class WorkflowCoordinator:
|
||||
"feature_plan_hash": state.feature_plan_hash,
|
||||
"feature_node_statuses": self._feature_node_statuses(task_id, None),
|
||||
})
|
||||
messages: list[dict[str, Any]] = [{"role": "system", "content": "You are the autonomous CAD author. Use exactly one offered structured tool call. Never emit Markdown plans or free-form JSON."}, {"role": "user", "content": json.dumps(content, ensure_ascii=False)}]
|
||||
return [*messages, *feedback[-2:]]
|
||||
guidance = self.author_guidance.select(
|
||||
phase=state.phase,
|
||||
atomic_id=state.pending_feature.atomic_id if state.pending_feature is not None else "",
|
||||
repair_required=state.repair_required,
|
||||
supported_atomic_ids=self.runtime.supported_atomic_ids(),
|
||||
)
|
||||
system = "You are the autonomous CAD author. Use exactly one offered structured tool call. Never emit Markdown plans or free-form JSON."
|
||||
if guidance.content:
|
||||
system += "\n\nThe following is non-authoritative CDSL author guidance. The current tool schema, operation contract, and server facts take precedence.\n\n" + guidance.content
|
||||
messages: list[dict[str, Any]] = [{"role": "system", "content": system}, {"role": "user", "content": json.dumps(content, ensure_ascii=False)}]
|
||||
return [*messages, *feedback[-2:]], guidance
|
||||
|
||||
def _user_clarifications(self, task_id: str) -> list[dict[str, str]]:
|
||||
clarifications: list[dict[str, str]] = []
|
||||
|
||||
@@ -6,6 +6,7 @@ from dataclasses import dataclass
|
||||
import shutil
|
||||
|
||||
from app.cad_agent.adapters.artifact_store import FileArtifactStore
|
||||
from app.cad_agent.adapters.author_guidance import FileAuthorGuidance
|
||||
from app.cad_agent.adapters.event_publisher import IdempotentInProcessPublisher
|
||||
from app.cad_agent.adapters.runtime import ProfileCadRuntime
|
||||
from app.cad_agent.adapters.review_gateway import RenderedReviewGateway
|
||||
@@ -17,7 +18,7 @@ from app.cad_agent.application.outbox import OutboxDispatcher
|
||||
from app.cad_agent.application.requirements import RequirementsCommandHandler
|
||||
from app.cad_agent.application.workflow import ModelIdentity, WorkflowConfig, WorkflowCoordinator
|
||||
from app.cad_agent.domain.verifier_registry import default_registry
|
||||
from app.settings import Settings
|
||||
from app.settings import BACKEND_ROOT, Settings
|
||||
from app.services.storage import WorkspaceStore
|
||||
|
||||
|
||||
@@ -65,5 +66,10 @@ def compose_v3(settings: Settings) -> V3Services:
|
||||
RenderedReviewGateway(models),
|
||||
requirements,
|
||||
actions,
|
||||
FileAuthorGuidance(
|
||||
BACKEND_ROOT / "agent" / "skills" / "cdsl-author-guidance",
|
||||
enabled=settings.agent_author_guidance_enabled,
|
||||
max_chars=settings.agent_author_guidance_max_chars,
|
||||
),
|
||||
)
|
||||
return V3Services(repository, artifacts, workflow, models, outbox)
|
||||
|
||||
@@ -15,6 +15,7 @@ from hashlib import sha256
|
||||
import json
|
||||
from pathlib import Path
|
||||
import secrets
|
||||
from statistics import median
|
||||
import subprocess
|
||||
import sys
|
||||
from typing import Any
|
||||
@@ -100,12 +101,15 @@ def _arguments() -> argparse.Namespace:
|
||||
parser.add_argument("--author-model")
|
||||
parser.add_argument("--review-provider")
|
||||
parser.add_argument("--review-model")
|
||||
parser.add_argument("--scenario", help="Run one fixture scenario by its stable ID for targeted regression validation.")
|
||||
parser.add_argument("--scenario", action="append", dest="scenarios", help="Run a fixture scenario by stable ID. Repeat this option to select a comparison set.")
|
||||
parser.add_argument("--repetitions", type=int, help="Run every selected scenario this many times.")
|
||||
parser.add_argument("--author-guidance", choices=("on", "off"), help="Override CDSL author guidance for this run.")
|
||||
parser.add_argument("--compare-guidance-reports", nargs=2, type=Path, metavar=("CONTROL", "TREATMENT"), help="Compare matched --author-guidance off/on report.json files without invoking providers.")
|
||||
parser.add_argument("--baseline-report", type=Path, help="Measured pre-v3 token baseline JSON for a release run.")
|
||||
return parser.parse_args()
|
||||
|
||||
|
||||
def _fixture(suite: str, scenario_id: str | None = None) -> list[dict[str, Any]]:
|
||||
def _fixture(suite: str, scenario_ids: list[str] | None = None) -> list[dict[str, Any]]:
|
||||
fixture_name = "comprehensive.json" if suite == "comprehensive" else "release.json"
|
||||
value = json.loads((Path(__file__).parent / "fixtures" / fixture_name).read_text(encoding="utf-8"))
|
||||
if fixture_name == "comprehensive.json":
|
||||
@@ -122,11 +126,14 @@ def _fixture(suite: str, scenario_id: str | None = None) -> list[dict[str, Any]]
|
||||
raise ValueError("Comprehensive fixture is not synchronized with its source document.")
|
||||
values = [item for item in value.get("scenarios") or () if isinstance(item, dict)]
|
||||
values = values[:2] if suite == "smoke" else values
|
||||
if scenario_id is None:
|
||||
if not scenario_ids:
|
||||
return values
|
||||
selected = [item for item in values if str(item.get("id") or "") == scenario_id]
|
||||
if not selected:
|
||||
raise ValueError(f"Unknown scenario {scenario_id!r} for suite {suite!r}")
|
||||
requested = list(dict.fromkeys(scenario_ids))
|
||||
available = {str(item.get("id") or "") for item in values}
|
||||
unknown = [scenario_id for scenario_id in requested if scenario_id not in available]
|
||||
if unknown:
|
||||
raise ValueError(f"Unknown scenario {unknown[0]!r} for suite {suite!r}")
|
||||
selected = [item for item in values if str(item.get("id") or "") in set(requested)]
|
||||
return selected
|
||||
|
||||
|
||||
@@ -567,12 +574,180 @@ def _safe_artifact_manifest(artifact_root: Path) -> dict[str, Any]:
|
||||
return {"schema_version": "cad.live-eval-artifact-manifest.v1", "artifact_root": str(artifact_root), "files": files}
|
||||
|
||||
|
||||
def _guidance_metadata(usage: dict[str, Any]) -> dict[str, Any]:
|
||||
"""Summarize author-only guidance audit metadata without retaining prompts."""
|
||||
records = [
|
||||
item for item in usage.get("records") or ()
|
||||
if isinstance(item, dict) and item.get("role") != "reviewer"
|
||||
]
|
||||
sections = sorted({
|
||||
section_id
|
||||
for item in records
|
||||
for section_id in item.get("guidance_section_ids") or ()
|
||||
if isinstance(section_id, str)
|
||||
})
|
||||
versions = sorted({
|
||||
str(item.get("guidance_version") or "")
|
||||
for item in records
|
||||
if str(item.get("guidance_version") or "")
|
||||
})
|
||||
return {
|
||||
"enabled": bool(records) and any(item.get("guidance_enabled") is True for item in records),
|
||||
"versions": versions,
|
||||
"section_ids": sections,
|
||||
"chars_total": sum(int(item.get("guidance_chars") or 0) for item in records),
|
||||
"fallback_reasons": sorted({
|
||||
str(item.get("guidance_fallback_reason") or "")
|
||||
for item in records
|
||||
if str(item.get("guidance_fallback_reason") or "")
|
||||
}),
|
||||
}
|
||||
|
||||
|
||||
def _has_unsupported_capability(row: dict[str, Any]) -> bool:
|
||||
"""Recognize engine-declared unsupported capability without hiding model errors."""
|
||||
for event in row.get("ledger") or ():
|
||||
if not isinstance(event, dict):
|
||||
continue
|
||||
for failure in event.get("operation_failures") or ():
|
||||
if isinstance(failure, dict) and "unsupported_" in str(failure.get("message") or ""):
|
||||
return True
|
||||
if "unsupported_" in str(event.get("message") or ""):
|
||||
return True
|
||||
return False
|
||||
|
||||
|
||||
def _guidance_metric_rows(report: dict[str, Any]) -> tuple[list[dict[str, Any]], list[dict[str, Any]]]:
|
||||
results = [item for item in report.get("results") or () if isinstance(item, dict)]
|
||||
capability_gaps = [
|
||||
item for item in results
|
||||
if item.get("outcome") == "validation_capability_gap" or _has_unsupported_capability(item)
|
||||
]
|
||||
return [item for item in results if item not in capability_gaps], capability_gaps
|
||||
|
||||
|
||||
def _guidance_metrics(rows: list[dict[str, Any]]) -> dict[str, Any]:
|
||||
if not rows:
|
||||
return {"eligible_runs": 0}
|
||||
author_calls = [
|
||||
sum(1 for item in (row.get("usage") or {}).get("records") or () if isinstance(item, dict) and item.get("role") != "reviewer")
|
||||
for row in rows
|
||||
]
|
||||
context_chars = [
|
||||
sum(int(item.get("context_chars") or 0) for item in (row.get("usage") or {}).get("records") or () if isinstance(item, dict) and item.get("role") != "reviewer")
|
||||
for row in rows
|
||||
]
|
||||
prompt_tokens = [
|
||||
sum(int(item.get("prompt_tokens") or 0) for item in (row.get("usage") or {}).get("records") or () if isinstance(item, dict) and item.get("role") != "reviewer")
|
||||
for row in rows
|
||||
]
|
||||
failure_layers: dict[str, int] = {}
|
||||
for row in rows:
|
||||
layer = str((row.get("failure_attribution") or {}).get("layer") or "passed")
|
||||
failure_layers[layer] = failure_layers.get(layer, 0) + 1
|
||||
def rate(predicate: Any) -> float:
|
||||
return sum(1 for row in rows if predicate(row)) / len(rows)
|
||||
return {
|
||||
"eligible_runs": len(rows),
|
||||
"executable_checkpoint_rate": rate(lambda row: bool(row.get("revision_ids"))),
|
||||
"completion_rate": rate(lambda row: str((row.get("projection") or {}).get("phase") or "") == "COMPLETED"),
|
||||
"deterministic_claim_success_rate": rate(lambda row: bool((row.get("checks") or {}).get("deterministic_claims_pass"))),
|
||||
"schema_or_decision_rejections": sum(int(row.get("schema_rejection_count") or 0) for row in rows),
|
||||
"cdsl_expression_failures": sum(1 for row in rows if str((row.get("failure_attribution") or {}).get("layer") or "") == "cdsl_expression"),
|
||||
"median_author_calls": median(author_calls),
|
||||
"median_author_context_chars": median(context_chars),
|
||||
"total_author_prompt_tokens": sum(prompt_tokens),
|
||||
"median_author_prompt_tokens": median(prompt_tokens),
|
||||
"failure_layers": dict(sorted(failure_layers.items())),
|
||||
}
|
||||
|
||||
|
||||
def compare_guidance_reports(control: dict[str, Any], treatment: dict[str, Any]) -> dict[str, Any]:
|
||||
"""Compare paired guidance-off/on runs without treating engine gaps as prompt results."""
|
||||
control_rows, control_gaps = _guidance_metric_rows(control)
|
||||
treatment_rows, treatment_gaps = _guidance_metric_rows(treatment)
|
||||
control_by_key = {(str(row.get("scenario") or ""), int(row.get("repetition") or 0)): row for row in control_rows}
|
||||
treatment_by_key = {(str(row.get("scenario") or ""), int(row.get("repetition") or 0)): row for row in treatment_rows}
|
||||
paired = sorted(set(control_by_key).intersection(treatment_by_key))
|
||||
control_only = sorted(set(control_by_key).difference(treatment_by_key))
|
||||
treatment_only = sorted(set(treatment_by_key).difference(control_by_key))
|
||||
control_pairs = [control_by_key[key] for key in paired]
|
||||
treatment_pairs = [treatment_by_key[key] for key in paired]
|
||||
control_metrics = _guidance_metrics(control_pairs)
|
||||
treatment_metrics = _guidance_metrics(treatment_pairs)
|
||||
same_runtime = (
|
||||
control.get("author") == treatment.get("author")
|
||||
and control.get("reviewer") == treatment.get("reviewer")
|
||||
and control.get("runtime_profile_sha256") == treatment.get("runtime_profile_sha256")
|
||||
and control.get("operation_contracts") == treatment.get("operation_contracts")
|
||||
)
|
||||
control_guidance = bool((control.get("author_guidance") or {}).get("enabled"))
|
||||
treatment_guidance = bool((treatment.get("author_guidance") or {}).get("enabled"))
|
||||
same_budgets = all(
|
||||
control_by_key[key].get("scenario_budget") == treatment_by_key[key].get("scenario_budget")
|
||||
for key in paired
|
||||
)
|
||||
calls_control = control_metrics.get("median_author_calls")
|
||||
calls_treatment = treatment_metrics.get("median_author_calls")
|
||||
calls_within_limit = (
|
||||
isinstance(calls_control, (int, float))
|
||||
and isinstance(calls_treatment, (int, float))
|
||||
and calls_treatment <= calls_control * 1.10
|
||||
)
|
||||
improved = (
|
||||
treatment_metrics.get("schema_or_decision_rejections", 0) < control_metrics.get("schema_or_decision_rejections", 0)
|
||||
or treatment_metrics.get("cdsl_expression_failures", 0) < control_metrics.get("cdsl_expression_failures", 0)
|
||||
)
|
||||
gates = {
|
||||
"complete_pairing": bool(paired) and not control_only and not treatment_only,
|
||||
"control_off_treatment_on": not control_guidance and treatment_guidance,
|
||||
"same_author_reviewer_runtime_and_contracts": same_runtime,
|
||||
"same_per_scenario_budgets": same_budgets,
|
||||
"checkpoint_rate_not_lower": treatment_metrics.get("executable_checkpoint_rate", -1) >= control_metrics.get("executable_checkpoint_rate", 0),
|
||||
"completion_rate_not_lower": treatment_metrics.get("completion_rate", -1) >= control_metrics.get("completion_rate", 0),
|
||||
"median_author_calls_within_ten_percent": calls_within_limit,
|
||||
"model_or_cdsl_failure_improved": improved,
|
||||
}
|
||||
return {
|
||||
"schema_version": "cad.author-guidance-comparison.v1",
|
||||
"status": "passed" if all(gates.values()) else "failed",
|
||||
"gates": gates,
|
||||
"paired_runs": [{"scenario": scenario, "repetition": repetition} for scenario, repetition in paired],
|
||||
"unpaired_runs": {
|
||||
"control_only": [{"scenario": scenario, "repetition": repetition} for scenario, repetition in control_only],
|
||||
"treatment_only": [{"scenario": scenario, "repetition": repetition} for scenario, repetition in treatment_only],
|
||||
},
|
||||
"control": control_metrics,
|
||||
"treatment": treatment_metrics,
|
||||
"excluded_capability_gaps": {
|
||||
"control": [{"scenario": item.get("scenario"), "repetition": item.get("repetition")} for item in control_gaps],
|
||||
"treatment": [{"scenario": item.get("scenario"), "repetition": item.get("repetition")} for item in treatment_gaps],
|
||||
},
|
||||
}
|
||||
|
||||
|
||||
def compare_guidance_report_paths(control_path: Path, treatment_path: Path) -> dict[str, Any]:
|
||||
try:
|
||||
control = json.loads(control_path.read_text(encoding="utf-8"))
|
||||
treatment = json.loads(treatment_path.read_text(encoding="utf-8"))
|
||||
except (OSError, json.JSONDecodeError) as error:
|
||||
return {"status": "failed", "error": f"GUIDANCE_COMPARISON_INPUT_INVALID: {type(error).__name__}"}
|
||||
if not isinstance(control, dict) or not isinstance(treatment, dict):
|
||||
return {"status": "failed", "error": "GUIDANCE_COMPARISON_INPUT_INVALID: report must be an object"}
|
||||
return compare_guidance_reports(control, treatment)
|
||||
|
||||
|
||||
async def _run(arguments: argparse.Namespace, report_root: Path) -> dict[str, Any]:
|
||||
try:
|
||||
scenarios = _fixture(arguments.suite, arguments.scenario)
|
||||
scenarios = _fixture(arguments.suite, arguments.scenarios)
|
||||
except ValueError as error:
|
||||
return {"status": "LIVE_EVAL_BLOCKED", "error": str(error)}
|
||||
repetitions = arguments.repetitions if arguments.repetitions is not None else 3 if arguments.suite == "release" else 1
|
||||
if repetitions < 1:
|
||||
return {"status": "LIVE_EVAL_BLOCKED", "error": "--repetitions must be at least 1"}
|
||||
settings = get_settings()
|
||||
if arguments.author_guidance is not None:
|
||||
settings = replace(settings, agent_author_guidance_enabled=arguments.author_guidance == "on")
|
||||
try:
|
||||
author_provider, author_model = settings.resolve_model(arguments.author_provider, arguments.author_model)
|
||||
if arguments.review_provider or arguments.review_model:
|
||||
@@ -628,7 +803,7 @@ async def _run(arguments: argparse.Namespace, report_root: Path) -> dict[str, An
|
||||
}
|
||||
results: list[dict[str, Any]] = []
|
||||
for scenario in scenarios:
|
||||
for repetition in range(1, 4 if arguments.suite == "release" else 2):
|
||||
for repetition in range(1, repetitions + 1):
|
||||
services.workflow.config = replace(
|
||||
services.workflow.config,
|
||||
# State transitions include local candidate recovery, so the
|
||||
@@ -716,6 +891,13 @@ async def _run(arguments: argparse.Namespace, report_root: Path) -> dict[str, An
|
||||
"terminal": terminal, "projection": projection, "usage": usage, "checks": checks,
|
||||
"acceptance_coverage": acceptance,
|
||||
"failure_attribution": failure_attribution,
|
||||
"guidance": _guidance_metadata(usage),
|
||||
"scenario_budget": {
|
||||
"max_author_turns": int(scenario["max_author_turns"]),
|
||||
"max_reviewer_turns": int(scenario["max_reviewer_turns"]),
|
||||
"max_total_calls": int(scenario["max_total_calls"]),
|
||||
"max_total_tokens": int(scenario["max_total_tokens"]),
|
||||
},
|
||||
"rejection_codes": _rejection_codes(events),
|
||||
"schema_rejection_count": len(_rejection_codes(events)),
|
||||
"retry_count": sum(1 for event in events if (event.get("payload") or {}).get("status") == "error"),
|
||||
@@ -745,6 +927,10 @@ async def _run(arguments: argparse.Namespace, report_root: Path) -> dict[str, An
|
||||
"author": {"provider": author_provider.id, "model": author_model.id},
|
||||
"author_request_identity": current_author_identity,
|
||||
"reviewer": {"provider": review_provider.id, "model": review_model.id},
|
||||
"author_guidance": {
|
||||
"enabled": settings.agent_author_guidance_enabled,
|
||||
"max_chars": settings.agent_author_guidance_max_chars,
|
||||
},
|
||||
"author_capability": author_capability,
|
||||
"reviewer_capability": reviewer_capability,
|
||||
"structured_output_mode": {
|
||||
@@ -759,6 +945,7 @@ async def _run(arguments: argparse.Namespace, report_root: Path) -> dict[str, An
|
||||
"verifier_schema_hash": verifier_schema_hash,
|
||||
"token_comparison": token_comparison,
|
||||
"baseline_report": str(baseline_path) if baseline_path else "",
|
||||
"repetitions": repetitions,
|
||||
"results": results,
|
||||
}
|
||||
|
||||
@@ -768,7 +955,11 @@ def main() -> int:
|
||||
report_root = BACKEND_ROOT / "live-evals" / datetime.now(timezone.utc).strftime("%Y%m%dT%H%M%SZ")
|
||||
report_root.mkdir(parents=True, exist_ok=True)
|
||||
try:
|
||||
result = asyncio.run(_run(arguments, report_root))
|
||||
result = (
|
||||
compare_guidance_report_paths(*arguments.compare_guidance_reports)
|
||||
if arguments.compare_guidance_reports
|
||||
else asyncio.run(_run(arguments, report_root))
|
||||
)
|
||||
except KeyboardInterrupt:
|
||||
# Let an explicit operator interruption retain its normal CLI
|
||||
# semantics. An external kill cannot be reported reliably either.
|
||||
|
||||
@@ -5,13 +5,60 @@ from __future__ import annotations
|
||||
from dataclasses import dataclass
|
||||
from typing import Any, Protocol
|
||||
|
||||
from .domain.state import TaskState
|
||||
from .domain.state import TaskPhase, TaskState
|
||||
|
||||
|
||||
class AdapterUnavailable(RuntimeError):
|
||||
"""A bounded external-service outage; handlers must preserve checkpoints."""
|
||||
|
||||
|
||||
@dataclass(frozen=True, slots=True)
|
||||
class AuthorGuidanceSelection:
|
||||
"""Non-authoritative author context selected from the local guidance corpus."""
|
||||
|
||||
version: str = ""
|
||||
section_ids: tuple[str, ...] = ()
|
||||
content: str = ""
|
||||
enabled: bool = False
|
||||
fallback_reason: str = ""
|
||||
|
||||
def usage_metadata(self) -> dict[str, object]:
|
||||
return {
|
||||
"guidance_version": self.version,
|
||||
"guidance_section_ids": list(self.section_ids),
|
||||
"guidance_chars": len(self.content),
|
||||
"guidance_enabled": self.enabled,
|
||||
"guidance_fallback_reason": self.fallback_reason,
|
||||
}
|
||||
|
||||
|
||||
class AuthorGuidance(Protocol):
|
||||
"""Select bounded local author guidance without interpreting user intent."""
|
||||
|
||||
def select(
|
||||
self,
|
||||
*,
|
||||
phase: TaskPhase,
|
||||
atomic_id: str,
|
||||
repair_required: bool,
|
||||
supported_atomic_ids: tuple[str, ...],
|
||||
) -> AuthorGuidanceSelection: ...
|
||||
|
||||
|
||||
class NullAuthorGuidance:
|
||||
"""Compatibility default that retains the pre-guidance author prompt."""
|
||||
|
||||
def select(
|
||||
self,
|
||||
*,
|
||||
phase: TaskPhase,
|
||||
atomic_id: str,
|
||||
repair_required: bool,
|
||||
supported_atomic_ids: tuple[str, ...],
|
||||
) -> AuthorGuidanceSelection:
|
||||
return AuthorGuidanceSelection(fallback_reason="guidance_not_configured")
|
||||
|
||||
|
||||
@dataclass(frozen=True, slots=True)
|
||||
class InvocationRecord:
|
||||
invocation_id: str
|
||||
|
||||
@@ -71,6 +71,8 @@ class Settings:
|
||||
agent_consecutive_no_progress_limit: int = 6
|
||||
agent_format_error_repeat_limit: int = 3
|
||||
agent_context_char_limit: int = 14000
|
||||
agent_author_guidance_enabled: bool = True
|
||||
agent_author_guidance_max_chars: int = 3600
|
||||
agent_render_cache: bool = True
|
||||
autonomous_generation: bool = True
|
||||
resume_running_tasks_on_startup: bool = True
|
||||
@@ -211,6 +213,8 @@ def get_settings() -> Settings:
|
||||
agent_consecutive_no_progress_limit=max(1, int(os.getenv("CDSL_AGENT_CONSECUTIVE_NO_PROGRESS_LIMIT", "6"))),
|
||||
agent_format_error_repeat_limit=max(1, int(os.getenv("CDSL_AGENT_FORMAT_ERROR_REPEAT_LIMIT", "3"))),
|
||||
agent_context_char_limit=max(4000, int(os.getenv("CDSL_AGENT_CONTEXT_CHAR_LIMIT", "14000"))),
|
||||
agent_author_guidance_enabled=_env_flag("CDSL_AGENT_AUTHOR_GUIDANCE_ENABLED", True),
|
||||
agent_author_guidance_max_chars=min(6000, max(1200, int(os.getenv("CDSL_AGENT_AUTHOR_GUIDANCE_MAX_CHARS", "3600")))),
|
||||
agent_render_cache=_env_flag("CDSL_AGENT_RENDER_CACHE", True),
|
||||
autonomous_generation=True,
|
||||
# Production instances recover durable runs by default. Test workers
|
||||
|
||||
@@ -0,0 +1,158 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import json
|
||||
from pathlib import Path
|
||||
import sys
|
||||
import tempfile
|
||||
import unittest
|
||||
|
||||
|
||||
ROOT = Path(__file__).resolve().parents[2]
|
||||
sys.path.insert(0, str(ROOT / "backend"))
|
||||
|
||||
from app.cad_agent.adapters.author_guidance import FileAuthorGuidance # noqa: E402
|
||||
from app.cad_agent.application.workflow import ModelIdentity, WorkflowConfig, WorkflowCoordinator # noqa: E402
|
||||
from app.cad_agent.domain.errors import ErrorCode, WorkflowError # noqa: E402
|
||||
from app.cad_agent.domain.state import TaskPhase, TaskState # noqa: E402
|
||||
|
||||
|
||||
GUIDANCE_ROOT = ROOT / "backend" / "agent" / "skills" / "cdsl-author-guidance"
|
||||
PROFILE = ROOT / "backend" / "engine" / "cdsl_engine" / "profile_schema.json"
|
||||
|
||||
|
||||
def atomic_ids() -> tuple[str, ...]:
|
||||
return tuple(json.loads(PROFILE.read_text(encoding="utf-8"))["operation_contracts"])
|
||||
|
||||
|
||||
class _Repository:
|
||||
def __init__(self, state: TaskState) -> None:
|
||||
self.state = state
|
||||
self.usage_records: list[dict] = []
|
||||
|
||||
def get_state(self, _task_id: str) -> TaskState:
|
||||
return self.state
|
||||
|
||||
def ledger_events(self, _task_id: str) -> list[dict]:
|
||||
return []
|
||||
|
||||
def record_usage(self, _task_id: str, payload: dict) -> None:
|
||||
self.usage_records.append(payload)
|
||||
|
||||
def record_tool_audit(self, _task_id: str, _payload: dict) -> None:
|
||||
pass
|
||||
|
||||
|
||||
class _Artifacts:
|
||||
def read_source_requirements(self, _task_id: str) -> str:
|
||||
return "Create a symmetric mounting plate."
|
||||
|
||||
def read_json(self, *_args: object) -> None:
|
||||
return None
|
||||
|
||||
|
||||
class _Runtime:
|
||||
def supported_atomic_ids(self) -> tuple[str, ...]:
|
||||
return atomic_ids()
|
||||
|
||||
|
||||
class AuthorGuidanceTests(unittest.TestCase):
|
||||
def test_manifest_covers_every_runtime_atomic_and_keeps_coordinate_core_at_minimum_budget(self) -> None:
|
||||
guidance = FileAuthorGuidance(GUIDANCE_ROOT, max_chars=1_200)
|
||||
covered: set[str] = set()
|
||||
for atomic_id in atomic_ids():
|
||||
selection = guidance.select(
|
||||
phase=TaskPhase.FEATURE_PENDING,
|
||||
atomic_id=atomic_id,
|
||||
repair_required=False,
|
||||
supported_atomic_ids=atomic_ids(),
|
||||
)
|
||||
self.assertTrue(selection.enabled, selection.fallback_reason)
|
||||
self.assertIn("00-author-contract", selection.section_ids)
|
||||
self.assertIn("03-coordinate-system-and-datums", selection.section_ids)
|
||||
self.assertLessEqual(len(selection.content), 1_200)
|
||||
self.assertIn("世界坐标", selection.content)
|
||||
covered.update(section_id for section_id in selection.section_ids if section_id.startswith("op-"))
|
||||
self.assertEqual(covered, {"op-extrude-add", "op-extrude-cut", "op-revolve", "op-hole", "op-reference", "op-pattern", "op-finish", "op-sphere"})
|
||||
|
||||
def test_phase_repair_and_budget_selection_are_stable(self) -> None:
|
||||
guidance = FileAuthorGuidance(GUIDANCE_ROOT, max_chars=3_600)
|
||||
planning = guidance.select(
|
||||
phase=TaskPhase.COMPILING_FEATURE_PLAN,
|
||||
atomic_id="",
|
||||
repair_required=False,
|
||||
supported_atomic_ids=atomic_ids(),
|
||||
)
|
||||
repair = guidance.select(
|
||||
phase=TaskPhase.AWAITING_ACTION,
|
||||
atomic_id="fillet",
|
||||
repair_required=True,
|
||||
supported_atomic_ids=atomic_ids(),
|
||||
)
|
||||
self.assertEqual(planning.section_ids[:2], ("00-author-contract", "03-coordinate-system-and-datums"))
|
||||
self.assertIn("02-parameters-and-derived-dimensions", planning.section_ids)
|
||||
self.assertEqual(repair.section_ids[:3], ("00-author-contract", "03-coordinate-system-and-datums", "op-finish"))
|
||||
self.assertIn("10-repair-and-best-effort", repair.section_ids)
|
||||
|
||||
def test_disabled_missing_and_invalid_corpus_fall_back_without_authoring_failure(self) -> None:
|
||||
common = {
|
||||
"phase": TaskPhase.FEATURE_PENDING,
|
||||
"atomic_id": "extrude_add_blind",
|
||||
"repair_required": False,
|
||||
"supported_atomic_ids": atomic_ids(),
|
||||
}
|
||||
self.assertEqual(FileAuthorGuidance(GUIDANCE_ROOT, enabled=False).select(**common).fallback_reason, "guidance_disabled")
|
||||
with tempfile.TemporaryDirectory() as temporary:
|
||||
root = Path(temporary)
|
||||
self.assertEqual(FileAuthorGuidance(root).select(**common).fallback_reason, "guidance_load_failed:FileNotFoundError")
|
||||
(root / "manifest.json").write_text("{}", encoding="utf-8")
|
||||
self.assertEqual(FileAuthorGuidance(root).select(**common).fallback_reason, "guidance_load_failed:ValueError")
|
||||
|
||||
def test_author_context_receives_guidance_but_keeps_the_existing_tool_instruction(self) -> None:
|
||||
state = TaskState("cad_123456abcdef", TaskPhase.DRAFTING_REQUIREMENTS_DOCUMENT, 1)
|
||||
workflow = WorkflowCoordinator(
|
||||
WorkflowConfig(max_turns=8, format_error_limit=2),
|
||||
_Repository(state),
|
||||
_Artifacts(),
|
||||
_Runtime(),
|
||||
object(),
|
||||
object(),
|
||||
object(),
|
||||
object(),
|
||||
FileAuthorGuidance(GUIDANCE_ROOT),
|
||||
)
|
||||
messages, selection = workflow._author_context(state.task_id, [])
|
||||
system = str(messages[0]["content"])
|
||||
self.assertTrue(selection.enabled)
|
||||
self.assertIn("Use exactly one offered structured tool call", system)
|
||||
self.assertIn("Coordinate System And Datums", system)
|
||||
self.assertIn("世界坐标", system)
|
||||
|
||||
def test_invalid_author_tool_call_retains_guidance_usage_metadata(self) -> None:
|
||||
state = TaskState("cad_123456abcdef", TaskPhase.DRAFTING_REQUIREMENTS_DOCUMENT, 1)
|
||||
repository = _Repository(state)
|
||||
class _Models:
|
||||
async def call_tool(self, **_kwargs: object) -> dict:
|
||||
return {"tool_calls": [], "usage": {"prompt_tokens": 3, "completion_tokens": 1, "total_tokens": 4}}
|
||||
workflow = WorkflowCoordinator(
|
||||
WorkflowConfig(max_turns=8, format_error_limit=2),
|
||||
repository,
|
||||
_Artifacts(),
|
||||
_Runtime(),
|
||||
_Models(),
|
||||
object(),
|
||||
object(),
|
||||
object(),
|
||||
FileAuthorGuidance(GUIDANCE_ROOT),
|
||||
)
|
||||
tool = {"type": "function", "function": {"name": "write_requirements_document", "parameters": {"type": "object"}}}
|
||||
result = asyncio.run(workflow._author_turn(state.task_id, ModelIdentity("provider", "model"), [tool], []))
|
||||
self.assertIsInstance(result, WorkflowError)
|
||||
self.assertEqual(result.code, ErrorCode.AUTHOR_FORMAT_INVALID)
|
||||
self.assertEqual(repository.usage_records[0]["guidance_enabled"], True)
|
||||
self.assertIn("03-coordinate-system-and-datums", repository.usage_records[0]["guidance_section_ids"])
|
||||
self.assertEqual(repository.usage_records[0]["retry_reason"], "invalid_tool_call")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
@@ -0,0 +1,71 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from pathlib import Path
|
||||
import sys
|
||||
import unittest
|
||||
|
||||
|
||||
ROOT = Path(__file__).resolve().parents[2]
|
||||
sys.path.insert(0, str(ROOT / "backend"))
|
||||
|
||||
from app.cad_agent.evals.live import _fixture, compare_guidance_reports # noqa: E402
|
||||
|
||||
|
||||
def _result(*, scenario: str, repetition: int, author_calls: int, context_chars: int, schema_rejections: int, failure_layer: str = "") -> dict:
|
||||
return {
|
||||
"scenario": scenario,
|
||||
"repetition": repetition,
|
||||
"outcome": "passed",
|
||||
"revision_ids": ["revision_001"],
|
||||
"projection": {"phase": "COMPLETED"},
|
||||
"checks": {"deterministic_claims_pass": True},
|
||||
"schema_rejection_count": schema_rejections,
|
||||
"failure_attribution": {"layer": failure_layer} if failure_layer else None,
|
||||
"scenario_budget": {"max_author_turns": 8, "max_reviewer_turns": 2, "max_total_calls": 10, "max_total_tokens": 1000},
|
||||
"usage": {"records": [{"context_chars": context_chars} for _ in range(author_calls)]},
|
||||
}
|
||||
|
||||
|
||||
def _report(results: list[dict]) -> dict:
|
||||
return {
|
||||
"author": {"provider": "author", "model": "model"},
|
||||
"reviewer": {"provider": "reviewer", "model": "review"},
|
||||
"runtime_profile_sha256": "a" * 64,
|
||||
"operation_contracts": [{"atomic_id": "extrude_add_blind", "contract_hash": "b" * 64}],
|
||||
"author_guidance": {"enabled": False, "max_chars": 3600},
|
||||
"results": results,
|
||||
}
|
||||
|
||||
|
||||
class LiveGuidanceComparisonTests(unittest.TestCase):
|
||||
def test_fixture_accepts_multiple_stable_scenarios_in_fixture_order(self) -> None:
|
||||
selected = _fixture("comprehensive", ["l_bracket", "circular_flange_pcd"])
|
||||
self.assertEqual([item["id"] for item in selected], ["circular_flange_pcd", "l_bracket"])
|
||||
|
||||
def test_comparison_enforces_matched_budget_and_quality_gates(self) -> None:
|
||||
control = _report([
|
||||
_result(scenario="part_a", repetition=1, author_calls=10, context_chars=1000, schema_rejections=2, failure_layer="cdsl_expression"),
|
||||
_result(scenario="part_a", repetition=2, author_calls=10, context_chars=1000, schema_rejections=1),
|
||||
])
|
||||
treatment = _report([
|
||||
_result(scenario="part_a", repetition=1, author_calls=11, context_chars=1300, schema_rejections=0),
|
||||
_result(scenario="part_a", repetition=2, author_calls=11, context_chars=1300, schema_rejections=0),
|
||||
])
|
||||
treatment["author_guidance"]["enabled"] = True
|
||||
comparison = compare_guidance_reports(control, treatment)
|
||||
self.assertEqual(comparison["status"], "passed")
|
||||
self.assertTrue(comparison["gates"]["median_author_calls_within_ten_percent"])
|
||||
self.assertTrue(comparison["gates"]["model_or_cdsl_failure_improved"])
|
||||
|
||||
def test_comparison_excludes_explicit_unsupported_runtime_capability(self) -> None:
|
||||
control = _report([_result(scenario="part_a", repetition=1, author_calls=10, context_chars=1000, schema_rejections=1)])
|
||||
treatment = _report([_result(scenario="part_a", repetition=1, author_calls=10, context_chars=1200, schema_rejections=0)])
|
||||
treatment["author_guidance"]["enabled"] = True
|
||||
treatment["results"][0]["ledger"] = [{"operation_failures": [{"message": "unsupported_draft"}]}]
|
||||
comparison = compare_guidance_reports(control, treatment)
|
||||
self.assertEqual(comparison["treatment"]["eligible_runs"], 0)
|
||||
self.assertEqual(comparison["excluded_capability_gaps"]["treatment"], [{"scenario": "part_a", "repetition": 1}])
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
@@ -0,0 +1,123 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
from pathlib import Path
|
||||
import sys
|
||||
import unittest
|
||||
|
||||
|
||||
ROOT = Path(__file__).resolve().parents[2]
|
||||
sys.path.insert(0, str(ROOT / "backend"))
|
||||
|
||||
from app.cad_agent.adapters.structured_llm import StructuredModelError, StructuredModelGateway # noqa: E402
|
||||
from app.settings import ProviderConfig, ProviderModel, Settings # noqa: E402
|
||||
|
||||
|
||||
def _settings(*, api_style: str = "responses", reasoning_effort: str = "medium") -> Settings:
|
||||
provider = ProviderConfig(
|
||||
"provider", "Provider", "https://example.invalid/v1", "key",
|
||||
(ProviderModel("model"),), reasoning_effort=reasoning_effort, api_style=api_style,
|
||||
)
|
||||
return Settings(
|
||||
task_root=ROOT / "tmp-tasks",
|
||||
conversation_root=ROOT / "tmp-conversations",
|
||||
library_root=ROOT / "backend" / "cdsl_library",
|
||||
engine_root=ROOT / "backend" / "engine" / "cdsl_engine",
|
||||
llm_base_url=provider.base_url,
|
||||
llm_api_key=provider.api_key,
|
||||
llm_model="model",
|
||||
llm_timeout_s=1,
|
||||
default_provider_id="provider",
|
||||
providers=(provider,),
|
||||
)
|
||||
|
||||
|
||||
def _tool() -> dict:
|
||||
return {
|
||||
"type": "function",
|
||||
"function": {
|
||||
"name": "write_document",
|
||||
"description": "Write one document.",
|
||||
"parameters": {"type": "object", "properties": {}, "additionalProperties": False},
|
||||
},
|
||||
}
|
||||
|
||||
|
||||
def _response(api_style: str) -> dict:
|
||||
if api_style == "responses":
|
||||
return {
|
||||
"output": [{"type": "function_call", "name": "write_document", "arguments": "{}"}],
|
||||
"usage": {"input_tokens": 5, "output_tokens": 3, "total_tokens": 8},
|
||||
}
|
||||
return {
|
||||
"choices": [{"message": {"tool_calls": [{"function": {"name": "write_document", "arguments": "{}"}}]}}],
|
||||
"usage": {"prompt_tokens": 5, "completion_tokens": 3, "total_tokens": 8},
|
||||
}
|
||||
|
||||
|
||||
class StructuredModelGatewayCompatibilityTests(unittest.TestCase):
|
||||
def test_thinking_tool_choice_retries_without_reasoning_before_relaxing_tool_choice(self) -> None:
|
||||
gateway = StructuredModelGateway(_settings())
|
||||
payloads: list[dict] = []
|
||||
|
||||
async def request(_provider: object, payload: dict) -> dict:
|
||||
payloads.append(payload)
|
||||
if len(payloads) == 1:
|
||||
raise StructuredModelError("Provider rejected structured request (400): Thinking mode does not support this tool_choice")
|
||||
return _response("responses")
|
||||
|
||||
gateway._request = request # type: ignore[method-assign]
|
||||
result = asyncio.run(gateway.call_tool(
|
||||
messages=[{"role": "system", "content": "Call the tool."}],
|
||||
tool=_tool(), provider_id="provider", model_id="model", required_tool_name="write_document",
|
||||
))
|
||||
|
||||
self.assertEqual(len(payloads), 2)
|
||||
self.assertEqual(payloads[0]["tool_choice"], {"type": "function", "name": "write_document"})
|
||||
self.assertEqual(payloads[0]["reasoning"], {"effort": "medium"})
|
||||
self.assertEqual(payloads[1]["tool_choice"], {"type": "function", "name": "write_document"})
|
||||
self.assertNotIn("reasoning", payloads[1])
|
||||
self.assertEqual(result["usage"]["structured_compatibility_mode"], "reasoning_disabled")
|
||||
|
||||
def test_persistent_thinking_rejection_uses_auto_with_the_same_single_tool(self) -> None:
|
||||
gateway = StructuredModelGateway(_settings(api_style="chat_completions", reasoning_effort=""))
|
||||
payloads: list[dict] = []
|
||||
|
||||
async def request(_provider: object, payload: dict) -> dict:
|
||||
payloads.append(payload)
|
||||
if len(payloads) < 3:
|
||||
raise StructuredModelError("Thinking mode does not support this tool_choice")
|
||||
return _response("chat_completions")
|
||||
|
||||
gateway._request = request # type: ignore[method-assign]
|
||||
result = asyncio.run(gateway.call_tool(
|
||||
messages=[{"role": "system", "content": "Call the tool."}],
|
||||
tool=_tool(), provider_id="provider", model_id="model", required_tool_name="write_document",
|
||||
))
|
||||
|
||||
self.assertEqual(len(payloads), 3)
|
||||
self.assertEqual(payloads[0]["tool_choice"]["function"]["name"], "write_document")
|
||||
self.assertEqual(payloads[1]["tool_choice"]["function"]["name"], "write_document")
|
||||
self.assertEqual(payloads[2]["tool_choice"], "auto")
|
||||
self.assertEqual(len(payloads[2]["tools"]), 1)
|
||||
self.assertEqual(result["usage"]["structured_compatibility_mode"], "single_tool_auto")
|
||||
|
||||
def test_unrelated_provider_rejection_is_not_retried(self) -> None:
|
||||
gateway = StructuredModelGateway(_settings())
|
||||
payloads: list[dict] = []
|
||||
|
||||
async def request(_provider: object, payload: dict) -> dict:
|
||||
payloads.append(payload)
|
||||
raise StructuredModelError("Provider rejected structured request (400): invalid model")
|
||||
|
||||
gateway._request = request # type: ignore[method-assign]
|
||||
with self.assertRaisesRegex(StructuredModelError, "invalid model"):
|
||||
asyncio.run(gateway.call_tool(
|
||||
messages=[{"role": "system", "content": "Call the tool."}],
|
||||
tool=_tool(), provider_id="provider", model_id="model", required_tool_name="write_document",
|
||||
))
|
||||
self.assertEqual(len(payloads), 1)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
Reference in New Issue
Block a user