chenlin/Mujoco_WASM
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat: release v1.0.2 LeKiwi 语言控制与网站嵌入
web-platform-ci / Standalone decision service (no cloud credentials) (push) Has been cancelled
Details
web-platform-ci / TypeScript, lint, unit, build (push) Has been cancelled
Details
web-platform-ci / Playwright E2E (push) Has been cancelled
Details
lekiwi-compatibility / cpu-compatibility (push) Has been cancelled
Details

Browse Source 
集成服务器托管模型、自然语言移动与有界抓放、内置 LeKiwi URL 导入和双摄像头；同步部署契约与指定域名 iframe 白名单，保留原有物理安全、会话及调用预算防护。

更新 npm 包及锁文件版本、CHANGELOG 与发布文档。提交前 typecheck、120 项定向前端测试和 44 项后端测试通过（3 项可选跳过）；真实 v2 云模型抓放仍待单独验收，不包含运行密钥或构建产物。
This commit is contained in:
cen617-code
2026-09-24 15:29:49 +08:00
parent f3a8a38acd
commit 0d986f60bd
93 changed files with 5166 additions and 1094 deletions
Download Patch File Download Diff File Expand all files Collapse all files
web_platform/ARCHITECTURE.md
+3 -1
View File
@@ -45,7 +45,9 @@ SimulationSession snapshot → app → viewer / 各业务面板
## Website 产品边界
`vite --mode website` 使用 `app/websiteConfig.ts` 关闭云端首版的训练/调参/外控入口，构建到 `website-dist/`，不包含 `tuning.html`；默认 local 构建和底层算法保留。`WebsiteModelSettings` 仅在展开设置后建立同源匿名会话，`DecisionClient.website()` 自动携带 Cookie/CSRF/配置版本，配置与模型请求仍遵守旧的任务失效机制。API_KEY 与设备码不写浏览器持久化。
`vite --mode website` 使用 `app/websiteConfig.ts` 关闭训练/调参/外控入口，构建到 `website-dist/`，不包含 `tuning.html`；默认 local 构建与旧连接/订阅界面保留。当前网站改用 `WebsiteRobotPanel`，无密钥或模型设置；显式执行才通过 `DecisionClient.website()` 创建 Cookie/CSRF 会话并调用 `/command`。固定密钥由后端受保护文件提供，跨会话调用限额由持久 SQLite 保存。
`RobotCommand` 是有限语义，不是任意代码：move/turn/pick_place/stop/clarify。`MotionDriver` 仅通过 `SimulationSession.startAgent` 的写入票据操作轮速，机械臂保持当前姿态；没有独立时钟或 qpos 传送。速度/加速度受限，距离/累计角度闭环，稳定误差符合后才完成；接触非平地几何、倾斜、异常速度/高度/时钟和超时停止。开始新动作以当前真实朝向/位置为基准，不重载模型；抓放仍显式使用固定台面技能链。停止、隐藏、卸载、控制权或等待期间场景变化会拒绝迟到结果。
后端 `web_server/web_sessions/web_config/model_catalog` 分离公网边界、逐用户内存状态、固定上游配置及模型目录，复用原 `Service` 的有界请求、契约和物理控制解耦。订阅由逐会话 `CodexAccount` 提供官方设备码登录，不共用进程级账号、不放开任意 CLI 执行。部署与验证见 `docs/website-deployment.md`。
web_platform/README.md
+4
View File
@@ -55,6 +55,10 @@ python3 -m http.server 8080 --directory web-platform-dist
访问 <http://127.0.0.1:8080/>。构建使用 `base: './'`，可部署到任意静态子目录。静态服务器需将 `.wasm` 返回为 `application/wasm`；Python 3.11+、Vite preview、Nginx 等常见服务器均支持。
## URL 自动导入内置机器人
访问 `/?robot=lekiwi-v1`（或 `http://localhost:5173/?robot=lekiwi-v1`），自动下载服务器内置 LeKiwi ZIP、选择对应 profile 并加载，无需上传文件或确认弹窗。默认暂停，不自动运行控制器。不带参数时不下载模型，原手动导入流程不变。参数仅接受允许列表内的 ID，不接受文件路径或外部 URL。[完整参数、部署与测试说明](../docs/builtin-robots.md)。
## 工程导入约定
- 文件夹和 ZIP 中必须至少有一个根元素为 `<mujoco>` 或 `<robot>` 的 `.xml`/`.urdf`。
web_platform/assets/robots/lekiwi-v1.zip
BIN
View File
Binary file not shown.
web_platform/e2e/builtinRobots.spec.ts
+160
View File
@@ -0,0 +1,160 @@
import { expect, test, type Page, type TestInfo } from '@playwright/test';
import { resolve } from 'node:path';
import { readFile, writeFile } from 'node:fs/promises';
import { Quaternion, Vector3 } from 'three';
// Vite dev also serves a tiny JS URL module at .zip?import&url; only count/intercept ZIP bytes.
const zipRequest = /\/[^/?]*lekiwi-v1[^/?]*\.zip$/;
async function verifyBuiltinCameras(page: Page, info: TestInfo) {
await page.getByRole('button', { name: '源代码', exact: true }).click();
const editor = page.getByRole('dialog', { name: '转换后的 MJCF 编辑器' });
const download = page.waitForEvent('download');
await editor.getByRole('button', { name: '下载', exact: true }).click();
const xml = await readFile((await (await download).path())!, 'utf8');
const cameras = await page.evaluate((source) => {
const doc = new DOMParser().parseFromString(source, 'application/xml');
return Array.from(doc.querySelectorAll('camera')).map((camera) => {
const frames = [];
for (
let element: Element | null = camera;
element?.tagName !== 'worldbody';
element = element.parentElement
) {
if (!element) throw new Error('Camera is not mounted in worldbody');
frames.unshift({
name: element.getAttribute('name'),
pos: (element.getAttribute('pos') ?? '0 0 0').split(/\s+/).map(Number),
quat: (element.getAttribute('quat') ?? '1 0 0 0').split(/\s+/).map(Number),
});
}
return { name: camera.getAttribute('name'), frames };
});
}, xml);
expect(cameras).toHaveLength(2);
const tilt = (10 * Math.PI) / 180;
const evidence = [];
for (const camera of cameras) {
const wrist = camera.name === '__platform_camera__wrist';
expect(camera.frames.at(-2)?.name).toBe(wrist ? 'Camera-Model-v3-1' : 'Camera-Model-v3');
const position = new Vector3();
const rotation = new Quaternion();
for (const frame of camera.frames) {
position.add(new Vector3().fromArray(frame.pos).applyQuaternion(rotation));
const [w, x, y, z] = frame.quat;
rotation.multiply(new Quaternion(x, y, z, w).normalize());
}
const forward = new Vector3(0, 0, -1).applyQuaternion(rotation);
const up = new Vector3(0, 1, 0).applyQuaternion(rotation);
// Independent source-URDF oracle, checked against the actual WASM-exported
// model including the profile's CAD yaw and full wrist kinematic chain.
const expectedPosition = wrist
? new Vector3(0.215391977166, 0.004249997308, 0.168290381278)
: new Vector3(0.13095590444, 0, 0.062079251628);
const expectedForward = wrist
? new Vector3(-Math.sin(tilt), 0, -Math.cos(tilt))
: new Vector3(Math.cos(tilt), 0, Math.sin(tilt));
const expectedUp = wrist
? new Vector3(Math.cos(tilt), 0, -Math.sin(tilt))
: new Vector3(-Math.sin(tilt), 0, Math.cos(tilt));
expect(position.distanceTo(expectedPosition)).toBeLessThan(2e-6);
expect(forward.distanceTo(expectedForward)).toBeLessThan(2e-6);
expect(up.distanceTo(expectedUp)).toBeLessThan(2e-6);
evidence.push({ ...camera, position, forward, up });
}
await writeFile(info.outputPath('dual-cameras.json'), JSON.stringify(evidence, null, 2));
await page.getByRole('button', { name: '关闭源代码编辑器' }).click();
}
test('URL 自动下载、配置 profile、加载真实 WASM；刷新可重入且不自动运行', async ({
page,
}, info) => {
const downloads: string[] = [];
const inference: string[] = [];
const errors: string[] = [];
page.on('pageerror', (e) => errors.push(e.message));
page.on('request', (r) => {
if (zipRequest.test(r.url())) downloads.push(r.url());
if (/\/api\/decision\/v1\/(command|plan|decide|test)$/.test(r.url())) inference.push(r.url());
});
await page.goto('/?robot=lekiwi-v1&utm_source=partner');
const play = page.getByRole('button', { name: '▶ 播放' });
await expect(play).toBeEnabled({ timeout: 90000 });
await expect(page.getByRole('dialog', { name: '配置 URDF 仿真组件' })).toHaveCount(0);
await expect(page.getByRole('button', { name: '关节拖动', exact: true })).toBeDisabled();
expect(downloads).toHaveLength(1);
await verifyBuiltinCameras(page, info);
const selector = page.getByRole('combobox', { name: '摄像头视角' });
const frame = page.getByLabel('摄像头画面', { exact: true });
await expect(selector.locator('option')).toHaveCount(2);
await expect(frame).toHaveAttribute('data-camera-body', 'Camera-Model-v3');
await page.screenshot({ path: info.outputPath('builtin-lekiwi-front.png') });
await selector.selectOption({ label: '腕部摄像头' });
await expect(frame).toHaveAttribute('data-camera-body', 'Camera-Model-v3-1');
await page.evaluate(
() =>
new Promise<void>((resolve) =>
requestAnimationFrame(() => requestAnimationFrame(() => resolve())),
),
);
await page.screenshot({ path: info.outputPath('builtin-lekiwi-wrist.png') });
await page.getByRole('button', { name: '隐藏画面' }).click();
await expect(frame).toHaveCount(0);
await page.getByRole('button', { name: '显示摄像头画面' }).click();
await expect(frame).toHaveAttribute('data-camera-body', 'Camera-Model-v3-1');
await expect(play).toBeEnabled();
expect(downloads).toHaveLength(1); // Switching/hiding never reloads the robot.
await page.reload();
await expect(play).toBeEnabled({ timeout: 90000 });
await expect(page.getByRole('dialog', { name: '配置 URDF 仿真组件' })).toHaveCount(0);
await expect(page.getByRole('button', { name: '关节拖动', exact: true })).toBeDisabled();
await expect(selector.locator('option')).toHaveCount(2);
await expect(frame).toHaveAttribute('data-camera-body', 'Camera-Model-v3');
expect(downloads).toHaveLength(2);
expect(inference).toEqual([]);
expect(errors).toEqual([]);
});
test('不带参数保持空白，不下载内置 ZIP', async ({ page }) => {
const downloads: string[] = [];
page.on('request', (r) => {
if (zipRequest.test(r.url())) downloads.push(r.url());
});
await page.goto('/');
await expect(page.getByRole('button', { name: '选择文件', exact: true })).toBeVisible();
await expect(page.getByRole('button', { name: '▶ 播放' })).toBeDisabled();
expect(downloads).toEqual([]);
});
for (const query of ['robot=unknown', 'robot=lekiwi-v1&robot=lekiwi-v1']) {
test(`非法参数不下载、不加载：${query}`, async ({ page }) => {
const downloads: string[] = [];
page.on('request', (r) => {
if (zipRequest.test(r.url())) downloads.push(r.url());
});
await page.goto(`/?${query}`);
await expect(page.getByRole('alert')).toContainText('内置机器人加载失败');
await page.getByRole('button', { name: '技术详情' }).click();
await expect(page.getByRole('alert')).toContainText('?robot=lekiwi-v1');
await expect(page.getByRole('button', { name: '▶ 播放' })).toBeDisabled();
expect(downloads).toEqual([]);
});
}
for (const status of [404, 200]) {
test(`下载失败或损坏包（${status}）可恢复手动导入`, async ({ page }) => {
await page.route(zipRequest, (route) =>
route.fulfill({ status, contentType: 'text/html', body: '<html>not a robot zip</html>' }),
);
await page.goto('/?robot=lekiwi-v1');
await expect(page.getByRole('alert')).toContainText('内置机器人加载失败');
await page.getByRole('button', { name: '技术详情' }).click();
await expect(page.getByRole('alert')).toContainText(status === 404 ? 'HTTP 404' : '文件不完整');
await page
.locator('input[type=file]')
.first()
.setInputFiles(resolve('web_platform/fixtures/python_controller/model.xml'));
await expect(page.getByRole('button', { name: '▶ 播放' })).toBeEnabled({ timeout: 60000 });
await expect(page.getByRole('alert')).toHaveCount(0);
});
}
web_platform/e2e/lekiwi.agent.live.spec.ts
+16 -57
View File
@@ -1,77 +1,36 @@
import { expect, test } from '@playwright/test';
import { resolve } from 'node:path';
import { readFile, writeFile } from 'node:fs/promises';
import { writeFile } from 'node:fs/promises';
import { startLiveDecisionService } from './fixtures/decisionService';
import type {} from '../physics/agent';
test.use({ trace: 'off', video: 'off' });
test('显式 opt-in：主工作台 DeepSeek flash + OpenRouter Jev 真实物理回合', async ({
test('显式 opt-in：旧 v1 Bearer 客户端真实 API 物理兼容（非 v2 产品验收）', async ({
page,
}, info) => {
test.skip(
process.env.LEKIWI_LIVE_API !== '1',
'需明确授权 .env 两个角色与一次付费回合；不自动运行',
);
test.skip(process.env.LEKIWI_LIVE_API !== '1', '需明确授权 .env 两个角色与一次付费回合');
const service = await startLiveDecisionService();
try {
await page.goto('/');
await page
.locator('input[type="file"]')
.first()
.setInputFiles(resolve('build/lekiwi/lekiwi-v1.zip'));
const dialog = page.getByRole('dialog', { name: '配置 URDF 仿真组件' });
await expect(dialog).toBeVisible({ timeout: 30000 });
await dialog.getByLabel('URDF 机器人 profile').selectOption('lekiwi-v1');
await dialog.getByRole('button', { name: '转换并加载' }).click();
await expect(page.getByRole('button', { name: '▶ 播放' })).toBeEnabled({ timeout: 60000 });
await page.getByRole('tab', { name: '控制台', exact: true }).click();
await page.getByRole('button', { name: 'LeKiwi 智能抓放', exact: true }).click();
const panel = page.getByLabel('LeKiwi 智能抓放任务', { exact: true });
await panel.getByText('模型连接与官方订阅登录', { exact: true }).click();
await panel.getByLabel('本机模型服务', { exact: true }).fill(service.endpoint);
await panel.getByLabel('本机服务令牌', { exact: true }).fill(service.token);
await panel.getByRole('button', { name: '连接本机模型服务', exact: true }).click();
await expect(panel).toContainText('本机桥已连接');
await expect(panel.getByLabel('LLM 型号', { exact: true })).toHaveValue('deepseek-flash');
await expect(panel.getByLabel('JEV 型号', { exact: true })).toHaveValue('typesafe/jev-1.13');
await expect(panel.getByLabel('LLM Base URL', { exact: true })).toHaveValue(
'https://api.deepseek.com',
await page.goto('/physics/agent.html');
await page.waitForFunction(() => Boolean(window.lekiwiAgent));
await page.evaluate(
(url) => window.lekiwiAgent.boot(url),
`/@fs${resolve('build/lekiwi/lekiwi-v1.zip')}`,
);
await expect(panel.getByLabel('JEV Base URL', { exact: true })).toHaveValue(
'https://openrouter.ai/api/alpha/decisions',
);
await panel.getByText('模型连接与官方订阅登录', { exact: true }).click();
await panel.getByLabel('智能任务决策模式').selectOption('api');
await panel.getByRole('button', { name: '开始任务（真实 API）', exact: true }).click();
const status = panel.getByTestId('agent-task-state');
await expect(status).toContainText(/物理验收成功|失败 \/ 安全停止|已取消/, { timeout: 240000 });
const pendingDownload = page.waitForEvent('download');
await panel.getByRole('button', { name: '导出任务证据', exact: true }).click();
const downloaded = await pendingDownload;
const data = JSON.parse(await readFile((await downloaded.path())!, 'utf8'));
const encoded = JSON.stringify(data, null, 2);
if (encoded.includes(service.token)) throw new Error('任务导出包含本机令牌');
await writeFile(info.outputPath('live-physical-evidence.json'), encoded);
console.log('真实 API 回合（非订阅）', {
state: data.state,
error: data.error,
llm: data.llmCalls,
jev: data.jevCalls,
simTime: data.sample.time,
transported: data.transported,
models: [...new Set(data.receipts.map((r: { model: string }) => r.model))],
const data = await page.evaluate((connection) => window.lekiwiAgent.task(0, connection), {
address: service.endpoint,
token: service.token,
});
expect(data.mode).toBe('api');
const encoded = JSON.stringify(data, null, 2);
if (encoded.includes(service.token)) throw new Error('证据包含服务令牌');
await writeFile(info.outputPath('v1-live-physical-evidence.json'), encoded);
expect(data.state, data.error).toBe('succeeded');
expect(data.succeeded).toBe(true);
expect(data.transported).toBeGreaterThanOrEqual(0.5);
expect(data.llmCalls).toBe(1);
expect(data.jevCalls).toBe(11);
expect(data.receipts).toHaveLength(12);
await page.screenshot({ path: info.outputPath('live-workbench.png') });
} catch (error) {
// Playwright action errors may quote input values. Close before automatic DOM diagnostics.
await page.close().catch(() => undefined);
// eslint-disable-next-line preserve-caught-error -- The original cause may contain the private bridge token.
// eslint-disable-next-line preserve-caught-error -- Private bridge credentials must not enter errors.
throw new Error(String(error).replaceAll(service.token, '[redacted]'));
} finally {
await service.stop();
web_platform/e2e/lekiwi.agent.subscription.spec.ts
+29 -88
View File
@@ -1,103 +1,44 @@
import { expect, test } from '@playwright/test';
import { resolve } from 'node:path';
import { readFile, writeFile } from 'node:fs/promises';
import { writeFile } from 'node:fs/promises';
import { subscriptionOperation } from './fixtures/subscriptionSession';
import type {} from '../physics/agent';
test.use({ trace: 'off', video: 'off' });
test('显式 opt-in：已人工登录的 ChatGPT 订阅 + Jev 主视口物理验收与退出', async ({
page,
}, info) => {
test.skip(
process.env.LEKIWI_LIVE_SUBSCRIPTION !== '1',
'需要用户在隔离官方页面完成登录；不自动发起登录或使用全局账号',
);
test('显式 opt-in：旧 v1 隔离订阅 API 兼容，非新版语言面板功能', async ({ page }, info) => {
test.skip(process.env.LEKIWI_LIVE_SUBSCRIPTION !== '1', '需人工完成隔离官方登录；不使用全局账号');
const session = await subscriptionOperation('connect');
const token = String(session.token);
const connection = { address: String(session.address), token: String(session.token) };
try {
expect(session.llmProtocol).toBe('codex');
const before = await subscriptionOperation('status');
expect(before.loggedIn).toBe(true);
const limitsBefore = await subscriptionOperation('limits');
await page.goto('/');
await page
.locator('input[type="file"]')
.first()
.setInputFiles(resolve('build/lekiwi/lekiwi-v1.zip'));
const dialog = page.getByRole('dialog', { name: '配置 URDF 仿真组件' });
await expect(dialog).toBeVisible({ timeout: 30000 });
await dialog.getByLabel('URDF 机器人 profile').selectOption('lekiwi-v1');
await dialog.getByRole('button', { name: '转换并加载' }).click();
await expect(page.getByRole('button', { name: '▶ 播放' })).toBeEnabled({ timeout: 60000 });
await page.getByRole('tab', { name: '控制台', exact: true }).click();
await page.getByRole('button', { name: 'LeKiwi 智能抓放', exact: true }).click();
const panel = page.getByLabel('LeKiwi 智能抓放任务', { exact: true });
await panel.getByText('模型连接与官方订阅登录', { exact: true }).click();
await panel.getByLabel('本机模型服务', { exact: true }).fill(String(session.address));
await panel.getByLabel('本机服务令牌', { exact: true }).fill(token);
await panel.getByRole('button', { name: '连接本机模型服务', exact: true }).click();
await expect(panel).toContainText('本机桥已连接');
await expect(panel.getByLabel('LLM 协议', { exact: true })).toHaveValue('codex');
await expect(panel.getByLabel('LLM API Key', { exact: true })).toHaveCount(0);
await panel.getByText(/ChatGPT 订阅 · 官方/).click();
await panel.getByRole('button', { name: '刷新账号', exact: true }).click();
await expect(panel).toContainText('ChatGPT 已登录（仅本次服务会话）');
await panel.getByRole('button', { name: '查询官方型号', exact: true }).click();
await expect(panel.getByLabel('LLM 型号', { exact: true })).toHaveValue(
String(session.llmModel),
expect((await subscriptionOperation('status')).loggedIn).toBe(true);
await page.goto('/physics/agent.html');
await page.waitForFunction(() => Boolean(window.lekiwiAgent));
await page.evaluate(
(url) => window.lekiwiAgent.boot(url),
`/@fs${resolve('build/lekiwi/lekiwi-v1.zip')}`,
);
await expect(panel.getByLabel('JEV 型号', { exact: true })).toHaveValue('typesafe/jev-1.13');
await panel.getByText('模型连接与官方订阅登录', { exact: true }).click();
await panel.getByLabel('智能任务决策模式').selectOption('api');
await panel.getByRole('button', { name: '开始任务（真实 API）', exact: true }).click();
await expect(panel.getByTestId('agent-task-state')).toContainText(
/物理验收成功|失败 \/ 安全停止|已取消/,
{ timeout: 240000 },
);
const download = page.waitForEvent('download');
await panel.getByRole('button', { name: '导出任务证据', exact: true }).click();
const data = JSON.parse(await readFile((await (await download).path())!, 'utf8'));
const after = await subscriptionOperation('status');
const evidence = {
source: 'real-chatgpt-subscription-plus-openrouter-jev',
before,
after,
limitsBefore,
...data,
};
const encoded = JSON.stringify(evidence, null, 2);
if (encoded.includes(token) || encoded.includes('auth.openai.com/'))
throw new Error('任务证据包含认证信息');
await writeFile(info.outputPath('subscription-physical-evidence.json'), encoded);
console.log('真实订阅回合', {
state: data.state,
error: data.error,
model: session.llmModel,
llm: data.llmCalls,
jev: data.jevCalls,
simTime: data.sample.time,
transported: data.transported,
});
const data = await page.evaluate((c) => window.lekiwiAgent.task(0, c), connection);
const encoded = JSON.stringify(data, null, 2);
if (encoded.includes(connection.token) || encoded.includes('auth.openai.com/'))
throw new Error('证据包含认证信息');
await writeFile(info.outputPath('v1-subscription-evidence.json'), encoded);
expect(data.state, data.error).toBe('succeeded');
expect(data.succeeded).toBe(true);
expect(data.transported).toBeGreaterThanOrEqual(0.5);
expect(data.llmCalls).toBe(1);
expect(data.jevCalls).toBe(11);
expect(data.receipts).toHaveLength(12);
expect(data.receipts.find((r: { kind: string }) => r.kind === 'plan').provider).toBe('codex');
expect(after.checkedModels).toContain(session.llmModel);
await page.screenshot({ path: info.outputPath('subscription-workbench.png') });
await panel.getByText('模型连接与官方订阅登录', { exact: true }).click();
await panel.getByRole('button', { name: '退出隔离账号', exact: true }).click();
await expect(panel).toContainText('已退出本次隔离会话');
const loggedOut = await subscriptionOperation('status');
expect(loggedOut.loggedIn).toBe(false);
await writeFile(
info.outputPath('subscription-logout.json'),
JSON.stringify(loggedOut, null, 2),
);
expect(data.receipts.find((r) => r.kind === 'plan')?.provider).toBe('codex');
const loggedOut = await page.evaluate(async (c) => {
const response = await fetch(c.address + '/api/decision/v1/codex/logout', {
method: 'POST',
headers: { Authorization: 'Bearer ' + c.token, 'Content-Type': 'application/json' },
body: '{}',
});
return response.ok;
}, connection);
expect(loggedOut).toBe(true);
expect((await subscriptionOperation('status')).loggedIn).toBe(false);
} catch (error) {
await page.close().catch(() => undefined);
// eslint-disable-next-line preserve-caught-error -- Playwright causes can contain the private bridge token.
throw new Error(String(error).replaceAll(token, '[redacted]'));
// eslint-disable-next-line preserve-caught-error -- Avoid exposing private bridge tokens.
throw new Error(String(error).replaceAll(connection.token, '[redacted]'));
}
});
web_platform/e2e/lekiwi.agent.workspace.spec.ts
+25 -63
View File
@@ -1,70 +1,32 @@
import { expect, test } from '@playwright/test';
import { resolve } from 'node:path';
import { readFile, writeFile } from 'node:fs/promises';
import { collectRobotResults, runWebsiteCommand } from './website.helpers';
test('主工作台：A 导入、mock 单技能暂停/继续、物理成功及证据导出', async ({ page }, info) => {
const errors: string[] = [];
let modelRequests = 0;
page.on('pageerror', (e) => errors.push(e.message));
page.on('request', (r) => {
if (/\/api\/decision\/v1\/(plan|decide)$/.test(r.url())) modelRequests++;
});
test('主工作台：加载预设、语言命名目标及连续自由坐标抓放', async ({ page }, info) => {
await collectRobotResults(page);
await page.goto('/');
await page
.locator('input[type="file"]')
.first()
.setInputFiles(resolve('build/lekiwi/lekiwi-v1.zip'));
const dialog = page.getByRole('dialog', { name: '配置 URDF 仿真组件' });
await expect(dialog).toBeVisible({ timeout: 30000 });
await dialog.getByLabel('URDF 机器人 profile').selectOption('lekiwi-v1');
await dialog.getByRole('button', { name: '转换并加载' }).click();
await expect(page.getByRole('button', { name: '▶ 播放' })).toBeEnabled({ timeout: 60000 });
await page.getByRole('tab', { name: '控制台', exact: true }).click();
await page.getByRole('button', { name: 'LeKiwi 智能抓放', exact: true }).click();
const panel = page.getByLabel('LeKiwi 智能抓放任务', { exact: true });
await expect(panel).toContainText('MOCK（不调用模型 API）');
await panel.getByRole('button', { name: '加载 A 任务场景', exact: true }).click();
await expect(panel).toContainText('A 抓放场景已加载并暂停', { timeout: 60000 });
await panel.getByRole('button', { name: '单技能步进', exact: true }).click();
const status = panel.getByTestId('agent-task-state');
await expect(status).toContainText('任务已暂停', { timeout: 60000 });
await expect(status).toContainText('当前技能：open');
await expect(status).toContainText('Jev 1/60');
await panel.getByRole('button', { name: '继续任务', exact: true }).click();
await expect(status).toContainText('物理验收成功', { timeout: 180000 });
await expect(status).toContainText('Jev 11/60');
const download = page.waitForEvent('download');
await panel.getByRole('button', { name: '导出任务证据', exact: true }).click();
const downloaded = await download;
const evidence = JSON.parse(await readFile((await downloaded.path())!, 'utf8'));
await writeFile(
info.outputPath('workbench-physical-evidence.json'),
JSON.stringify(evidence, null, 2),
await page.getByRole('button', { name: '机器人语言控制', exact: true }).click();
const panel = page.getByLabel('机器人语言控制', { exact: true });
await expect(panel.locator('input[type=password], details')).toHaveCount(0);
await panel.getByRole('button', { name: '载入预设场景' }).click();
await expect(panel.getByRole('status')).toContainText('预设已载入并暂停', { timeout: 90000 });
const first = await runWebsiteCommand(page, '把方块搬到 B 区', info, 'named-target');
expect(first.goal).toEqual([0.257, 0.75, 0.128]);
expect(first.sample.onGoalSupport).toBe(true);
expect(first.transported).toBeGreaterThan(0.7);
expect(first.llmCalls).toBe(2);
expect(first.jevCalls).toBe(14);
expect(first.qualityScore).toBe(75);
const second = await runWebsiteCommand(
page,
'把方块搬到 (0.25,0.50) 米',
info,
'coordinate-target',
);
expect(evidence.mode).toBe('mock');
expect(evidence.succeeded).toBe(true);
expect(evidence.transported).toBeGreaterThanOrEqual(0.5);
expect(evidence.sample.onGoalSupport).toBe(true);
expect(modelRequests).toBe(0);
expect(errors).toEqual([]);
await page.screenshot({ path: info.outputPath('agent-workbench.png') });
await panel.getByRole('button', { name: '重置任务场景', exact: true }).click();
await expect(panel.getByTestId('agent-task-state')).toHaveCount(0, { timeout: 60000 });
await panel.getByLabel('目标 Y', { exact: true }).fill('0.617');
await panel.getByRole('button', { name: '单技能步进', exact: true }).click();
await expect(status).toContainText('任务已暂停', { timeout: 60000 });
const changedDownload = page.waitForEvent('download');
await panel.getByRole('button', { name: '导出任务证据', exact: true }).click();
const changed = JSON.parse(await readFile((await (await changedDownload).path())!, 'utf8'));
expect(changed.goal).toEqual([0.257, 0.617, 0.128]);
expect(changed.runId).not.toBe(evidence.runId);
expect(changed.sample.time).toBeLessThan(2.01);
expect(changed.sample.object[1]).toBeLessThan(0.02); // New source episode, not the old placed cube.
await writeFile(info.outputPath('edited-goal-episode.json'), JSON.stringify(changed, null, 2));
await panel.getByLabel('目标 Y', { exact: true }).fill('0.615');
await expect(status).toContainText('已取消');
await expect(status).toContainText('目标编辑使旧计划失效');
expect(modelRequests).toBe(0);
expect(errors).toEqual([]);
expect(second.goal).toEqual([0.25, 0.5, 0.128]);
expect(second.sample.time).toBeGreaterThan(first.sample.time);
expect(second.transported).toBeGreaterThan(0.2);
expect(second.runId).not.toBe(first.runId);
await expect(page.getByRole('button', { name: '强化学习任务', exact: true })).toBeVisible();
await page.screenshot({ path: info.outputPath('language-workbench.png') });
});
web_platform/e2e/lekiwi.cameras.spec.ts
+104
View File
@@ -0,0 +1,104 @@
import { expect, test } from '@playwright/test';
import { writeFile } from 'node:fs/promises';
// Isolated development-only kinematic oracle; no UI/production pose-write API.
test('真实 WASM 双相机：腕部随臂运动，前置保持底盘固连，转动底盘两者同步', async ({
page,
}, info) => {
await page.goto('/physics/runner.html');
const result = await page.evaluate(async () => {
const paths = [
'/src/simulation/PhysicsAdapter.ts',
'/src/project/importer.ts',
'/src/project/builtinRobots.ts',
];
const { MainThreadPhysicsAdapter } = (await import(
/* @vite-ignore */ paths[0]
)) as typeof import('../src/simulation/PhysicsAdapter');
const { importBrowserFiles } = (await import(
/* @vite-ignore */ paths[1]
)) as typeof import('../src/project/importer');
const { BUILTIN_ROBOTS, builtinRobotEnhancements, downloadBuiltinRobot } = (await import(
/* @vite-ignore */ paths[2]
)) as typeof import('../src/project/builtinRobots');
const robot = BUILTIN_ROBOTS[0];
const source = await importBrowserFiles([
await downloadBuiltinRobot(robot, new AbortController().signal),
]);
const adapter = new MainThreadPhysicsAdapter();
try {
await adapter.load(source, robot.entryPath, {
robotProfileId: robot.profileId,
enhancements: builtinRobotEnhancements(robot.id),
});
const session = adapter.session!;
const { model, data, module } = session;
const jointAddress = (name: string) => {
for (let id = 0; id < model.njnt; id++) {
const joint = model.jnt(id);
try {
if (joint.name === name) return Number(model.jnt_qposadr[id]);
} finally {
joint.delete();
}
}
throw new Error(`Missing joint ${name}`);
};
const read = () =>
Array.from({ length: model.ncam }, (_, id) => {
const camera = model.cam(id);
const body = model.body(Number(model.cam_bodyid[id]));
try {
return {
name: camera.name,
body: body.name,
position: Array.from(data.cam_xpos.slice(id * 3, id * 3 + 3), Number),
forward: [2, 5, 8].map((offset) => -Number(data.cam_xmat[id * 9 + offset])),
};
} finally {
camera.delete();
body.delete();
}
});
const before = read();
data.qpos[jointAddress('arm_wrist_flex')] += 0.35;
module.mj_forward(model, data);
const wristMoved = read();
const base = jointAddress('__platform_base_freejoint__');
data.qpos[base + 3] = Math.SQRT1_2;
data.qpos[base + 4] = 0;
data.qpos[base + 5] = 0;
data.qpos[base + 6] = Math.SQRT1_2;
module.mj_forward(model, data);
return { before, wristMoved, baseTurned: read(), time: Number(data.time) };
} finally {
adapter.dispose();
}
});
expect(result.before).toHaveLength(2);
const front = (items: typeof result.before) =>
items.find((camera) => camera.body === 'Camera-Model-v3')!;
const wrist = (items: typeof result.before) =>
items.find((camera) => camera.body === 'Camera-Model-v3-1')!;
const distance = (a: number[], b: number[]) => Math.hypot(...a.map((value, i) => value - b[i]));
expect(distance(front(result.before).position, front(result.wristMoved).position)).toBeLessThan(
1e-10,
);
expect(distance(front(result.before).forward, front(result.wristMoved).forward)).toBeLessThan(
1e-10,
);
expect(
distance(wrist(result.before).position, wrist(result.wristMoved).position),
).toBeGreaterThan(0.005);
expect(distance(wrist(result.before).forward, wrist(result.wristMoved).forward)).toBeGreaterThan(
0.1,
);
for (const before of result.wristMoved) {
const after = result.baseTurned.find((camera) => camera.name === before.name)!;
const turn = ([x, y, z]: number[]) => [-y, x, z];
expect(distance(after.position, turn(before.position))).toBeLessThan(1e-8);
expect(distance(after.forward, turn(before.forward))).toBeLessThan(1e-8);
}
expect(result.time).toBe(0);
await writeFile(info.outputPath('dual-camera-kinematics.json'), JSON.stringify(result, null, 2));
});
web_platform/e2e/lekiwi.language.spec.ts
+67
View File
@@ -0,0 +1,67 @@
import { test, expect } from '@playwright/test';
import { resolve } from 'node:path';
import { writeFile } from 'node:fs/promises';
import type {} from '../physics/agent';
test('v2 物理：连续两次搬运，不重载或移动支撑面', async ({ page }, info) => {
await page.goto('/physics/agent.html');
await page.waitForFunction(() => Boolean(window.lekiwiAgent));
await page.evaluate(
(url) => window.lekiwiAgent.boot(url),
`/@fs${resolve('build/lekiwi/lekiwi-v1.zip')}`,
);
const results = await page.evaluate(() =>
window.lekiwiAgent.languagePhysical([
[0.257, 0.75, 0.128],
[0.25, 0.5, 0.128],
]),
);
await writeFile(info.outputPath('continuous-physical.json'), JSON.stringify(results, null, 2));
for (const r of results) {
console.log('language physical', {
status: r.status,
error: r.error,
goal: r.goal,
sample: r.sample,
transported: r.transported,
});
expect(r.status, r.error).toBe('succeeded');
expect(r.succeeded).toBe(true);
expect(r.sample.onGoalSupport).toBe(true);
expect(r.maxJointSpeed).toBeLessThanOrEqual(2);
}
await page.evaluate(() => window.lekiwiAgent.dispose());
});
test('v2 真实故障：夹爪锁开不会宣称成功', async ({ page }, info) => {
await page.goto('/physics/agent.html');
await page.waitForFunction(() => Boolean(window.lekiwiAgent));
await page.evaluate(
(url) => window.lekiwiAgent.boot(url),
`/@fs${resolve('build/lekiwi/lekiwi-v1.zip')}`,
);
const results = await page.evaluate(() =>
window.lekiwiAgent.languagePhysical([[0.257, 0.75, 0.128]], 'empty'),
);
await writeFile(info.outputPath('empty-grasp.json'), JSON.stringify(results, null, 2));
expect(results[0].status).toBe('failed');
expect(results[0].error).toBe('empty_grasp');
expect(results[0].transported).toBe(0);
expect(results[0].succeeded).toBe(false);
await page.evaluate(() => window.lekiwiAgent.dispose());
});
test('v2 标定：连续台面边界与停靠姿态 scratch IK', async ({ page }, info) => {
await page.goto('/physics/agent.html');
await page.waitForFunction(() => Boolean(window.lekiwiAgent));
await page.evaluate(
(url) => window.lekiwiAgent.boot(url),
`/@fs${resolve('build/lekiwi/lekiwi-v1.zip')}`,
);
const result = await page.evaluate(() => window.lekiwiAgent.calibrateLanguage());
await writeFile(info.outputPath('calibration.json'), JSON.stringify(result, null, 2));
expect(result.unchanged).toBe(true);
for (const check of result.checks)
expect(check.residual, JSON.stringify(check.target)).toBeLessThan(0.004);
await page.evaluate(() => window.lekiwiAgent.dispose());
});
web_platform/e2e/website.embed.production.spec.ts
+113
View File
@@ -0,0 +1,113 @@
import { expect, test } from '@playwright/test';
const site = 'https://cadworld-sim.robotquan.com';
const approvedParent = 'https://cadworld.robotquan.com';
test.skip(process.env.CADWORLD_PRODUCTION_TEST !== '1', '真实站点非付费检查须显式启用');
test('生产嵌入响应头：仅允许自身与指定同事域名，API 仍拒绝父页面来源', async ({ request }) => {
const response = await request.get(site);
expect(response.status()).toBe(200);
const headers = response.headers();
expect(headers['x-frame-options']).toBeUndefined();
expect(headers['content-security-policy']).toContain(`frame-ancestors 'self' ${approvedParent};`);
expect(headers['content-security-policy']).toContain("object-src 'none'");
expect(headers['x-content-type-options']).toBe('nosniff');
expect(headers['strict-transport-security']).toContain('max-age=31536000');
const forbidden = await request.post(`${site}/api/decision/v1/session`, {
headers: { Origin: approvedParent, 'Sec-Fetch-Site': 'same-site' },
data: {},
});
expect(forbidden.status()).toBe(403);
expect((await forbidden.json()).error).toBe('origin_forbidden');
expect(forbidden.headers()['access-control-allow-origin']).toBeUndefined();
});
test('生产嵌入：允许来源真实加载应用、Strict Cookie 会话和 CSRF 正常，无推理', async ({
page,
context,
}, info) => {
let inference = 0;
await page.route('**/api/decision/v1/**', async (route) => {
if (/\/(command|plan|decide|test)$/.test(new URL(route.request().url()).pathname)) {
inference++;
await route.abort();
} else await route.continue();
});
// Only the parent HTML is synthetic. The iframe, assets and session API are live.
const parentUrl = `${approvedParent}/__cadworld_embed_acceptance__`;
await page.route(parentUrl, (route) =>
route.fulfill({
contentType: 'text/html',
body: `<html><body style="margin:0"><iframe title="CADWorld Sim" src="${site}/" style="width:100%;height:95vh;border:0" allow="fullscreen"></iframe></body></html>`,
}),
);
await page.goto(parentUrl);
const frame = page.frameLocator('iframe');
await frame.getByRole('tab', { name: '控制台', exact: true }).click();
await frame.getByRole('button', { name: '机器人语言控制', exact: true }).click();
await expect(frame.getByRole('textbox', { name: '机器人指令' })).toBeEditable();
const embedded = page.frames().find((f) => f.url() === `${site}/`);
expect(embedded).toBeTruthy();
const sessionResult = await embedded!.evaluate(async () => {
const prefix = '/api/decision/v1';
const bootstrap = await fetch(prefix + '/session', {
method: 'POST',
headers: { 'Content-Type': 'application/json' },
body: '{}',
});
const session = await bootstrap.json();
const status = await fetch(prefix + '/status');
const csrfGuard = await fetch(prefix + '/cancel', {
method: 'POST',
headers: { 'Content-Type': 'application/json' },
body: '{}',
});
const destroyed = await fetch(prefix + '/session', {
method: 'DELETE',
headers: { 'Content-Type': 'application/json', 'X-CSRF-Token': session.csrfToken },
body: '{}',
});
const expired = await fetch(prefix + '/status');
return {
bootstrap: bootstrap.status,
ready: session.ready,
status: status.status,
csrfGuard: csrfGuard.status,
destroyed: destroyed.status,
expired: expired.status,
};
});
expect(sessionResult).toEqual({
bootstrap: 200,
ready: true,
status: 200,
csrfGuard: 403,
destroyed: 200,
expired: 401,
});
expect(
(await context.cookies(site)).filter((c) => c.name.includes('session')).map((c) => c.name),
).toHaveLength(0);
expect(inference).toBe(0);
await page.screenshot({ path: info.outputPath('approved-iframe.png') });
});
test('生产嵌入：未授权同站子域名仍被 CSP 拒绝', async ({ page }) => {
const violations: string[] = [];
page.on('console', (message) => {
if (message.type() === 'error') violations.push(message.text());
});
const parentUrl = 'https://unapproved.robotquan.com/__cadworld_embed_acceptance__';
await page.route(parentUrl, (route) =>
route.fulfill({
contentType: 'text/html',
body: `<iframe src="${site}/" title="forbidden"></iframe>`,
}),
);
await page.goto(parentUrl);
await expect.poll(() => violations.some((text) => text.includes('frame-ancestors'))).toBe(true);
await expect(
page.frameLocator('iframe').getByRole('tab', { name: '控制台', exact: true }),
).toHaveCount(0);
});
web_platform/e2e/website.helpers.ts
+53
View File
@@ -0,0 +1,53 @@
import { expect, type Page, type TestInfo } from '@playwright/test';
import { resolve } from 'node:path';
import { writeFile } from 'node:fs/promises';
export async function collectRobotResults(page: Page) {
await page.addInitScript(() => {
window.addEventListener('robot-task-result', (event) => {
sessionStorage.setItem('test-robot-result', JSON.stringify((event as CustomEvent).detail));
});
});
}
export async function readRobotResult(page: Page) {
await page.waitForFunction(() => sessionStorage.getItem('test-robot-result') !== null);
return page.evaluate(() => JSON.parse(sessionStorage.getItem('test-robot-result')!));
}
export async function loadWebsiteRobot(page: Page) {
await collectRobotResults(page);
await page.goto('/');
await page
.locator('input[type=file]')
.first()
.setInputFiles(resolve('build/lekiwi/lekiwi-v1.zip'));
const dialog = page.getByRole('dialog', { name: '配置 URDF 仿真组件' });
await expect(dialog).toBeVisible({ timeout: 30000 });
await dialog.getByLabel('URDF 机器人 profile').selectOption('lekiwi-v1');
await dialog.getByRole('button', { name: '转换并加载' }).click();
await expect(page.getByRole('button', { name: '▶ 播放' })).toBeEnabled({ timeout: 90000 });
await page.getByRole('tab', { name: '控制台', exact: true }).click();
await page.getByRole('button', { name: '机器人语言控制', exact: true }).click();
await expect(page.locator('input[type=password]')).toHaveCount(0);
return page.getByLabel('机器人语言控制', { exact: true });
}
export async function runWebsiteCommand(
page: Page,
instruction: string,
info: TestInfo,
name: string,
) {
const panel = page.getByLabel('机器人语言控制', { exact: true });
await page.evaluate(() => sessionStorage.removeItem('test-robot-result'));
await panel.getByRole('textbox', { name: '机器人指令' }).fill(instruction);
const response = page.waitForResponse((r) => new URL(r.url()).pathname.endsWith('/command'));
await panel.getByRole('button', { name: '执行指令', exact: true }).click();
expect((await response).status()).toBe(200);
await expect(panel.getByTestId('robot-command-state')).toContainText(/已完成|安全停止|已停止/, {
timeout: 240000,
});
const data = await readRobotResult(page);
await writeFile(info.outputPath(`${name}.json`), JSON.stringify(data, null, 2));
expect(data.state, data.error).toBe('succeeded');
expect(data.succeeded).toBe(true);
return data;
}
web_platform/e2e/website.live.spec.ts
+88
View File
@@ -0,0 +1,88 @@
import { expect, test } from '@playwright/test';
import { writeFile } from 'node:fs/promises';
import { loadWebsiteRobot, runWebsiteCommand } from './website.helpers';
test('显式授权：公网内置模型，前进/转向/抓放，无手填密钥或自动重试', async ({ page }, info) => {
test.skip(process.env.CADWORLD_LIVE_API !== '1', '仅经用户授权后运行，会消耗服务器公共额度');
test.setTimeout(360000);
const calls = { command: 0, plan: 0, decide: 0, blocked: 0 };
let csrf = '',
cleanup = false;
await page.route('**/api/decision/v1/**', async (route) => {
const endpoint = new URL(route.request().url()).pathname.split('/').at(-1);
if (endpoint === 'command' || endpoint === 'plan' || endpoint === 'decide') {
const limit = { command: 3, plan: 1, decide: 13 };
if (calls[endpoint] >= limit[endpoint]) {
calls.blocked++;
await route.abort();
return;
}
calls[endpoint]++;
} else if (endpoint === 'test') {
calls.blocked++;
await route.abort();
return;
}
await route.continue();
});
let capture: Promise<void> | undefined;
const captureSession = () =>
page
.waitForResponse((r) => r.url().endsWith('/session') && r.request().method() === 'POST')
.then(async (r) => {
csrf = (await r.json()).csrfToken;
})
.catch(() => undefined);
// Observe capture on all outcomes; no pre-existing keys/Cookies or storage state used.
try {
await loadWebsiteRobot(page);
capture = captureSession();
const move = await runWebsiteCommand(page, '前进0.5米', info, 'live-forward');
expect(move.distance).toBeGreaterThan(0.49);
expect(move.distance).toBeLessThan(0.51);
const turn = await runWebsiteCommand(page, '左转90度', info, 'live-turn');
expect(turn.turnDegrees).toBeGreaterThan(89);
expect(turn.turnDegrees).toBeLessThan(91);
expect(turn.start.x).toBeCloseTo(move.sample.x, 2);
const panel = page.getByLabel('机器人语言控制', { exact: true });
await panel.getByRole('button', { name: '载入预设场景' }).click();
await expect(panel.getByRole('status')).toContainText('预设已载入', { timeout: 90000 });
const grasp = await runWebsiteCommand(page, '把方块搬到 B 区', info, 'live-pick-place');
expect(grasp.transported).toBeGreaterThanOrEqual(0.5);
expect(grasp.sample.onGoalSupport).toBe(true);
expect(calls).toEqual({ command: 3, plan: 1, decide: 13, blocked: 0 });
await page.screenshot({ path: info.outputPath('hosted-control.png') });
} finally {
await capture;
if (csrf)
cleanup = await page
.evaluate(
async (token) =>
(
await fetch('/api/decision/v1/session', {
method: 'DELETE',
headers: { 'Content-Type': 'application/json', 'X-CSRF-Token': token },
body: '{}',
})
).status === 200,
csrf,
)
.catch(() => false);
await page.close();
await writeFile(
info.outputPath('hosted-live-summary.json'),
JSON.stringify(
{
calls,
sessionDestroyed: cleanup,
maximumUpstreamCalls: { llm: 4, jev: 16 },
retries: 0,
credentialInput: false,
},
null,
2,
),
);
}
expect(cleanup).toBe(true);
});
web_platform/e2e/website.production.spec.ts
+74 -95
View File
@@ -1,29 +1,15 @@
import { expect, test } from '@playwright/test';
import { resolve } from 'node:path';
import { existsSync } from 'node:fs';
import { readFile, writeFile } from 'node:fs/promises';
test.skip(
process.env.CADWORLD_PRODUCTION_TEST !== '1',
'真实站点检查须显式 opt-in；不使用任何现存密钥',
);
test('生产 HTTPS：模型导入、Pyodide、精简设置、安全会话与订阅不可用边界', async ({
page,
}, info) => {
test.skip(process.env.CADWORLD_PRODUCTION_TEST !== '1', '真实站点非付费检查须显式启用');
test('生产 HTTPS：Pyodide 与精简语言界面，不触发推理', async ({ page }) => {
let inference = 0;
const errors: string[] = [];
const external: string[] = [];
page.on('pageerror', (e) => errors.push(e.message));
page.on('request', (r) => {
if (
r.url().startsWith('http') &&
new URL(r.url()).origin !== 'https://cadworld-sim.robotquan.com'
)
external.push(r.url());
if (/\/api\/decision\/v1\/(command|plan|decide|test)$/.test(r.url())) inference++;
});
const response = await page.goto('/');
expect(response?.status()).toBe(200);
await expect(page).toHaveTitle(/CADWorld Sim/);
expect((await page.goto('/'))?.status()).toBe(200);
await page
.locator('input[type=file]')
.first()
@@ -40,102 +26,95 @@ test('生产 HTTPS：模型导入、Pyodide、精简设置、安全会话与订
});
await page.getByRole('button', { name: '启用', exact: true }).click();
await expect(page.getByRole('button', { name: /Python 脚本控制/ })).toContainText('运行');
await page.getByRole('button', { name: 'LeKiwi 智能抓放', exact: true }).click();
await page.getByText('模型设置 · API_KEY / ChatGPT 订阅', { exact: true }).click();
await expect(page.getByRole('button', { name: '保存设置' })).toBeEnabled({ timeout: 20000 });
await expect(page.getByLabel('本机服务令牌')).toHaveCount(0);
const cookie = (await page.context().cookies()).find((c) => c.name === '__Host-cadworld-session');
expect(cookie?.secure).toBe(true);
expect(cookie?.httpOnly).toBe(true);
expect(cookie?.sameSite).toBe('Strict');
await page.screenshot({ path: info.outputPath('website-api-settings.png') });
await page.getByRole('button', { name: 'ChatGPT 订阅', exact: true }).click();
// No login or model invocation. Availability must be explicit; user's login stays interactive.
await expect(page.getByText(/当前服务器可能无法连接官方服务/)).toBeVisible();
await expect(page.getByRole('button', { name: '登录 ChatGPT 订阅', exact: true })).toBeVisible();
await page.screenshot({ path: info.outputPath('website-subscription-entry.png') });
await page.getByText('连接测试与凭据管理', { exact: true }).click();
await page.getByRole('button', { name: '清除本次凭据', exact: true }).click();
await expect(page.getByText(/本次凭据已清除/)).toBeVisible();
await page.getByRole('button', { name: '机器人语言控制', exact: true }).click();
await expect(page.locator('input[type=password]')).toHaveCount(0);
await expect(page.getByRole('textbox', { name: '机器人指令' })).toBeEditable();
await page.getByRole('button', { name: '执行指令', exact: true }).click();
await expect(page.getByText('请先导入并加载 LeKiwi v1 机器人', { exact: true })).toBeVisible();
expect(inference).toBe(0);
expect(errors).toEqual([]);
expect(external).toEqual([]);
});
test('生产 v2：空工作台载入预设、文字目标及双相机，无付费推理', async ({ page }, info) => {
let inference = 0;
const errors: string[] = [];
page.on('pageerror', (error) => errors.push(error.message));
await page.route('**/api/decision/v1/**', async (route) => {
if (/\/(command|plan|decide|test)$/.test(new URL(route.request().url()).pathname)) {
inference++;
await route.abort();
} else await route.continue();
});
expect((await page.goto('/'))?.status()).toBe(200);
await page.getByRole('tab', { name: '控制台', exact: true }).click();
await page.getByRole('button', { name: '机器人语言控制', exact: true }).click();
const panel = page.getByLabel('机器人语言控制', { exact: true });
await expect(panel.getByRole('combobox', { name: '预设场景' })).toBeVisible();
await expect(panel.locator('input[type=password], details')).toHaveCount(0);
await panel.getByRole('button', { name: '载入预设场景' }).click();
await expect(panel.getByRole('status')).toContainText('预设已载入并暂停', { timeout: 120000 });
await expect(page.getByRole('button', { name: '▶ 播放' })).toBeEnabled();
await expect(page.getByRole('combobox', { name: '摄像头视角' }).locator('option')).toHaveCount(2);
await panel.getByRole('textbox', { name: '机器人指令' }).fill('把方块搬到 (0.25,0.50) 米');
await expect(panel.getByRole('button', { name: '执行指令', exact: true })).toBeEnabled();
await panel.getByRole('textbox', { name: '机器人指令' }).fill('停止');
await panel.getByRole('button', { name: '执行指令', exact: true }).click();
await expect(panel.getByRole('status')).toContainText('用户停止');
expect(inference).toBe(0);
expect(errors).toEqual([]);
await page.screenshot({ path: info.outputPath('language-v2-production.png') });
});
test('官方订阅入口网络探测：不登录账号、不推理，完成后销毁会话', async ({ page }, info) => {
test('生产默认配置受保护、订阅关闭、静态目录无密钥；退出销毁会话，无推理', async ({ page }) => {
await page.goto('/');
const evidence = await page.evaluate(async () => {
const result = await page.evaluate(async () => {
const prefix = '/api/decision/v1';
const headers = { 'Content-Type': 'application/json' };
const bootstrap = await fetch(prefix + '/session', { method: 'POST', headers, body: '{}' });
const session = await bootstrap.json();
const protectedHeaders = {
const secured = {
...headers,
'X-CSRF-Token': session.csrfToken,
'X-Config-Version': String(session.configVersion),
};
const response = await fetch(prefix + '/codex/login', {
const config = await fetch(prefix + '/configuration', {
method: 'PUT',
headers: secured,
body: '{}',
});
const subscription = await fetch(prefix + '/codex/login', {
method: 'POST',
headers: protectedHeaders,
headers: secured,
body: '{}',
});
const result = await response.json();
const cleared = await fetch(prefix + '/session', {
const env = await fetch('/.env');
const secret = await fetch('/run/secrets/model-keys.env');
const destroy = await fetch(prefix + '/session', {
method: 'DELETE',
headers: protectedHeaders,
headers: secured,
body: '{}',
});
const expired = await fetch(prefix + '/status');
return {
status: response.status,
error: typeof result.error === 'string' ? result.error : undefined,
deviceFlowOffered: Boolean(result.verificationUrl && result.userCode),
cleared: cleared.status,
accountLoginPerformed: false,
inferenceCalls: 0,
ready: session.ready,
llm: session.configuration.llm.model,
keyTypes: [typeof session.configuration.llm.hasKey, typeof session.configuration.jev.hasKey],
locked: config.status,
subscription: subscription.status,
env: env.status,
secret: secret.status,
destroyed: destroy.status,
expired: expired.status,
};
});
expect(evidence.cleared).toBe(200);
expect([200, 409, 429, 502, 504]).toContain(evidence.status);
if (evidence.status !== 200) expect(evidence.error).toMatch(/^[a-z0-9_]+$/);
await writeFile(
info.outputPath('subscription-network-status.json'),
JSON.stringify(evidence, null, 2),
);
});
test('公网真实 WASM A 抓放（明确 mock，不调用付费 API）', async ({ page }, info) => {
test.skip(!existsSync('build/lekiwi/lekiwi-v1.zip'), '无本地 A 资产');
let modelCalls = 0;
page.on('request', (r) => {
if (/\/api\/decision\/v1\/(plan|decide|test)$/.test(r.url())) modelCalls++;
expect(result).toEqual({
ready: true,
llm: 'deepseek-flash',
keyTypes: ['boolean', 'boolean'],
locked: 403,
subscription: 403,
env: 404,
secret: 404,
destroyed: 200,
expired: 401,
});
await page.goto('/');
await page
.locator('input[type=file]')
.first()
.setInputFiles(resolve('build/lekiwi/lekiwi-v1.zip'));
const dialog = page.getByRole('dialog', { name: '配置 URDF 仿真组件' });
await expect(dialog).toBeVisible({ timeout: 30000 });
await dialog.getByLabel('URDF 机器人 profile').selectOption('lekiwi-v1');
await dialog.getByRole('button', { name: '转换并加载' }).click();
await expect(page.getByRole('button', { name: '▶ 播放' })).toBeEnabled({ timeout: 90000 });
await page.getByRole('tab', { name: '控制台', exact: true }).click();
await page.getByRole('button', { name: 'LeKiwi 智能抓放', exact: true }).click();
const panel = page.getByLabel('LeKiwi 智能抓放任务', { exact: true });
await panel.getByRole('button', { name: '开始任务（mock）', exact: true }).click();
await expect(panel.getByTestId('agent-task-state')).toContainText('物理验收成功', {
timeout: 180000,
});
await panel.getByText('高级操作与证据导出', { exact: true }).click();
const download = page.waitForEvent('download');
await panel.getByRole('button', { name: '导出任务证据', exact: true }).click();
const data = JSON.parse(await readFile((await (await download).path())!, 'utf8'));
expect(data.succeeded).toBe(true);
expect(data.transported).toBeGreaterThanOrEqual(0.5);
expect(data.mode).toBe('mock');
expect(modelCalls).toBe(0);
await writeFile(
info.outputPath('production-mock-physical-evidence.json'),
JSON.stringify(data, null, 2),
);
await page.screenshot({ path: info.outputPath('production-physical-success.png') });
});
web_platform/e2e/website.spec.ts
+48 -114
View File
@@ -1,119 +1,53 @@
import { expect, test } from '@playwright/test';
import { resolve } from 'node:path';
import { existsSync } from 'node:fs';
import { readFile, writeFile } from 'node:fs/promises';
import { writeFile } from 'node:fs/promises';
import { loadWebsiteRobot, runWebsiteCommand, readRobotResult } from './website.helpers';
const prefix = '/api/decision/v1';
test('网站生产构建：精简设置、同源 Cookie/CSRF、双浏览器隔离与假 HTTP 上游', async ({
page,
browser,
}, info) => {
const errors: string[] = [];
page.on('pageerror', (e) => errors.push(e.message));
await page.goto('/');
await expect(page).toHaveTitle(/CADWorld Sim/);
await page
.locator('input[type=file]')
.first()
.setInputFiles(resolve('web_platform/fixtures/python_controller/model.xml'));
await expect(page.getByRole('button', { name: '▶ 播放' })).toBeEnabled({ timeout: 60000 });
await page.getByRole('tab', { name: '控制台', exact: true }).click();
await expect(page.getByRole('button', { name: '强化学习任务', exact: true })).toHaveCount(0);
await expect(page.getByRole('button', { name: '开源项目 / 外部控制', exact: true })).toHaveCount(
0,
);
await page.getByRole('button', { name: 'LeKiwi 智能抓放', exact: true }).click();
await page.getByText('模型设置 · API_KEY / ChatGPT 订阅', { exact: true }).click();
await expect(page.getByRole('button', { name: '保存设置' })).toBeEnabled();
await expect(page.getByLabel('本机服务令牌')).toHaveCount(0);
await page.getByLabel('LLM API_KEY', { exact: true }).fill('e2e-llm-fixture-not-real');
await page.getByLabel('Jev API_KEY（OpenRouter）').fill('e2e-jev-fixture-not-real');
await page.getByRole('button', { name: '保存设置', exact: true }).click();
await expect(page.getByRole('status').filter({ hasText: '设置已保存' })).toBeVisible();
await expect(page.getByLabel('LLM API_KEY', { exact: true })).toHaveValue('');
await page.getByText('连接测试与凭据管理', { exact: true }).click();
await page.getByRole('button', { name: '测试 LLM（可能计费）', exact: true }).click();
await expect(page.getByText(/LLM 连接测试通过/)).toBeVisible();
const other = await browser.newContext();
try {
const second = await other.newPage();
await second.goto('/');
const fresh = await second.evaluate(async (prefix) => {
const response = await fetch(prefix + '/session', {
method: 'POST',
headers: { 'Content-Type': 'application/json' },
body: '{}',
});
return response.json();
}, prefix);
expect(fresh.ready).toBe(false);
const state = await page.evaluate(
async (prefix) => (await fetch(prefix + '/status')).json(),
prefix,
);
expect(state.ready).toBe(true);
const forbidden = await page.evaluate(
async (prefix) =>
(
await fetch(prefix + '/configuration', {
method: 'PUT',
headers: { 'Content-Type': 'application/json' },
body: '{}',
})
).status,
prefix,
);
expect(forbidden).toBe(403);
expect(await page.evaluate(() => document.cookie)).not.toContain('cadworld');
expect(
await page.evaluate(() => JSON.stringify({ ...localStorage, ...sessionStorage })),
).not.toContain('e2e-llm');
} finally {
await other.close();
}
await page.getByRole('button', { name: 'ChatGPT 订阅', exact: true }).click();
await expect(page.getByRole('button', { name: '登录 ChatGPT 订阅', exact: true })).toBeVisible();
await page.screenshot({ path: info.outputPath('website-settings.png') });
expect(errors).toEqual([]);
});
test('网站 A 模型：真实 WASM、显式 mock 抓放和证据导出（无远端模型调用）', async ({
test('服务器内置设置＋假 HTTP 上游：不同语言改变真实 WASM 位移与角度，不重载机器人', async ({
page,
}, info) => {
test.skip(!existsSync('build/lekiwi/lekiwi-v1.zip'), '需要本地 A 模型，不自动公开或下载资产');
let calls = 0;
page.on('request', (r) => {
if (/\/api\/decision\/v1\/(plan|decide)$/.test(r.url())) calls++;
});
await page.goto('/');
await page
.locator('input[type=file]')
.first()
.setInputFiles(resolve('build/lekiwi/lekiwi-v1.zip'));
const dialog = page.getByRole('dialog', { name: '配置 URDF 仿真组件' });
await expect(dialog).toBeVisible({ timeout: 30000 });
await dialog.getByLabel('URDF 机器人 profile').selectOption('lekiwi-v1');
await dialog.getByRole('button', { name: '转换并加载' }).click();
await expect(page.getByRole('button', { name: '▶ 播放' })).toBeEnabled({ timeout: 60000 });
await page.getByRole('tab', { name: '控制台', exact: true }).click();
await page.getByRole('button', { name: 'LeKiwi 智能抓放', exact: true }).click();
const panel = page.getByLabel('LeKiwi 智能抓放任务', { exact: true });
await panel.getByRole('button', { name: '开始任务（mock）', exact: true }).click();
await expect(panel.getByTestId('agent-task-state')).toContainText('物理验收成功', {
timeout: 180000,
});
await panel.getByText('高级操作与证据导出', { exact: true }).click();
const downloaded = page.waitForEvent('download');
await panel.getByRole('button', { name: '导出任务证据', exact: true }).click();
const evidence = JSON.parse(await readFile((await (await downloaded).path())!, 'utf8'));
expect(evidence.mode).toBe('mock');
expect(evidence.succeeded).toBe(true);
expect(evidence.transported).toBeGreaterThanOrEqual(0.5);
expect(evidence.sample.onGoalSupport).toBe(true);
expect(calls).toBe(0);
await writeFile(
info.outputPath('website-physical-evidence.json'),
JSON.stringify(evidence, null, 2),
);
await page.screenshot({ path: info.outputPath('website-grasp.png') });
const panel = await loadWebsiteRobot(page);
const forward = await runWebsiteCommand(page, '前进0.5米', info, 'forward');
expect(forward.distance).toBeGreaterThan(0.49);
expect(forward.distance).toBeLessThan(0.51);
const turn = await runWebsiteCommand(page, '左转90度', info, 'turn');
expect(turn.turnDegrees).toBeGreaterThan(89);
expect(turn.turnDegrees).toBeLessThan(91);
expect(turn.start.x).toBeCloseTo(forward.sample.x, 2);
const next = await runWebsiteCommand(page, '前进0.2米', info, 'forward-new-heading');
expect(next.sample.y - next.start.y).toBeGreaterThan(0.19);
const right = await runWebsiteCommand(page, '右转90度', info, 'turn-right');
expect(right.turnDegrees).toBeGreaterThan(-91);
expect(right.turnDegrees).toBeLessThan(-89);
const back = await runWebsiteCommand(page, '后退0.2米', info, 'backward');
expect(back.sample.x - back.start.x).toBeLessThan(-0.19);
await panel.getByRole('textbox', { name: '机器人指令' }).fill('转动90度');
await panel.getByRole('button', { name: '执行指令', exact: true }).click();
await expect(panel.getByRole('status')).toContainText('请说明左转还是右转');
await expect(panel.getByTestId('robot-command-state')).toHaveCount(0);
await expect(page.getByRole('button', { name: '▶ 播放' })).toBeVisible();
});
test('内置设置＋假 HTTP 上游：保留完整单方块真实物理抓放，不手填密钥', async ({ page }, info) => {
const panel = await loadWebsiteRobot(page);
await panel.getByRole('button', { name: '载入预设场景' }).click();
await expect(panel.getByRole('status')).toContainText('预设已载入', { timeout: 90000 });
const data = await runWebsiteCommand(page, '把方块搬到 B 区', info, 'pick-place');
expect(data.transported).toBeGreaterThanOrEqual(0.5);
expect(data.sample.onGoalSupport).toBe(true);
expect(data.receipts).toHaveLength(14);
});
test('正在运动时停止：撤销控制权并暂停物理，不把中断当成功', async ({ page }, info) => {
const panel = await loadWebsiteRobot(page);
await panel.getByRole('textbox', { name: '机器人指令' }).fill('前进0.5米');
await panel.getByRole('button', { name: '执行指令', exact: true }).click();
await expect(panel.getByTestId('robot-command-state')).toContainText('执行中', {
timeout: 30000,
});
await panel.getByRole('button', { name: '停止', exact: true }).click();
await expect(panel.getByTestId('robot-command-state')).toContainText('已停止');
await expect(page.getByRole('button', { name: '▶ 播放' })).toBeVisible();
const data = await readRobotResult(page);
expect(data.state).toBe('cancelled');
expect(data.succeeded).toBe(false);
expect(data.distance).toBeLessThan(0.1);
await writeFile(info.outputPath('cancelled-motion.json'), JSON.stringify(data, null, 2));
});
web_platform/physics/agent.ts
+101
View File
@@ -1,3 +1,10 @@
import {
composeLanguageScene,
LANGUAGE_SCENE,
resolveTarget,
} from '../src/mobile/agent/LanguageScene';
import { PickPlaceExecutor } from '../src/mobile/agent/PickPlaceExecutor';
import { LeKiwiIK } from '../src/mobile/agent/LeKiwiIK';
import { MainThreadPhysicsAdapter } from '../src/simulation/PhysicsAdapter';
import { importBrowserFiles } from '../src/project/importer';
import type { ProjectManifest } from '../src/project/types';
@@ -29,6 +36,100 @@ const api = {
if (!response.ok) throw new Error(`A 资产 HTTP ${response.status}`);
source = await importBrowserFiles([new File([await response.blob()], 'lekiwi-v1.zip')]);
},
async calibrateLanguage() {
if (!source) throw new Error('尚未导入 A 资产');
await adapter.load(source, 'URDF/LeKiwi.urdf', {
robotProfileId: 'lekiwi-v1',
configureRobotRuntime: false,
mobileRobot: PICK_PLACE_ROBOT,
urdfMode: 'mjcf',
baseMode: 'floating',
map: { kind: 'none' },
sceneComposer: (xml) => composeLanguageScene(xml, true),
});
adapter.releaseRetired();
const session = adapter.session!;
const ik = new LeKiwiIK(session);
const before = Array.from(session.data.qpos);
try {
const targets = [
LANGUAGE_SCENE.source,
...LANGUAGE_SCENE.targets.map((t) => t.position),
[0.237, 0.365, 0.128],
[0.277, 0.865, 0.128],
];
const checks = targets.flatMap((p) => {
resolveTarget('coordinates', p);
const pose = { x: p[0] - 0.257, y: p[1] - 0.015, yaw: 0 };
return [p[2], 0.195, 0.205, 0.223].map((z) => ({
target: [p[0], p[1], z],
residual: ik.solve([p[0], p[1], z], pose).residual,
}));
});
return { checks, unchanged: before.every((v, i) => session.data.qpos[i] === v) };
} finally {
ik.dispose();
}
},
async languagePhysical(goals: number[][], fault: false | 'stop' | 'empty' = false) {
await api.calibrateLanguage();
const session = adapter.session!;
const results = [];
for (const goal of goals) {
let skills!: PickPlaceExecutor;
let cancelled = false;
session.startAgent((write) => {
const grip = new ModelBindings(session.model).scalarActuator(
PICK_PLACE_ROBOT.gripperActuators[0].name,
PICK_PLACE_ROBOT.gripperJoint,
'position',
).actuatorId;
skills = new PickPlaceExecutor(
session,
(id, value) =>
write(id, fault === 'empty' && id === grip ? PICK_PLACE_ROBOT.gripperOpen : value),
resolveTarget('coordinates', goal),
false,
);
return {
beforeStep: () => {
if (cancelled) return false;
skills.beforeStep();
return skills.status === 'running';
},
afterStep: (dt) => skills.afterStep(dt),
hold: () => skills.hold(),
cancel: () => {
cancelled = true;
},
dispose: () => skills.dispose(),
};
});
session.setPaused(false);
let frames = 0;
while (skills.status === 'running' && !cancelled) {
session.singleStep();
if (++frames % 20 === 0) session.module.mj_forward(session.model, session.data);
if (fault === 'stop' && skills.phase === 'carry' && skills.evaluator.transported > 0.08)
session.stopAgent('测试运行中停止');
if (frames % 200 === 0) await new Promise<void>((resolve) => setTimeout(resolve, 0));
}
results.push({
status: skills.status,
error: skills.error,
goal,
sample: skills.sample,
transported: skills.evaluator.transported,
succeeded: skills.evaluator.succeeded,
maxJointSpeed: skills.maxJointSpeed,
trace: skills.trace,
cancelled,
});
session.stopAgent('回合结束');
if (skills.status !== 'succeeded') break;
}
return results;
},
async baseline(seed: number, governed = false) {
const epoch = ++generation;
task?.dispose();
web_platform/playwright.agent.config.ts
+2 -5
View File
@@ -1,4 +1,5 @@
import { defineConfig } from '@playwright/test';
import languageConfig from './playwright.language.config';
export default defineConfig({
testDir: './e2e',
testMatch: /lekiwi\.agent(?:\.(?:workspace|live|faults|lifecycle|subscription))?\.spec\.ts/,
@@ -6,9 +7,5 @@ export default defineConfig({
timeout: 300_000,
workers: 1,
use: { baseURL: 'http://127.0.0.1:4176' },
webServer: {
command: 'npm run dev --prefix .. -- --host 127.0.0.1 --port 4176 --strictPort',
url: 'http://127.0.0.1:4176',
reuseExistingServer: false,
},
webServer: languageConfig.webServer,
});
web_platform/playwright.builtin-robots.config.ts
+32
View File
@@ -0,0 +1,32 @@
import { defineConfig } from '@playwright/test';
const production = process.env.CADWORLD_PRODUCTION_TEST === '1';
const dev = process.env.CADWORLD_BUILTIN_DEV === '1';
export default defineConfig({
testDir: './e2e',
testMatch: 'builtinRobots.spec.ts',
workers: 1,
timeout: 180_000,
use: {
baseURL: production ? 'https://cadworld-sim.robotquan.com' : 'http://127.0.0.1:4182',
viewport: { width: 1440, height: 1000 },
trace: production ? 'off' : 'retain-on-failure',
screenshot: 'only-on-failure',
launchOptions: {
args: [
'--use-angle=swiftshader',
'--enable-unsafe-swiftshader',
...(production
? ['--host-resolver-rules=MAP cadworld-sim.robotquan.com 47.93.31.109']
: []),
],
},
},
webServer: production
? undefined
: {
command: `cd .. && npm run ${dev ? 'dev:website' : 'preview -- --mode website'} ${dev ? '--' : ''} --host 127.0.0.1 --port 4182 --strictPort`,
url: 'http://127.0.0.1:4182',
reuseExistingServer: false,
},
});
web_platform/playwright.config.ts
+1 -1
View File
@@ -1,7 +1,7 @@
import { defineConfig } from '@playwright/test';
export default defineConfig({
testDir: './e2e',
testIgnore: /(?:lekiwi.*|mobile|website.*)\.spec\.ts/,
testIgnore: /(?:lekiwi.*|mobile|website.*|builtinRobots)\.spec\.ts/,
timeout: 120_000,
use: { baseURL: 'http://127.0.0.1:4173' },
webServer: [
web_platform/playwright.language.config.ts
+30
View File
@@ -0,0 +1,30 @@
import { defineConfig } from '@playwright/test';
export default defineConfig({
testDir: './e2e',
testMatch: /lekiwi\.(?:language|agent\.workspace)\.spec\.ts/,
outputDir: '../build/e2e/lekiwi-language',
timeout: 300_000,
workers: 1,
use: {
baseURL: 'http://127.0.0.1:4176',
actionTimeout: 20000,
screenshot: 'only-on-failure',
trace: 'retain-on-failure',
viewport: { width: 1440, height: 1000 },
launchOptions: { args: ['--use-angle=swiftshader', '--enable-unsafe-swiftshader'] },
},
webServer: [
{
command:
'cd .. && CADWORLD_E2E=1 CADWORLD_E2E_ORIGIN=http://127.0.0.1:4176 CADWORLD_E2E_PORT=8771 .venv/bin/python -m decision_server.tests.website_fixture',
url: 'http://127.0.0.1:8771/healthz',
reuseExistingServer: false,
},
{
command:
'CADWORLD_DEV_API=http://127.0.0.1:8771 npm run dev --prefix .. -- --host 127.0.0.1 --port 4176 --strictPort',
url: 'http://127.0.0.1:4176',
reuseExistingServer: false,
},
],
});
web_platform/playwright.production.config.ts
+1 -1
View File
@@ -1,7 +1,7 @@
import { defineConfig } from '@playwright/test';
export default defineConfig({
testDir: './e2e',
testMatch: 'website.production.spec.ts',
testMatch: ['website.production.spec.ts', 'website.embed.production.spec.ts'],
workers: 1,
timeout: 240_000,
use: {
web_platform/playwright.website-live.config.ts
+9
View File
@@ -0,0 +1,9 @@
import { defineConfig } from '@playwright/test';
import production from './playwright.production.config';
export default defineConfig(production, {
testMatch: 'website.live.spec.ts',
retries: 0,
timeout: 360_000,
use: { trace: 'off', video: 'off', screenshot: 'off' },
});
web_platform/src/app/App.tsx
+129 -37
View File
@@ -50,6 +50,13 @@ import {
normalizeProjectPath,
ProjectImportError,
} from '../project/importer';
import {
builtinRobotFromSearch,
BUILTIN_ROBOTS,
builtinRobotEnhancements,
downloadBuiltinRobot,
type BuiltinRobot,
} from '../project/builtinRobots';
import {
MainThreadPhysicsAdapter,
type UrdfBaseMode,
@@ -57,6 +64,7 @@ import {
type UrdfLoadMode,
} from '../simulation/PhysicsAdapter';
import type { ActuatorParameters, SimulationSession } from '../simulation/SimulationSession';
import { composeLanguageScene } from '../mobile/agent/LanguageScene';
import { AgentTaskPanel } from '../mobile/agent/AgentTaskPanel';
import {
composePickPlaceScene,
@@ -66,7 +74,7 @@ import {
import type { DataRecorderConfig } from '../telemetry/DataRecorder';
import type { ControllerCommand, ControllerStatus } from '../controller/types';
import type { RLCommand, RLPolicyStatus } from '../rl/types';
import type { MuJoCoViewer, InteractionMode } from '../viewer/MuJoCoViewer';
import type { MuJoCoViewer, InteractionMode, ViewerCamera } from '../viewer/MuJoCoViewer';
import {
DEFAULT_VIEWER_DISPLAY_OPTIONS,
type ViewerDisplayOptions,
@@ -91,6 +99,7 @@ import { EntrySelectionDialog } from './components/EntrySelectionDialog';
import { ErrorRecoveryPanel } from './components/ErrorRecoveryPanel';
import { useSidebarLayout } from './hooks/useSidebarLayout';
import { ViewportOverlayLayout } from './components/ViewportOverlayLayout';
import { SensorCameraOverlay } from './components/SensorCameraOverlay';
import { SimulationControls } from './components/SimulationControls';
import { ViewerDisplayPopover } from './components/ViewerDisplayPopover';
import { ViewportHUD } from './components/ViewportHUD';
@@ -107,7 +116,7 @@ import {
LayoutSettingsDialog,
type LayoutPreset,
} from './components/LayoutSettingsDialog';
import { Button, ConfirmDialog, IconButton } from '../components/ui';
import { ConfirmDialog, IconButton } from '../components/ui';
import { PanelWidthBudgetContext } from '../components/ui/panelWidthBudget';
import { DiagnosticsDrawer } from './components/DiagnosticsDrawer';
import { ToolbarOverflowMenu } from './components/ToolbarOverflowMenu';
@@ -310,6 +319,10 @@ export function App() {
const lastAgentConfig = useRef<PickPlaceTaskConfig>({ seed: 0 });
const [agentScene, setAgentScene] = useState(false);
const sensorCameraFrame = useRef<HTMLDivElement>(null);
const [sensorCameraState, setSensorCameraState] = useState<{
cameras: readonly ViewerCamera[];
selectedId: number;
}>({ cameras: [], selectedId: -1 });
const orientationHost = useRef<HTMLDivElement>(null);
const [mapCommitState, setMapCommitState] = useState<'idle' | 'submitting' | 'failed'>('idle');
const [forceScale, setForceScale] = useState(50),
@@ -463,6 +476,9 @@ export function App() {
.then(({ MuJoCoViewer: Viewer }) => {
if (!active) return null;
const next = new Viewer(host, {
onSensorCamerasChanged: (cameras, selectedId) => {
if (active) setSensorCameraState({ cameras, selectedId });
},
onSelection: (selection) => {
if (selection && compiledMapBodyInteraction.current(selection.bodyName)) {
state.setSelection(null);
@@ -625,7 +641,9 @@ export function App() {
requestedPolicy?: { data: Uint8Array; path: string },
requestedMobile?: RobotConfig,
requestedTask?: PickPlaceTaskConfig,
signal?: AbortSignal,
) => {
signal?.throwIfAborted();
if (!manifest.current || loadInFlight.current) return false;
// 普通模型重载始终使用上次成功应用的地图基线。只有场景提交入口可以显式
// 传入草稿实例，防止切换模型/URDF 模式时把一半场景静默提前提交。
@@ -672,7 +690,9 @@ export function App() {
urdfMode: requestedMode ?? urdfModeRef.current,
baseMode: requestedTask ? 'floating' : baseModeRef.current,
enhancements: requestedTask
? { addActuators: true, addSensors: false, sensorType: 'camera' }
? requestedTask.preset === 'language-v2'
? builtinRobotEnhancements('lekiwi-v1')
: { addActuators: true, addSensors: false, sensorType: 'camera' }
: urdfEnhancementsRef.current,
robotProfileId: mobileRobot
? mobileRobot.recipe === 'lekiwi-v1'
@@ -682,7 +702,10 @@ export function App() {
mobileRobot,
configureRobotRuntime: requestedTask ? false : undefined,
sceneComposer: requestedTask
? (xml) => composePickPlaceScene(xml, requestedTask.seed, true, requestedTask.goal)
? (xml) =>
requestedTask.preset === 'language-v2'
? composeLanguageScene(xml, true)
: composePickPlaceScene(xml, requestedTask.seed, true, requestedTask.goal)
: undefined,
mapAssets: mobileRobot ? [] : sceneAssets,
map: requestedTask ? { kind: 'none' } : undefined,
@@ -705,6 +728,7 @@ export function App() {
});
adapter.current.setSpeed(useAppStore.getState().speed);
try {
signal?.throwIfAborted();
activeViewer.attach(adapter.current.session);
attachedViewer = activeViewer;
sessionSwapped = true;
@@ -777,6 +801,7 @@ export function App() {
summarizeMapSceneDraft(placedMapAssetsRef.current, appliedMapAssetsRef.current)
.changedIds,
);
signal?.throwIfAborted();
adapter.current.releaseRetired();
setTrainingDeployment(requestedDeployment);
agentSceneSession.current = requestedTask
@@ -885,7 +910,7 @@ export function App() {
}
};
const ingest = useCallback(
async (files: File[], lockOwned = false) => {
async (files: File[], lockOwned = false, builtin?: BuiltinRobot, signal?: AbortSignal) => {
if (importInFlight.current && !lockOwned) return;
importInFlight.current = true;
externalControl.disconnect('正在导入工程，请重新连接并授权');
@@ -917,6 +942,12 @@ export function App() {
setImportProgress({ title: '正在导入工程', label, detail: path, value });
},
);
signal?.throwIfAborted();
if (builtin) {
if (!next.entries.some((entry) => entry.path === builtin.entryPath))
throw new Error(`内置机器人缺少入口：${builtin.entryPath}`);
next.selectedEntry = builtin.entryPath;
}
setImportProgress({
title: '正在导入工程',
label: '处理模型资源与入口',
@@ -925,6 +956,14 @@ export function App() {
});
manifest.current = next;
robotProfilesRef.current = new Map();
if (builtin) {
robotProfilesRef.current.set(builtin.entryPath, builtin.profileId);
urdfModeRef.current = 'mjcf';
setUrdfMode('mjcf');
baseModeRef.current = 'floating';
setBaseMode('floating');
urdfEnhancementsRef.current = builtinRobotEnhancements(builtin.id);
}
setProjectMaps(next.maps);
setCommittedEditorDocuments(manifestEditorDocuments(next));
setProjectSidebarTab('assets');
@@ -968,8 +1007,10 @@ export function App() {
};
setNotifications((items) => [importNotice, ...items].slice(0, 20));
setToast(importNotice);
if (next.selectedEntry) await requestLoadEntry(next.selectedEntry);
if (builtin) await loadEntry(builtin.entryPath);
else if (next.selectedEntry) await requestLoadEntry(next.selectedEntry);
} catch (error) {
if (signal?.aborted) return;
state.setDiagnostic(
diagnostic(
error instanceof ProjectImportError && /ZIP/.test(error.message) ? 'ZIP' : '导入',
@@ -994,8 +1035,51 @@ export function App() {
state.setLoading(false);
}
},
[requestLoadEntry],
[requestLoadEntry, loadEntry],
);
useEffect(() => {
const controller = new AbortController();
let ownsImport = false;
const start = async () => {
// React StrictMode replays effects. Defer acquisition so its discarded mount
// neither downloads the ZIP nor leaves the shared import lock occupied.
await Promise.resolve();
if (controller.signal.aborted) return;
try {
const builtin = builtinRobotFromSearch(window.location.search);
if (!builtin || importInFlight.current || manifest.current) return;
ownsImport = true;
importInFlight.current = true;
state.setLoading(true);
state.setDiagnostic(undefined);
setImportProgress({
title: `正在加载内置机器人 ${builtin.label}`,
label: '下载服务器机器人资源',
detail: '首次加载约 22 MB，下载后自动导入仿真',
value: 0.02,
});
const signal = AbortSignal.any([controller.signal, AbortSignal.timeout(120_000)]);
const file = await downloadBuiltinRobot(builtin, signal);
if (!controller.signal.aborted) await ingest([file], true, builtin, controller.signal);
} catch (error) {
if (!controller.signal.aborted)
state.setDiagnostic({
category: '导入',
summary: '内置机器人加载失败',
detail: `${error instanceof Error ? error.message : String(error)}。可刷新页面重试，或移除 robot 参数后手动导入。`,
at: Date.now(),
});
} finally {
if (ownsImport) {
importInFlight.current = false;
setImportProgress(undefined);
state.setLoading(false);
}
}
};
void start();
return () => controller.abort();
}, [ingest]);
const removeProject = () => {
if (state.projectName) setRemoveConfirmOpen(true);
};
@@ -2207,7 +2291,33 @@ export function App() {
setSelectedPolicyPath(path);
void loadPolicyBytes(file.data, path).catch(() => {});
};
const prepareAgentScene = async (config: PickPlaceTaskConfig): Promise<SimulationSession> => {
const prepareAgentScene = async (
config: PickPlaceTaskConfig,
signal?: AbortSignal,
): Promise<SimulationSession> => {
if (loadInFlight.current || importInFlight.current) throw new Error('模型正在加载，请稍后重试');
signal?.throwIfAborted();
const hasRobot = manifest.current?.entries.some(
(e) =>
e.format === 'urdf' &&
(robotProfilesRef.current.get(e.path) === 'lekiwi-v1' ||
/(^|\/)URDF\/LeKiwi\.urdf$/.test(e.path)),
);
if (!hasRobot && config.preset === 'language-v2') {
if (
manifest.current &&
!window.confirm('载入预设将替换当前工程。请先保存未保存内容，是否继续？')
)
throw new Error('已取消替换工程');
const builtin = BUILTIN_ROBOTS[0];
const bounded = signal
? AbortSignal.any([signal, AbortSignal.timeout(120000)])
: AbortSignal.timeout(120000);
const file = await downloadBuiltinRobot(builtin, bounded);
bounded.throwIfAborted();
await ingest([file], false, builtin, bounded);
bounded.throwIfAborted();
}
const entry = manifest.current?.entries.find(
(e) =>
e.format === 'urdf' &&
@@ -2224,6 +2334,7 @@ export function App() {
undefined,
PICK_PLACE_ROBOT,
config,
signal,
))
)
throw new Error('A 任务场景加载失败，请查看编译诊断');
@@ -2982,38 +3093,19 @@ export function App() {
)
}
camera={
Boolean(state.snapshot?.model.ncam) &&
(showSensorCamera ? (
<div
aria-label="摄像头画面"
ref={(node) => {
Boolean(state.snapshot?.model.ncam) && (
<SensorCameraOverlay
cameras={sensorCameraState.cameras}
selectedId={sensorCameraState.selectedId}
visible={showSensorCamera}
onVisibleChange={setShowSensorCamera}
onSelect={(id) => viewer.current?.setSensorCamera(id)}
frameRef={(node) => {
sensorCameraFrame.current = node;
viewer.current?.setSensorCameraViewportElement(node);
}}
className="sensor-camera-frame overflow-hidden rounded-lg border border-border-strong shadow-xl"
>
<div className="pointer-events-auto absolute inset-x-0 top-0 flex h-7 items-center justify-between bg-black/65 px-2 text-xs font-medium text-white">
<span className="flex items-center gap-1">
<Camera className="h-3 w-3" />
摄像头
</span>
<button
type="button"
className="rounded px-1.5 py-0.5 hover:bg-white/15 focus-visible:outline-none focus-visible:ring-2 focus-visible:ring-white/60"
onClick={() => setShowSensorCamera(false)}
>
隐藏画面
</button>
</div>
</div>
) : (
<Button
icon={<Camera className="h-3.5 w-3.5" />}
onClick={() => setShowSensorCamera(true)}
>
显示摄像头画面
</Button>
))
/>
)
}
/>
<MapAssetDropIndicator target={mapAssetDropTarget} />
web_platform/src/app/components/ModelControlsSidebar.test.tsx
+15
View File
@@ -83,6 +83,21 @@ function props(
}
describe('ModelControlsSidebar', () => {
it('从空白工作台载入机器人不卸载语言控制工具', () => {
const tools = <input aria-label="任务草稿" defaultValue="" />;
const view = render(
<ModelControlsSidebar
{...props({ snapshot: undefined, workspaceTool: 'controls', workspaceTools: tools })}
/>,
);
const input = screen.getByRole('textbox', { name: '任务草稿' });
fireEvent.change(input, { target: { value: '把方块搬到 B 区' } });
view.rerender(
<ModelControlsSidebar {...props({ workspaceTool: 'controls', workspaceTools: tools })} />,
);
expect(screen.getByRole('textbox', { name: '任务草稿' })).toBe(input);
expect(input).toHaveValue('把方块搬到 B 区');
});
it('没有选择时仅显示收敛的场景摘要', () => {
render(<ModelControlsSidebar {...props()} />);
expect(screen.getByText('未选择对象')).toBeVisible();
web_platform/src/app/components/ModelControlsSidebar.tsx
+10 -4
View File
@@ -102,14 +102,20 @@ export function ModelControlsSidebar(props: ModelControlsProps) {
return (
<SidebarPanel title={title} side="right" visible={props.visible} icon={icon}>
{tabs}
{props.workspaceTool ? (
props.workspaceTools
) : (
<div
hidden={props.workspaceTool !== null}
role="tabpanel"
aria-label="检查器"
className="min-h-0 flex-1 overflow-auto panel-scroll"
>
<div className="m-2.5 flex h-14 items-center justify-center gap-2 border border-dashed border-border-subtle text-xs text-text-tertiary">
<PanelRight className="h-4 w-4" aria-hidden="true" />
检查器待命
</div>
)}
</div>
<div hidden={!props.workspaceTool} className="min-h-0 flex-1 overflow-auto panel-scroll">
{props.workspaceTools}
</div>
</SidebarPanel>
);
web_platform/src/app/components/SensorCameraOverlay.test.tsx
+54
View File
@@ -0,0 +1,54 @@
import { render, screen, fireEvent } from '@testing-library/react';
import { createRef } from 'react';
import { SensorCameraOverlay } from './SensorCameraOverlay';
const cameras = [
{ id: 0, name: '__platform_camera__', bodyName: 'Camera-Model-v3' },
{ id: 1, name: '__platform_camera__wrist', bodyName: 'Camera-Model-v3-1' },
];
it('双摄像头选择器明确前置/腕部及挂载点，切换只发送选择事件', () => {
const onSelect = vi.fn(),
onVisibleChange = vi.fn();
const props = {
cameras,
selectedId: 0,
visible: true,
frameRef: createRef<HTMLDivElement>(),
onSelect,
onVisibleChange,
};
const { rerender } = render(<SensorCameraOverlay {...props} />);
expect(screen.getByRole('option', { name: '前置摄像头' })).toHaveValue('0');
expect(screen.getByRole('option', { name: '腕部摄像头' })).toHaveValue('1');
expect(screen.getByLabelText('摄像头视角')).toHaveAttribute('title', 'Camera-Model-v3');
fireEvent.change(screen.getByLabelText('摄像头视角'), { target: { value: '1' } });
expect(onSelect).toHaveBeenCalledExactlyOnceWith(1);
rerender(<SensorCameraOverlay {...props} selectedId={1} />);
expect(screen.getByLabelText('摄像头画面')).toHaveAttribute(
'data-camera-body',
'Camera-Model-v3-1',
);
fireEvent.click(screen.getByRole('button', { name: '隐藏画面' }));
expect(onVisibleChange).toHaveBeenCalledExactlyOnceWith(false);
rerender(<SensorCameraOverlay {...props} selectedId={1} visible={false} />);
expect(props.frameRef.current).toBeNull();
expect(screen.queryByLabelText('摄像头视角')).not.toBeInTheDocument();
fireEvent.click(screen.getByRole('button', { name: '显示摄像头画面' }));
expect(onVisibleChange).toHaveBeenLastCalledWith(true);
});
it('普通单相机不显示多余选择器', () => {
render(
<SensorCameraOverlay
cameras={[{ id: 0, name: 'custom_camera', bodyName: 'head' }]}
selectedId={0}
visible
frameRef={null}
onSelect={vi.fn()}
onVisibleChange={vi.fn()}
/>,
);
expect(screen.getByText('custom_camera')).toBeVisible();
expect(screen.queryByRole('combobox')).not.toBeInTheDocument();
});
web_platform/src/app/components/SensorCameraOverlay.tsx
+74
View File
@@ -0,0 +1,74 @@
import type { Ref } from 'react';
import { Camera } from 'lucide-react';
import { Button } from '../../components/ui';
import type { ViewerCamera } from '../../viewer/MuJoCoViewer';
function cameraLabel(camera: ViewerCamera): string {
if (camera.bodyName === 'Camera-Model-v3') return '前置摄像头';
if (camera.bodyName === 'Camera-Model-v3-1') return '腕部摄像头';
return camera.name;
}
/** One live viewport with a camera selector, avoiding extra CAD render passes. */
export function SensorCameraOverlay({
cameras,
selectedId,
visible,
frameRef,
onSelect,
onVisibleChange,
}: {
cameras: readonly ViewerCamera[];
selectedId: number;
visible: boolean;
frameRef: Ref<HTMLDivElement>;
onSelect(id: number): void;
onVisibleChange(visible: boolean): void;
}) {
if (!visible)
return (
<Button icon={<Camera className="h-3.5 w-3.5" />} onClick={() => onVisibleChange(true)}>
显示摄像头画面
</Button>
);
const selected = cameras.find((camera) => camera.id === selectedId);
return (
<div
aria-label="摄像头画面"
data-camera-name={selected?.name}
data-camera-body={selected?.bodyName}
ref={frameRef}
className="sensor-camera-frame overflow-hidden rounded-lg border border-border-strong shadow-xl"
>
<div className="pointer-events-auto absolute inset-x-0 top-0 flex h-7 items-center gap-1 bg-black/65 px-2 text-xs font-medium text-white">
<Camera className="h-3 w-3 shrink-0" />
{cameras.length > 1 ? (
<select
aria-label="摄像头视角"
title={selected?.bodyName}
className="min-w-0 flex-1 rounded bg-black/50 text-xs text-white focus-visible:outline-none focus-visible:ring-2 focus-visible:ring-white/60"
value={selectedId}
onChange={(event) => onSelect(Number(event.target.value))}
>
{cameras.map((camera) => (
<option key={camera.id} value={camera.id}>
{cameraLabel(camera)}
</option>
))}
</select>
) : (
<span className="min-w-0 flex-1 truncate" title={selected?.bodyName}>
{selected ? cameraLabel(selected) : '摄像头'}
</span>
)}
<button
type="button"
className="shrink-0 rounded px-1.5 py-0.5 hover:bg-white/15 focus-visible:outline-none focus-visible:ring-2 focus-visible:ring-white/60"
onClick={() => onVisibleChange(false)}
>
隐藏画面
</button>
</div>
</div>
);
}
web_platform/src/app/components/WorkspaceToolsPanel.tsx
+8 -6
View File
@@ -139,11 +139,6 @@ export function WorkspaceToolsPanel({
const resolvedPolicyStatus = policyStatus ?? snapshot?.rlPolicy;
const controls = snapshot ? (
<>
{agentTask && (
<CollapsibleSection title="LeKiwi 智能抓放" defaultOpen={false} keepMounted>
{agentTask}
</CollapsibleSection>
)}
{websiteCapabilities.externalControl && externalControl && (
<CollapsibleSection
title="开源项目 / 外部控制"
@@ -356,7 +351,14 @@ export function WorkspaceToolsPanel({
aria-label={active === 'controls' ? '控制台' : '数据录制'}
className="min-h-0 flex-1 overflow-y-auto panel-scroll"
>
<div hidden={active !== 'controls'}>{controls}</div>
<div hidden={active !== 'controls'}>
{agentTask && (
<CollapsibleSection title="机器人语言控制" defaultOpen={false} keepMounted>
{agentTask}
</CollapsibleSection>
)}
{controls}
</div>
<div hidden={active !== 'data'}>
{active === 'data' &&
(snapshot ? (
web_platform/src/mobile/agent/AgentTaskController.ts
+33 -16
View File
@@ -1,3 +1,5 @@
import { TaskSupervisor } from './TaskSupervisor';
import { LANGUAGE_VERSION } from './LanguageScene';
import type { AgentDriver } from '../../simulation/AgentDriver';
import type { DeterministicBaseline } from './DeterministicBaseline';
import type { DecisionProvider } from './DecisionClient';
@@ -5,8 +7,9 @@ import { RequestGate, type RequestStamp } from './RequestGate';
import {
AGENT_VERSION,
SKILLS,
LANGUAGE_SKILLS,
type AgentVersion,
observation,
validateJevDecision,
validatePlan,
skillErrorCode,
type JevDecision,
@@ -56,6 +59,9 @@ export class AgentTaskController implements AgentDriver {
error = '';
plan?: Plan;
lastDecision?: JevDecision;
readonly supervisor = new TaskSupervisor();
readonly version: AgentVersion;
private readonly skillSequence: Skill[];
readonly results: SkillResult[] = [];
readonly failures: { skill: Skill; code: string; sequence: number }[] = [];
readonly recoveries: Partial<Record<Skill, number>> = {};
@@ -90,8 +96,15 @@ export class AgentTaskController implements AgentDriver {
sceneRevision?: number;
now?: () => number;
limits?: Partial<Limits>;
version?: AgentVersion;
initialLlmCalls?: number;
} = {},
) {
this.version = options.version ?? AGENT_VERSION;
this.skillSequence = this.version === LANGUAGE_VERSION ? LANGUAGE_SKILLS : SKILLS;
this.llmCalls = options.initialLlmCalls ?? 0;
if (![0, 1].includes(this.llmCalls)) throw new Error('初始任务调用数非法');
this.jevCalls = this.llmCalls; // /command includes one Jev intent gate in the same run.
this.runId = options.runId ?? crypto.randomUUID();
this.sceneRevision = options.sceneRevision ?? 0;
this.now = options.now ?? (() => performance.now());
@@ -112,7 +125,9 @@ export class AgentTaskController implements AgentDriver {
}
snapshot() {
return {
version: AGENT_VERSION,
version: this.version,
qualityScore: this.supervisor.qualityScore,
scoreScope: 'latest-skill-boundary',
goal: [...this.skills.goal],
mode: this.provider.mode,
wallMs: (this.finishedAt ?? this.now()) - this.began,
@@ -179,7 +194,7 @@ export class AgentTaskController implements AgentDriver {
sequence: this.sequence,
});
this.results.push({
version: AGENT_VERSION,
version: this.version,
skill: this.skills.phase,
status: 'failed',
code: skillErrorCode(this.skills.error),
@@ -212,7 +227,7 @@ export class AgentTaskController implements AgentDriver {
this.planRevision++;
const remaining = this.recoveryPlan
? [...SKILLS]
: SKILLS.slice(SKILLS.indexOf(this.skills.candidate));
: this.skillSequence.slice(this.skillSequence.indexOf(this.skills.candidate));
const stamp = this.stamp();
const obs = observation(
this.skills.sample,
@@ -220,6 +235,8 @@ export class AgentTaskController implements AgentDriver {
this.skills.goal,
stamp,
this.skills.phase,
[],
this.version,
);
const epoch = this.epoch;
this.state = 'planning';
@@ -248,6 +265,8 @@ export class AgentTaskController implements AgentDriver {
this.skills.goal,
stamp,
this.skills.phase,
[],
this.version,
);
const epoch = this.epoch;
this.state = 'deciding';
@@ -272,7 +291,7 @@ export class AgentTaskController implements AgentDriver {
private apply(outcome: Outcome): void {
if (outcome.kind === 'error') throw new Error(outcome.error);
if (outcome.kind === 'plan') {
this.plan = validatePlan(outcome.value, outcome.remaining);
this.plan = validatePlan(outcome.value, outcome.remaining, this.version);
this.needPlan = false;
if (this.recoveryPlan) {
this.recover();
@@ -280,11 +299,13 @@ export class AgentTaskController implements AgentDriver {
}
return;
}
const value = validateJevDecision(outcome.value, outcome.candidates);
const value = this.supervisor.inspect(
outcome.value,
outcome.candidates,
this.version,
this.skills.evaluator.secure,
);
this.lastDecision = value;
if (value.choice === 'stop' || value.recovery === 'stop') throw new Error('Jev 请求安全停止');
if (value.grasp === 'secure' && !this.skills.evaluator.secure)
throw new Error('Jev 抓持判断与本地物理证据矛盾');
const failed = this.skills.status === 'failed';
if (value.recovery === 'replan') {
this.replan(failed);
@@ -301,11 +322,7 @@ export class AgentTaskController implements AgentDriver {
if (policy !== 'retry') throw new Error('当前计划不允许直接重试');
this.recover();
} else if (value.recovery !== 'continue') throw new Error('无故障时不能重复技能');
if (
['carry', 'stop-base'].includes(value.choice) &&
(value.grasp !== 'secure' || !this.skills.evaluator.secure)
)
throw new Error('搬运前未取得一致抓持证据');
if (value.choice === 'stop') throw new Error('Jev 请求安全停止');
this.skills.authorize(value.choice);
this.state = 'executing';
}
@@ -368,7 +385,7 @@ export class AgentTaskController implements AgentDriver {
else if (this.skills.waiting || this.skills.status === 'succeeded') {
if (this.skills.status === 'succeeded' && !this.skills.evaluator.succeeded) {
this.results.push({
version: AGENT_VERSION,
version: this.version,
skill: this.skills.phase,
status: 'failed',
code: 'placement_unverified',
@@ -378,7 +395,7 @@ export class AgentTaskController implements AgentDriver {
return;
}
this.results.push({
version: AGENT_VERSION,
version: this.version,
skill: this.skills.phase,
status: 'completed',
code: 'none',
web_platform/src/mobile/agent/AgentTaskPanel.test.tsx
+99
View File
@@ -0,0 +1,99 @@
import { cleanup, fireEvent, render, screen, waitFor } from '@testing-library/react';
import { afterEach, expect, it, vi } from 'vitest';
import { AgentTaskPanel, type AgentWorkbenchBridge } from './AgentTaskPanel';
import { DecisionClient } from './DecisionClient';
import * as motion from './MotionDriver';
import type { SimulationSession } from '../../simulation/SimulationSession';
vi.mock('../../app/websiteConfig', () => ({ WEBSITE_MODE: true }));
afterEach(() => {
cleanup();
vi.restoreAllMocks();
});
function fixture(session?: SimulationSession) {
const bridge: AgentWorkbenchBridge = {
loadScene: vi.fn(),
getSession: vi.fn(() => session),
isTaskScene: vi.fn(),
publish: vi.fn(),
};
render(<AgentTaskPanel bridge={bridge} loading={false} />);
return bridge;
}
it('网站只显示语言控制，不显示密钥、模型模式或订阅配置，示例不自动执行', () => {
const bootstrap = vi.spyOn(DecisionClient, 'website');
const bridge = fixture();
expect(screen.queryByText(/API_KEY|ChatGPT|deepseek|MOCK/)).not.toBeInTheDocument();
expect(screen.getByRole('textbox', { name: '机器人指令' })).not.toBeDisabled();
expect(screen.queryByRole('button', { name: '左转90度' })).not.toBeInTheDocument();
expect(screen.getByRole('button', { name: '载入预设场景' })).toBeInTheDocument();
fireEvent.change(screen.getByRole('textbox', { name: '机器人指令' }), {
target: { value: '左转90度' },
});
expect(screen.getByRole('textbox', { name: '机器人指令' })).toHaveValue('左转90度');
expect(bootstrap).not.toHaveBeenCalled();
expect(bridge.loadScene).not.toHaveBeenCalled();
});
it('没加载机器人拒绝执行且不付费；停止指令不依赖网络', async () => {
const bootstrap = vi.spyOn(DecisionClient, 'website');
fixture();
fireEvent.click(screen.getByRole('button', { name: '执行指令' }));
await waitFor(() => expect(screen.getByRole('status')).toHaveTextContent('请先导入'));
fireEvent.change(screen.getByRole('textbox', { name: '机器人指令' }), {
target: { value: '停止' },
});
fireEvent.click(screen.getByRole('button', { name: '执行指令' }));
expect(bootstrap).not.toHaveBeenCalled();
});
it('其他控制器仍占用时在发起模型请求前拒绝，不为无法执行的动作付费', async () => {
const bootstrap = vi.spyOn(DecisionClient, 'website');
vi.spyOn(motion, 'motionIO').mockReturnValue({ sample: vi.fn(), write: vi.fn(), hold: vi.fn() });
fixture({ controlOwner: 'python' } as unknown as SimulationSession);
fireEvent.click(screen.getByRole('button', { name: '执行指令' }));
await waitFor(() => expect(screen.getByRole('status')).toHaveTextContent('请先停止当前脚本'));
expect(bootstrap).not.toHaveBeenCalled();
});
for (const intervention of ['stop', 'reload'] as const) {
it(`等待期间 ${intervention} 后，即使上游忽略取消并迟到回复也不会驱动机器人`, async () => {
const startAgent = vi.fn();
const session = {
data: { time: 0, qpos: [0], ctrl: [0] },
paused: true,
controlOwner: 'manual',
setPaused: vi.fn(),
startAgent,
} as unknown as SimulationSession;
vi.spyOn(motion, 'motionIO').mockReturnValue({
sample: vi.fn(),
write: vi.fn(),
hold: vi.fn(),
});
let complete!: (value: unknown) => void;
const command = vi.fn(
() =>
new Promise((resolve) => {
complete = resolve;
}),
);
vi.spyOn(DecisionClient, 'website').mockResolvedValue({
call: vi.fn(async () => ({ ready: true })),
command,
} as unknown as DecisionClient);
const bridge = fixture(session);
fireEvent.change(screen.getByRole('textbox', { name: '机器人指令' }), {
target: { value: '前进0.5米' },
});
fireEvent.click(screen.getByRole('button', { name: '执行指令' }));
await waitFor(() => expect(command).toHaveBeenCalledTimes(1));
if (intervention === 'stop') fireEvent.click(screen.getByRole('button', { name: /^停止$/ }));
else vi.mocked(bridge.getSession).mockReturnValue(undefined);
complete({ command: { action: 'move', value: 0.5, summary: '前进' } });
await waitFor(() =>
expect(screen.getByRole('status')).toHaveTextContent(
intervention === 'stop' ? '用户停止' : '旧指令未执行',
),
);
expect(startAgent).not.toHaveBeenCalled();
});
}
web_platform/src/mobile/agent/AgentTaskPanel.tsx
+6 -410
View File
@@ -1,418 +1,14 @@
import { useEffect, useRef, useState } from 'react';
import { Button } from '../../components/ui';
import { WEBSITE_MODE } from '../../app/websiteConfig';
import { downloadBytes } from '../../project/cachedFiles';
import type { SimulationSession } from '../../simulation/SimulationSession';
import { AgentTaskController } from './AgentTaskController';
import { DecisionClient } from './DecisionClient';
import { DeterministicBaseline } from './DeterministicBaseline';
import { MockDecisionProvider } from './MockDecisionProvider';
import { ModelConnectionPanel } from './ModelConnectionPanel';
import {
PICK_PLACE,
TASK_GOAL_LIMITS,
seedPositions,
type PickPlaceTaskConfig,
} from './PickPlaceScene';
import type { PickPlaceTaskConfig } from './PickPlaceScene';
import { RobotLanguagePanel } from './RobotLanguagePanel';
export interface AgentWorkbenchBridge {
loadScene(config: PickPlaceTaskConfig): Promise<SimulationSession>;
loadScene(config: PickPlaceTaskConfig, signal?: AbortSignal): Promise<SimulationSession>;
getSession(): SimulationSession | undefined;
isTaskScene(): boolean;
publish(): void;
}
const states = {
idle: '准备',
planning: '等待 LLM 规划',
deciding: '等待 Jev 判定',
executing: '执行技能',
paused: '任务已暂停',
succeeded: '物理验收成功',
failed: '失败 / 安全停止',
cancelled: '已取消',
};
export function AgentTaskPanel({
bridge,
loading,
}: {
bridge: AgentWorkbenchBridge;
loading: boolean;
}) {
const liveBridge = useRef(bridge);
useEffect(() => {
liveBridge.current = bridge;
}, [bridge]);
const task = useRef<AgentTaskController | undefined>(undefined);
const ownSession = useRef<SimulationSession | undefined>(undefined);
const [snapshot, setSnapshot] = useState<ReturnType<AgentTaskController['snapshot']>>();
const [client, setClient] = useState<DecisionClient>();
const [settingsOpen, setSettingsOpen] = useState(false);
const [mode, setMode] = useState<'mock' | 'api'>('mock');
const [instruction, setInstruction] = useState(
'把方块抓起，使用底盘搬运到指定支撑台，放下、松开并撤离。',
);
const [goal, setGoal] = useState<number[]>([...PICK_PLACE.goal.position]);
const [busy, setBusy] = useState(false);
const [message, setMessage] = useState('需要 A / lekiwi-v1 资产；不会自动下载或替换机器人。');
const epoch = useRef(0);
const sceneDirty = useRef(true);
const alive = useRef(true);
function publish() {
const current = task.current;
if (current) setSnapshot(current.snapshot());
liveBridge.current.publish();
}
function stop(reason = '用户停止任务') {
epoch.current++;
const current = task.current,
session = ownSession.current;
if (current) current.cancel(reason);
if (
session &&
session === liveBridge.current.getSession() &&
session.snapshot().controlOwner === 'agent'
) {
session.stopAgent(reason);
session.setPaused(true);
}
if (alive.current) publish();
}
useEffect(() => {
alive.current = true;
const timer = setInterval(() => {
const current = task.current,
session = ownSession.current;
if (!current) return;
setSnapshot(current.snapshot());
if (
current.terminal &&
session === liveBridge.current.getSession() &&
session?.snapshot().controlOwner === 'agent'
) {
session.stopAgent('任务已结束');
session.setPaused(true);
liveBridge.current.publish();
}
}, 250);
const invalidatePending = () => {
epoch.current++;
};
return () => {
alive.current = false;
invalidatePending();
clearInterval(timer);
const session = ownSession.current;
if (
session === liveBridge.current.getSession() &&
session?.snapshot().controlOwner === 'agent'
) {
session.stopAgent('任务面板已卸载');
session.setPaused(true);
}
task.current?.dispose();
};
}, []);
async function load() {
stop('任务场景重新加载');
const generation = epoch.current;
setBusy(true);
try {
seedPositions(0, goal);
const session = await liveBridge.current.loadScene({ seed: 0, goal: [...goal] });
if (generation !== epoch.current || !alive.current) return;
ownSession.current = session;
task.current = undefined;
sceneDirty.current = false;
setSnapshot(undefined);
setMessage('A 抓放场景已加载并暂停；用户地图草稿保留，但不用于本次空旷平地任务。');
} catch (error) {
if (generation === epoch.current && alive.current)
setMessage(error instanceof Error ? error.message : '加载失败');
} finally {
if (alive.current) setBusy(false);
}
}
async function start(single = false) {
if (busy || loading) return;
if (mode === 'api' && !client) {
setMessage(
WEBSITE_MODE
? '请在模型设置中保存 LLM 与 Jev 连接；不会自动转为演示。'
: '真实模式必须先连接本机模型服务；不会转为 mock。',
);
return;
}
if (task.current) {
stop('开始新回合');
task.current.dispose();
task.current = undefined;
sceneDirty.current = true;
}
setSnapshot(undefined);
const generation = ++epoch.current;
setBusy(true);
try {
seedPositions(0, goal);
let session = liveBridge.current.getSession();
if (sceneDirty.current || task.current || !liveBridge.current.isTaskScene())
session = await liveBridge.current.loadScene({ seed: 0, goal: [...goal] });
if (generation !== epoch.current || !alive.current) return;
if (!session) throw new Error('尚未加载任务场景');
const provider = mode === 'api' ? client! : new MockDecisionProvider();
session.startAgent((write) => {
const skills = new DeterministicBaseline(session!, 0, write, true, goal);
try {
const controller = new AgentTaskController(skills, provider, instruction);
if (single) controller.resume(true);
task.current = controller;
return controller;
} catch (error) {
skills.dispose();
throw error;
}
});
ownSession.current = session;
sceneDirty.current = false;
session.setPaused(false);
setMessage(
mode === 'mock'
? '显式 mock 决策；执行与验收为真实 MuJoCo 物理。'
: '真实模型模式，可能计费；失败不降级规则。',
);
publish();
} catch (error) {
if (generation === epoch.current && alive.current)
setMessage(error instanceof Error ? error.message : '启动失败');
} finally {
if (alive.current) setBusy(false);
}
}
function resume(single = false) {
const current = task.current;
if (current && !current.terminal && ownSession.current === liveBridge.current.getSession()) {
current.resume(single);
publish();
} else void start(single);
}
function changeGoal(axis: number, value: number) {
stop('目标编辑使旧计划失效');
sceneDirty.current = true;
setGoal((previous) => previous.map((v, i) => (i === axis ? value : v)));
// Any edit requires a new scene before starting, even if values are subsequently restored.
task.current?.dispose();
setMessage('目标已编辑；开始时会重新加载场景与方块，旧计划不可复用。');
}
const running = Boolean(
snapshot && !['succeeded', 'failed', 'cancelled', 'paused'].includes(snapshot.state),
);
const locked = busy || loading;
const receipts = client?.receipts.filter((r) => r.stamp.runId === snapshot?.runId) ?? [];
const costs = receipts.filter((r) => r.usage.cost !== undefined);
const tokens = receipts.filter((r) =>
['total_tokens', 'input_tokens', 'prompt_tokens'].some((key) => r.usage[key] !== undefined),
);
return (
<div className="space-y-3 text-xs" aria-label="LeKiwi 智能抓放任务">
<p>
仅仿真 · MuJoCo 结构化真值（非视觉） · A /
lekiwi-v1。空旷平地单方块，不承诺障碍导航或任意目标泛化。
</p>
<p>任务模式：{mode === 'mock' ? 'MOCK（不调用模型 API）' : '真实 LLM + Jev（可能计费）'}</p>
<label className="block">
决策模式
<select
aria-label="智能任务决策模式"
disabled={locked}
value={mode}
onChange={(e) => {
stop('决策模式改变');
setMode(e.target.value as 'mock' | 'api');
}}
className="w-full bg-surface p-1"
>
<option value="mock">显式 mock LLM + Jev</option>
<option value="api">真实模型服务（无自动回退）</option>
</select>
</label>
<label className="block">
中文任务指令
<textarea
className="w-full rounded border border-border bg-surface p-2"
rows={3}
maxLength={2000}
value={instruction}
disabled={locked || running}
onChange={(e) => {
stop('任务指令修改');
setInstruction(e.target.value);
}}
/>
</label>
<details open={!WEBSITE_MODE}>
<summary>高级：目标位置与实验范围</summary>
<p>语言计划必须遵循安全技能顺序；放置位置以以下场景坐标为准。</p>
<fieldset disabled={locked || running} className="space-y-1">
<legend>目标位置（米，暂停后编辑；重新载入回合）</legend>
{[0, 1].map((axis) => (
<label key={axis} className="block">
目标 {axis ? 'Y' : 'X'}
<input
className="ml-2 w-24 bg-surface p-1"
type="number"
min={TASK_GOAL_LIMITS.min[axis]}
max={TASK_GOAL_LIMITS.max[axis]}
step="0.001"
value={Number.isNaN(goal[axis]) ? '' : goal[axis]}
onChange={(e) => changeGoal(axis, e.target.valueAsNumber)}
/>
</label>
))}
<p>Z 固定 {goal[2]}；坐标编辑为受限实验，默认冻结场景已验收。</p>
</fieldset>
</details>
<div className="flex flex-wrap gap-2">
<Button disabled={locked || running} onClick={() => void start()}>
开始任务（{mode === 'api' ? '真实 API' : 'mock'}）
</Button>
<Button
disabled={locked || !running}
onClick={() => {
task.current?.pause();
publish();
}}
>
暂停任务
</Button>
<Button disabled={locked || snapshot?.state !== 'paused'} onClick={() => resume()}>
继续任务
</Button>
<Button onClick={() => stop()}>停止任务</Button>
</div>
<details open={!WEBSITE_MODE}>
<summary>高级操作与证据导出</summary>
<div className="mt-2 flex flex-wrap gap-2">
<Button disabled={locked || running} onClick={() => void load()}>
加载 A 任务场景
</Button>
<Button disabled={locked || running} onClick={() => resume(true)}>
单技能步进
</Button>
<Button disabled={locked} onClick={() => void load()}>
重置任务场景
</Button>
<Button
disabled={!snapshot}
onClick={() => {
const current = task.current;
if (!current) return;
downloadBytes(
new TextEncoder().encode(
JSON.stringify(
{
...current.snapshot(),
results: current.results,
failures: current.failures,
trace: (current.skills as DeterministicBaseline).trace,
receipts,
},
null,
2,
),
),
`lekiwi-agent-${current.snapshot().runId}.json`,
);
}}
>
导出任务证据
</Button>
</div>
</details>
<p role="status">{message}</p>
{snapshot && (
<div data-testid="agent-task-state" className="space-y-1">
<p>
状态：{states[snapshot.state]} · 当前技能：{snapshot.phase}
</p>
{snapshot.error && (
<p role="alert" className="text-warning">
{snapshot.error}
</p>
)}
<details open={!WEBSITE_MODE}>
<summary>运行详情与物理证据</summary>
<p>
规划 {snapshot.llmCalls}/3 · Jev {snapshot.jevCalls}/60 · 重规划 {snapshot.replans}/2
</p>
<p>
墙钟 {(snapshot.wallMs / 1000).toFixed(1)} s · 仿真 {snapshot.sample.time.toFixed(2)}{' '}
s · 安全抓持 {snapshot.secure ? '是' : '否'} · 有效搬运{' '}
{snapshot.transported.toFixed(3)} m
</p>
<p>
双指力 {snapshot.sample.fingerForces.map((v) => v.toFixed(2)).join(' / ')} N ·
目标支撑 {snapshot.sample.onGoalSupport ? '是' : '否'}
</p>
<p>
Jev：
{snapshot.decision
? `${snapshot.decision.choice} / ${snapshot.decision.grasp} / ${snapshot.decision.diagnosis} / ${snapshot.decision.recovery}`
: '尚未判定'}
</p>
<p>
已返回费用：
{costs.length
? `$${costs.reduce((sum, r) => sum + r.usage.cost, 0).toFixed(6)}（部分服务报告；非完整账单）`
: '未报告，不估算为零'}
</p>
<p>
已报告 Token：
{tokens.length
? tokens.reduce(
(sum, r) =>
sum +
(r.usage.total_tokens ??
(r.usage.input_tokens ?? r.usage.prompt_tokens ?? 0) +
(r.usage.output_tokens ?? r.usage.completion_tokens ?? 0)),
0,
)
: '未报告'}{' '}
· 最近调用耗时{' '}
{receipts.at(-1)?.elapsedMs === undefined
? '未报告'
: `${receipts.at(-1)!.elapsedMs!.toFixed(0)} ms`}
</p>
<p>最近模型：{receipts.at(-1)?.model ?? (mode === 'mock' ? 'mock' : '尚无有效响应')}</p>
</details>
{snapshot.plan && (
<details open>
<summary>LLM 受约束计划</summary>
<p>{snapshot.plan.summary}</p>
<ol className="list-inside list-decimal">
{snapshot.plan.steps.map((s) => (
<li key={s.skill}>
{s.skill} · 失败时 {s.onFailure}
</li>
))}
</ol>
</details>
)}
</div>
)}
<p>
全局暂停/隐藏/重载会取消任务；任务级暂停可继续。重置前请导出当前证据。
{!WEBSITE_MODE && 'RL/外控与任务互斥，后台训练作业不会被停止。'}
</p>
<details onToggle={(event) => setSettingsOpen(event.currentTarget.open)}>
<summary>
{WEBSITE_MODE ? '模型设置 · API_KEY / ChatGPT 订阅' : '模型连接与官方订阅登录'}
</summary>
{(!WEBSITE_MODE || settingsOpen) && (
<ModelConnectionPanel
onClient={setClient}
onInvalidate={() => stop('模型配置或账号操作使任务失效')}
/>
)}
</details>
</div>
);
export function AgentTaskPanel(props: { bridge: AgentWorkbenchBridge; loading: boolean }) {
return <RobotLanguagePanel {...props} />;
}
web_platform/src/mobile/agent/DecisionClient.ts
+45 -2
View File
@@ -1,4 +1,5 @@
import type { RequestStamp, Stamped } from './RequestGate';
import { validateRobotCommand, type SceneContext } from './RobotCommand';
import {
validateContract,
validateJevDecision,
@@ -206,6 +207,45 @@ export class DecisionClient implements DecisionProvider {
signal.removeEventListener('abort', cancel);
}
}
async command(
instruction: string,
stamp: RequestStamp,
signal: AbortSignal,
sceneContext?: SceneContext,
) {
const cancel = () => {
void this.call(
'/cancel',
'POST',
{ runId: stamp.runId, requestId: stamp.requestId },
AbortSignal.timeout(2000),
).catch(() => undefined);
};
if (signal.aborted) throw new Error('指令已取消');
signal.addEventListener('abort', cancel, { once: true });
try {
const response = (await this.call(
'/command',
'POST',
{ instruction, stamp, ...(sceneContext ? { sceneContext } : {}) },
signal,
)) as {
stamp: unknown;
value: unknown;
usage?: unknown;
};
const actual = validateContract<RequestStamp>('Stamp', response.stamp);
if ((Object.keys(stamp) as (keyof RequestStamp)[]).some((key) => actual[key] !== stamp[key]))
throw new Error('过期指令响应已拒绝');
return {
command: validateRobotCommand(response.value),
usage: response.usage,
stamp: actual,
};
} finally {
signal.removeEventListener('abort', cancel);
}
}
async plan(
obs: Observation,
instruction: string,
@@ -213,7 +253,7 @@ export class DecisionClient implements DecisionProvider {
signal: AbortSignal,
): Promise<Stamped<Plan>> {
const result = await this.request('plan', obs, { instruction, remaining }, signal);
return { stamp: result.stamp, value: validatePlan(result.value, remaining) };
return { stamp: result.stamp, value: validatePlan(result.value, remaining, obs.version) };
}
async decide(
obs: Observation,
@@ -222,6 +262,9 @@ export class DecisionClient implements DecisionProvider {
signal: AbortSignal,
): Promise<Stamped<JevDecision>> {
const result = await this.request('decide', obs, { candidates, failure }, signal);
return { stamp: result.stamp, value: validateJevDecision(result.value, candidates) };
return {
stamp: result.stamp,
value: validateJevDecision(result.value, candidates, obs.version),
};
}
}
web_platform/src/mobile/agent/DeterministicBaseline.ts
+74 -18
View File
@@ -1,3 +1,4 @@
import languageScene from '../../../../contracts/lekiwi-language-scene-v2.json';
import type { SimulationSession } from '../../simulation/SimulationSession';
import { ModelBindings } from '../../simulation/ModelBindings';
import { LeKiwiIK } from './LeKiwiIK';
@@ -17,7 +18,18 @@ const phases = [
'retreat',
'settle',
] as const;
export type BaselinePhase = (typeof phases)[number];
export type BaselinePhase = (typeof phases)[number] | 'stow' | 'approach';
export interface PickPlaceExecution {
source: number[];
goal: number[];
sourceDock: number[];
goalDock: number[];
supportGeom: string;
transportMin: number;
approachTimeout: number;
carryTimeout: number;
episodeTimeout: number;
}
const durations = [2, 6, 3, 3, 4, 22, 1, 4, 3, 4, 1];
const clamp = (v: number, lo: number, hi: number) => Math.max(lo, Math.min(hi, v));
@@ -44,6 +56,7 @@ export class DeterministicBaseline {
private readonly velocities = Array<number>(5).fill(0);
private readonly baseVelocity = [0, 0, 0];
private readonly translation: readonly number[];
private readonly executionPhases: readonly BaselinePhase[];
private opening: number;
private phaseIndex = 0;
private phaseStart: number;
@@ -64,9 +77,13 @@ export class DeterministicBaseline {
private readonly writeServo: (actuator: number, value: number) => void,
private readonly gated = false,
goal?: readonly number[],
private readonly execution?: PickPlaceExecution,
) {
const b = new ModelBindings(session.model);
this.positions = seedPositions(seed, goal);
this.executionPhases = execution ? ['stow', 'approach', ...phases] : phases;
this.positions = execution
? { object: execution.source, goal: execution.goal }
: seedPositions(seed, goal);
this.translation = goal
? this.positions.goal.slice(0, 2).map((v, i) => v - this.positions.object[i])
: P.baseTranslation;
@@ -79,7 +96,7 @@ export class DeterministicBaseline {
this.wheels = R.baseActuators.map((name, i) =>
b.scalarActuator(name, R.baseJoints[i], 'velocity'),
);
this.sampler = new PhysicalSampler(session);
this.sampler = new PhysicalSampler(session, execution?.supportGeom, Boolean(execution));
let ik: LeKiwiIK | undefined;
try {
this.ik = ik = new LeKiwiIK(session);
@@ -93,16 +110,20 @@ export class DeterministicBaseline {
this.servo = [...this.sample.joints];
this.opening = this.sample.opening;
this.start = this.phaseStart = this.nextControl = this.sample.time;
this.evaluator = new PickPlaceEvaluator(this.positions.object, this.positions.goal);
this.evaluator = new PickPlaceEvaluator(
this.positions.object,
this.positions.goal,
execution?.transportMin,
);
}
get phase(): BaselinePhase {
return phases[this.phaseIndex];
return this.executionPhases[this.phaseIndex];
}
get goal(): readonly number[] {
return this.positions.goal;
}
get candidate(): BaselinePhase {
return phases[this.phaseIndex + (this.completedPhase ? 1 : 0)] ?? 'settle';
return this.executionPhases[this.phaseIndex + (this.completedPhase ? 1 : 0)] ?? 'settle';
}
/** Re-check facts even if both models recommend continuation. No model can grant this gate. */
authorize(skill: BaselinePhase): void {
@@ -117,7 +138,10 @@ export class DeterministicBaseline {
throw new Error('empty_grasp');
if (['carry', 'stop-base'].includes(skill) && !this.evaluator.secure)
throw new Error('grasp_unverified');
if (skill === 'place' && (this.evaluator.transported < P.transportMin || s.baseSpeed > 0.01))
if (
skill === 'place' &&
(this.evaluator.transported < this.evaluator.transportMin || s.baseSpeed > 0.01)
)
throw new Error('transport_unverified');
if (skill === 'release' && !s.onGoalSupport) throw new Error('placement_unsupported');
if (
@@ -135,7 +159,7 @@ export class DeterministicBaseline {
return (
this.gated &&
this.status === 'failed' &&
this.phaseIndex <= phases.indexOf('verify') &&
this.phaseIndex <= this.executionPhases.indexOf('verify') &&
['alignment_timeout', 'empty_grasp', 'grasp_unverified'].includes(this.error) &&
s.supported &&
s.objectSpeed < P.placementSpeedMax &&
@@ -154,7 +178,7 @@ export class DeterministicBaseline {
this.servo.splice(0, this.servo.length, ...this.sample.joints);
this.armStallTime.fill(0);
this.graspSource = [...this.sample.object];
this.phaseIndex = 0;
this.phaseIndex = this.executionPhases.indexOf('open');
this.phaseStart = this.nextControl = this.sample.time;
this.completedPhase = false;
this.waiting = true;
@@ -166,6 +190,12 @@ export class DeterministicBaseline {
const source = this.graspSource,
goal = this.positions.goal;
switch (this.phase) {
case 'stow':
return [
this.baseStart[0] + P.object.position[0],
this.baseStart[1] + P.object.position[1],
P.retreatHeight,
];
case 'pregrasp':
return [source[0], source[1], P.pregraspHeight];
case 'descend':
@@ -231,9 +261,11 @@ export class DeterministicBaseline {
R.gripperClosed + this.opening * (R.gripperOpen - R.gripperClosed),
);
const desired = [0, 0, 0];
if (this.phase === 'carry') {
const dx = this.baseStart[0] + this.translation[0] - this.sample.base[0];
const dy = this.baseStart[1] + this.translation[1] - this.sample.base[1];
if (this.phase === 'carry' || this.phase === 'approach') {
const dock =
this.phase === 'approach' ? this.execution!.sourceDock : this.execution?.goalDock;
const dx = (dock?.[0] ?? this.baseStart[0] + this.translation[0]) - this.sample.base[0];
const dy = (dock?.[1] ?? this.baseStart[1] + this.translation[1]) - this.sample.base[1];
const scale = Math.min(1, P.baseSpeed / Math.max(1e-12, Math.hypot(dx, dy)));
const c = Math.cos(this.sample.yaw),
s = Math.sin(this.sample.yaw);
@@ -282,19 +314,43 @@ export class DeterministicBaseline {
if (this.sample.peakJointSpeed > P.jointSpeedStop) throw new Error('joint_velocity');
if (this.sample.base[2] < 0.03 || this.sample.base[2] > 0.1 || this.sample.tilt > 0.35)
throw new Error('base_unstable');
if (this.sample.time - this.start > P.episodeTimeout) throw new Error('episode_timeout');
const supportForce = this.sample.fingerSupportForce ?? 0;
if (
this.sample.forbiddenContact ||
supportForce > languageScene.fingerSupportForceMax ||
(supportForce > 0.2 &&
(this.sample.baseSpeed > languageScene.contactBaseSpeedMax ||
!['descend', 'close', 'verify', 'place', 'release'].includes(this.phase)))
)
throw new Error('collision');
if (this.sample.time - this.start > (this.execution?.episodeTimeout ?? P.episodeTimeout))
throw new Error('episode_timeout');
if (this.phase === 'carry' || this.phase === 'stop-base') {
this.lostGraspTime = this.evaluator.secure ? 0 : this.lostGraspTime + dt;
if (this.lostGraspTime > 0.2) throw new Error('lost_grasp');
}
let ready = this.sample.time - this.phaseStart >= durations[this.phaseIndex];
if (this.phase === 'carry') {
const duration =
this.phase === 'stow'
? 6
: this.phase === 'approach'
? this.execution!.approachTimeout
: this.phase === 'carry' && this.execution
? this.execution.carryTimeout
: durations[phases.indexOf(this.phase as (typeof phases)[number])];
let ready = this.sample.time - this.phaseStart >= duration;
if (this.phase === 'carry' || this.phase === 'approach') {
const dock =
this.phase === 'approach' ? this.execution!.sourceDock : this.execution?.goalDock;
const distance = Math.hypot(
...this.sample.base
.slice(0, 2)
.map((v, i) => v - this.baseStart[i] - this.translation[i]),
.map((v, i) => v - (dock?.[i] ?? this.baseStart[i] + this.translation[i])),
);
if (distance < 0.007) ready = true;
if (
distance < 0.007 &&
(!this.execution || (this.sample.baseSpeed < 0.01 && Math.abs(this.sample.yaw) < 0.015))
)
ready = true;
else if (ready) throw new Error('navigation_timeout');
}
if (!ready) return;
@@ -304,7 +360,7 @@ export class DeterministicBaseline {
if (this.phase === 'close' && this.sample.fingerForces.some((f) => f < P.fingerForceMin))
throw new Error('empty_grasp');
if (this.phase === 'verify' && !this.evaluator.secure) throw new Error('grasp_unverified');
if (this.phase === 'carry' && this.evaluator.transported < P.transportMin)
if (this.phase === 'carry' && this.evaluator.transported < this.evaluator.transportMin)
throw new Error('transport_unverified');
if (this.phase === 'place' && !this.sample.onGoalSupport)
throw new Error('placement_unsupported');
web_platform/src/mobile/agent/LanguageScene.test.ts
+43
View File
@@ -0,0 +1,43 @@
import { describe, expect, it } from 'vitest';
import { LANGUAGE_VERSION, resolveTarget } from './LanguageScene';
import { validateRobotCommand } from './RobotCommand';
describe('语言任务目标', () => {
it('命名区与连续坐标使用同一支撑面，不修改场景', () => {
expect(resolveTarget('B')).toEqual({ position: [0.257, 0.75, 0.128], supportId: 'table' });
expect(resolveTarget('coordinates', [0.25, 0.5])).toEqual({
position: [0.25, 0.5, 0.128],
supportId: 'table',
});
expect(resolveTarget('coordinates', [0.257, 0.015, 0.128]).supportId).toBe('source');
});
it.each([
[0.2, 0.5],
[0.25, 0.9],
[0.25, 0.5, 0.3],
[NaN, 0.5],
[Infinity, 0.5],
])('rejects unsafe %s', (...p) => {
expect(() => resolveTarget('coordinates', p)).toThrow();
});
it('拒绝模型覆盖命名区、未知目标与不匹配支撑', () => {
const c = {
version: LANGUAGE_VERSION,
action: 'pick_place',
value: 0,
summary: '搬运',
objectId: 'block',
targetId: 'B',
position: [],
supportId: 'table',
};
expect(validateRobotCommand(c).targetId).toBe('B');
for (const invalid of [
{ position: [0.25, 0.5] },
{ targetId: 'C' },
{ supportId: 'source' },
{ code: 'run()' },
])
expect(() => validateRobotCommand({ ...c, ...invalid })).toThrow();
});
});
web_platform/src/mobile/agent/LanguageScene.ts
+70
View File
@@ -0,0 +1,70 @@
import config from '../../../../contracts/lekiwi-language-scene-v2.json';
import { composePickPlaceScene } from './PickPlaceScene';
export const LANGUAGE_SCENE = config;
export const LANGUAGE_VERSION = 'lekiwi-language-v2' as const;
export type SupportId = 'source' | 'table';
export type TargetId = 'A' | 'B' | 'coordinates';
export interface ResolvedTarget {
position: number[];
supportId: SupportId;
}
/** Positions refer to the centre of the released block, not the tabletop. */
export function resolveTarget(id: TargetId, coordinates: readonly number[] = []): ResolvedTarget {
const named = config.targets.find((t) => t.id === id);
if (id !== 'coordinates' && !named) throw new Error('未知目标区');
if (id !== 'coordinates' && coordinates.length) throw new Error('命名目标不能覆写坐标');
const p = named ? [...named.position] : [...coordinates];
if (![2, 3].includes(p.length) || p.some((v) => !Number.isFinite(v)))
throw new Error('请提供世界坐标 XY 或 XYZ，单位为米');
const margin = config.objectHalfSize + config.edgeMargin;
const supports = config.supports.filter((s) =>
s.center.every((v, i) => Math.abs(p[i] - v) <= s.halfSize[i] - margin + 1e-9),
);
if (supports.length !== 1) throw new Error('目标没有安全支撑面或过于靠近台面边缘');
const support = supports[0];
const z = support.height + config.objectHalfSize;
if (p.length === 3 && Math.abs(p[2] - z) > 0.001)
throw new Error(`目标 Z 必须为物体中心高度 ${z.toFixed(3)} 米`);
return { position: [p[0], p[1], z], supportId: support.id as SupportId };
}
export function composeLanguageScene(source: Uint8Array, alreadyMobile = false): Uint8Array {
const doc = new DOMParser().parseFromString(
new TextDecoder().decode(composePickPlaceScene(source, 0, alreadyMobile)),
'application/xml',
);
for (const s of config.supports) {
const geom = doc.querySelector(`geom[name="${s.geom}"]`)!;
geom.setAttribute('pos', [...s.center, s.height / 2].join(' '));
geom.setAttribute('size', [...s.halfSize, s.height / 2].join(' '));
}
// Keep the legacy binding for the mobile adapter, but hide its obsolete fixed-goal disk.
doc.querySelector('geom[name="__mm_goal_geom"]')!.setAttribute('rgba', '0 0 0 0');
const world = doc.querySelector('mujoco > worldbody')!;
for (const t of config.targets) {
const site = doc.createElement('site');
site.setAttribute('name', `__language_${t.id}`);
site.setAttribute('type', 'box');
site.setAttribute('size', '.018 .025 .0005');
site.setAttribute('pos', `${t.position[0]} ${t.position[1]} .1106`);
site.setAttribute('rgba', '0 0 0 0');
world.append(site);
// The web viewer renders geoms, not MuJoCo sites. Markers never participate in contact.
const marker = doc.createElement('geom');
for (const [name, value] of Object.entries({
name: `__language_marker_${t.id}`,
type: 'box',
group: '1',
size: '.018 .025 .0005',
pos: `${t.position[0]} ${t.position[1]} .1106`,
contype: '0',
conaffinity: '0',
rgba: t.id === 'A' ? '.2 .8 .3 .7' : '.2 .4 1 .7',
}))
marker.setAttribute(name, value);
world.append(marker);
}
return new TextEncoder().encode(new XMLSerializer().serializeToString(doc));
}
web_platform/src/mobile/agent/LeKiwiIK.ts
+14 -1
View File
@@ -50,7 +50,10 @@ export class LeKiwiIK {
throw error;
}
}
solve(target: readonly number[]): { targets: number[]; residual: number; iterations: number } {
solve(
target: readonly number[],
basePose?: { x: number; y: number; yaw: number },
): { targets: number[]; residual: number; iterations: number } {
if (this.disposed) throw new Error('IK 已释放');
if (target.length !== 3 || target.some((v) => !Number.isFinite(v) || Math.abs(v) > 2))
throw new Error('IK 目标超出标定范围');
@@ -58,6 +61,16 @@ export class LeKiwiIK {
// Refresh views at each call; compiling another candidate can grow WASM memory.
const qpos: Float64Array = this.scratch.qpos;
qpos.set(data.qpos);
if (basePose) {
if (Object.values(basePose).some((v) => !Number.isFinite(v) || Math.abs(v) > 4))
throw new Error('非法预检底盘位姿');
const bindings = new ModelBindings(model);
const joint = bindings.resolve(bindings.joints, PICK_PLACE_ROBOT.baseJointName);
const address = Number(model.jnt_qposadr[joint]);
qpos[address] = basePose.x;
qpos[address + 1] = basePose.y;
qpos.set([Math.cos(basePose.yaw / 2), 0, 0, Math.sin(basePose.yaw / 2)], address + 3);
}
const q = this.joints.map((j) => qpos[j.qposAddress]);
const jp: Float64Array = this.positionJacobian.GetView(),
jr: Float64Array = this.rotationJacobian.GetView();
web_platform/src/mobile/agent/MockDecisionProvider.ts
+5 -3
View File
@@ -1,7 +1,6 @@
import type { DecisionProvider } from './DecisionClient';
import type { Stamped } from './RequestGate';
import {
AGENT_VERSION,
PRECONDITIONS,
type Observation,
type Plan,
@@ -22,7 +21,7 @@ export class MockDecisionProvider implements DecisionProvider {
return {
stamp: { ...obs.stamp },
value: {
version: AGENT_VERSION,
version: obs.version,
objectId: 'block',
goalId: 'placement',
summary: 'MOCK：固定规范顺序，成功仅由物理评估决定',
@@ -44,7 +43,10 @@ export class MockDecisionProvider implements DecisionProvider {
return {
stamp: { ...obs.stamp },
value: {
version: AGENT_VERSION,
version: obs.version,
...(obs.version === 'lekiwi-language-v2'
? { noul: 'allow' as const, score: 'good' as const, reason: 'verified_progress' as const }
: {}),
choice: candidates[0],
grasp: obs.evidence.secure
? 'secure'
web_platform/src/mobile/agent/MotionDriver.test.ts
+88
View File
@@ -0,0 +1,88 @@
import { expect, it } from 'vitest';
import { MotionDriver, type MotionIO, type MotionPose } from './MotionDriver';
import { validateRobotCommand } from './RobotCommand';
function environment(yaw = 0) {
const pose: MotionPose = {
time: 0,
x: 0,
y: 0,
z: 0,
yaw,
tilt: 0,
speed: 0,
angularSpeed: 0,
collision: false,
};
let velocity = [0, 0, 0];
const io: MotionIO = {
sample: () => ({ ...pose }),
write: (x, y, w) => {
velocity = [x, y, w];
},
hold: () => {
velocity = [0, 0, 0];
},
};
function step(d: MotionDriver) {
if (!d.beforeStep()) return;
const dt = 0.002,
[x, y, w] = velocity;
pose.x += (x * Math.cos(pose.yaw) - y * Math.sin(pose.yaw)) * dt;
pose.y += (x * Math.sin(pose.yaw) + y * Math.cos(pose.yaw)) * dt;
pose.yaw = Math.atan2(Math.sin(pose.yaw + w * dt), Math.cos(pose.yaw + w * dt));
pose.time += dt;
pose.speed = Math.hypot(x, y);
pose.angularSpeed = Math.abs(w);
d.afterStep(dt);
}
return { io, pose, step, velocity: () => velocity };
}
for (const [action, value, yaw] of [
['move', 0.5, 0],
['move', -0.2, Math.PI / 2],
['turn', 90, 0],
['turn', -90, -3],
['turn', 180, 3],
] as const) {
it(`闭环 ${action} ${value} 在起始朝向 ${yaw} 下以物理误差验收，跨 ±π 不反转`, () => {
const e = environment(yaw),
c = new MotionDriver(e.io, { action, value, summary: '测试' });
for (let i = 0; i < 15000 && !c.terminal; i++) e.step(c);
expect(c.state, c.error).toBe('succeeded');
if (action === 'move') {
expect(
Math.hypot(c.sample.x - value * Math.cos(yaw), c.sample.y - value * Math.sin(yaw)),
).toBeLessThan(0.008);
} else expect(c.snapshot().turnDegrees).toBeCloseTo(value, 0);
expect(e.velocity()).toEqual([0, 0, 0]);
});
}
it('接触/姿态异常不由模型绕过，停止后旧控制器无法继续推进', () => {
const e = environment(),
c = new MotionDriver(e.io, { action: 'move', value: 0.5, summary: '前进' });
e.step(c);
e.pose.collision = true;
e.step(c);
expect(c.state).toBe('failed');
expect(c.beforeStep()).toBe(false);
expect(e.velocity()).toEqual([0, 0, 0]);
const f = environment(),
d = new MotionDriver(f.io, { action: 'turn', value: 90, summary: '左转' });
d.cancel('停止');
expect(d.beforeStep()).toBe(false);
expect(d.snapshot().succeeded).toBe(false);
});
it('静止/打滑不算完成，超时拒绝；非法输出不能启动', () => {
const e = environment(),
c = new MotionDriver(e.io, { action: 'move', value: 0.5, summary: '前进' });
for (let i = 0; i < 16000 && !c.terminal; i++) {
c.beforeStep();
e.pose.time += 0.002;
c.afterStep(0.002);
}
expect(c.state).toBe('failed');
expect(c.snapshot().succeeded).toBe(false);
for (const value of [NaN, Infinity, 0, 2, true])
expect(() => validateRobotCommand({ action: 'move', value, summary: '测试' })).toThrow();
});
web_platform/src/mobile/agent/MotionDriver.ts
+230
View File
@@ -0,0 +1,230 @@
import type { AgentDriver } from '../../simulation/AgentDriver';
import type { SimulationSession } from '../../simulation/SimulationSession';
import { ModelBindings } from '../../simulation/ModelBindings';
import { LEKIWI_PROFILE as P, bodyToWheels } from '../../robot/profiles/lekiwi';
import { validateRobotCommand, type RobotCommand } from './RobotCommand';
export interface MotionPose {
time: number;
x: number;
y: number;
z: number;
yaw: number;
tilt: number;
speed: number;
angularSpeed: number;
collision: boolean;
}
export interface MotionIO {
sample(): MotionPose;
write(vx: number, vy: number, wz: number): void;
hold(): void;
}
const clamp = (x: number, max: number) => Math.max(-max, Math.min(max, x));
const wrap = (x: number) => Math.atan2(Math.sin(x), Math.cos(x));
/** Bounded closed-loop motion on the existing physics clock; never sets qpos or steps physics. */
export class MotionDriver implements AgentDriver {
state: 'running' | 'succeeded' | 'failed' | 'cancelled' = 'running';
error = '';
readonly start: MotionPose;
readonly target: { x: number; y: number; yaw: number };
sample: MotionPose;
readonly trace: MotionPose[] = [];
private yaw: number;
private readonly velocity = [0, 0, 0];
private nextControl: number;
private stable = 0;
private readonly wallStart = performance.now();
constructor(
readonly io: MotionIO,
readonly command: RobotCommand,
) {
validateRobotCommand(command);
if (!['move', 'turn'].includes(command.action)) throw new Error('不是移动指令');
this.sample = this.start = io.sample();
this.check(this.sample);
if (this.sample.speed > 0.03 || this.sample.angularSpeed > 0.05)
throw new Error('请先让底盘停稳');
this.yaw = this.sample.yaw;
this.nextControl = this.sample.time;
this.target = {
x: this.start.x + (command.action === 'move' ? command.value * Math.cos(this.start.yaw) : 0),
y: this.start.y + (command.action === 'move' ? command.value * Math.sin(this.start.yaw) : 0),
yaw: this.start.yaw + (command.action === 'turn' ? (command.value * Math.PI) / 180 : 0),
};
}
get terminal() {
return this.state !== 'running';
}
private check(s: MotionPose) {
if (Object.entries(s).some(([key, v]) => key !== 'collision' && !Number.isFinite(v)))
throw new Error('物理状态无效');
if (s.tilt > 0.15 || s.speed > 0.3 || s.angularSpeed > 1.3)
throw new Error('姿态或速度超限，安全停止');
if (s.collision) throw new Error('接触障碍物，安全停止；当前不支持避障导航');
}
private fail(error: unknown) {
this.state = 'failed';
this.error = error instanceof Error ? error.message : '动作失败';
this.hold();
}
beforeStep(): boolean {
if (this.terminal) return false;
try {
if (performance.now() - this.wallStart > 60000 || this.sample.time - this.start.time > 30)
throw new Error('动作超时，已停止');
if (this.sample.time + 1e-9 < this.nextControl) return true;
this.nextControl = this.sample.time + 1 / 30;
const dx = this.target.x - this.sample.x,
dy = this.target.y - this.sample.y;
const c = Math.cos(this.sample.yaw),
s = Math.sin(this.sample.yaw);
const desired = [
clamp(1.5 * (c * dx + s * dy), 0.08),
clamp(1.5 * (-s * dx + c * dy), 0.08),
clamp(2 * (this.target.yaw - this.yaw), 0.5),
];
this.velocity.forEach((v, i) => {
this.velocity[i] += clamp(desired[i] - v, (i === 2 ? 1 : 0.15) / 30);
});
this.io.write(this.velocity[0], this.velocity[1], this.velocity[2]);
if (this.trace.length < 1500) this.trace.push({ ...this.sample });
return true;
} catch (error) {
this.fail(error);
return false;
}
}
afterStep(dt: number): void {
if (this.terminal) return;
try {
const next = this.io.sample();
this.check(next);
if (Math.abs(next.z - this.start.z) > 0.1)
throw new Error('底盘高度变化超限，仅支持平地动作');
if (
!Number.isFinite(dt) ||
dt <= 0 ||
dt > 0.02 ||
Math.abs(next.time - this.sample.time - dt) > 1e-6
)
throw new Error('物理时钟不连续');
if (
Math.hypot(next.x - this.sample.x, next.y - this.sample.y) > 0.3 * dt + 1e-4 ||
Math.abs(wrap(next.yaw - this.sample.yaw)) > 1.3 * dt + 1e-4
)
throw new Error('物理位姿突变');
this.yaw += wrap(next.yaw - this.sample.yaw);
this.sample = next;
const distanceError = Math.hypot(next.x - this.target.x, next.y - this.target.y);
const angleError = Math.abs(this.yaw - this.target.yaw);
this.stable =
distanceError < 0.008 &&
angleError < 0.015 &&
next.speed < 0.012 &&
next.angularSpeed < 0.02
? this.stable + dt
: 0;
if (this.stable >= 0.3) {
this.state = 'succeeded';
this.hold();
}
} catch (error) {
this.fail(error);
}
}
hold(): void {
this.velocity.fill(0);
this.io.hold();
}
cancel(reason = '用户停止'): void {
if (!this.terminal) {
this.state = 'cancelled';
this.error = reason;
}
}
dispose(): void {
this.cancel('控制器已释放');
}
snapshot() {
return {
state: this.state,
error: this.error,
command: this.command,
start: this.start,
target: this.target,
sample: this.sample,
distance: Math.hypot(this.sample.x - this.start.x, this.sample.y - this.start.y),
turnDegrees: ((this.yaw - this.start.yaw) * 180) / Math.PI,
succeeded: this.state === 'succeeded',
};
}
}
export function motionIO(
session: SimulationSession,
write: (id: number, value: number) => void,
): MotionIO {
if (session.describeRobot()?.profileId !== P.id && session.mobile?.env.config.id !== P.id)
throw new Error('请先导入并加载 LeKiwi v1 机器人');
const m = session.model,
b = new ModelBindings(m);
const base = b.resolve(b.bodies, P.baseBody);
const free = Array.from({ length: m.njnt }, (_, i) => i).find(
(i) => Number(m.jnt_type[i]) === 0 && Number(m.jnt_bodyid[i]) === base,
);
if (free === undefined) throw new Error('底盘自由关节不匹配');
const dof = Number(m.jnt_dofadr[free]);
const arm = P.arm.map((s) => b.scalarActuator(`${s.joint}_servo`, s.joint, 'position'));
const wheels = P.wheels.map((name) => b.scalarActuator(`${name}_servo`, name, 'velocity'));
const armTargets = arm.map((j) => Number(session.data.qpos[j.qposAddress]));
const robotBodies = new Set<number>([base]);
for (let i = base + 1; i < m.nbody; i++)
if (robotBodies.has(Number(m.body_parentid[i]))) robotBodies.add(i);
const holdArm = () => arm.forEach((j, i) => write(j.actuatorId, armTargets[i]));
return {
sample() {
const d = session.data,
pos = d.xpos.subarray(base * 3, base * 3 + 3);
const [w, x, y, z] = Array.from(d.xquat.subarray(base * 4, base * 4 + 4), Number);
let collision = false;
const contacts = d.contact;
try {
for (let i = 0; i < d.ncon; i++) {
const ct = contacts.get(i)!;
try {
const a = robotBodies.has(Number(m.geom_bodyid[ct.geom1])),
c = robotBodies.has(Number(m.geom_bodyid[ct.geom2]));
// Flat floor planes are the only permitted outside contacts. This is NOT a navigation system.
if (a !== c && Number(m.geom_type[a ? ct.geom2 : ct.geom1]) !== 0) collision = true;
} finally {
ct.delete();
}
}
} finally {
contacts.delete();
}
return {
time: Number(d.time),
x: Number(pos[0]),
y: Number(pos[1]),
z: Number(pos[2]),
yaw: Math.atan2(2 * (w * z + x * y), 1 - 2 * (y * y + z * z)),
tilt: Math.acos(Math.max(-1, Math.min(1, 1 - 2 * (x * x + y * y)))),
speed: Math.hypot(Number(d.qvel[dof]), Number(d.qvel[dof + 1])),
angularSpeed: Math.abs(Number(d.qvel[dof + 5])),
collision,
};
},
write(vx, vy, wz) {
holdArm();
const values = bodyToWheels(vx, vy, wz);
wheels.forEach((j, i) => write(j.actuatorId, values[i]));
},
hold() {
holdArm();
wheels.forEach((j) => write(j.actuatorId, 0));
},
};
}
web_platform/src/mobile/agent/PhysicalEvidence.ts
+31 -4
View File
@@ -18,6 +18,8 @@ export interface PhysicalSample {
fingerForces: [number, number];
supported: boolean;
onGoalSupport: boolean;
forbiddenContact?: boolean;
fingerSupportForce?: number;
}
/** Geom-to-finger aggregation: many hull contacts on ONE finger never count as two. */
@@ -34,7 +36,11 @@ export class PhysicalSampler {
private readonly goalSupport: number;
private readonly site: number;
private disposed = false;
constructor(private readonly session: SimulationSession) {
constructor(
private readonly session: SimulationSession,
goalSupportName = '__agent_support_1',
private readonly checkCollisions = false,
) {
const { model, module } = session,
b = new ModelBindings(model);
this.arm = R.armJoints.map((joint) => b.scalarJoint(joint.name));
@@ -49,7 +55,7 @@ export class PhysicalSampler {
this.fingerBodies = ['Wrist_Roll_08c-v1', 'Moving_Jaw_08d-v1'].map((name) =>
Number(model.body_weldid[b.resolve(b.bodies, name)]),
);
this.goalSupport = module.mj_name2id(model, 5, '__agent_support_1');
this.goalSupport = module.mj_name2id(model, 5, goalSupportName);
this.site = module.mj_name2id(model, 6, P.tcpSite);
if (this.site < 0 || this.goalSupport < 0) throw new Error('缺少抓放场景');
this.force = new module.DoubleBuffer(6);
@@ -59,7 +65,9 @@ export class PhysicalSampler {
const { model: m, data: d, module } = this.session;
const forces: [number, number] = [0, 0];
let supported = false,
onGoalSupport = false;
onGoalSupport = false,
forbiddenContact = false,
fingerSupportForce = 0;
const contacts = d.contact;
try {
for (let i = 0; i < d.ncon; i++) {
@@ -67,6 +75,23 @@ export class PhysicalSampler {
try {
const a = Number(m.geom_bodyid[contact.geom1]),
b = Number(m.geom_bodyid[contact.geom2]);
if (this.checkCollisions) {
// The task scene has only world supports, robot bodies and the free block.
const robotA = a !== 0 && a !== this.objectBody;
const robotB = b !== 0 && b !== this.objectBody;
if (robotA !== robotB) {
const outside = robotA ? contact.geom2 : contact.geom1;
const body = robotA ? a : b;
const objectContact = (robotA ? b : a) === this.objectBody;
const finger = this.fingerBodies.includes(Number(m.body_weldid[body]));
if (Number(m.geom_type[outside]) !== 0 && !(objectContact && finger)) {
module.mj_contactForce(m, d, i, this.force);
const force = Number(this.force.GetView()[0]);
if (finger && !objectContact) fingerSupportForce += Math.max(0, force);
else if (force > 0.2) forbiddenContact = true;
}
}
}
if (a !== this.objectBody && b !== this.objectBody) continue;
const other = a === this.objectBody ? b : a;
const geom = a === this.objectBody ? contact.geom2 : contact.geom1;
@@ -110,6 +135,7 @@ export class PhysicalSampler {
fingerForces: forces,
supported,
onGoalSupport,
...(this.checkCollisions ? { forbiddenContact, fingerSupportForce } : {}),
};
if (
[
@@ -153,6 +179,7 @@ export class PickPlaceEvaluator {
constructor(
readonly initialObject: readonly number[],
readonly goal: readonly number[],
readonly transportMin = P.transportMin,
) {}
update(sample: PhysicalSample, dt: number): void {
if (!Number.isFinite(dt) || dt <= 0 || dt > P.controlDt) throw new Error('非法物理采样步长');
@@ -194,7 +221,7 @@ export class PickPlaceEvaluator {
sample.opening >= 0.65 && sample.fingerForces.every((force) => force < P.fingerForceMin);
const placed =
this.hasGrasped &&
this.transported >= P.transportMin &&
this.transported >= this.transportMin &&
released &&
sample.onGoalSupport &&
Math.hypot(...sample.object.map((v, i) => v - this.goal[i])) < P.placementTolerance &&
web_platform/src/mobile/agent/PickPlaceExecutor.ts
+19
View File
@@ -0,0 +1,19 @@
import type { SimulationSession } from '../../simulation/SimulationSession';
import { DeterministicBaseline, type PickPlaceExecution } from './DeterministicBaseline';
import { planPickPlace } from './TaskPlanner';
import type { ResolvedTarget } from './LanguageScene';
/** Version-two adapter over the same rate-limited servos and physical evidence. */
export class PickPlaceExecutor extends DeterministicBaseline {
readonly taskConfig: PickPlaceExecution;
constructor(
session: SimulationSession,
write: (id: number, value: number) => void,
target: ResolvedTarget,
gated = true,
) {
const execution = planPickPlace(session, target);
super(session, 0, write, gated, undefined, execution);
this.taskConfig = execution;
}
}
web_platform/src/mobile/agent/PickPlaceScene.ts
+1
View File
@@ -6,6 +6,7 @@ export const PICK_PLACE = preset;
export const PICK_PLACE_ROBOT = MOBILE_ROBOTS.find((robot) => robot.id === preset.robotId)!;
export const TASK_GOAL_LIMITS = { min: [0.237, 0.565, 0.128], max: [0.277, 0.665, 0.128] } as const;
export interface PickPlaceTaskConfig {
preset?: 'language-v2';
seed: number;
goal?: readonly number[];
}
web_platform/src/mobile/agent/RobotCommand.ts
+61
View File
@@ -0,0 +1,61 @@
import { LANGUAGE_VERSION, resolveTarget, type TargetId } from './LanguageScene';
import { validateLanguageData } from './protocol';
export interface SceneContext {
version: typeof LANGUAGE_VERSION;
sceneId: 'lekiwi-language-scene-v2';
source: 'mujoco-ground-truth';
units: 'SI';
frame: 'world-z-up';
object: number[];
base: number[];
yaw: number;
}
export interface RobotCommand {
version?: typeof LANGUAGE_VERSION;
objectId?: 'block' | 'none';
targetId?: TargetId | 'none';
position?: number[];
supportId?: 'source' | 'table' | 'none';
action: 'move' | 'turn' | 'pick_place' | 'stop' | 'clarify';
value: number;
summary: string;
}
export function validateRobotCommand(input: unknown): RobotCommand {
if (!input || typeof input !== 'object' || Array.isArray(input)) throw new Error('指令响应无效');
const c = input as RobotCommand;
if (c.version === LANGUAGE_VERSION) {
const v = validateLanguageData<RobotCommand>('Command', input);
if (v.action === 'pick_place') {
if (v.objectId !== 'block' || v.value !== 0 || v.targetId === 'none')
throw new Error('非法抓放任务');
const resolved = resolveTarget(v.targetId as TargetId, v.position!);
if (resolved.supportId !== v.supportId) throw new Error('目标支撑面不匹配');
} else {
if (
v.objectId !== 'none' ||
v.targetId !== 'none' ||
v.supportId !== 'none' ||
v.position!.length
)
throw new Error('指令包含无关目标');
validateRobotCommand({ action: v.action, value: v.value, summary: v.summary });
}
return v;
}
if (
Object.keys(c).sort().join(',') !== 'action,summary,value' ||
!['move', 'turn', 'pick_place', 'stop', 'clarify'].includes(c.action) ||
typeof c.value !== 'number' ||
!Number.isFinite(c.value) ||
typeof c.summary !== 'string' ||
!c.summary.length ||
c.summary.length > 200 ||
(c.action === 'move'
? Math.abs(c.value) < 0.01 || Math.abs(c.value) > 1
: c.action === 'turn'
? Math.abs(c.value) < 1 || Math.abs(c.value) > 180
: c.value !== 0)
)
throw new Error('指令超出安全范围，未执行');
return { action: c.action, value: c.value, summary: c.summary };
}
web_platform/src/mobile/agent/RobotLanguagePanel.tsx
+341
View File
@@ -0,0 +1,341 @@
import { useCallback, useEffect, useRef, useState } from 'react';
import { Button } from '../../components/ui';
import type { SimulationSession } from '../../simulation/SimulationSession';
import type { AgentWorkbenchBridge } from './AgentTaskPanel';
import { DecisionClient } from './DecisionClient';
import { AgentTaskController } from './AgentTaskController';
import { PickPlaceExecutor } from './PickPlaceExecutor';
import { PhysicalSampler } from './PhysicalEvidence';
import { MotionDriver, motionIO } from './MotionDriver';
import { LANGUAGE_VERSION, resolveTarget, type TargetId } from './LanguageScene';
import type { SceneContext } from './RobotCommand';
import type { RequestStamp } from './RequestGate';
type Driver = MotionDriver | AgentTaskController;
const labels: Record<string, string> = {
idle: '准备',
planning: '理解与规划中',
deciding: '安全检查中',
executing: '执行中',
running: '执行中',
paused: '已暂停',
succeeded: '已完成',
failed: '安全停止',
cancelled: '已停止',
};
const errors: Record<string, string> = {
shared_budget_exceeded: '公共服务额度已用完，请稍后再试。',
session_hourly_budget_exceeded: '本会话调用额度已用完。',
ip_rate_limit: '操作过于频繁，请稍后再试。',
server_busy: '服务繁忙，请稍后再试。',
request_timeout: '理解指令超时，未执行。',
command_out_of_range: '动作范围：距离 0.01–1 米，角度 1–180 度。',
command_mismatch: '模型理解的数字、方向或单位与指令不一致，未执行。',
command_not_approved: '指令未通过监督检查，请明确对象和目的地。',
target_unsupported: '目标没有安全支撑面或太靠近边缘。',
target_height_mismatch: 'Z 是方块中心高度，应为 0.128 米。',
target_support_mismatch: '目标与支撑面不匹配。',
session_expired: '会话过期，请重新执行。',
};
/** One shared local/hosted UI. Configuration and credentials stay on the server. */
export function RobotLanguagePanel({
bridge,
loading,
}: {
bridge: AgentWorkbenchBridge;
loading: boolean;
}) {
const live = useRef(bridge);
useEffect(() => {
live.current = bridge;
}, [bridge]);
const client = useRef<DecisionClient | undefined>(undefined);
const driver = useRef<Driver | undefined>(undefined);
const own = useRef<SimulationSession | undefined>(undefined);
const pending = useRef<AbortController | undefined>(undefined);
const epoch = useRef(0),
alive = useRef(true);
const reported = useRef<Driver | undefined>(undefined);
const receipt = useRef<unknown>(undefined);
const [instruction, setInstruction] = useState('把方块搬到 B 区');
const [busy, setBusy] = useState(false);
const [message, setMessage] = useState('载入预设后输入指令。仅仿真，不支持通用避障。');
const [snapshot, setSnapshot] = useState<ReturnType<Driver['snapshot']>>();
const running =
snapshot && !['succeeded', 'failed', 'cancelled', 'paused'].includes(snapshot.state);
const report = useCallback((current: Driver) => {
if (reported.current === current || !current.terminal) return;
reported.current = current;
// Read-only bounded result telemetry; no credentials or simulation write handles.
window.dispatchEvent(
new CustomEvent('robot-task-result', {
detail: {
...current.snapshot(),
commandReceipt: receipt.current,
receipts:
client.current?.receipts.filter(
(r) =>
current instanceof AgentTaskController &&
r.stamp.runId === current.snapshot().runId,
) ?? [],
},
}),
);
}, []);
const stop = useCallback(
(reason = '用户停止') => {
epoch.current++;
pending.current?.abort();
driver.current?.cancel(reason);
const session = own.current;
if (session && session === live.current.getSession() && session.controlOwner === 'agent')
session.stopAgent(reason);
if (alive.current) {
setBusy(false);
setMessage(reason);
if (driver.current) {
setSnapshot(driver.current.snapshot());
report(driver.current);
}
live.current.publish();
}
},
[report],
);
useEffect(() => {
alive.current = true;
const timer = setInterval(() => {
const current = driver.current,
session = own.current;
if (!current) return;
setSnapshot(current.snapshot());
if (current.terminal) {
report(current);
if (session === live.current.getSession() && session?.controlOwner === 'agent') {
session.stopAgent('任务结束');
live.current.publish();
}
}
}, 100);
const hide = () => {
if (document.hidden) stop('页面已隐藏，指令取消');
};
const leave = () => stop('页面已离开，指令取消');
document.addEventListener('visibilitychange', hide);
window.addEventListener('pagehide', leave);
return () => {
alive.current = false;
stop('语言控制面板已卸载');
driver.current?.dispose();
clearInterval(timer);
document.removeEventListener('visibilitychange', hide);
window.removeEventListener('pagehide', leave);
};
}, [report, stop]);
async function load() {
if (busy || loading) return;
stop('正在载入预设');
driver.current?.dispose();
driver.current = undefined;
setSnapshot(undefined);
const generation = epoch.current;
const abort = new AbortController();
pending.current = abort;
setBusy(true);
try {
const session = await live.current.loadScene(
{ seed: 0, preset: 'language-v2' },
abort.signal,
);
if (generation !== epoch.current || !alive.current || abort.signal.aborted) return;
own.current = session;
setMessage('单方块搬运预设已载入并暂停。A 区 Y=0.45，B 区 Y=0.75。');
live.current.publish();
} catch (e) {
if (generation === epoch.current && alive.current)
setMessage(e instanceof Error ? e.message : '预设加载失败');
} finally {
if (generation === epoch.current && alive.current) setBusy(false);
}
}
async function execute() {
if (busy || loading || !instruction.trim()) return;
const stopping = /^(停止|停下|停车|stop)[。！!\s]*$/i.test(instruction.trim());
stop(stopping ? '用户停止' : '准备新指令');
if (stopping) return;
const generation = ++epoch.current;
const abort = new AbortController();
pending.current = abort;
setBusy(true);
setSnapshot(undefined);
receipt.current = undefined;
driver.current?.dispose();
driver.current = undefined;
setMessage('正在理解指令，机器人保持暂停…');
try {
const session = live.current.getSession();
if (!session) throw new Error('请先导入 LeKiwi 或载入预设场景');
motionIO(session, () => undefined).sample();
if (session.controlOwner !== 'manual') throw new Error('请先停止当前脚本或其他控制器');
session.setPaused(true);
own.current = session;
live.current.publish();
const time = Number(session.data.time),
qpos = Array.from(session.data.qpos),
ctrl = Array.from(session.data.ctrl);
let sceneContext: SceneContext | undefined;
if (
live.current.isTaskScene() &&
session.module.mj_name2id(session.model, 6, '__language_A') >= 0
) {
const sampler = new PhysicalSampler(session);
try {
const s = sampler.sample();
sceneContext = {
version: LANGUAGE_VERSION,
sceneId: 'lekiwi-language-scene-v2',
source: 'mujoco-ground-truth',
units: 'SI',
frame: 'world-z-up',
object: s.object,
base: s.base,
yaw: s.yaw,
};
} finally {
sampler.dispose();
}
}
if (!sceneContext && /方块|抓|搬|放|pick|place/i.test(instruction))
throw new Error('请先点击“载入预设场景”，不会自动重置当前场景执行演示');
const signal = AbortSignal.any([abort.signal, AbortSignal.timeout(65000)]);
const c = client.current ?? (await DecisionClient.website(signal));
client.current = c;
const status = (await c.call('/status', 'GET', undefined, signal)) as { ready: boolean };
if (!status.ready)
throw new Error('模型服务未配置，请按 decision_server/README.md 启动同源模型服务');
const stamp: RequestStamp = {
runId: crypto.randomUUID(),
requestId: crypto.randomUUID(),
sceneRevision: generation,
sequence: 0,
planRevision: 0,
};
const result = await c.command(instruction, stamp, signal, sceneContext);
if (generation !== epoch.current || !alive.current || signal.aborted) return;
if (
document.hidden ||
session !== live.current.getSession() ||
!session.paused ||
session.controlOwner !== 'manual' ||
Number(session.data.time) !== time ||
qpos.some((v, i) => session.data.qpos[i] !== v) ||
ctrl.some((v, i) => session.data.ctrl[i] !== v)
)
throw new Error('等待期间场景或控制权已变化，旧指令未执行');
const command = result.command;
receipt.current = result;
if (command.action === 'stop' || command.action === 'clarify') {
setMessage(command.summary);
return;
}
if (command.action === 'pick_place') {
if (
command.version !== LANGUAGE_VERSION ||
!sceneContext ||
!command.targetId ||
command.targetId === 'none'
)
throw new Error('抓放任务缺少明确目标，请载入预设并指定 A/B 区或坐标');
const target = resolveTarget(command.targetId as TargetId, command.position);
session.startAgent((write) => {
const skills = new PickPlaceExecutor(session, write, target);
try {
return (driver.current = new AgentTaskController(skills, c, instruction, {
version: LANGUAGE_VERSION,
runId: stamp.runId,
sceneRevision: stamp.sceneRevision,
initialLlmCalls: 1,
}));
} catch (e) {
skills.dispose();
throw e;
}
});
} else
session.startAgent(
(write) => (driver.current = new MotionDriver(motionIO(session, write), command)),
);
session.setPaused(false);
setSnapshot(driver.current!.snapshot());
setMessage(command.summary);
live.current.publish();
} catch (e) {
if (generation === epoch.current && alive.current) {
const code =
e instanceof Error ? (e.name === 'TimeoutError' ? 'request_timeout' : e.message) : '';
if (code === 'session_expired' || code === 'configuration_changed')
client.current = undefined;
setMessage(
errors[code] ??
(/^[a-z0-9_]+$/.test(code) ? '模型服务不可用，未执行指令。' : code || '指令未执行'),
);
}
} finally {
if (generation === epoch.current && alive.current) setBusy(false);
}
}
return (
<div className="space-y-3 text-xs" aria-label="机器人语言控制">
<label className="block">
预设场景{' '}
<select
aria-label="预设场景"
disabled={busy || loading || Boolean(running)}
className="rounded border border-border bg-surface p-1"
>
<option>单方块搬运 · A/B 区</option>
</select>
</label>
<Button disabled={busy || loading || Boolean(running)} onClick={() => void load()}>
载入预设场景
</Button>
<label className="block">
机器人指令
<textarea
className="w-full rounded border border-border bg-surface p-2"
rows={3}
maxLength={500}
value={instruction}
disabled={busy || loading || Boolean(running)}
onChange={(e) => {
stop('指令已修改');
setInstruction(e.target.value);
}}
/>
</label>
<p>
例如：把方块搬到 B 区；搬到 (0.25,0.50) 米。世界坐标，目标台 X 0.237–0.277、Y
0.365–0.865，Z（物体中心）0.128 米。
</p>
<div className="flex gap-2">
<Button
disabled={busy || loading || Boolean(running) || !instruction.trim()}
onClick={() => void execute()}
>
执行指令
</Button>
<Button onClick={() => stop()}>停止</Button>
</div>
<p role="status">{message}</p>
{snapshot && (
<div data-testid="robot-command-state">
<p>状态：{labels[snapshot.state] ?? snapshot.state}</p>
{snapshot.error && <p role="alert">{snapshot.error}</p>}
</div>
)}
</div>
);
}
web_platform/src/mobile/agent/TaskPlanner.ts
+74
View File
@@ -0,0 +1,74 @@
import type { SimulationSession } from '../../simulation/SimulationSession';
import { PhysicalSampler } from './PhysicalEvidence';
import { LeKiwiIK } from './LeKiwiIK';
import { LANGUAGE_SCENE as C, resolveTarget, type ResolvedTarget } from './LanguageScene';
import type { PickPlaceExecution } from './DeterministicBaseline';
import { PICK_PLACE as P } from './PickPlaceScene';
/** Bounded, obstacle-free docking corridor, not a general navigation planner. */
export function planPickPlace(
session: SimulationSession,
target: ResolvedTarget,
): PickPlaceExecution {
if (session.module.mj_name2id(session.model, 6, '__language_A') < 0)
throw new Error('请先载入语言控制预设场景');
const sampler = new PhysicalSampler(session, '__agent_support_1', true);
let ik: LeKiwiIK | undefined;
try {
ik = new LeKiwiIK(session);
const sample = sampler.sample();
const resolved = resolveTarget('coordinates', target.position);
if (resolved.supportId !== target.supportId) throw new Error('目标支撑面不匹配');
resolveTarget('coordinates', sample.object);
if (
sample.objectSpeed > 0.05 ||
sample.baseSpeed > 0.01 ||
sample.fingerForces.some((f) => f >= P.fingerForceMin)
)
throw new Error('请先停稳；物体必须静止在支撑面上且未被夹持');
if (sample.forbiddenContact || Math.abs(sample.yaw) > 0.15 || sample.tilt > 0.15)
throw new Error('当前姿态不在预设安全停靠走廊，请停止并载入预设');
const source = [...sample.object];
const goal = [...target.position];
const sourceDock = source.slice(0, 2).map((v, i) => v - C.dockOffset[i]);
const goalDock = goal.slice(0, 2).map((v, i) => v - C.dockOffset[i]);
// The corridor is convex and disjoint from the tables with a 0.15 m base envelope.
for (const p of [sample.base, sourceDock, goalDock]) {
if (p.slice(0, 2).some((v, i) => v < C.baseBounds.min[i] || v > C.baseBounds.max[i]))
throw new Error('底盘路径超出预设净空走廊');
for (const s of C.supports)
if (p[0] + 0.15 + 0.01 >= s.center[0] - s.halfSize[0])
throw new Error('底盘与台面净空不足');
}
const transfer = Math.hypot(...goal.slice(0, 2).map((v, i) => v - source[i]));
if (transfer < C.minTransfer || transfer > C.maxTransfer)
throw new Error('搬运距离必须在 0.08–1 米内，不能原地冒充完成');
const approach = Math.hypot(...sourceDock.map((v, i) => v - sample.base[i]));
for (const [point, dock] of [
[source, sourceDock],
[goal, goalDock],
]) {
for (const z of [point[2], P.pregraspHeight, P.carryHeight, P.retreatHeight]) {
if (ik.solve([point[0], point[1], z], { x: dock[0], y: dock[1], yaw: 0 }).residual > 0.004)
throw new Error('抓取或放置位姿不可达');
}
}
// The held block bottom stays above every support by > 0.06 m.
if (C.supports.some((s) => P.carryHeight - C.objectHalfSize - s.height < 0.04))
throw new Error('持物搬运净空不足');
return {
source,
goal,
sourceDock,
goalDock,
supportGeom: C.supports.find((s) => s.id === target.supportId)!.geom,
transportMin: Math.max(0.03, transfer - 0.025),
approachTimeout: approach / P.baseSpeed + 6,
carryTimeout: transfer / P.baseSpeed + 6,
episodeTimeout: Math.min(C.episodeTimeout, (approach + transfer) / P.baseSpeed + 65),
};
} finally {
ik?.dispose();
sampler.dispose();
}
}
web_platform/src/mobile/agent/TaskSupervisor.test.ts
+36
View File
@@ -0,0 +1,36 @@
import { describe, it, expect } from 'vitest';
import { TaskSupervisor } from './TaskSupervisor';
import { LANGUAGE_VERSION } from './LanguageScene';
const decision = {
version: LANGUAGE_VERSION,
choice: 'carry',
grasp: 'secure',
diagnosis: 'none',
recovery: 'continue',
noul: 'allow',
score: 'excellent',
reason: 'verified_progress',
};
describe('Choice / Noul / Score', () => {
it('allows only agreement with physical evidence; grades are not probability', () => {
const s = new TaskSupervisor();
expect(s.inspect(decision, ['carry', 'stop'], LANGUAGE_VERSION, true).choice).toBe('carry');
expect(s.qualityScore).toBe(100);
expect(() => s.inspect(decision, ['carry'], LANGUAGE_VERSION, false)).toThrow('物理证据');
});
it.each(['deny', 'uncertain'])('Noul %s stops even with excellent score', (noul) => {
expect(() =>
new TaskSupervisor().inspect({ ...decision, noul }, ['carry'], LANGUAGE_VERSION, true),
).toThrow('Noul');
});
it('rejects illegal candidate, missing fields and arbitrary scores', () => {
const s = new TaskSupervisor();
expect(() => s.inspect(decision, ['open'], LANGUAGE_VERSION, true)).toThrow();
expect(() =>
s.inspect({ ...decision, score: 100 }, ['carry'], LANGUAGE_VERSION, true),
).toThrow();
expect(() =>
s.inspect({ ...decision, noul: undefined }, ['carry'], LANGUAGE_VERSION, true),
).toThrow();
});
});
web_platform/src/mobile/agent/TaskSupervisor.ts
+32
View File
@@ -0,0 +1,32 @@
import { LANGUAGE_VERSION } from './LanguageScene';
import { validateJevDecision, type AgentVersion, type JevDecision, type Skill } from './protocol';
export const QUALITY_POINTS = { poor: 0, partial: 25, good: 75, excellent: 100 } as const;
/** Supervisory advice can veto but cannot grant physical authority. */
export class TaskSupervisor {
last?: JevDecision;
get qualityScore(): number | undefined {
const score = this.last?.score;
return score && score !== 'unavailable' ? QUALITY_POINTS[score] : undefined;
}
inspect(
value: unknown,
candidates: readonly (Skill | 'stop')[],
version: AgentVersion,
secure: boolean,
): JevDecision {
const decision = validateJevDecision(value, candidates, version);
this.last = decision;
if (version === LANGUAGE_VERSION && decision.noul !== 'allow')
throw new Error(`Noul 拦截：${decision.reason ?? 'insufficient_evidence'}`);
if (decision.choice === 'stop' || decision.recovery === 'stop')
throw new Error('Jev 请求安全停止');
if (decision.grasp === 'secure' && !secure) throw new Error('Jev 抓持判断与本地物理证据矛盾');
if (
['carry', 'stop-base'].includes(decision.choice) &&
(decision.grasp !== 'secure' || !secure)
)
throw new Error('搬运前未取得一致抓持证据');
return decision;
}
}
web_platform/src/mobile/agent/WebsiteModelSettings.test.tsx
+38 -2
View File
@@ -14,7 +14,9 @@ const status = {
jev: { hasKey: false },
},
};
function fixture() {
function fixture(
login: { verificationUrl: string; userCode: string } | Error = new Error('codex_rpc_timeout'),
) {
const call = vi.fn(async (path: string) => {
if (path === '/models')
return {
@@ -25,7 +27,10 @@ function fixture() {
],
};
if (path === '/configuration') return { ...status, ready: true };
if (path === '/codex/login') throw new Error('codex_rpc_timeout');
if (path === '/codex/login') {
if (login instanceof Error) throw login;
return login;
}
return status;
});
const client = { call, receipts: [], mode: 'api' } as unknown as DecisionClient;
@@ -77,6 +82,37 @@ it('换提供方清空密钥，保留订阅入口并明确网络不可用', asyn
await screen.findByText(/官方订阅服务连接超时/);
expect(call.mock.calls.filter(([path]) => path === '/configuration')).toHaveLength(0);
});
it('订阅失败在按钮旁说明未取得链接，不假装打开页面或启动付费调用', async () => {
const { call } = fixture(new Error('codex_rpc_failed'));
render(<WebsiteModelSettings onClient={vi.fn()} onInvalidate={vi.fn()} />);
await waitFor(() => expect(screen.getByRole('button', { name: '保存设置' })).not.toBeDisabled());
fireEvent.click(screen.getByRole('button', { name: 'ChatGPT 订阅' }));
fireEvent.click(screen.getByRole('button', { name: '登录 ChatGPT 订阅' }));
await waitFor(() =>
expect(screen.getByRole('status', { name: '订阅操作状态' })).toHaveTextContent(
'当前没有验证链接可打开',
),
);
expect(screen.queryByRole('link', { name: '打开 OpenAI 官方验证页' })).not.toBeInTheDocument();
expect(
call.mock.calls.some(([path]) =>
['/plan', '/test', '/decide', '/configuration'].includes(path),
),
).toBe(false);
});
it('设备码成功后展示官方链接，明确需要用户点击而非自动弹窗', async () => {
fixture({ verificationUrl: 'https://auth.openai.com/codex/device', userCode: 'TEST-CODE' });
render(<WebsiteModelSettings onClient={vi.fn()} onInvalidate={vi.fn()} />);
await waitFor(() => expect(screen.getByRole('button', { name: '保存设置' })).not.toBeDisabled());
fireEvent.click(screen.getByRole('button', { name: 'ChatGPT 订阅' }));
fireEvent.click(screen.getByRole('button', { name: '登录 ChatGPT 订阅' }));
expect(await screen.findByRole('link', { name: '打开 OpenAI 官方验证页' })).toHaveAttribute(
'href',
'https://auth.openai.com/codex/device',
);
expect(screen.getByText('TEST-CODE')).toBeVisible();
expect(screen.getByText(/不会自动弹窗/)).toBeVisible();
});
it('同源客户端自动携带 Cookie/CSRF，但不给原本机模式放开任意远端', async () => {
const fetcher = vi.fn<typeof fetch>(async (_url, opts) => {
if (String(_url).endsWith('/session'))
web_platform/src/mobile/agent/WebsiteModelSettings.tsx
+18 -5
View File
@@ -28,8 +28,9 @@ const errors: Record<string, string> = {
ip_rate_limit: '请求次数已达上限，请稍后再试。',
subscription_capacity: '订阅登录名额暂满，请稍后再试；不会自动切换付费 API。',
codex_chatgpt_login_required: '请先完成官方订阅登录。',
codex_rpc_failed: '当前订阅服务不可用，可能是网络或账号限制；不会切换付费 API。',
codex_rpc_timeout: '官方订阅服务连接超时；API 模式仍可手动使用。',
codex_rpc_failed:
'服务器未能获取官方登录码，当前没有验证链接可打开。可能是服务器网络或账号限制；不会切换付费 API。',
codex_rpc_timeout: '官方订阅服务连接超时，未获取登录码或验证链接；API 模式仍可手动使用。',
codex_not_installed: '当前服务器未启用订阅服务。',
model_unavailable: '当前模型不可用，请重新选择；不会自动替换。',
csrf_required: '会话验证失效，请重新打开模型设置。',
@@ -238,6 +239,15 @@ export function WebsiteModelSettings({ onClient, onInvalidate }: Props) {
}
});
}
const statusNotice = (
<p
role="status"
aria-label={mode === 'codex' ? '订阅操作状态' : '模型设置状态'}
className="break-words"
>
{busy ? '正在连接…' : notice}
</p>
);
const matching = models.filter((m) =>
`${m.provider} ${m.id} ${m.name}`.toLowerCase().includes(search.toLowerCase()),
);
@@ -328,12 +338,17 @@ export function WebsiteModelSettings({ onClient, onInvalidate }: Props) {
) : (
<div className="space-y-2">
<p>官方订阅登录（实验性）。当前服务器可能无法连接官方服务；不会自动转付费 API。</p>
<p>
点击登录先由服务器申请设备码；成功后请点击下方的官方验证链接，不会自动弹窗。仅浏览器能够打开
ChatGPT，并不代表服务器也能连接。
</p>
<div className="flex flex-wrap gap-2">
<Button onClick={() => void account('login')}>登录 ChatGPT 订阅</Button>
<Button onClick={() => void account('status')}>刷新登录状态</Button>
<Button onClick={() => void account('cancel')}>取消登录</Button>
<Button onClick={() => void account('logout')}>退出订阅</Button>
</div>
{statusNotice}
{device && (
<div className="space-y-1">
<a
@@ -427,9 +442,7 @@ export function WebsiteModelSettings({ onClient, onInvalidate }: Props) {
</div>
</details>
</fieldset>
<p role="status" className="break-words">
{busy ? '正在连接…' : notice}
</p>
{mode === 'api' && statusNotice}
{dirty && <p className="text-warning">设置未保存，真实任务暂不可用。</p>}
</div>
);
web_platform/src/mobile/agent/WebsiteRobotPanel.tsx
+2
View File
@@ -0,0 +1,2 @@
/** Compatibility export: local and hosted builds use the same product panel. */
export { RobotLanguagePanel as WebsiteRobotPanel } from './RobotLanguagePanel';
web_platform/src/mobile/agent/protocol.ts
+74 -37
View File
@@ -1,11 +1,16 @@
import schema from '../../../../contracts/lekiwi-agent-v1.schema.json';
import languageSchema from '../../../../contracts/lekiwi-language-task-v2.schema.json';
import { LANGUAGE_VERSION } from './LanguageScene';
import type { BaselinePhase } from './DeterministicBaseline';
import type { PhysicalSample, PickPlaceEvaluator } from './PhysicalEvidence';
import type { RequestStamp } from './RequestGate';
export const AGENT_VERSION = 'lekiwi-agent-v1' as const;
export type AgentVersion = typeof AGENT_VERSION | typeof LANGUAGE_VERSION;
export type Skill = BaselinePhase;
export const PRECONDITIONS = {
stow: 'scene-ready',
approach: 'arm-safe',
open: 'scene-ready',
pregrasp: 'base-stopped',
descend: 'tcp-above',
@@ -18,23 +23,27 @@ export const PRECONDITIONS = {
retreat: 'released',
settle: 'retreat',
} as const satisfies Record<Skill, string>;
export const SKILLS = Object.keys(PRECONDITIONS) as Skill[];
export const LANGUAGE_SKILLS = Object.keys(PRECONDITIONS) as Skill[];
export const SKILLS: Skill[] = LANGUAGE_SKILLS.filter((s) => s !== 'stow' && s !== 'approach');
export interface Plan {
version: typeof AGENT_VERSION;
version: AgentVersion;
objectId: 'block';
goalId: 'placement';
summary: string;
steps: { skill: Skill; precondition: string; onFailure: 'retry' | 'replan' | 'stop' }[];
}
export interface JevDecision {
version: typeof AGENT_VERSION;
version: AgentVersion;
noul?: 'allow' | 'deny' | 'uncertain';
score?: 'poor' | 'partial' | 'good' | 'excellent' | 'unavailable';
reason?: 'none' | 'unsafe' | 'insufficient_evidence' | 'tracking_error' | 'verified_progress';
choice: Skill | 'stop';
grasp: 'secure' | 'empty' | 'slipping' | 'uncertain';
diagnosis: 'none' | 'empty' | 'slipping' | 'misaligned' | 'unreachable' | 'stalled' | 'uncertain';
recovery: 'continue' | 'retry' | 'replan' | 'stop';
}
export interface Observation {
version: typeof AGENT_VERSION;
version: AgentVersion;
stamp: RequestStamp;
source: 'mujoco-ground-truth';
units: 'SI';
@@ -57,7 +66,7 @@ export interface Observation {
safety: string[];
}
export interface SkillResult {
version: typeof AGENT_VERSION;
version: AgentVersion;
skill: Skill;
status: 'running' | 'completed' | 'failed';
code: string;
@@ -85,14 +94,14 @@ type ContractName = keyof typeof schema.$defs;
const definitions = schema.$defs as unknown as Record<ContractName, Node>;
/** The intentionally small JSON Schema subset used by this version; fail closed
* on data, not an eval/template engine. Python validates this same schema. */
function check(node: Node, value: unknown, path: string): void {
function check(node: Node, value: unknown, path: string, defs = definitions): void {
const fail = () => {
throw new Error(`${path}: 智能任务契约不匹配`);
};
if (node.$ref) {
const name = node.$ref.replace('#/$defs/', '') as ContractName;
if (!Object.hasOwn(definitions, name)) fail();
return check(definitions[name], value, path);
if (!Object.hasOwn(defs, name)) fail();
return check(defs[name], value, path, defs);
}
if (Object.hasOwn(node, 'const') && value !== node.const) fail();
if (node.enum && !node.enum.includes(value)) fail();
@@ -120,7 +129,7 @@ function check(node: Node, value: unknown, path: string): void {
if (values.length < (node.minItems ?? 0) || values.length > (node.maxItems ?? Infinity)) fail();
if (node.uniqueItems && new Set(values.map((v) => JSON.stringify(v))).size !== values.length)
fail();
if (node.items) values.forEach((v, i) => check(node.items!, v, `${path}[${i}]`));
if (node.items) values.forEach((v, i) => check(node.items!, v, `${path}[${i}]`, defs));
} else if (node.type === 'object') {
if (value === null || typeof value !== 'object' || Array.isArray(value)) fail();
const obj = value as Record<string, unknown>;
@@ -128,17 +137,39 @@ function check(node: Node, value: unknown, path: string): void {
for (const key of Object.keys(obj)) {
if (!Object.hasOwn(node.properties ?? {}, key)) {
if (node.additionalProperties === false) fail();
} else check(node.properties![key], obj[key], `${path}.${key}`);
} else check(node.properties![key], obj[key], `${path}.${key}`, defs);
}
}
}
export function validateContract<T>(name: ContractName, value: unknown): T {
export function validateContract<T>(
name: ContractName,
value: unknown,
version: AgentVersion = AGENT_VERSION,
): T {
if ((JSON.stringify(value)?.length ?? 0) > 65536) throw new Error('智能任务消息超长');
check(definitions[name], value, name);
const defs =
version === LANGUAGE_VERSION
? (languageSchema.$defs as unknown as typeof definitions)
: definitions;
check(defs[name], value, name, defs);
return structuredClone(value) as T;
}
export function validatePlan(value: unknown, remaining: readonly Skill[] = SKILLS): Plan {
const plan = validateContract<Plan>('Plan', value);
export function validateLanguageData<T>(name: 'Command' | 'SceneContext', value: unknown): T {
if ((JSON.stringify(value)?.length ?? 0) > 65536) throw new Error('语言任务消息超长');
check(
languageSchema.$defs[name] as unknown as Node,
value,
name,
languageSchema.$defs as unknown as typeof definitions,
);
return structuredClone(value) as T;
}
export function validatePlan(
value: unknown,
remaining: readonly Skill[] = SKILLS,
version: AgentVersion = AGENT_VERSION,
): Plan {
const plan = validateContract<Plan>('Plan', value, version);
if (
plan.steps.length !== remaining.length ||
plan.steps.some(
@@ -151,8 +182,9 @@ export function validatePlan(value: unknown, remaining: readonly Skill[] = SKILL
export function validateJevDecision(
value: unknown,
candidates: readonly (Skill | 'stop')[],
version: AgentVersion = AGENT_VERSION,
): JevDecision {
const decision = validateContract<JevDecision>('JevDecision', value);
const decision = validateContract<JevDecision>('JevDecision', value, version);
if (!candidates.includes(decision.choice)) throw new Error('Jev 选择不在当前候选中');
return decision;
}
@@ -163,30 +195,35 @@ export function observation(
stamp: RequestStamp,
phase: Skill,
safety: string[] = [],
version: AgentVersion = AGENT_VERSION,
): Observation {
return validateContract<Observation>('Observation', {
version: AGENT_VERSION,
stamp,
source: 'mujoco-ground-truth',
units: 'SI',
frame: 'world-z-up',
time: sample.time,
phase,
base: { position: sample.base, yaw: sample.yaw },
joints: sample.joints,
opening: sample.opening,
tcp: sample.tcp,
object: { id: 'block', position: sample.object, speed: sample.objectSpeed },
goal: { id: 'placement', position: [...goal] },
evidence: {
fingerForces: sample.fingerForces,
supported: sample.supported,
onGoalSupport: sample.onGoalSupport,
secure: evaluator.secure,
transported: evaluator.transported,
return validateContract<Observation>(
'Observation',
{
version,
stamp,
source: 'mujoco-ground-truth',
units: 'SI',
frame: 'world-z-up',
time: sample.time,
phase,
base: { position: sample.base, yaw: sample.yaw },
joints: sample.joints,
opening: sample.opening,
tcp: sample.tcp,
object: { id: 'block', position: sample.object, speed: sample.objectSpeed },
goal: { id: 'placement', position: [...goal] },
evidence: {
fingerForces: sample.fingerForces,
supported: sample.supported,
onGoalSupport: sample.onGoalSupport,
secure: evaluator.secure,
transported: evaluator.transported,
},
safety,
},
safety,
});
version,
);
}
export const PLAN_OUTPUT_SCHEMA = schema.$defs.Plan;
export function skillErrorCode(error: string): string {
web_platform/src/project/builtinRobots.test.ts
+166
View File
@@ -0,0 +1,166 @@
import { File as NodeFile } from 'node:buffer';
import { webcrypto } from 'node:crypto';
import { readFileSync } from 'node:fs';
import { resolve } from 'node:path';
import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest';
import { Euler, Quaternion, Vector3 } from 'three';
import {
BUILTIN_ROBOTS,
builtinRobotFromSearch,
builtinRobotEnhancements,
downloadBuiltinRobot,
} from './builtinRobots';
const robot = BUILTIN_ROBOTS[0];
const zip = readFileSync(resolve('web_platform/assets/robots/lekiwi-v1.zip'));
const fetchMock = vi.fn();
beforeEach(() => {
vi.stubGlobal('fetch', fetchMock);
vi.stubGlobal('File', NodeFile);
vi.stubGlobal('crypto', webcrypto);
});
afterEach(() => {
vi.unstubAllGlobals();
vi.resetAllMocks();
});
describe('URL 内置机器人允许列表', () => {
it('无参数时不导入，其他参数不影响已知机器人', () => {
expect(builtinRobotFromSearch('')).toBeUndefined();
expect(builtinRobotFromSearch('?utm_source=partner')).toBeUndefined();
expect(builtinRobotFromSearch('?utm_source=partner&robot=lekiwi-v1')).toBe(robot);
expect(fetchMock).not.toHaveBeenCalled();
});
it.each([
'?robot=',
'?robot=unknown',
'?robot=constructor',
'?robot=__proto__',
'?robot=https://example.com/robot.zip',
'?robot=../../.env',
'?robot=lekiwi-v1&robot=lekiwi-v1',
])('拒绝无效参数 %s，不请求任意地址', (search) => {
expect(() => builtinRobotFromSearch(search)).toThrow(/robot/);
expect(fetchMock).not.toHaveBeenCalled();
});
});
describe('LeKiwi 双摄像头预设', () => {
it.each(['Camera-Model-v3', 'Camera-Model-v3-1'])(
'从源 URDF 的 %s 推导独立光心与姿态',
(bodyName) => {
const options = builtinRobotEnhancements('lekiwi-v1');
expect(options.cameras).toHaveLength(2);
const camera = options.cameras!.find((camera) => camera.cameraMountBody === bodyName)!;
const document = new DOMParser().parseFromString(
readFileSync(resolve('robot_profiles/lekiwi-reference.urdf'), 'utf8'),
'application/xml',
);
const visual = document.querySelector(`link[name="${bodyName}"] > visual > origin`)!;
const [roll, pitch, yaw] = visual.getAttribute('rpy')!.split(/\s+/).map(Number);
const rotation = new Quaternion().setFromEuler(new Euler(roll, pitch, yaw, 'ZYX'));
const origin = new Vector3().fromArray(visual.getAttribute('xyz')!.split(/\s+/).map(Number));
// The CAD lens apex is mesh (0,0,21.5 mm); keep the pinhole 1 mm clear.
const lens = new Vector3(0, 0, 0.0225).applyQuaternion(rotation).add(origin);
const [w, x, y, z] = camera.cameraQuaternion!;
const actual = new Quaternion(x, y, z, w).normalize();
expect(options.addActuators).toBe(false);
expect(options.addSensors).toBe(true);
expect(new Vector3(...camera.cameraPosition!).distanceTo(lens)).toBeLessThan(1e-10);
expect(
new Vector3(0, 0, -1)
.applyQuaternion(actual)
.distanceTo(new Vector3(0, 0, 1).applyQuaternion(rotation)),
).toBeLessThan(1e-10);
expect(
new Vector3(0, 1, 0)
.applyQuaternion(actual)
.distanceTo(new Vector3(0, -1, 0).applyQuaternion(rotation)),
).toBeLessThan(1e-10);
},
);
it('每次加载取得独立的参数副本', () => {
const first = builtinRobotEnhancements('lekiwi-v1');
for (const camera of first.cameras!) {
camera.cameraPosition![0] = 999;
camera.cameraQuaternion![0] = 999;
}
const second = builtinRobotEnhancements('lekiwi-v1');
for (const camera of second.cameras!) {
expect(camera.cameraPosition![0]).not.toBe(999);
expect(camera.cameraQuaternion![0]).not.toBe(999);
}
});
});
describe('内置 ZIP 下载', () => {
it('发布资产大小与 SHA256 正确，使用可取消的匿名同源请求', async () => {
fetchMock.mockResolvedValue(new Response(zip));
const signal = new AbortController().signal;
const file = await downloadBuiltinRobot(robot, signal);
expect(file.name).toBe('lekiwi-v1.zip');
expect(file.size).toBe(robot.bytes);
expect(file.type).toBe('application/zip');
expect(fetchMock).toHaveBeenCalledExactlyOnceWith(robot.url, {
signal,
credentials: 'omit',
redirect: 'error',
});
});
it('HTTP 失败明确报错', async () => {
fetchMock.mockResolvedValue(new Response('missing', { status: 404 }));
await expect(downloadBuiltinRobot(robot, new AbortController().signal)).rejects.toThrow(
'HTTP 404',
);
});
it('拒绝不完整文件或 HTML 回退', async () => {
fetchMock.mockResolvedValue(new Response('<html>not a zip</html>'));
await expect(downloadBuiltinRobot(robot, new AbortController().signal)).rejects.toThrow(
'文件不完整',
);
});
it('拒绝同等大小但内容被更换的包', async () => {
const corrupt = new Uint8Array(zip);
corrupt[0] ^= 1;
fetchMock.mockResolvedValue(new Response(corrupt));
await expect(downloadBuiltinRobot(robot, new AbortController().signal)).rejects.toThrow(
'校验失败',
);
});
it('拒绝 Content-Length 超过 ZIP 限额', async () => {
fetchMock.mockResolvedValue(
new Response('', { headers: { 'Content-Length': String(129 * 1024 * 1024) } }),
);
await expect(downloadBuiltinRobot(robot, new AbortController().signal)).rejects.toThrow(
'下载大小限制',
);
});
it('没有 Content-Length 时也限制流大小并取消读取', async () => {
const cancel = vi.fn();
fetchMock.mockResolvedValue(
new Response(
new ReadableStream({
start(controller) {
controller.enqueue(new Uint8Array(robot.bytes + 1));
},
cancel,
}),
),
);
await expect(downloadBuiltinRobot(robot, new AbortController().signal)).rejects.toThrow(
'文件大小不匹配',
);
expect(cancel).toHaveBeenCalledOnce();
});
it('下载后取消不能继续导入', async () => {
const controller = new AbortController();
fetchMock.mockImplementation(() => {
controller.abort();
return Promise.resolve(new Response(zip));
});
await expect(downloadBuiltinRobot(robot, controller.signal)).rejects.toMatchObject({
name: 'AbortError',
});
});
});
web_platform/src/project/builtinRobots.ts
+94
View File
@@ -0,0 +1,94 @@
import lekiwiZipUrl from '../../assets/robots/lekiwi-v1.zip?url';
import { DEFAULT_IMPORT_LIMITS } from './types';
import type { UrdfEnhancementOptions } from './urdfToMjcf';
/** Only these server-packaged assets may be selected by a public URL. */
export const BUILTIN_ROBOTS = [
{
id: 'lekiwi-v1',
label: 'LeKiwi v1',
url: lekiwiZipUrl,
fileName: 'lekiwi-v1.zip',
entryPath: 'URDF/LeKiwi.urdf',
profileId: 'lekiwi-v1',
bytes: 22040277,
sha256: 'a10ac577ea49cdf87f324f3f6e9a7a887a1e3c0254ae638059711fef9e385b04',
},
] as const;
export type BuiltinRobot = (typeof BUILTIN_ROBOTS)[number];
export function builtinRobotEnhancements(id: BuiltinRobot['id']): UrdfEnhancementOptions {
switch (id) {
case 'lekiwi-v1':
return {
// The profile supplies the nine servos; do not generate generic motors.
addActuators: false,
addSensors: true,
sensorType: 'camera',
// Both source STLs have +Z optical axes and a lens apex at z=21.5 mm.
// Transform a pinhole 1 mm ahead of each lens using its own URDF visual
// origin. MJCF -Z looks forward; mesh +X is right and -Y is image up.
cameras: [
{
name: '__platform_camera__',
cameraMountBody: 'Camera-Model-v3',
// Canonical +X forward, retaining the CAD's 10° upward tilt.
cameraPosition: [-0.01216119999, 0.02458924444, -0.01207162489],
cameraQuaternion: [0.413175911167, 0.492403876506, 0.492403876506, -0.586824088833],
},
{
name: '__platform_camera__wrist',
cameraMountBody: 'Camera-Model-v3-1',
// Wrist-local optical pose: follows the arm, not the chassis.
cameraPosition: [0.024189561811, -0.017694392834, -0.00015312743],
cameraQuaternion: [0.459991520356, -0.040244043349, -0.883635489162, -0.077308087991],
},
],
};
}
}
export function builtinRobotFromSearch(search: string): BuiltinRobot | undefined {
const ids = new URLSearchParams(search).getAll('robot');
if (!ids.length) return undefined;
if (ids.length !== 1) throw new Error('robot 参数只能指定一次，请使用 ?robot=lekiwi-v1');
const robot = BUILTIN_ROBOTS.find((item) => item.id === ids[0]);
if (!robot) throw new Error('未知的内置机器人，请使用 ?robot=lekiwi-v1');
return robot;
}
export async function downloadBuiltinRobot(
robot: BuiltinRobot,
signal: AbortSignal,
): Promise<File> {
const response = await fetch(robot.url, { signal, credentials: 'omit', redirect: 'error' });
if (!response.ok) throw new Error(`内置机器人下载失败（HTTP ${response.status}），请刷新重试`);
const declaredBytes = Number(response.headers.get('Content-Length'));
if (declaredBytes > DEFAULT_IMPORT_LIMITS.maxZipBytes)
throw new Error('内置机器人 ZIP 超过下载大小限制');
// Bound the stream even when Content-Length is absent or incorrect.
if (!response.body) throw new Error('内置机器人下载为空，请刷新重试');
const reader = response.body.getReader();
const parts: Uint8Array<ArrayBuffer>[] = [];
let size = 0;
try {
while (true) {
const { done, value } = await reader.read();
if (done) break;
size += value.byteLength;
if (size > robot.bytes) throw new Error('内置机器人文件大小不匹配，请联系管理员');
parts.push(new Uint8Array(value));
}
} finally {
await reader.cancel();
reader.releaseLock();
}
signal.throwIfAborted();
if (size !== robot.bytes) throw new Error('内置机器人文件不完整，请刷新重试');
const file = new File(parts, robot.fileName, { type: 'application/zip' });
const digest = await crypto.subtle.digest('SHA-256', await file.arrayBuffer());
const hash = Array.from(new Uint8Array(digest), (b) => b.toString(16).padStart(2, '0')).join('');
if (hash !== robot.sha256) throw new Error('内置机器人校验失败，请联系管理员');
signal.throwIfAborted();
return file;
}
web_platform/src/project/urdfToMjcf.test.ts
+101 -1
View File
@@ -1,4 +1,8 @@
import { enhanceConvertedMjcf, groundConvertedMjcf } from './urdfToMjcf';
import {
enhanceConvertedMjcf,
groundConvertedMjcf,
type UrdfEnhancementOptions,
} from './urdfToMjcf';
const encode = (value: string) => new TextEncoder().encode(value);
const decode = (value: Uint8Array) => new TextDecoder().decode(value);
@@ -152,6 +156,102 @@ describe('enhanceConvertedMjcf', () => {
expect(repeatedDocument.querySelectorAll('sensor > accelerometer')).toHaveLength(1);
});
it('支持固定相机的任意安装倾角，归一化 wxyz 且不混用 xyaxes', () => {
const result = enhanceConvertedMjcf(
encode('<mujoco><worldbody><body name="camera"/></worldbody></mujoco>'),
{
addActuators: false,
addSensors: true,
sensorType: 'camera',
cameraMountBody: 'camera',
cameraDirection: '+X',
cameraQuaternion: [2, 0, 0, 0],
},
);
const camera = new DOMParser()
.parseFromString(decode(result.data), 'application/xml')
.querySelector('camera');
expect(camera?.getAttribute('quat')).toBe('1 0 0 0');
expect(camera?.hasAttribute('xyaxes')).toBe(false);
});
it.each([
[0, 0, 0, 0],
[NaN, 0, 0, 1],
[Infinity, 0, 0, 1],
[1e308, 1e308, 1e308, 1e308],
])('拒绝无效相机四元数 %j', (...values) => {
expect(() =>
enhanceConvertedMjcf(
encode('<mujoco><worldbody><body name="camera"/></worldbody></mujoco>'),
{
addActuators: false,
addSensors: true,
sensorType: 'camera',
cameraQuaternion: values as [number, number, number, number],
},
),
).toThrow('四元数');
});
it('显式固连 body 不存在时拒绝猜测其他挂载点', () => {
expect(() =>
enhanceConvertedMjcf(encode('<mujoco><worldbody><body name="head"/></worldbody></mujoco>'), {
addActuators: false,
addSensors: true,
sensorType: 'camera',
cameraMountBody: 'missing-camera',
}),
).toThrow('摄像头固连 Body 不存在');
});
it('独立固连双相机、保留已有相机并可幂等补齐第二路，不额外注入通用相机', () => {
const options: UrdfEnhancementOptions = {
addActuators: false,
addSensors: true,
sensorType: 'camera',
cameras: [
{ name: '__platform_camera__', cameraMountBody: 'front', cameraDirection: '+X' },
{ name: '__platform_camera__wrist', cameraMountBody: 'wrist', cameraDirection: '-Z' },
],
};
const source = encode(
'<mujoco><worldbody><body name="base"><body name="front"/><body name="wrist"/></body></worldbody></mujoco>',
);
const first = enhanceConvertedMjcf(source, {
...options,
cameras: options.cameras!.slice(0, 1),
});
const second = enhanceConvertedMjcf(first.data, options);
expect(second.cameraCount).toBe(1);
const repeated = enhanceConvertedMjcf(second.data, options);
expect(repeated.cameraCount).toBe(0);
expect(repeated.cameraAdded).toBe(false);
const doc = new DOMParser().parseFromString(decode(repeated.data), 'application/xml');
expect(doc.querySelectorAll('camera')).toHaveLength(2);
expect(doc.querySelector('body[name="front"] > camera')?.getAttribute('name')).toBe(
'__platform_camera__',
);
expect(doc.querySelector('body[name="wrist"] > camera')?.getAttribute('xyaxes')).toBe(
'0 -1 0 1 0 0',
);
expect(doc.querySelectorAll('sensor > gyro')).toHaveLength(1);
expect(doc.querySelectorAll('sensor > accelerometer')).toHaveLength(1);
const disabled = enhanceConvertedMjcf(source, { ...options, addSensors: false });
expect(disabled.cameraCount).toBe(0);
});
it.each(['', 'duplicate'])('拒绝空或重复的显式相机名称：%s', (name) => {
expect(() =>
enhanceConvertedMjcf(encode('<mujoco><worldbody><body name="head"/></worldbody></mujoco>'), {
addActuators: false,
addSensors: true,
sensorType: 'camera',
cameras: [{ name }, { name }],
}),
).toThrow('摄像头名称');
});
it('关闭选项时不修改 actuator 和 camera', () => {
const result = enhanceConvertedMjcf(
encode(
web_platform/src/project/urdfToMjcf.ts
+53 -19
View File
@@ -10,18 +10,25 @@ function numbers(value: string | undefined, count: number): number[] {
export type UrdfBaseMode = 'floating' | 'fixed';
export type CameraDirection = '+X' | '-X' | '+Y' | '-Y' | '+Z' | '-Z';
export interface UrdfEnhancementOptions {
addActuators: boolean;
addSensors: boolean;
sensorType: 'camera';
export interface UrdfCameraOptions {
cameraMountBody?: string;
cameraPosition?: [number, number, number];
cameraDirection?: CameraDirection;
/** MuJoCo wxyz orientation in the mount body; overrides the axis-only direction. */
cameraQuaternion?: [number, number, number, number];
}
export interface UrdfEnhancementOptions extends UrdfCameraOptions {
addActuators: boolean;
addSensors: boolean;
sensorType: 'camera';
/** Explicit named cameras replace the legacy single-camera fields when supplied. */
cameras?: Array<UrdfCameraOptions & { name: string }>;
}
export interface UrdfEnhancementResult {
data: Uint8Array;
actuatorCount: number;
cameraAdded: boolean;
cameraCount: number;
imuAdded: boolean;
unitreeGo2wTuned: boolean;
}
@@ -93,7 +100,7 @@ function tuneUnitreeGo2w(document: Document): boolean {
return true;
}
/** 为转换后的 MJCF 按需补充可直接控制的关节电机和一台自动取景相机。 */
/** 为转换后的 MJCF 按需补充关节电机、IMU 与固连摄像头。 */
export function enhanceConvertedMjcf(
data: Uint8Array,
options: UrdfEnhancementOptions,
@@ -180,16 +187,31 @@ export function enhanceConvertedMjcf(
imuAdded = true;
}
}
let cameraAdded = false;
if (
options.addSensors &&
options.sensorType === 'camera' &&
!document.querySelector('camera[name^="__platform_camera__"]')
) {
let cameraCount = 0;
const cameras = options.cameras ?? [{ ...options, name: '__platform_camera__' }];
const existingNames = new Set(
Array.from(document.querySelectorAll('camera[name]')).map((camera) =>
camera.getAttribute('name'),
),
);
const requestedNames = new Set<string>();
for (const cameraOptions of options.addSensors && options.sensorType === 'camera'
? cameras
: []) {
if (!cameraOptions.name.trim() || requestedNames.has(cameraOptions.name))
throw new Error('摄像头名称不能为空或重复');
requestedNames.add(cameraOptions.name);
if (
existingNames.has(cameraOptions.name) ||
(!options.cameras && document.querySelector('camera[name^="__platform_camera__"]'))
)
continue;
const bodies = Array.from(worldbody.querySelectorAll('body')),
preferred = options.cameraMountBody
? bodies.find((body) => body.getAttribute('name') === options.cameraMountBody)
preferred = cameraOptions.cameraMountBody
? bodies.find((body) => body.getAttribute('name') === cameraOptions.cameraMountBody)
: undefined;
if (cameraOptions.cameraMountBody && !preferred)
throw new Error(`摄像头固连 Body 不存在：${cameraOptions.cameraMountBody}`);
const mount =
preferred ??
bodies.find((body) =>
@@ -205,26 +227,38 @@ export function enhanceConvertedMjcf(
'+Z': '0 1 0 1 0 0',
'-Z': '0 -1 0 1 0 0',
},
position = options.cameraPosition ?? [0.1, 0, 0.05],
direction = options.cameraDirection ?? '+X';
position = cameraOptions.cameraPosition ?? [0.1, 0, 0.05],
direction = cameraOptions.cameraDirection ?? '+X';
const camera = document.createElement('camera');
camera.setAttribute('name', uniqueName(document, 'camera[name]', '__platform_camera__'));
camera.setAttribute('name', cameraOptions.name);
camera.setAttribute('mode', 'fixed');
camera.setAttribute(
'pos',
position.map((value) => (Number.isFinite(value) ? value : 0)).join(' '),
);
camera.setAttribute('xyaxes', directions[direction]);
if (cameraOptions.cameraQuaternion) {
const quaternion = cameraOptions.cameraQuaternion;
const norm = Math.hypot(...quaternion);
if (
quaternion.length !== 4 ||
!quaternion.every(Number.isFinite) ||
!Number.isFinite(norm) ||
norm < 1e-12
)
throw new Error('摄像头旋转四元数必须为非零、有限的 wxyz');
camera.setAttribute('quat', quaternion.map((value) => value / norm).join(' '));
} else camera.setAttribute('xyaxes', directions[direction]);
camera.setAttribute('fovy', '45');
camera.setAttribute('resolution', '640 480');
mount.append(camera);
cameraAdded = true;
cameraCount += 1;
}
}
return {
data: encoder.encode(new XMLSerializer().serializeToString(document)),
actuatorCount,
cameraAdded,
cameraAdded: cameraCount > 0,
cameraCount,
imuAdded,
unitreeGo2wTuned,
};
web_platform/src/simulation/PhysicsAdapter.ts
+3 -1
View File
@@ -377,7 +377,9 @@ export class MainThreadPhysicsAdapter implements PhysicsAdapter {
);
if (enhanced.cameraAdded)
warnings.push(
`已将 640×480 摄像头固连到 ${enhancements.cameraMountBody || '自动选择的头部/末端 body'}，局部位置 ${(enhancements.cameraPosition ?? [0.1, 0, 0.05]).join(' ')} m，朝向 ${enhancements.cameraDirection ?? '+X'}`,
enhancements.cameras
? `已添加 ${enhanced.cameraCount} 路 640×480 固定摄像头：${enhancements.cameras.map((camera) => `${camera.name} → ${camera.cameraMountBody}`).join('；')}`
: `已将 640×480 摄像头固连到 ${enhancements.cameraMountBody || '自动选择的头部/末端 body'}，局部位置 ${(enhancements.cameraPosition ?? [0.1, 0, 0.05]).join(' ')} m，朝向 ${enhancements.cameraQuaternion ? `四元数 wxyz ${enhancements.cameraQuaternion.join(' ')}` : (enhancements.cameraDirection ?? '+X')}`,
);
} finally {
intermediate.dispose();
web_platform/src/viewer/MuJoCoViewer.overlay.test.ts
+42
View File
@@ -1,5 +1,47 @@
import { Matrix4, PerspectiveCamera, Vector3 } from 'three';
import { MuJoCoViewer } from './MuJoCoViewer';
it('切换两路相机使用各自 MuJoCo 世界位姿与视场角，逐帧跟随运动但不推进物理', () => {
const onSensorCamerasChanged = vi.fn();
const data = {
cam_xpos: [1, 2, 3, 4, 5, 6],
cam_xmat: [1, 0, 0, 0, 1, 0, 0, 0, 1, 0, 0, 1, 0, 1, 0, -1, 0, 0],
};
const sensorCamera = new PerspectiveCamera();
const viewer = Object.assign(Object.create(MuJoCoViewer.prototype), {
session: { model: { ncam: 2, cam_fovy: [45, 60] }, data },
sensorCameras: [
{ id: 0, name: 'front' },
{ id: 1, name: 'wrist' },
],
sensorCameraId: 0,
callbacks: { onSensorCamerasChanged },
sensorCamera,
sensorRotation: new Matrix4(),
modelExtent: 2,
}) as MuJoCoViewer;
const update = () =>
(viewer as unknown as { updateSensorCamera(): boolean }).updateSensorCamera();
expect(update()).toBe(true);
expect(sensorCamera.position.toArray()).toEqual([1, 2, 3]);
expect(viewer.setSensorCamera(1)).toBe(true);
expect(update()).toBe(true);
expect(sensorCamera.position.toArray()).toEqual([4, 5, 6]);
expect(sensorCamera.fov).toBe(60);
expect(
sensorCamera.getWorldDirection(new Vector3()).distanceTo(new Vector3(-1, 0, 0)),
).toBeLessThan(1e-12);
data.cam_xpos.splice(3, 3, 7, 8, 9);
update();
expect(sensorCamera.position.toArray()).toEqual([7, 8, 9]);
expect(viewer.setSensorCamera(NaN)).toBe(false);
expect(viewer.setSensorCamera(2)).toBe(false);
expect(viewer.setSensorCamera(0)).toBe(true);
update();
expect(sensorCamera.position.toArray()).toEqual([1, 2, 3]);
expect(onSensorCamerasChanged).toHaveBeenCalledTimes(2);
});
/** 不创建 WebGL 上下文；验证布局槽位到实际 WebGL 子视口的坐标换算。 */
describe('MuJoCoViewer 摄像头槽位', () => {
it('以 DOM 矩形定位渲染与裁剪，正确处理画布缩放并恢复主视口', () => {
web_platform/src/viewer/MuJoCoViewer.ts
+35 -3
View File
@@ -27,6 +27,11 @@ import { ParametricMapPreviewLayer } from './ParametricMapPreviewLayer';
export type InteractionMode = 'select' | 'joint' | 'force';
export type ViewerTheme = 'light' | 'dark';
export interface ViewerCamera {
id: number;
name: string;
bodyName: string;
}
export interface ViewerSelection {
bodyId: number;
geomId: number;
@@ -42,6 +47,7 @@ interface ViewerCallbacks {
onError(error: Error): void;
onNavigationTarget?(target: [number, number]): void;
onNavigationMode?(active: boolean): void;
onSensorCamerasChanged?(cameras: readonly ViewerCamera[], selectedId: number): void;
onMapEditorSelect(id: string | null): void;
onMapEditorPreviewSelect(mapAssetId: string, objectId: string): void;
onParametricMapSelect(id: string | null): void;
@@ -87,6 +93,7 @@ export class MuJoCoViewer {
private showPerceptionRays = true;
private perceptionLines?: THREE.LineSegments<THREE.BufferGeometry, THREE.LineBasicMaterial>;
private sensorCameraId = -1;
private sensorCameras: ViewerCamera[] = [];
private modelExtent = 2;
private session: SimulationSession | null = null;
private option: MjvOption | null = null;
@@ -269,6 +276,9 @@ export class MuJoCoViewer {
}
attach(session: SimulationSession | null): void {
const previousCamera = this.sensorCameras.find(
(camera) => camera.id === this.sensorCameraId,
)?.name;
this.releaseModel();
this.runtimeErrorReported = false;
this.snapshotDirty = false;
@@ -300,13 +310,24 @@ export class MuJoCoViewer {
for (let id = 0; id < session.model.ncam; id += 1) {
const camera = session.model.cam(id);
try {
if (this.sensorCameraId < 0 || camera.name.startsWith('__platform_camera__'))
this.sensorCameraId = id;
if (camera.name.startsWith('__platform_camera__')) break;
const body = session.model.body(Number(session.model.cam_bodyid[id]));
try {
this.sensorCameras.push({ id, name: camera.name || `camera_${id}`, bodyName: body.name });
} finally {
body.delete();
}
} finally {
camera.delete();
}
}
this.sensorCameraId =
(
this.sensorCameras.find((camera) => camera.name === previousCamera) ??
this.sensorCameras.find((camera) => camera.name === '__platform_camera__') ??
this.sensorCameras.find((camera) => camera.name.startsWith('__platform_camera__')) ??
this.sensorCameras[0]
)?.id ?? -1;
this.callbacks.onSensorCamerasChanged?.(this.sensorCameras, this.sensorCameraId);
this.fitCamera(session);
this.visualizationHelpers.attach(session, this.modelExtent);
}
@@ -422,6 +443,15 @@ export class MuJoCoViewer {
setShowSensorCamera(value: boolean): void {
this.showSensorCamera = value;
}
/** Display-only selection: does not reload, step physics, or change control ownership. */
setSensorCamera(id: number): boolean {
if (!Number.isInteger(id) || !this.sensorCameras.some((camera) => camera.id === id))
return false;
this.sensorCameraId = id;
this.callbacks.onSensorCamerasChanged?.(this.sensorCameras, id);
return true;
}
setTheme(theme: ViewerTheme): void {
const light = theme === 'light';
const background =
@@ -1305,6 +1335,8 @@ export class MuJoCoViewer {
this.option = null;
this.session = null;
this.sensorCameraId = -1;
this.sensorCameras = [];
this.callbacks?.onSensorCamerasChanged?.([], -1);
this.modelExtent = 2;
this.modelHasVisuals = false;
this.visualizationHelpers.attach(null, this.modelExtent);
web_platform/vite.config.ts
+6 -12
View File
@@ -120,20 +120,14 @@ export default defineConfig(({ mode }) => ({
server: {
open: mode !== 'website',
fs: { allow: ['..'] },
proxy:
mode === 'website'
? {
'/api/decision/v1': { target: process.env.CADWORLD_DEV_API ?? 'http://127.0.0.1:8768' },
}
: undefined,
proxy: {
'/api/decision/v1': { target: process.env.CADWORLD_DEV_API ?? 'http://127.0.0.1:8768' },
},
},
preview: {
proxy:
mode === 'website'
? {
'/api/decision/v1': { target: process.env.CADWORLD_DEV_API ?? 'http://127.0.0.1:8768' },
}
: undefined,
proxy: {
'/api/decision/v1': { target: process.env.CADWORLD_DEV_API ?? 'http://127.0.0.1:8768' },
},
},
test: {
globals: true,