feat: release v1.0.1 CADWorld 网站与 LeKiwi 智能抓放
web-platform-ci / Standalone decision service (no cloud credentials) (push) Has been cancelled
web-platform-ci / TypeScript, lint, unit, build (push) Has been cancelled
web-platform-ci / Playwright E2E (push) Has been cancelled
lekiwi-compatibility / cpu-compatibility (push) Has been cancelled
web-platform-ci / Standalone decision service (no cloud credentials) (pull_request) Has been cancelled
web-platform-ci / TypeScript, lint, unit, build (pull_request) Has been cancelled
web-platform-ci / Playwright E2E (pull_request) Has been cancelled
lekiwi-compatibility / cpu-compatibility (pull_request) Has been cancelled

集成同源 BYOK 会话隔离、精简模型设置、官方订阅入口和 HTTPS 发布运维;保留本地训练/调参与控制能力。同步 npm 版本及 CHANGELOG,记录公网真实 API 验收仍待用户凭据。
This commit is contained in:
2026-09-24 09:57:41 +08:00
parent 3ad29356c9
commit f3a8a38acd
194 changed files with 32918 additions and 236 deletions
+15
View File
@@ -10,6 +10,7 @@ src/
├── components/ui/ 无业务依赖的共享 UI 基础组件
├── controller/ Python 控制器运行时、类型和面板
├── map/ 地图加载、生成、编辑和面板
├── mobile/ 移动操作 RL 底层;agent/ 为独立语义契约、物理技能与任务 UI
├── project/ 工程导入、转换、文件树和工作区
├── rl/ ONNX 策略运行时、任务绑定、类型和面板
├── robot/ 通用机器人契约、profile、实时协议与外部控制面板
@@ -42,6 +43,20 @@ SimulationSession snapshot → app → viewer / 各业务面板
主工作台由 `index.html → src/main.tsx` 启动;自调参工作台由 Vite MPA 入口 `tuning.html → src/tuning/main.tsx` 启动,避免把 MuJoCo/Three.js 主应用依赖打入监控页面。两页仅通过训练 HTTP API和严格同源的短消息交接训练服务凭据/策略导入请求,不在 URL 中传 token。
## Website 产品边界
`vite --mode website` 使用 `app/websiteConfig.ts` 关闭云端首版的训练/调参/外控入口,构建到 `website-dist/`,不包含 `tuning.html`;默认 local 构建和底层算法保留。`WebsiteModelSettings` 仅在展开设置后建立同源匿名会话,`DecisionClient.website()` 自动携带 Cookie/CSRF/配置版本,配置与模型请求仍遵守旧的任务失效机制。API_KEY 与设备码不写浏览器持久化。
后端 `web_server/web_sessions/web_config/model_catalog` 分离公网边界、逐用户内存状态、固定上游配置及模型目录,复用原 `Service` 的有界请求、契约和物理控制解耦。订阅由逐会话 `CodexAccount` 提供官方设备码登录,不共用进程级账号、不放开任意 CLI 执行。部署与验证见 `docs/website-deployment.md`。
## LeKiwi 智能任务
主工作台控制台同时承载 RL 与 `mobile/agent/AgentTaskPanel`;独立 `mobile.html` 构建入口已移除,`tuning.html` 和物理夹具保留。`App.loadEntry()` 仍负责同一适配器/查看器的事务与回滚;智能任务只添加场景配置,不另建 RobotManager。任务场景不应用/清空地图草稿,RL 切回时重新编译 RL 场景。
`AgentTaskController` → gated 本地技能 → 同一个 `SimulationSession.advance()`;模型 Promise 只写邮箱,票据限定同步物理窗口。任务级暂停可继续,全局暂停/外力拖动/地图编辑/模型重载撤销旧请求。完整 CAD 软件渲染时 agent 仅降低绘制频率,空闲帧仍调用同一个 advance,不新建时钟、不减面。
`DecisionClient` 仅访问本机 `decision_server`;模型 key 只在服务内存,页面临时输入保存后清空,令牌不进入持久存储/任务证据。模型连接、官方 Codex 登录均无隐式付费来源回退。算法、网络和认证 UI 位于各自业务模块,不嵌入 App 的加载算法。验收边界见 [智能抓放](../docs/lekiwi-agent.md)。
## 机器人接口边界
`robot/types + validation` 定义跨语言契约,`robot_profiles/` 是唯一参数源;可信 `RobotAdapter` 封装物理模型,`ModelBindings` 为机器人/Python/Go2 共用索引。`ControlArbiter` 在 Session 边界授予身份票据,所有写入都经过所有者检查,不依赖 UI 灰化。
+14 -1
View File
@@ -1,4 +1,17 @@
# 自定义训练与前视射线避障
# 统一强化学习训练控制台
## LeKiwi / LeKiwi Bundle
1. 主工作台导入机器人 ZIP,在 URDF 选项选对应的「移动操作训练场景」。也可先加载旧 LeKiwi v1 profile,再从控制台选择任务;开始训练会自动加载抓取/搬运场景。
2. 启动配置了 `--mobile-python` 的训练服务([独立依赖配置](../training_server/README.md#lekiwi-一键训练与-go2-共用控制台)),在「控制台 → 强化学习任务」连接。面板按当前机器人选择任务,也允许手动切换。
3. 默认先选「底盘接近」,配置示教初始化步数(0 可禁用)、位置随机化、独立评估回合,再配置迭代数、环境数(1–64)、每环境采样步数、设备、种子及物体/目标坐标。控制周期固定0.02s,Go2 地形/射线/基础策略/自调参配置隐藏。点击「发起本地训练」自动同步场景并创建后台作业,**不再下载训练包**。
4. 共享日志、Loss/平均奖励曲线、进度与停止按钮。完成后点击「导入策略」自动拉取 ONNX 与 deployment,核验形状、任务、变体及 SHA-256,在当前主视口开始推断。暂停/重置/移除策略沿用主控制台。
5. 根据独立评估决定续训:点击「接续此作业(保留权重)」可同阶段继续。至少10回合、≥80%成功率、无安全终止才允许依次升级到末端接近、抓取放置;场景资产与已评估初态分布必须匹配。未达标策略可以调试导入,但不是成品策略。控制采用 v2 限速增量目标,旧68维 ONNX 会拒绝加载。详见 [训练课程与实测结果](../docs/mobile-training-curriculum.md)。
两种变体不能互用策略。移动操作任务使用专用物体/目标/地面,不使用 Go2 地图编辑配置;总采样步数为 `迭代 × 环境数 × 每环境采样步数`。物理在 CPU、PPO 网络可选 CPU/单 GPU;短训练只验证运行链路,不保证抓取收敛。独立 `/mobile.html` 产品页已移除;保留主工作台 RL、`tuning.html` 与 `/physics/mobile.html` 夹具。新增的 [LeKiwi 智能抓放](../docs/lekiwi-agent.md) 使用独立任务契约,切回训练会重建 RL 场景,不把双支撑台任务作为原训练快照。普通作业状态在服务重启后不保留,与现有 Go2 一致。
## Go2 自定义训练与前视射线避障
1. 启动仓库本地训练服务,在「控制台 → 强化学习任务」输入令牌并连接。
2. 选择「前视射线避障导航」,设置地形、种子、障碍物参数、FOV、探测/安全距离及避障奖励权重,发起训练。界面显示进度、最近价值/策略/熵损失和原始日志。
@@ -0,0 +1,86 @@
import { spawn } from 'node:child_process';
import { randomBytes } from 'node:crypto';
import { mkdtemp, rm } from 'node:fs/promises';
import { tmpdir } from 'node:os';
import { join, resolve } from 'node:path';
import { stopProcess } from './controlBridge';
/** Explicit live-test helper: keys stay in Python memory, only the ephemeral bridge
* token crosses to the browser. Tighten (never bypass) production admission limits. */
export async function startLiveDecisionService() {
const token = randomBytes(32).toString('hex');
const directory = await mkdtemp(join(tmpdir(), 'lekiwi-live-decision-'));
const script = `
import asyncio, json, os, signal, sys
from pathlib import Path
from aiohttp import web
from decision_server.server import create_app, STATE
from decision_server.credentials import deepseek_llm, openrouter_jev
from decision_server.protocol import DecisionError
async def main():
app = create_app(Path(sys.argv[1]), os.environ.pop('DECISION_SMOKE_TOKEN'), origins={'http://127.0.0.1:4176'}, port=0)
service = app[STATE]
service.connections.values['llm'] = deepseek_llm(Path('.env'))
service.connections.values['jev'] = openrouter_jev(Path('.env'))
original = service.admit
counts = {'llm': 0, 'jev': 0}
def admit(role, stamp):
if counts[role] >= {'llm': 1, 'jev': 12}[role]:
raise DecisionError('live_smoke_budget', 429)
original(role, stamp)
counts[role] += 1
service.admit = admit
runner = web.AppRunner(app, access_log=None, handler_cancellation=True)
await runner.setup()
site = web.TCPSite(runner, '127.0.0.1', 0)
await site.start()
port = site._server.sockets[0].getsockname()[1]
service.hosts = {f'127.0.0.1:{port}'}
stop = asyncio.Event()
for sig in (signal.SIGTERM, signal.SIGINT):
asyncio.get_running_loop().add_signal_handler(sig, stop.set)
print('DECISION_READY ' + json.dumps({'endpoint': f'http://127.0.0.1:{port}'}), flush=True)
try:
await stop.wait()
finally:
await runner.cleanup()
asyncio.run(main())
`;
const child = spawn(resolve('.venv/bin/python'), ['-u', '-c', script, directory], {
env: { ...process.env, DECISION_SMOKE_TOKEN: token, PYTHONNOUSERSITE: '1', PYTHONPATH: '' },
stdio: ['ignore', 'pipe', 'ignore'],
});
const stop = async () => {
if (child.pid) await stopProcess(child);
await rm(directory, { recursive: true, force: true });
};
try {
const endpoint = await new Promise<string>((resolve, reject) => {
let output = '';
const timer = setTimeout(
() => reject(new Error('真实模型服务启动超时(不输出认证日志)')),
15000,
);
child.once('error', () => {
clearTimeout(timer);
reject(new Error('真实模型服务启动失败'));
});
child.once('exit', (code) => {
clearTimeout(timer);
reject(new Error(`真实模型服务退出:${code}`));
});
child.stdout.on('data', (chunk: Buffer) => {
output = (output + chunk.toString()).slice(-8192);
const match = output.match(/DECISION_READY (\{[^\n]+\})/);
if (match) {
clearTimeout(timer);
resolve((JSON.parse(match[1]) as { endpoint: string }).endpoint);
}
});
});
return { endpoint, token, stop };
} catch (error) {
await stop();
throw error;
}
}
@@ -0,0 +1,43 @@
import { createConnection } from 'node:net';
import { readFile } from 'node:fs/promises';
/** Same-user 0600 Unix channel. Never print the connect response or persist its token. */
export async function subscriptionOperation(operation: 'connect' | 'status' | 'limits' | 'stop') {
const pointer = JSON.parse(
await readFile('build/lekiwi-agent/subscription-session.json', 'utf8'),
) as { socket?: string; status: string };
if (!pointer.socket || pointer.status === 'closed')
throw new Error('隔离订阅会话已关闭,请重新发起官方登录');
return new Promise<Record<string, unknown>>((resolve, reject) => {
const channel = createConnection(pointer.socket!);
let buffer = '';
channel.setTimeout(20000);
channel.once('connect', () => channel.write(JSON.stringify(operation) + '\n'));
channel.on('data', (chunk) => {
buffer += String(chunk);
if (buffer.length > 16384) {
channel.destroy();
reject(new Error('订阅管理响应过大'));
return;
}
if (buffer.includes('\n')) {
try {
const result = JSON.parse(buffer.split('\n')[0]) as Record<string, unknown>;
if (result.error) reject(new Error('订阅管理请求失败'));
else resolve(result);
} catch {
reject(new Error('订阅管理响应非法'));
}
channel.end();
}
});
channel.once('timeout', () => {
channel.destroy();
reject(new Error('订阅管理请求超时'));
});
channel.once('error', () => reject(new Error('无法连接隔离订阅会话')));
channel.once('end', () => {
if (!buffer.includes('\n')) reject(new Error('订阅管理连接提前关闭'));
});
});
}
@@ -0,0 +1,104 @@
import { test, expect } from '@playwright/test';
import { resolve } from 'node:path';
import { writeFile } from 'node:fs/promises';
import type {} from '../physics/agent';
for (const fault of [
'empty-grasp',
'empty-once',
'alignment',
'slip',
'jam',
'navigation',
'unreachable',
'overspeed',
] as const) {
test(`真实 WASM 故障:${fault} 不可冒充成功/无限恢复`, async ({ page }, info) => {
await page.goto('/physics/agent.html');
await page.waitForFunction(() => Boolean(window.lekiwiAgent));
await page.evaluate(
(url) => window.lekiwiAgent.boot(url),
`/@fs${resolve('build/lekiwi/lekiwi-v1.zip')}`,
);
try {
const r = await page.evaluate(
(fault) => window.lekiwiAgent.task(0, undefined, { fault }),
fault,
);
await writeFile(info.outputPath('fault-evidence.json'), JSON.stringify(r, null, 2));
console.log('WASM fault', {
fault,
state: r.state,
error: r.error,
failures: r.failures,
recoveries: r.recoveries,
simTime: r.simTime,
});
expect(r.mode).toBe('mock');
expect(r.state).toBe(fault === 'empty-once' ? 'succeeded' : 'failed');
expect(r.succeeded).toBe(fault === 'empty-once');
expect(r.simTime).toBeLessThanOrEqual(70.01);
expect(r.llmCalls).toBe(1);
if (fault === 'empty-grasp' || fault === 'empty-once' || fault === 'alignment') {
const skill = fault === 'alignment' ? 'pregrasp' : 'close';
const code = fault === 'alignment' ? 'alignment_timeout' : 'empty_grasp';
const recovered = fault === 'empty-once';
expect(r.recoveries[skill]).toBe(recovered ? 1 : 2);
expect(r.failures.map((f) => f.code)).toEqual(Array(recovered ? 1 : 3).fill(code));
if (!recovered) expect(r.trace.every((s) => !s.secure)).toBe(true);
else {
expect(r.transported).toBeGreaterThanOrEqual(0.5);
expect(r.sample.onGoalSupport).toBe(true);
}
} else {
expect(r.recoveries).toEqual({});
expect(r.failures).toHaveLength(1);
expect(r.failures[0].code).toBe(
{
slip: 'lost_grasp',
jam: 'joint_stall',
navigation: 'navigation_timeout',
unreachable: 'unreachable',
overspeed: 'joint_velocity',
}[fault],
);
}
if (fault === 'slip') {
expect(Math.max(...r.trace.map((s) => s.transported))).toBeGreaterThan(0.12);
expect(r.transported).toBe(0);
}
} finally {
await page.evaluate(() => window.lekiwiAgent.dispose());
}
});
}
for (const goal of [
[0.237, 0.565, 0.128],
[0.277, 0.665, 0.128],
]) {
test(`真实 WASM 受限目标角点 ${goal.slice(0, 2).join(',')}(非任意目标泛化)`, async ({
page,
}, info) => {
await page.goto('/physics/agent.html');
await page.waitForFunction(() => Boolean(window.lekiwiAgent));
await page.evaluate(
(url) => window.lekiwiAgent.boot(url),
`/@fs${resolve('build/lekiwi/lekiwi-v1.zip')}`,
);
try {
const r = await page.evaluate(
(goal) => window.lekiwiAgent.task(0, undefined, { goal }),
goal,
);
await writeFile(info.outputPath('goal-evidence.json'), JSON.stringify(r, null, 2));
expect(r.state, r.error).toBe('succeeded');
expect(r.goal).toEqual(goal);
expect(r.succeeded).toBe(true);
expect(r.transported).toBeGreaterThanOrEqual(0.5);
expect(Math.hypot(...r.sample.object.map((v, i) => v - goal[i]))).toBeLessThan(0.04);
} finally {
await page.evaluate(() => window.lekiwiAgent.dispose());
}
});
}
@@ -0,0 +1,46 @@
import { test, expect } from '@playwright/test';
import { resolve } from 'node:path';
import { writeFile } from 'node:fs/promises';
import type {} from '../physics/agent';
test('8 次 A 场景与 IK scratch 加载/释放:验证原生句柄,不以 heap 高水位判泄漏', async ({
page,
}, info) => {
await page.goto('/physics/agent.html');
await page.waitForFunction(() => Boolean(window.lekiwiAgent));
await page.evaluate(
(url) => window.lekiwiAgent.boot(url),
`/@fs${resolve('build/lekiwi/lekiwi-v1.zip')}`,
);
const rows = await page.evaluate(() => window.lekiwiAgent.resources());
await writeFile(info.outputPath('resource-evidence.json'), JSON.stringify(rows, null, 2));
expect(rows).toHaveLength(8);
expect(rows.every((row) => row.deleted.length === 6 && row.deleted.every(Boolean))).toBe(true);
console.log(
'A + scratch heap capacity MiB',
rows.map((row) => row.heapBytes / 1024 / 1024),
);
});
for (const action of ['pause-task', 'pause-global', 'stop', 'force', 'reload'] as const) {
test(`真实 WASM 等待规划时 ${action}:迟到结果不写控制/不步进`, async ({ page }, info) => {
await page.goto('/physics/agent.html');
await page.waitForFunction(() => Boolean(window.lekiwiAgent));
await page.evaluate(
(url) => window.lekiwiAgent.boot(url),
`/@fs${resolve('build/lekiwi/lekiwi-v1.zip')}`,
);
try {
const result = await page.evaluate((action) => window.lekiwiAgent.lifecycle(action), action);
await writeFile(
info.outputPath('late-response-evidence.json'),
JSON.stringify(result, null, 2),
);
expect(result.before).toEqual(result.after);
expect(result.before.time).toBe(0);
expect(result.jevCalls).toBe(0);
expect(result.state).toBe(action === 'pause-task' ? 'paused' : 'cancelled');
expect(result.owner).toBe(action === 'pause-task' ? 'agent' : 'manual');
} finally {
await page.evaluate(() => window.lekiwiAgent.dispose());
}
});
}
@@ -0,0 +1,79 @@
import { expect, test } from '@playwright/test';
import { resolve } from 'node:path';
import { readFile, writeFile } from 'node:fs/promises';
import { startLiveDecisionService } from './fixtures/decisionService';
test.use({ trace: 'off', video: 'off' });
test('显式 opt-in:主工作台 DeepSeek flash + OpenRouter Jev 真实物理回合', async ({
page,
}, info) => {
test.skip(
process.env.LEKIWI_LIVE_API !== '1',
'需明确授权 .env 两个角色与一次付费回合;不自动运行',
);
const service = await startLiveDecisionService();
try {
await page.goto('/');
await page
.locator('input[type="file"]')
.first()
.setInputFiles(resolve('build/lekiwi/lekiwi-v1.zip'));
const dialog = page.getByRole('dialog', { name: '配置 URDF 仿真组件' });
await expect(dialog).toBeVisible({ timeout: 30000 });
await dialog.getByLabel('URDF 机器人 profile').selectOption('lekiwi-v1');
await dialog.getByRole('button', { name: '转换并加载' }).click();
await expect(page.getByRole('button', { name: '▶ 播放' })).toBeEnabled({ timeout: 60000 });
await page.getByRole('tab', { name: '控制台', exact: true }).click();
await page.getByRole('button', { name: 'LeKiwi 智能抓放', exact: true }).click();
const panel = page.getByLabel('LeKiwi 智能抓放任务', { exact: true });
await panel.getByText('模型连接与官方订阅登录', { exact: true }).click();
await panel.getByLabel('本机模型服务', { exact: true }).fill(service.endpoint);
await panel.getByLabel('本机服务令牌', { exact: true }).fill(service.token);
await panel.getByRole('button', { name: '连接本机模型服务', exact: true }).click();
await expect(panel).toContainText('本机桥已连接');
await expect(panel.getByLabel('LLM 型号', { exact: true })).toHaveValue('deepseek-flash');
await expect(panel.getByLabel('JEV 型号', { exact: true })).toHaveValue('typesafe/jev-1.13');
await expect(panel.getByLabel('LLM Base URL', { exact: true })).toHaveValue(
'https://api.deepseek.com',
);
await expect(panel.getByLabel('JEV Base URL', { exact: true })).toHaveValue(
'https://openrouter.ai/api/alpha/decisions',
);
await panel.getByText('模型连接与官方订阅登录', { exact: true }).click();
await panel.getByLabel('智能任务决策模式').selectOption('api');
await panel.getByRole('button', { name: '开始任务(真实 API)', exact: true }).click();
const status = panel.getByTestId('agent-task-state');
await expect(status).toContainText(/物理验收成功|失败 \/ 安全停止|已取消/, { timeout: 240000 });
const pendingDownload = page.waitForEvent('download');
await panel.getByRole('button', { name: '导出任务证据', exact: true }).click();
const downloaded = await pendingDownload;
const data = JSON.parse(await readFile((await downloaded.path())!, 'utf8'));
const encoded = JSON.stringify(data, null, 2);
if (encoded.includes(service.token)) throw new Error('任务导出包含本机令牌');
await writeFile(info.outputPath('live-physical-evidence.json'), encoded);
console.log('真实 API 回合(非订阅)', {
state: data.state,
error: data.error,
llm: data.llmCalls,
jev: data.jevCalls,
simTime: data.sample.time,
transported: data.transported,
models: [...new Set(data.receipts.map((r: { model: string }) => r.model))],
});
expect(data.mode).toBe('api');
expect(data.state, data.error).toBe('succeeded');
expect(data.succeeded).toBe(true);
expect(data.transported).toBeGreaterThanOrEqual(0.5);
expect(data.llmCalls).toBe(1);
expect(data.jevCalls).toBe(11);
expect(data.receipts).toHaveLength(12);
await page.screenshot({ path: info.outputPath('live-workbench.png') });
} catch (error) {
// Playwright action errors may quote input values. Close before automatic DOM diagnostics.
await page.close().catch(() => undefined);
// eslint-disable-next-line preserve-caught-error -- The original cause may contain the private bridge token.
throw new Error(String(error).replaceAll(service.token, '[redacted]'));
} finally {
await service.stop();
}
});
+103
View File
@@ -0,0 +1,103 @@
import { test, expect } from '@playwright/test';
import { resolve } from 'node:path';
import { writeFile } from 'node:fs/promises';
import preset from '../../contracts/lekiwi-pick-place-v1.json' with { type: 'json' };
import type {} from '../physics/agent';
for (const seed of preset.frozenSeeds) {
test(`规则物理基线:冻结种子 ${seed.id},双指抓持/搬运/释放/撤离`, async ({ page }, info) => {
const errors: string[] = [];
page.on('pageerror', (error) => errors.push(error.message));
await page.goto('/physics/agent.html');
await page.waitForFunction(() => Boolean(window.lekiwiAgent));
await page.evaluate(
(url) => window.lekiwiAgent.boot(url),
`/@fs${resolve('build/lekiwi/lekiwi-v1.zip')}`,
);
try {
const result = await page.evaluate((seed) => window.lekiwiAgent.baseline(seed), seed.id);
await writeFile(info.outputPath('physical-evidence.json'), JSON.stringify(result, null, 2));
console.log('baseline', { ...result, trace: `${result.trace.length} control samples` });
expect(result.status, result.error).toBe('succeeded');
expect(result.succeeded).toBe(true);
expect(result.transported).toBeGreaterThanOrEqual(preset.transportMin);
expect(result.maxJointSpeed).toBeLessThanOrEqual(preset.jointSpeedStop);
const goal = preset.goal.position.map((v, i) => v + (seed.offset[i] ?? 0));
expect(Math.hypot(...result.final.object.map((v, i) => v - goal[i]))).toBeLessThan(
preset.placementTolerance,
);
expect(result.final.objectSpeed).toBeLessThan(preset.placementSpeedMax);
expect(result.final.onGoalSupport).toBe(true);
expect(result.final.fingerForces.every((force) => force < preset.fingerForceMin)).toBe(true);
expect(result.final.opening).toBeGreaterThan(0.65);
expect(
Math.hypot(...result.final.object.map((v, i) => v - result.final.tcp[i])),
).toBeGreaterThanOrEqual(preset.retreatMin);
expect(
result.trace.filter((s) => s.phase === 'carry' && s.secure && !s.supported).length,
).toBeGreaterThan(100);
expect(errors).toEqual([]);
} finally {
await page.evaluate(() => window.lekiwiAgent.dispose());
}
});
}
for (const seed of preset.frozenSeeds) {
test(`mock 决策闭环:冻结种子 ${seed.id},逐技能门禁与独立物理成功`, async ({ page }, info) => {
await page.goto('/physics/agent.html');
await page.waitForFunction(() => Boolean(window.lekiwiAgent));
await page.evaluate(
(url) => window.lekiwiAgent.boot(url),
`/@fs${resolve('build/lekiwi/lekiwi-v1.zip')}`,
);
try {
const result = await page.evaluate((id) => window.lekiwiAgent.task(id), seed.id);
await writeFile(info.outputPath('mock-closed-loop.json'), JSON.stringify(result, null, 2));
console.log('mock closed loop', {
state: result.state,
error: result.error,
transported: result.transported,
llm: result.llmCalls,
jev: result.jevCalls,
simTime: result.simTime,
});
expect(result.ownerAtStart).toBe('agent');
expect(result.mode).toBe('mock');
expect(result.state, result.error).toBe('succeeded');
expect(result.succeeded).toBe(true);
expect(result.transported).toBeGreaterThanOrEqual(preset.transportMin);
expect(result.maxJointSpeed).toBeLessThanOrEqual(preset.jointSpeedStop);
expect(result.llmCalls).toBe(1);
expect(result.jevCalls).toBe(11);
expect(result.results).toHaveLength(11);
expect(result.failures).toEqual([]);
expect(result.sample.onGoalSupport).toBe(true);
expect(result.sample.fingerForces.every((v) => v < preset.fingerForceMin)).toBe(true);
expect(result.sample.opening).toBeGreaterThanOrEqual(0.65);
expect(
Math.hypot(...result.sample.object.map((v, i) => v - result.sample.tcp[i])),
).toBeGreaterThanOrEqual(preset.retreatMin);
} finally {
await page.evaluate(() => window.lekiwiAgent.dispose());
}
});
}
test('agent 控制权下同一物理时钟完成基线,不建立第二个环境循环', async ({ page }, info) => {
await page.goto('/physics/agent.html');
await page.waitForFunction(() => Boolean(window.lekiwiAgent));
await page.evaluate(
(url) => window.lekiwiAgent.boot(url),
`/@fs${resolve('build/lekiwi/lekiwi-v1.zip')}`,
);
try {
const result = await page.evaluate(() => window.lekiwiAgent.baseline(0, true));
await writeFile(info.outputPath('governed-baseline.json'), JSON.stringify(result, null, 2));
expect(result.ownerAtStart).toBe('agent');
expect(result.status, result.error).toBe('succeeded');
expect(result.transported).toBeGreaterThanOrEqual(0.5);
} finally {
await page.evaluate(() => window.lekiwiAgent.dispose());
}
});
@@ -0,0 +1,103 @@
import { expect, test } from '@playwright/test';
import { resolve } from 'node:path';
import { readFile, writeFile } from 'node:fs/promises';
import { subscriptionOperation } from './fixtures/subscriptionSession';
test.use({ trace: 'off', video: 'off' });
test('显式 opt-in:已人工登录的 ChatGPT 订阅 + Jev 主视口物理验收与退出', async ({
page,
}, info) => {
test.skip(
process.env.LEKIWI_LIVE_SUBSCRIPTION !== '1',
'需要用户在隔离官方页面完成登录;不自动发起登录或使用全局账号',
);
const session = await subscriptionOperation('connect');
const token = String(session.token);
try {
expect(session.llmProtocol).toBe('codex');
const before = await subscriptionOperation('status');
expect(before.loggedIn).toBe(true);
const limitsBefore = await subscriptionOperation('limits');
await page.goto('/');
await page
.locator('input[type="file"]')
.first()
.setInputFiles(resolve('build/lekiwi/lekiwi-v1.zip'));
const dialog = page.getByRole('dialog', { name: '配置 URDF 仿真组件' });
await expect(dialog).toBeVisible({ timeout: 30000 });
await dialog.getByLabel('URDF 机器人 profile').selectOption('lekiwi-v1');
await dialog.getByRole('button', { name: '转换并加载' }).click();
await expect(page.getByRole('button', { name: '▶ 播放' })).toBeEnabled({ timeout: 60000 });
await page.getByRole('tab', { name: '控制台', exact: true }).click();
await page.getByRole('button', { name: 'LeKiwi 智能抓放', exact: true }).click();
const panel = page.getByLabel('LeKiwi 智能抓放任务', { exact: true });
await panel.getByText('模型连接与官方订阅登录', { exact: true }).click();
await panel.getByLabel('本机模型服务', { exact: true }).fill(String(session.address));
await panel.getByLabel('本机服务令牌', { exact: true }).fill(token);
await panel.getByRole('button', { name: '连接本机模型服务', exact: true }).click();
await expect(panel).toContainText('本机桥已连接');
await expect(panel.getByLabel('LLM 协议', { exact: true })).toHaveValue('codex');
await expect(panel.getByLabel('LLM API Key', { exact: true })).toHaveCount(0);
await panel.getByText(/ChatGPT 订阅 · 官方/).click();
await panel.getByRole('button', { name: '刷新账号', exact: true }).click();
await expect(panel).toContainText('ChatGPT 已登录(仅本次服务会话)');
await panel.getByRole('button', { name: '查询官方型号', exact: true }).click();
await expect(panel.getByLabel('LLM 型号', { exact: true })).toHaveValue(
String(session.llmModel),
);
await expect(panel.getByLabel('JEV 型号', { exact: true })).toHaveValue('typesafe/jev-1.13');
await panel.getByText('模型连接与官方订阅登录', { exact: true }).click();
await panel.getByLabel('智能任务决策模式').selectOption('api');
await panel.getByRole('button', { name: '开始任务(真实 API)', exact: true }).click();
await expect(panel.getByTestId('agent-task-state')).toContainText(
/物理验收成功|失败 \/ 安全停止|已取消/,
{ timeout: 240000 },
);
const download = page.waitForEvent('download');
await panel.getByRole('button', { name: '导出任务证据', exact: true }).click();
const data = JSON.parse(await readFile((await (await download).path())!, 'utf8'));
const after = await subscriptionOperation('status');
const evidence = {
source: 'real-chatgpt-subscription-plus-openrouter-jev',
before,
after,
limitsBefore,
...data,
};
const encoded = JSON.stringify(evidence, null, 2);
if (encoded.includes(token) || encoded.includes('auth.openai.com/'))
throw new Error('任务证据包含认证信息');
await writeFile(info.outputPath('subscription-physical-evidence.json'), encoded);
console.log('真实订阅回合', {
state: data.state,
error: data.error,
model: session.llmModel,
llm: data.llmCalls,
jev: data.jevCalls,
simTime: data.sample.time,
transported: data.transported,
});
expect(data.state, data.error).toBe('succeeded');
expect(data.succeeded).toBe(true);
expect(data.transported).toBeGreaterThanOrEqual(0.5);
expect(data.llmCalls).toBe(1);
expect(data.jevCalls).toBe(11);
expect(data.receipts).toHaveLength(12);
expect(data.receipts.find((r: { kind: string }) => r.kind === 'plan').provider).toBe('codex');
expect(after.checkedModels).toContain(session.llmModel);
await page.screenshot({ path: info.outputPath('subscription-workbench.png') });
await panel.getByText('模型连接与官方订阅登录', { exact: true }).click();
await panel.getByRole('button', { name: '退出隔离账号', exact: true }).click();
await expect(panel).toContainText('已退出本次隔离会话');
const loggedOut = await subscriptionOperation('status');
expect(loggedOut.loggedIn).toBe(false);
await writeFile(
info.outputPath('subscription-logout.json'),
JSON.stringify(loggedOut, null, 2),
);
} catch (error) {
await page.close().catch(() => undefined);
// eslint-disable-next-line preserve-caught-error -- Playwright causes can contain the private bridge token.
throw new Error(String(error).replaceAll(token, '[redacted]'));
}
});
@@ -0,0 +1,70 @@
import { expect, test } from '@playwright/test';
import { resolve } from 'node:path';
import { readFile, writeFile } from 'node:fs/promises';
test('主工作台:A 导入、mock 单技能暂停/继续、物理成功及证据导出', async ({ page }, info) => {
const errors: string[] = [];
let modelRequests = 0;
page.on('pageerror', (e) => errors.push(e.message));
page.on('request', (r) => {
if (/\/api\/decision\/v1\/(plan|decide)$/.test(r.url())) modelRequests++;
});
await page.goto('/');
await page
.locator('input[type="file"]')
.first()
.setInputFiles(resolve('build/lekiwi/lekiwi-v1.zip'));
const dialog = page.getByRole('dialog', { name: '配置 URDF 仿真组件' });
await expect(dialog).toBeVisible({ timeout: 30000 });
await dialog.getByLabel('URDF 机器人 profile').selectOption('lekiwi-v1');
await dialog.getByRole('button', { name: '转换并加载' }).click();
await expect(page.getByRole('button', { name: '▶ 播放' })).toBeEnabled({ timeout: 60000 });
await page.getByRole('tab', { name: '控制台', exact: true }).click();
await page.getByRole('button', { name: 'LeKiwi 智能抓放', exact: true }).click();
const panel = page.getByLabel('LeKiwi 智能抓放任务', { exact: true });
await expect(panel).toContainText('MOCK(不调用模型 API)');
await panel.getByRole('button', { name: '加载 A 任务场景', exact: true }).click();
await expect(panel).toContainText('A 抓放场景已加载并暂停', { timeout: 60000 });
await panel.getByRole('button', { name: '单技能步进', exact: true }).click();
const status = panel.getByTestId('agent-task-state');
await expect(status).toContainText('任务已暂停', { timeout: 60000 });
await expect(status).toContainText('当前技能:open');
await expect(status).toContainText('Jev 1/60');
await panel.getByRole('button', { name: '继续任务', exact: true }).click();
await expect(status).toContainText('物理验收成功', { timeout: 180000 });
await expect(status).toContainText('Jev 11/60');
const download = page.waitForEvent('download');
await panel.getByRole('button', { name: '导出任务证据', exact: true }).click();
const downloaded = await download;
const evidence = JSON.parse(await readFile((await downloaded.path())!, 'utf8'));
await writeFile(
info.outputPath('workbench-physical-evidence.json'),
JSON.stringify(evidence, null, 2),
);
expect(evidence.mode).toBe('mock');
expect(evidence.succeeded).toBe(true);
expect(evidence.transported).toBeGreaterThanOrEqual(0.5);
expect(evidence.sample.onGoalSupport).toBe(true);
expect(modelRequests).toBe(0);
expect(errors).toEqual([]);
await page.screenshot({ path: info.outputPath('agent-workbench.png') });
await panel.getByRole('button', { name: '重置任务场景', exact: true }).click();
await expect(panel.getByTestId('agent-task-state')).toHaveCount(0, { timeout: 60000 });
await panel.getByLabel('目标 Y', { exact: true }).fill('0.617');
await panel.getByRole('button', { name: '单技能步进', exact: true }).click();
await expect(status).toContainText('任务已暂停', { timeout: 60000 });
const changedDownload = page.waitForEvent('download');
await panel.getByRole('button', { name: '导出任务证据', exact: true }).click();
const changed = JSON.parse(await readFile((await (await changedDownload).path())!, 'utf8'));
expect(changed.goal).toEqual([0.257, 0.617, 0.128]);
expect(changed.runId).not.toBe(evidence.runId);
expect(changed.sample.time).toBeLessThan(2.01);
expect(changed.sample.object[1]).toBeLessThan(0.02); // New source episode, not the old placed cube.
await writeFile(info.outputPath('edited-goal-episode.json'), JSON.stringify(changed, null, 2));
await panel.getByLabel('目标 Y', { exact: true }).fill('0.615');
await expect(status).toContainText('已取消');
await expect(status).toContainText('目标编辑使旧计划失效');
expect(modelRequests).toBe(0);
expect(errors).toEqual([]);
await expect(page.getByRole('button', { name: '强化学习任务', exact: true })).toBeVisible();
});
@@ -0,0 +1,76 @@
import { test, expect } from '@playwright/test';
import { resolve } from 'node:path';
import { writeFile } from 'node:fs/promises';
test('完整 CAD 主视口:60 秒渲染/物理性能记录(不承诺固定 FPS)', async ({ page }, info) => {
const errors: string[] = [];
page.on('pageerror', (error) => errors.push(error.message));
await page.goto('/');
const started = performance.now();
await page
.getByLabel('打开文件', { exact: true })
.setInputFiles(resolve('build/lekiwi/lekiwi-v1.zip'));
const dialog = page.getByRole('dialog', { name: '配置 URDF 仿真组件' });
await expect(dialog).toBeVisible({ timeout: 30000 });
await dialog.getByLabel('URDF 机器人 profile').selectOption('lekiwi-v1');
await dialog.getByRole('button', { name: '转换并加载' }).click();
await expect(page.getByRole('button', { name: '▶ 播放' })).toBeEnabled({ timeout: 60000 });
const loadMs = performance.now() - started;
await page.getByRole('button', { name: '▶ 播放' }).click();
const result = await page.evaluate(async () => {
const path = '/src/stores/useAppStore.ts';
const { useAppStore } = (await import(
/* @vite-ignore */ path
)) as typeof import('../src/stores/useAppStore');
const canvas = document.querySelector('main canvas') as HTMLCanvasElement;
const gl = canvas.getContext('webgl2')!,
ext = gl.getExtension('WEBGL_debug_renderer_info');
const graphics = {
size: [canvas.width, canvas.height],
renderer: ext ? String(gl.getParameter(ext.UNMASKED_RENDERER_WEBGL)) : 'unknown',
};
const start = performance.now(),
frames: number[] = [],
steps: number[] = [];
const longTasks: number[] = [];
const observer = new PerformanceObserver((list) => {
longTasks.push(...list.getEntries().map((e) => e.duration));
});
observer.observe({ type: 'longtask' });
let previous = start;
await new Promise<void>((resolve) => {
const tick = (now: number) => {
frames.push(now - previous);
previous = now;
steps.push(useAppStore.getState().stepMs);
if (now - start >= 60000) resolve();
else requestAnimationFrame(tick);
};
requestAnimationFrame(tick);
});
observer.disconnect();
const p95 = (values: number[]) =>
[...values].sort((a, b) => a - b)[Math.floor(values.length * 0.95)] ?? 0;
const state = useAppStore.getState();
return {
graphics,
frames: frames.length,
fps: (1000 * frames.length) / (previous - start),
frameP95Ms: p95(frames),
stepP95Ms: p95(steps),
longTasks,
heap: state.snapshot?.wasmHeapBytes,
simTime: state.snapshot?.time,
diagnostic: state.diagnostic?.summary,
};
});
await writeFile(
info.outputPath('full-cad-performance.json'),
JSON.stringify({ loadMs, ...result }, null, 2),
);
console.log('full CAD performance', { loadMs, ...result });
await page.screenshot({ path: info.outputPath('full-cad.png') });
expect(errors).toEqual([]);
expect(result.diagnostic).toBeUndefined();
expect(result.simTime).toBeGreaterThan(0);
});
+58
View File
@@ -0,0 +1,58 @@
import { expect, test } from '@playwright/test';
import { resolve } from 'node:path';
import { writeFile } from 'node:fs/promises';
import type {} from '../physics/runner';
test('完整 314244 面三轮:真实 WASM、随轴运动、ZIP 重导与有界资源', async ({ page }, info) => {
const errors: string[] = [];
page.on('pageerror', (error) => errors.push(error.message));
await page.goto('/physics/runner.html');
await page.waitForFunction(() => Boolean(window.lekiwiPhysics));
await page.evaluate((path) => window.lekiwiPhysics.boot(path), `/@fs${resolve('build/lekiwi')}`);
try {
const before = await page.evaluate(() => window.lekiwiPhysics.visualStats());
// The three original filenames contain identical bytes; one content-addressed
// OBJ is shared by three independent visual instances.
expect(before.generatedMeshes).toHaveLength(1);
expect(before.wheels).toHaveLength(3);
for (const wheel of before.wheels) {
expect(wheel.faces).toBe(314244);
expect(wheel.vertices).toBeGreaterThan(150000);
expect(wheel.centerError).toBeLessThan(1e-6);
expect(wheel.scale).toEqual([0.001, 0.001, 0.001]);
expect(wheel.collision).toBe(false);
expect(wheel.mass).toBe(0);
}
await page.evaluate(() => window.lekiwiPhysics.drive(0, 0.1, 0, 0.25));
const after = await page.evaluate(() => window.lekiwiPhysics.visualStats());
for (let i = 0; i < 3; i++) {
expect(after.wheels[i].centerError).toBeLessThan(1e-6);
expect(
after.wheels[i].rotation.some((v, j) => Math.abs(v - before.wheels[i].rotation[j]) > 0.01),
).toBe(true);
}
const traces: Awaited<ReturnType<typeof window.lekiwiPhysics.packageRoundtrip>>[] = [];
for (let i = 0; i < 8; i++) {
const result = await page.evaluate(() => window.lekiwiPhysics.packageRoundtrip());
expect(result.oldModelDeleted).toBe(true);
expect(result.oldDataDeleted).toBe(true);
expect(result.workspaces).toHaveLength(1);
expect(result.generatedMeshes).toHaveLength(1);
expect(result.wheels.every((wheel) => wheel.faces === 314244 && !wheel.collision)).toBe(true);
traces.push(result);
}
console.log(
'mesh resources',
traces.map((t) => ({ heap: t.heap, bytes: t.projectBytes, xml: t.xmlFiles })),
);
await writeFile(
info.outputPath('full-mesh-roundtrip.json'),
JSON.stringify({ before, after, traces }, null, 2),
);
expect(traces.every((t) => t.projectBytes === traces[0].projectBytes)).toBe(true);
expect(traces[7].heap).toBeLessThanOrEqual(traces[5].heap + 32 * 1024 * 1024);
expect(errors).toEqual([]);
} finally {
await page.evaluate(() => window.lekiwiPhysics.dispose());
}
});
+187
View File
@@ -0,0 +1,187 @@
import { expect, test } from '@playwright/test';
import { spawn } from 'node:child_process';
import { createServer } from 'node:net';
import { existsSync } from 'node:fs';
import { resolve } from 'node:path';
import { mkdir, readFile, readdir, writeFile } from 'node:fs/promises';
const mobilePython = process.env.MOBILE_TRAINING_PYTHON ?? resolve('build/venvs/mobile/bin/python');
const bundle =
process.env.MOBILE_BUNDLE_ZIP ??
resolve('../Reference_Projects/LeKiwi/New_urdf/robot_urdf_bundle (1).zip');
for (const [taskId, file] of [
['MobileManipulator-LeKiwi-v1', resolve('build/lekiwi/lekiwi-v1.zip')],
['MobileManipulator-LeKiwi-Bundle', bundle],
]) {
test(`主控制台真实一键训练及策略导入:${taskId}`, async ({ page }, info) => {
test.skip(
!existsSync(mobilePython) || !existsSync(file),
'需要 MuJoCo 3.11/SB3 独立环境及机器人资产',
);
const port = await new Promise<number>((resolvePort) => {
const listener = createServer();
listener.listen(0, '127.0.0.1', () => {
const port = (listener.address() as { port: number }).port;
listener.close(() => resolvePort(port));
});
});
const root = info.outputPath('training');
await mkdir(root, { recursive: true });
const token = 'oneclick-test-token';
const endpoint = `http://127.0.0.1:${port}`;
const process = spawn(
resolve('.venv/bin/python'),
[
'-u',
'training_server/server.py',
'--port',
String(port),
'--trainer-root',
root,
'--mobile-python',
mobilePython,
'--token',
token,
],
{ stdio: ['ignore', 'pipe', 'pipe'] },
);
let output = '';
process.stdout.on('data', (data) => {
output += String(data);
});
process.stderr.on('data', (data) => {
output += String(data);
});
const errors: string[] = [];
page.on('pageerror', (error) => errors.push(error.message));
try {
await expect
.poll(
async () => {
try {
return (
await fetch(`${endpoint}/api/training/health`, {
headers: { Authorization: `Bearer ${token}` },
})
).ok;
} catch {
return false;
}
},
{ timeout: 30000 },
)
.toBe(true);
await page.goto('/');
await page.locator('input[type="file"]').first().setInputFiles(file);
const dialog = page.getByRole('dialog', { name: '配置 URDF 仿真组件' });
await expect(dialog).toBeVisible({ timeout: 30000 });
// v1 also covers migration from the existing external-control profile at start.
await dialog
.getByLabel('URDF 机器人 profile')
.selectOption(taskId === 'MobileManipulator-LeKiwi-v1' ? 'lekiwi-v1' : taskId);
await dialog.getByRole('button', { name: '转换并加载' }).click();
await expect(page.getByRole('button', { name: '▶ 播放' })).toBeEnabled({ timeout: 60000 });
await page.getByRole('tab', { name: '控制台', exact: true }).click();
if (taskId === 'MobileManipulator-LeKiwi-v1') {
await page.getByRole('button', { name: 'LeKiwi 智能抓放', exact: true }).click();
const agent = page.getByLabel('LeKiwi 智能抓放任务', { exact: true });
await agent.getByRole('button', { name: '单技能步进', exact: true }).click();
await expect(agent.getByTestId('agent-task-state')).toContainText('任务已暂停', {
timeout: 60000,
});
}
await page.getByRole('button', { name: '强化学习任务', exact: true }).click();
await page.getByLabel('本地训练服务地址').fill(endpoint);
await page.getByLabel('训练服务访问令牌').fill(token);
await page.getByRole('button', { name: '连接', exact: true }).click();
await expect(page.getByLabel('训练任务')).toHaveValue(taskId);
await page.getByLabel('训练迭代', { exact: true }).fill('2');
await page.getByLabel('每环境采样步数').fill('8');
await expect(page.getByLabel('移动操作训练阶段')).toHaveValue('navigate');
await page.getByLabel('独立评估回合').fill('2');
await page.getByLabel('导航启动示教步数(仅初训)').fill('0');
const jobResponse = page.waitForResponse(
(response) =>
response.url() === `${endpoint}/api/training/jobs` &&
response.request().method() === 'POST',
);
await page.getByRole('button', { name: '发起本地训练' }).click();
const created = (await (await jobResponse).json()) as { id: string };
await expect(page.getByRole('button', { name: '导入策略', exact: true })).toBeEnabled({
timeout: 60000,
});
await page.getByRole('button', { name: '导入策略', exact: true }).click();
await expect
.poll(
async () =>
page.evaluate(async () => {
const path = '/src/stores/useAppStore.ts';
const { useAppStore } = (await import(
/* @vite-ignore */ path
)) as typeof import('../src/stores/useAppStore');
return useAppStore.getState().snapshot?.rlPolicy?.inferenceCount ?? 0;
}),
{ timeout: 30000 },
)
.toBeGreaterThan(0);
const job = await (
await fetch(`${endpoint}/api/training/jobs/${created.id}`, {
headers: { Authorization: `Bearer ${token}` },
})
).json();
expect(job.state).toBe('succeeded');
expect(job.logs.join('\n')).toContain('Mean reward:');
expect(job.deployment.input.shape).toEqual([1, 92]);
expect(job.deployment.trainingStage).toBe('navigate');
expect(job.deployment.evaluation.episodes).toBe(2);
expect(job.deployment.evaluation.safetyStops).toBe(0);
expect(job.deployment.evaluation.maxJointVelocity).toBeLessThan(2);
expect(job.deployment.output.shape).toEqual([1, 12]);
if (taskId === 'MobileManipulator-LeKiwi-v1') {
const scenes = (await readdir(root, { recursive: true })).filter((name) =>
name.endsWith('.training.xml'),
);
expect(scenes.length).toBeGreaterThan(0);
for (const scene of scenes)
expect(await readFile(resolve(root, scene), 'utf8')).not.toContain('__agent_');
await expect(page.getByTestId('agent-task-state')).toContainText('已取消');
}
await writeFile(info.outputPath('job.json'), JSON.stringify(job, null, 2));
await page.screenshot({ path: info.outputPath('unified-training.png') });
expect(errors).toEqual([]);
if (taskId === 'MobileManipulator-LeKiwi-v1') {
await page.getByRole('button', { name: '接续此作业(保留权重)' }).click();
await expect(page.getByLabel('接续作业 ID')).toHaveValue(created.id);
const resumedResponse = page.waitForResponse(
(response) =>
response.url() === `${endpoint}/api/training/jobs` &&
response.request().method() === 'POST',
);
await page.getByRole('button', { name: '发起本地训练' }).click();
const response = await resumedResponse;
expect(response.status()).toBe(202);
const resumed = await response.json();
await expect(page.getByRole('button', { name: '导入策略', exact: true })).toBeEnabled({
timeout: 60000,
});
const second = await (
await fetch(`${endpoint}/api/training/jobs/${resumed.id}`, {
headers: { Authorization: `Bearer ${token}` },
})
).json();
expect(second.state).toBe('succeeded');
expect(second.deployment.initialization).toBe('resumed-checkpoint');
expect(second.deployment.trainedTimesteps).toBeGreaterThan(job.deployment.trainedTimesteps);
await writeFile(info.outputPath('resumed-job.json'), JSON.stringify(second, null, 2));
}
} finally {
await page.close();
process.kill('SIGTERM');
await new Promise<void>((resolveExit) => {
if (process.exitCode !== null) resolveExit();
else process.once('exit', () => resolveExit());
});
await writeFile(info.outputPath('server.log'), output.replaceAll(token, '<redacted>'));
}
});
}
+137
View File
@@ -0,0 +1,137 @@
import { expect, test } from '@playwright/test';
import { createHash } from 'node:crypto';
import { readFile, writeFile } from 'node:fs/promises';
import { readdirSync } from 'node:fs';
import { resolve } from 'node:path';
import type {} from '../physics/runner';
import type { ProjectManifest } from '../src/project/types';
const assetBase = `/@fs${resolve(process.cwd(), 'build/lekiwi')}`;
const hash = (text: string) => createHash('sha256').update(text).digest('hex');
test('等价 URDF 变体:真实 WASM 控制、输入溯源及 MJCF 导出重载', async ({ page }, info) => {
const original = await readFile('robot_profiles/lekiwi-reference.urdf', 'utf8');
const variant = original
.replace('name="LeKiwi"', 'name="LeKiwi 换色变体"')
.replace('</visual>', '<material name="red"><color rgba="1 0 0 1"/></material></visual>');
await page.route('**/URDF/LeKiwi.urdf', (route) =>
route.fulfill({ body: variant, contentType: 'application/xml' }),
);
await page.goto('/physics/runner.html');
await page.waitForFunction(() => Boolean(window.lekiwiPhysics));
const boot = await page.evaluate((path) => window.lekiwiPhysics.boot(path), assetBase);
expect(boot.nu).toBe(9);
expect(boot.njnt).toBe(46);
const stable = await page.evaluate(() => window.lekiwiPhysics.step(3));
expect(stable.finite).toBe(true);
const drive = await page.evaluate(() => window.lekiwiPhysics.drive(0.1, 0, 0, 2));
expect(drive.position[0] - stable.position[0]).toBeGreaterThan(0.15);
const xml = await page.evaluate(() => window.lekiwiPhysics.xml());
expect(xml).toContain(`name="platform_robot_source_sha256" data="${hash(original)}"`);
expect(xml).toContain(`name="platform_robot_input_sha256" data="${hash(variant)}"`);
const reloaded = await page.evaluate(() => window.lekiwiPhysics.reimport());
expect(reloaded.robot?.profileId).toBe('lekiwi-v1');
expect(reloaded.nu).toBe(9);
expect(await page.evaluate(() => window.lekiwiPhysics.xml())).toContain(hash(variant));
await writeFile(info.outputPath('variant.xml'), xml);
await writeFile(
info.outputPath('variant-control.json'),
JSON.stringify({ boot, stable, drive, reloaded }, null, 2),
);
await page.evaluate(() => window.lekiwiPhysics.dispose());
});
test('同名底盘 STL 被改动也拒绝旧 profile,不只校验机械臂网格', async ({ page }) => {
await page.route('**/URDF/meshes/base_plate_layer1-v5.stl', (route) =>
route.fulfill({ body: 'changed mesh bytes' }),
);
await page.goto('/physics/runner.html');
await page.waitForFunction(() => Boolean(window.lekiwiPhysics));
await expect(page.evaluate((path) => window.lekiwiPhysics.boot(path), assetBase)).rejects.toThrow(
/SHA-256 不匹配.*base_plate_layer1/,
);
await page.evaluate(() => window.lekiwiPhysics.dispose());
});
// Optional local evidence: assets stay outside the repository. Setting the env
// explicitly makes missing/invalid samples fail, rather than silently skipping.
const localDirectory = process.env.LEKIWI_VARIANT_DIR;
const localSamples = localDirectory
? readdirSync(localDirectory)
.filter((file) => file.endsWith('.zip'))
.sort()
: [];
if (localDirectory && !localSamples.length) throw new Error('LEKIWI_VARIANT_DIR 中没有 ZIP 样本');
for (const name of localSamples) {
test(`用户变体包诊断(不适配旧控制):${name}`, async ({ page }, info) => {
await page.route('**/__local_variant.zip', (route) =>
route.fulfill({ path: resolve(localDirectory!, name), contentType: 'application/zip' }),
);
await page.goto('/physics/runner.html');
const result = await page.evaluate(async () => {
const importerPath = '/src/project/importer.ts';
const profilesPath = '/src/project/robotProfiles/index.ts';
const physicsPath = '/src/simulation/PhysicsAdapter.ts';
const registryPath = '/src/robot/registry.ts';
const { importBrowserFiles, prepareProjectForMujoco } = await import(
/* @vite-ignore */ importerPath
);
const { prepareRobotProject } = await import(/* @vite-ignore */ profilesPath);
const { MainThreadPhysicsAdapter } = await import(/* @vite-ignore */ physicsPath);
const { sha256 } = await import(/* @vite-ignore */ registryPath);
const bytes = await (await fetch('/__local_variant.zip')).arrayBuffer();
const manifest: ProjectManifest = await importBrowserFiles([
new File([bytes], 'variant.zip'),
]);
const entry = manifest.selectedEntry!;
const data = manifest.files.find((file) => file.path === entry)!.data;
const sourceHash = await sha256(data);
const doc = new DOMParser().parseFromString(
new TextDecoder().decode(data),
'application/xml',
);
const movable = Array.from(doc.querySelectorAll('robot > joint')).filter(
(j) => j.getAttribute('type') !== 'fixed',
);
let profileError = '';
try {
await prepareRobotProject(manifest, entry, 'lekiwi-v1');
} catch (error) {
profileError = String(error);
}
const prepared = await prepareProjectForMujoco(manifest, entry);
const fixedText = new TextDecoder().decode(
prepared.manifest.files.find((file: { path: string }) => file.path === entry).data,
);
const adapter = new MainThreadPhysicsAdapter();
let genericCompileError = '';
try {
await adapter.load(manifest, entry, { urdfMode: 'mjcf', baseMode: 'floating' });
} catch (error) {
genericCompileError = String(error);
} finally {
adapter.dispose();
}
return {
entry,
sourceHash,
profileError,
genericCompileError,
links: doc.querySelectorAll('robot > link').length,
movable: movable.map((j) => j.getAttribute('name')),
warnings: prepared.warnings,
unresolvedPackageUri: fixedText.includes('package://'),
sourceUnmodified: (await sha256(data)) === sourceHash,
};
});
expect(result.links).toBe(35);
expect(result.movable).toHaveLength(10);
expect(result.profileError).toContain('需要 9 个活动关节,实际 10 个');
expect(result.profileError).toContain('Link7');
expect(result.unresolvedPackageUri).toBe(false);
expect(result.sourceUnmodified).toBe(true);
// This is an import repair, not an unreviewed wheel-mesh simplification.
expect(result.genericCompileError).toMatch(/STL|faces/i);
await writeFile(info.outputPath('local-variant-audit.json'), JSON.stringify(result, null, 2));
});
}
+181
View File
@@ -0,0 +1,181 @@
import { expect, test, type Page } from '@playwright/test';
import { existsSync } from 'node:fs';
import { mkdir, readFile, writeFile } from 'node:fs/promises';
import { resolve } from 'node:path';
import type {} from '../physics/mobile';
const original = resolve('build/lekiwi/lekiwi-v1.zip');
const bundle =
process.env.MOBILE_BUNDLE_ZIP ??
resolve('../Reference_Projects/LeKiwi/New_urdf/robot_urdf_bundle (1).zip');
async function exportRollout(page: Page, zipName: string, traceName: string) {
await page.evaluate(() => window.mobilePhysics.reset());
await mkdir('build/mobile-validation', { recursive: true });
const downloadPromise = page.waitForEvent('download');
await page.evaluate(() => window.mobilePhysics.download());
await (await downloadPromise).saveAs(`build/mobile-validation/${zipName}`);
const actions = Array.from({ length: 12 }, (_, step) => {
const a = new Array<number>(12).fill(0);
a[0] = 0.1;
a[3] = 0.03 * Math.sin(step);
a[11] = 1;
return a;
});
const rollout = [];
for (const action of actions)
rollout.push(await page.evaluate((a) => window.mobilePhysics.step(1, a), action));
await writeFile(
`build/mobile-validation/${traceName}`,
JSON.stringify({ actions, rollout }, null, 2),
);
}
test('真实 WASM:两种机器人、控制、拖动、失败回滚、重复切换与训练包导出', async ({ page }) => {
test.skip(!existsSync(original) || !existsSync(bundle), '需要本地两个机器人 ZIP');
const errors: string[] = [];
page.on('pageerror', (e) => errors.push(e.message));
await page.route('**/original.zip', (route) => route.fulfill({ path: original }));
await page.route('**/bundle.zip', (route) => route.fulfill({ path: bundle }));
await page.goto('/physics/mobile.html');
await page.waitForFunction(() => Boolean(window.mobilePhysics));
const traces = [];
for (const id of [
'lekiwi-v1',
'lekiwi-bundle',
'lekiwi-v1',
'lekiwi-bundle',
'lekiwi-v1',
'lekiwi-bundle',
'lekiwi-v1',
'lekiwi-bundle',
// Complete CAD meshes increase allocator/cache warm-up; exercise extra
// cycles without raising the peak-memory bound or weakening final stability.
'lekiwi-v1',
'lekiwi-bundle',
'lekiwi-v1',
'lekiwi-bundle',
]) {
const initial = await page.evaluate(
(id) => window.mobilePhysics.boot(id === 'lekiwi-v1' ? '/original.zip' : '/bundle.zip', id),
id,
);
console.log('mobile heap', id, initial.heap / 1048576);
expect(initial.heap).toBeLessThan(1600 * 1048576);
expect(initial.observation).toHaveLength(92);
expect(initial.observation.every(Number.isFinite)).toBe(true);
expect(initial.ctrl).toHaveLength(9);
expect(initial.retiredModelDeleted).toBe(true);
expect(initial.retiredDataDeleted).toBe(true);
const stable = await page.evaluate(() => window.mobilePhysics.step(10));
expect(stable.time).toBeCloseTo(0.2, 8);
expect(stable.qpos.every(Number.isFinite)).toBe(true);
const action = new Array<number>(12).fill(0);
action[0] = 0.3;
action[3] = 0.2;
action[11] = 1;
const moved = await page.evaluate((a) => window.mobilePhysics.step(10, a, true), action);
expect(moved.ctrl.some((v, i) => Math.abs(v - initial.ctrl[i]) > 0.01)).toBe(true);
expect(moved.qpos.some((v, i) => Math.abs(v - stable.qpos[i]) > 0.001)).toBe(true);
action[0] = 0;
action[11] = -1;
const closed = await page.evaluate((a) => window.mobilePhysics.step(80, a, true), action);
expect(closed.state[29]).toBeLessThan(initial.state[29] - 0.2);
expect(closed.qpos.every(Number.isFinite)).toBe(true);
expect(await page.evaluate(() => window.mobilePhysics.workspaces())).toHaveLength(1);
const edited = await page.evaluate(() => window.mobilePhysics.move('goal', [0.4, -0.2, 0.019]));
expect(edited.state.slice(47, 50)).toEqual([0.4, -0.2, 0.019]);
expect(edited.stage).toBe('reach');
traces.push({ initial, stable, moved, closed, edited });
if (traces.length === 1)
await exportRollout(page, 'lekiwi-training.zip', 'lekiwi-rollout.json');
}
const previous = await page.evaluate(() => window.mobilePhysics.state());
const failure = await page.evaluate(async () => {
try {
await window.mobilePhysics.boot('/original.zip', 'lekiwi-bundle');
return '';
} catch (e) {
return String(e);
}
});
expect(failure).toContain('Link1');
expect((await page.evaluate(() => window.mobilePhysics.state())).robotId).toBe(previous.robotId);
expect(await page.evaluate(() => window.mobilePhysics.workspaces())).toHaveLength(1);
const compileFailure = await page.evaluate(async () => {
try {
await window.mobilePhysics.invalidCandidate('/bundle.zip');
return '';
} catch (e) {
return String(e);
}
});
expect(compileFailure).toContain('missing_actuator');
expect(await page.evaluate(() => window.mobilePhysics.workspaces())).toHaveLength(1);
expect(
(await page.evaluate(() => window.mobilePhysics.step(1))).observation.every(Number.isFinite),
).toBe(true);
await writeFile(
'build/mobile-validation/heap-warmup.json',
JSON.stringify(
traces.map((t) => ({ robot: t.initial.robotId, heap: t.initial.heap })),
null,
2,
),
);
expect(traces[11].initial.heap).toBe(traces[9].initial.heap);
await exportRollout(page, 'bundle-training.zip', 'rollout.json');
await writeFile('build/mobile-validation/traces.json', JSON.stringify(traces, null, 2));
await page.evaluate(() => {
window.mobilePhysics.reset();
window.mobilePhysics.show();
});
await page.waitForFunction(() => document.querySelector('canvas')!.width > 500);
// Use actual camera projection, not hard-coded screenshot pixel coordinates.
for (const entity of ['object', 'goal'] as const) {
const before = await page.evaluate(() => window.mobilePhysics.state());
const p = await page.evaluate((entity) => window.mobilePhysics.screen(entity), entity);
await page.keyboard.down('Shift');
await page.mouse.move(p.x, p.y);
await page.mouse.down();
await page.mouse.move(p.x + 45, p.y + 12, { steps: 5 });
await page.mouse.up();
await page.keyboard.up('Shift');
const after = await page.evaluate(() => window.mobilePhysics.state());
const offset = entity === 'object' ? 37 : 47;
expect(
Math.hypot(
after.state[offset] - before.state[offset],
after.state[offset + 1] - before.state[offset + 1],
),
).toBeGreaterThan(0.01);
}
expect(errors).toEqual([]);
await page.evaluate(() => window.mobilePhysics.dispose());
});
// 独立产品页已移除;导入/控制/暂停/重置/状态 UI 在主工作台用例验收:
// lekiwi.agent.workspace.spec.ts。这里保留两模型物理、训练快照及 ORT 夹具。
test('真实 ORT WASM:导出的 smoke 权重锁步执行(不代表训练成功)', async ({ page }) => {
const model = resolve('build/mobile-validation/smoke.onnx');
test.skip(!existsSync(model) || !existsSync(bundle), '先按文档导出 smoke.onnx');
const metadata = JSON.parse(await readFile('build/mobile-validation/smoke.json', 'utf8'));
await page.route('**/bundle.zip', (route) => route.fulfill({ path: bundle }));
await page.route('**/smoke.onnx', (route) => route.fulfill({ path: model }));
await page.goto('/physics/mobile.html');
await page.waitForFunction(() => Boolean(window.mobilePhysics));
await page.evaluate(() => window.mobilePhysics.boot('/bundle.zip', 'lekiwi-bundle'));
await page.evaluate((metadata) => window.mobilePhysics.policy('/smoke.onnx', metadata), metadata);
await expect
.poll(
async () => (await page.evaluate(() => window.mobilePhysics.tick(performance.now()))).time,
{ timeout: 30_000, intervals: [20] },
)
.toBeGreaterThanOrEqual(0.1);
const status = await page.evaluate(() => window.mobilePhysics.tick(performance.now()));
expect(status.metrics?.error).toBe('');
expect(status.metrics!.inferenceCount).toBeGreaterThanOrEqual(5);
await writeFile('build/mobile-validation/onnx-metrics.json', JSON.stringify(status, null, 2));
await page.evaluate(() => window.mobilePhysics.dispose());
});
+141
View File
@@ -0,0 +1,141 @@
import { expect, test } from '@playwright/test';
import { resolve } from 'node:path';
import { existsSync } from 'node:fs';
import { readFile, writeFile } from 'node:fs/promises';
test.skip(
process.env.CADWORLD_PRODUCTION_TEST !== '1',
'真实站点检查须显式 opt-in;不使用任何现存密钥',
);
test('生产 HTTPS:模型导入、Pyodide、精简设置、安全会话与订阅不可用边界', async ({
page,
}, info) => {
const errors: string[] = [];
const external: string[] = [];
page.on('pageerror', (e) => errors.push(e.message));
page.on('request', (r) => {
if (
r.url().startsWith('http') &&
new URL(r.url()).origin !== 'https://cadworld-sim.robotquan.com'
)
external.push(r.url());
});
const response = await page.goto('/');
expect(response?.status()).toBe(200);
await expect(page).toHaveTitle(/CADWorld Sim/);
await page
.locator('input[type=file]')
.first()
.setInputFiles(resolve('web_platform/fixtures/python_controller/model.xml'));
await expect(page.getByRole('button', { name: '▶ 播放' })).toBeEnabled({ timeout: 90000 });
await page.getByRole('tab', { name: '控制台', exact: true }).click();
await expect(page.getByRole('button', { name: '强化学习任务', exact: true })).toHaveCount(0);
await page.getByRole('button', { name: /Python 脚本控制/ }).click();
await page
.locator('input[accept=".py,text/x-python"]')
.setInputFiles(resolve('web_platform/fixtures/python_controller/balance.py'));
await expect(page.getByText('倒立摆 PD 平衡控制', { exact: true })).toBeVisible({
timeout: 90000,
});
await page.getByRole('button', { name: '启用', exact: true }).click();
await expect(page.getByRole('button', { name: /Python 脚本控制/ })).toContainText('运行');
await page.getByRole('button', { name: 'LeKiwi 智能抓放', exact: true }).click();
await page.getByText('模型设置 · API_KEY / ChatGPT 订阅', { exact: true }).click();
await expect(page.getByRole('button', { name: '保存设置' })).toBeEnabled({ timeout: 20000 });
await expect(page.getByLabel('本机服务令牌')).toHaveCount(0);
const cookie = (await page.context().cookies()).find((c) => c.name === '__Host-cadworld-session');
expect(cookie?.secure).toBe(true);
expect(cookie?.httpOnly).toBe(true);
expect(cookie?.sameSite).toBe('Strict');
await page.screenshot({ path: info.outputPath('website-api-settings.png') });
await page.getByRole('button', { name: 'ChatGPT 订阅', exact: true }).click();
// No login or model invocation. Availability must be explicit; user's login stays interactive.
await expect(page.getByText(/当前服务器可能无法连接官方服务/)).toBeVisible();
await expect(page.getByRole('button', { name: '登录 ChatGPT 订阅', exact: true })).toBeVisible();
await page.screenshot({ path: info.outputPath('website-subscription-entry.png') });
await page.getByText('连接测试与凭据管理', { exact: true }).click();
await page.getByRole('button', { name: '清除本次凭据', exact: true }).click();
await expect(page.getByText(/本次凭据已清除/)).toBeVisible();
expect(errors).toEqual([]);
expect(external).toEqual([]);
});
test('官方订阅入口网络探测:不登录账号、不推理,完成后销毁会话', async ({ page }, info) => {
await page.goto('/');
const evidence = await page.evaluate(async () => {
const prefix = '/api/decision/v1';
const headers = { 'Content-Type': 'application/json' };
const bootstrap = await fetch(prefix + '/session', { method: 'POST', headers, body: '{}' });
const session = await bootstrap.json();
const protectedHeaders = {
...headers,
'X-CSRF-Token': session.csrfToken,
'X-Config-Version': String(session.configVersion),
};
const response = await fetch(prefix + '/codex/login', {
method: 'POST',
headers: protectedHeaders,
body: '{}',
});
const result = await response.json();
const cleared = await fetch(prefix + '/session', {
method: 'DELETE',
headers: protectedHeaders,
body: '{}',
});
return {
status: response.status,
error: typeof result.error === 'string' ? result.error : undefined,
deviceFlowOffered: Boolean(result.verificationUrl && result.userCode),
cleared: cleared.status,
accountLoginPerformed: false,
inferenceCalls: 0,
};
});
expect(evidence.cleared).toBe(200);
expect([200, 409, 429, 502, 504]).toContain(evidence.status);
if (evidence.status !== 200) expect(evidence.error).toMatch(/^[a-z0-9_]+$/);
await writeFile(
info.outputPath('subscription-network-status.json'),
JSON.stringify(evidence, null, 2),
);
});
test('公网真实 WASM A 抓放(明确 mock,不调用付费 API)', async ({ page }, info) => {
test.skip(!existsSync('build/lekiwi/lekiwi-v1.zip'), '无本地 A 资产');
let modelCalls = 0;
page.on('request', (r) => {
if (/\/api\/decision\/v1\/(plan|decide|test)$/.test(r.url())) modelCalls++;
});
await page.goto('/');
await page
.locator('input[type=file]')
.first()
.setInputFiles(resolve('build/lekiwi/lekiwi-v1.zip'));
const dialog = page.getByRole('dialog', { name: '配置 URDF 仿真组件' });
await expect(dialog).toBeVisible({ timeout: 30000 });
await dialog.getByLabel('URDF 机器人 profile').selectOption('lekiwi-v1');
await dialog.getByRole('button', { name: '转换并加载' }).click();
await expect(page.getByRole('button', { name: '▶ 播放' })).toBeEnabled({ timeout: 90000 });
await page.getByRole('tab', { name: '控制台', exact: true }).click();
await page.getByRole('button', { name: 'LeKiwi 智能抓放', exact: true }).click();
const panel = page.getByLabel('LeKiwi 智能抓放任务', { exact: true });
await panel.getByRole('button', { name: '开始任务(mock)', exact: true }).click();
await expect(panel.getByTestId('agent-task-state')).toContainText('物理验收成功', {
timeout: 180000,
});
await panel.getByText('高级操作与证据导出', { exact: true }).click();
const download = page.waitForEvent('download');
await panel.getByRole('button', { name: '导出任务证据', exact: true }).click();
const data = JSON.parse(await readFile((await (await download).path())!, 'utf8'));
expect(data.succeeded).toBe(true);
expect(data.transported).toBeGreaterThanOrEqual(0.5);
expect(data.mode).toBe('mock');
expect(modelCalls).toBe(0);
await writeFile(
info.outputPath('production-mock-physical-evidence.json'),
JSON.stringify(data, null, 2),
);
await page.screenshot({ path: info.outputPath('production-physical-success.png') });
});
+119
View File
@@ -0,0 +1,119 @@
import { expect, test } from '@playwright/test';
import { resolve } from 'node:path';
import { existsSync } from 'node:fs';
import { readFile, writeFile } from 'node:fs/promises';
const prefix = '/api/decision/v1';
test('网站生产构建:精简设置、同源 Cookie/CSRF、双浏览器隔离与假 HTTP 上游', async ({
page,
browser,
}, info) => {
const errors: string[] = [];
page.on('pageerror', (e) => errors.push(e.message));
await page.goto('/');
await expect(page).toHaveTitle(/CADWorld Sim/);
await page
.locator('input[type=file]')
.first()
.setInputFiles(resolve('web_platform/fixtures/python_controller/model.xml'));
await expect(page.getByRole('button', { name: '▶ 播放' })).toBeEnabled({ timeout: 60000 });
await page.getByRole('tab', { name: '控制台', exact: true }).click();
await expect(page.getByRole('button', { name: '强化学习任务', exact: true })).toHaveCount(0);
await expect(page.getByRole('button', { name: '开源项目 / 外部控制', exact: true })).toHaveCount(
0,
);
await page.getByRole('button', { name: 'LeKiwi 智能抓放', exact: true }).click();
await page.getByText('模型设置 · API_KEY / ChatGPT 订阅', { exact: true }).click();
await expect(page.getByRole('button', { name: '保存设置' })).toBeEnabled();
await expect(page.getByLabel('本机服务令牌')).toHaveCount(0);
await page.getByLabel('LLM API_KEY', { exact: true }).fill('e2e-llm-fixture-not-real');
await page.getByLabel('Jev API_KEY(OpenRouter)').fill('e2e-jev-fixture-not-real');
await page.getByRole('button', { name: '保存设置', exact: true }).click();
await expect(page.getByRole('status').filter({ hasText: '设置已保存' })).toBeVisible();
await expect(page.getByLabel('LLM API_KEY', { exact: true })).toHaveValue('');
await page.getByText('连接测试与凭据管理', { exact: true }).click();
await page.getByRole('button', { name: '测试 LLM(可能计费)', exact: true }).click();
await expect(page.getByText(/LLM 连接测试通过/)).toBeVisible();
const other = await browser.newContext();
try {
const second = await other.newPage();
await second.goto('/');
const fresh = await second.evaluate(async (prefix) => {
const response = await fetch(prefix + '/session', {
method: 'POST',
headers: { 'Content-Type': 'application/json' },
body: '{}',
});
return response.json();
}, prefix);
expect(fresh.ready).toBe(false);
const state = await page.evaluate(
async (prefix) => (await fetch(prefix + '/status')).json(),
prefix,
);
expect(state.ready).toBe(true);
const forbidden = await page.evaluate(
async (prefix) =>
(
await fetch(prefix + '/configuration', {
method: 'PUT',
headers: { 'Content-Type': 'application/json' },
body: '{}',
})
).status,
prefix,
);
expect(forbidden).toBe(403);
expect(await page.evaluate(() => document.cookie)).not.toContain('cadworld');
expect(
await page.evaluate(() => JSON.stringify({ ...localStorage, ...sessionStorage })),
).not.toContain('e2e-llm');
} finally {
await other.close();
}
await page.getByRole('button', { name: 'ChatGPT 订阅', exact: true }).click();
await expect(page.getByRole('button', { name: '登录 ChatGPT 订阅', exact: true })).toBeVisible();
await page.screenshot({ path: info.outputPath('website-settings.png') });
expect(errors).toEqual([]);
});
test('网站 A 模型:真实 WASM、显式 mock 抓放和证据导出(无远端模型调用)', async ({
page,
}, info) => {
test.skip(!existsSync('build/lekiwi/lekiwi-v1.zip'), '需要本地 A 模型,不自动公开或下载资产');
let calls = 0;
page.on('request', (r) => {
if (/\/api\/decision\/v1\/(plan|decide)$/.test(r.url())) calls++;
});
await page.goto('/');
await page
.locator('input[type=file]')
.first()
.setInputFiles(resolve('build/lekiwi/lekiwi-v1.zip'));
const dialog = page.getByRole('dialog', { name: '配置 URDF 仿真组件' });
await expect(dialog).toBeVisible({ timeout: 30000 });
await dialog.getByLabel('URDF 机器人 profile').selectOption('lekiwi-v1');
await dialog.getByRole('button', { name: '转换并加载' }).click();
await expect(page.getByRole('button', { name: '▶ 播放' })).toBeEnabled({ timeout: 60000 });
await page.getByRole('tab', { name: '控制台', exact: true }).click();
await page.getByRole('button', { name: 'LeKiwi 智能抓放', exact: true }).click();
const panel = page.getByLabel('LeKiwi 智能抓放任务', { exact: true });
await panel.getByRole('button', { name: '开始任务(mock)', exact: true }).click();
await expect(panel.getByTestId('agent-task-state')).toContainText('物理验收成功', {
timeout: 180000,
});
await panel.getByText('高级操作与证据导出', { exact: true }).click();
const downloaded = page.waitForEvent('download');
await panel.getByRole('button', { name: '导出任务证据', exact: true }).click();
const evidence = JSON.parse(await readFile((await (await downloaded).path())!, 'utf8'));
expect(evidence.mode).toBe('mock');
expect(evidence.succeeded).toBe(true);
expect(evidence.transported).toBeGreaterThanOrEqual(0.5);
expect(evidence.sample.onGoalSupport).toBe(true);
expect(calls).toBe(0);
await writeFile(
info.outputPath('website-physical-evidence.json'),
JSON.stringify(evidence, null, 2),
);
await page.screenshot({ path: info.outputPath('website-grasp.png') });
});
+10
View File
@@ -0,0 +1,10 @@
<!doctype html>
<html>
<head>
<meta charset="UTF-8" />
<title>LeKiwi 真实接触验收夹具</title>
</head>
<body>
<script type="module" src="./agent.ts"></script>
</body>
</html>
+356
View File
@@ -0,0 +1,356 @@
import { MainThreadPhysicsAdapter } from '../src/simulation/PhysicsAdapter';
import { importBrowserFiles } from '../src/project/importer';
import type { ProjectManifest } from '../src/project/types';
import { composePickPlaceScene, PICK_PLACE_ROBOT } from '../src/mobile/agent/PickPlaceScene';
import { DeterministicBaseline } from '../src/mobile/agent/DeterministicBaseline';
import { AgentTaskController } from '../src/mobile/agent/AgentTaskController';
import { MockDecisionProvider } from '../src/mobile/agent/MockDecisionProvider';
import { DecisionClient } from '../src/mobile/agent/DecisionClient';
import { ModelBindings } from '../src/simulation/ModelBindings';
import { PICK_PLACE as P } from '../src/mobile/agent/PickPlaceScene';
type Fault =
| 'empty-grasp'
| 'empty-once'
| 'alignment'
| 'slip'
| 'jam'
| 'navigation'
| 'unreachable'
| 'overspeed';
const adapter = new MainThreadPhysicsAdapter();
let source: ProjectManifest | undefined;
let active: DeterministicBaseline | undefined;
let task: AgentTaskController | undefined;
let generation = 0;
const api = {
async boot(url: string) {
const response = await fetch(url);
if (!response.ok) throw new Error(`A 资产 HTTP ${response.status}`);
source = await importBrowserFiles([new File([await response.blob()], 'lekiwi-v1.zip')]);
},
async baseline(seed: number, governed = false) {
const epoch = ++generation;
task?.dispose();
task = undefined;
active?.dispose();
active = undefined;
if (!source) throw new Error('尚未导入 A 资产');
await adapter.load(source, 'URDF/LeKiwi.urdf', {
robotProfileId: 'lekiwi-v1',
configureRobotRuntime: false,
mobileRobot: governed ? PICK_PLACE_ROBOT : undefined,
urdfMode: 'mjcf',
baseMode: 'floating',
map: { kind: 'none' },
sceneComposer: (xml) => composePickPlaceScene(xml, seed, governed),
});
adapter.releaseRetired();
const session = adapter.session!;
let baseline!: DeterministicBaseline;
let ownerAtStart = 'manual';
if (governed) {
session.startAgent((write) => {
baseline = new DeterministicBaseline(session, seed, write);
let cancelled = false;
return {
beforeStep: () => {
if (cancelled) return false;
baseline.beforeStep();
return baseline.status === 'running';
},
afterStep: (dt) => {
baseline.afterStep(dt);
if (baseline.status !== 'running') session.stopAgent(baseline.status);
},
hold: () => baseline.hold(),
cancel: () => {
cancelled = true;
},
dispose: () => baseline.dispose(),
};
});
session.setPaused(false);
ownerAtStart = session.snapshot().controlOwner!;
} else
baseline = new DeterministicBaseline(session, seed, (id, value) =>
session.setActuator(id, value),
);
active = baseline;
const opt = session.model.opt,
dt = Number(opt.timestep);
opt.delete();
let steps = 0;
const began = performance.now();
while (baseline.status === 'running') {
if (epoch !== generation) throw new Error('基线已取消');
if (!governed) baseline.beforeStep();
if (baseline.status !== 'running') break;
session.singleStep();
if (++steps % 20 === 0) session.module.mj_forward(session.model, session.data);
if (!governed) baseline.afterStep(dt);
if (steps % 200 === 0) await new Promise<void>((resolve) => setTimeout(resolve, 0));
}
const result = {
seed,
ownerAtStart,
status: baseline.status,
error: baseline.error,
steps,
simTime: Number(session.data.time),
wallMs: performance.now() - began,
transported: baseline.evaluator.transported,
succeeded: baseline.evaluator.succeeded,
maxJointSpeed: baseline.maxJointSpeed,
final: baseline.sample,
trace: baseline.trace,
};
baseline.dispose();
active = undefined;
return result;
},
async task(
seed: number,
connection?: { address: string; token: string },
options: { fault?: Fault; goal?: number[] } = {},
) {
if (connection && options.fault) throw new Error('故障夹具禁止真实 API');
const epoch = ++generation;
task?.dispose();
active?.dispose();
if (!source) throw new Error('尚未导入 A 资产');
await adapter.load(source, 'URDF/LeKiwi.urdf', {
robotProfileId: 'lekiwi-v1',
configureRobotRuntime: false,
mobileRobot: PICK_PLACE_ROBOT,
urdfMode: 'mjcf',
baseMode: 'floating',
map: { kind: 'none' },
sceneComposer: (xml) => composePickPlaceScene(xml, seed, true, options.goal),
});
adapter.releaseRetired();
const session = adapter.session!;
const provider = connection
? new DecisionClient(connection.address, connection.token)
: new MockDecisionProvider();
let controller!: AgentTaskController;
let skills!: DeterministicBaseline;
session.startAgent((write) => {
const b = new ModelBindings(session.model),
r = PICK_PLACE_ROBOT;
const grip = b.scalarActuator(
r.gripperActuators[0].name,
r.gripperJoint,
'position',
).actuatorId;
const jam = b.scalarActuator(r.armActuators[1], r.armJoints[1].name, 'position').actuatorId;
const wheels = new Set(
r.baseActuators.map(
(name, i) => b.scalarActuator(name, r.baseJoints[i], 'velocity').actuatorId,
),
);
let jamTarget = 0,
slipAt: number | undefined,
slipOpening = 0;
// Test-only actuator faults: no live qpos writes, attachments or object teleportation.
const disturbedWrite = (id: number, value: number) => {
if (options.fault === 'jam' && id === jam) value = jamTarget;
if (options.fault === 'overspeed' && id === jam) value = r.armJoints[1].max;
if (options.fault === 'navigation' && wheels.has(id)) value = 0;
if (
id === grip &&
(options.fault === 'empty-grasp' ||
(options.fault === 'empty-once' && controller.failures.length === 0))
)
value = r.gripperOpen;
if (id === grip && options.fault === 'slip') {
if (
slipAt === undefined &&
skills.phase === 'carry' &&
skills.evaluator.transported > 0.12
) {
slipAt = skills.sample.time;
slipOpening = skills.sample.opening;
}
if (slipAt !== undefined) {
const opening = Math.min(
1,
slipOpening + P.gripperOpeningRate * (skills.sample.time - slipAt),
);
value = r.gripperClosed + opening * (r.gripperOpen - r.gripperClosed);
}
}
write(id, value);
};
skills = new DeterministicBaseline(session, seed, disturbedWrite, true, options.goal);
jamTarget = skills.sample.joints[1];
if (options.fault === 'unreachable' || options.fault === 'alignment') {
const solve = skills.ik.solve.bind(skills.ik);
// Actual scratch-data IK, not fake success/failure results or live qpos writes.
skills.ik.solve = (target) =>
solve(
options.fault === 'unreachable' ? [2, 2, 2] : [target[0] + 0.02, target[1], target[2]],
);
}
controller = new AgentTaskController(
skills,
provider,
'将方块抓起,使用底盘搬运到指定支撑台并释放、撤离。',
);
return controller;
});
task = controller;
active = skills;
session.setPaused(false);
const ownerAtStart = session.snapshot().controlOwner;
const began = performance.now();
let frames = 0;
while (!controller.terminal) {
if (epoch !== generation) throw new Error('任务已取消');
session.singleStep();
if (++frames % 20 === 0 && controller.state === 'executing')
session.module.mj_forward(session.model, session.data);
if (frames % 200 === 0) await new Promise<void>((resolve) => setTimeout(resolve, 0));
}
const result = {
...controller.snapshot(),
fault: options.fault ?? null,
ownerAtStart,
seed,
simTime: Number(session.data.time),
wallMs: performance.now() - began,
maxJointSpeed: skills.maxJointSpeed,
results: controller.results,
failures: controller.failures,
trace: skills.trace,
receipts: provider instanceof DecisionClient ? provider.receipts : [],
};
session.stopAgent('任务夹具已结束');
task = undefined;
active = undefined;
return result;
},
async lifecycle(action: 'pause-task' | 'pause-global' | 'stop' | 'force' | 'reload') {
task?.dispose();
active?.dispose();
if (!source) throw new Error('尚未导入 A 资产');
const load = () =>
adapter.load(source!, 'URDF/LeKiwi.urdf', {
robotProfileId: 'lekiwi-v1',
configureRobotRuntime: false,
mobileRobot: PICK_PLACE_ROBOT,
urdfMode: 'mjcf',
baseMode: 'floating',
map: { kind: 'none' },
sceneComposer: (xml) => composePickPlaceScene(xml, 0, true),
});
await load();
adapter.releaseRetired();
const session = adapter.session!,
provider = new MockDecisionProvider();
let release!: () => void;
const delayed = new Promise<void>((resolve) => {
release = resolve;
});
const plan = provider.plan.bind(provider);
// Deliberately ignore abort AFTER creating the reply: prove the local epoch gate,
// not just HTTP cancellation, prevents a broken provider from moving real WASM.
provider.plan = async (...args) => {
const reply = await plan(...args);
await delayed;
return reply;
};
let controller!: AgentTaskController, skills!: DeterministicBaseline;
session.startAgent((write) => {
skills = new DeterministicBaseline(session, 0, write, true);
controller = new AgentTaskController(skills, provider, '抓取并搬运方块');
return controller;
});
task = controller;
active = skills;
session.setPaused(false);
session.singleStep();
await new Promise<void>((resolve) => setTimeout(resolve, 0));
if (controller.state !== 'planning') throw new Error('生命周期夹具未进入等待规划');
if (action === 'pause-task') controller.pause();
if (action === 'pause-global') session.setPaused(true);
if (action === 'stop') session.stopAgent('夹具停止');
if (action === 'force') session.setExternalForce(1, [1, 0, 0]);
if (action === 'reload') {
await load();
adapter.releaseRetired();
}
const current = adapter.session!;
const before = {
time: Number(current.data.time),
ctrl: Array.from(current.data.ctrl),
qpos: Array.from(current.data.qpos),
};
release();
await new Promise<void>((resolve) => setTimeout(resolve, 20));
for (let i = 0; i < 10; i++) current.advance(performance.now() + i * 20);
const after = {
time: Number(current.data.time),
ctrl: Array.from(current.data.ctrl),
qpos: Array.from(current.data.qpos),
};
const result = {
action,
source: 'delayed-mock-plus-real-WASM',
state: controller.state,
owner: current.controlOwner,
jevCalls: controller.jevCalls,
before,
after,
};
controller.cancel('生命周期验收结束');
return result;
},
async resources() {
const rows: { heapBytes: number; resources: { isDeleted(): boolean }[] }[] = [];
for (let i = 0; i < 8; i++) {
await api.lifecycle('stop');
const current = adapter.session!;
// Test-only reflection retains handles, not their typed-array views, to verify
// native deletions rather than inferring a leak from linear-memory capacity.
const ik = active!.ik as unknown as {
scratch: { isDeleted(): boolean };
positionJacobian: { isDeleted(): boolean };
rotationJacobian: { isDeleted(): boolean };
};
const sampler = active! as unknown as { sampler: { force: { isDeleted(): boolean } } };
rows.push({
heapBytes: (current.data.qpos as Float64Array).buffer.byteLength,
resources: [
current.model,
current.data,
ik.scratch,
ik.positionJacobian,
ik.rotationJacobian,
sampler.sampler.force,
],
});
task?.dispose();
active?.dispose();
}
api.dispose();
return rows.map((row) => ({
heapBytes: row.heapBytes,
deleted: row.resources.map((resource) => resource.isDeleted()),
}));
},
dispose() {
generation++;
task?.dispose();
task = undefined;
active?.dispose();
active = undefined;
adapter.dispose();
},
};
declare global {
interface Window {
lekiwiAgent: typeof api;
}
}
window.lekiwiAgent = api;
+10
View File
@@ -0,0 +1,10 @@
<!doctype html>
<html lang="zh-CN">
<head>
<meta charset="UTF-8" />
<title>Mobile WASM contract runner</title>
</head>
<body>
<script type="module" src="./mobile.ts"></script>
</body>
</html>
+136
View File
@@ -0,0 +1,136 @@
import { RobotManager } from '../src/mobile/RobotManager';
import { MOBILE_ROBOTS } from '../src/mobile/RobotDescriptor';
import type { MobilePolicyMetadata } from '../src/mobile/ONNXPolicyRunner';
import { MuJoCoViewer } from '../src/viewer/MuJoCoViewer';
import { TaskDragController } from '../src/mobile/TaskDragController';
import { Vector3 } from 'three';
let viewer: MuJoCoViewer | undefined;
let drag: TaskDragController | undefined;
const manager = new RobotManager({
attach(session) {
viewer?.attach(session);
},
});
function snapshot() {
const env = manager.env!;
const observation = env.observe();
return {
robotId: env.config.id,
state: Array.from(env.state),
observation: Array.from(observation),
ctrl: Array.from(env.session.data.ctrl, Number),
qpos: Array.from(env.session.data.qpos, Number),
time: Number(env.session.data.time),
reward: env.kernel.result.reward,
stage: env.kernel.result.info.stage,
revision: env.revision,
heap: env.wasmHeapBytes,
version: env.session.module.mj_versionString(),
workspace: manager.adapter.workspace!.root,
};
}
const api = {
async boot(url: string, id: string) {
const config = MOBILE_ROBOTS.find((c) => c.id === id)!;
const response = await fetch(url);
if (!response.ok) throw new Error(`asset HTTP ${response.status}`);
const previous = manager.adapter.session;
await manager.loadZip(new File([await response.blob()], 'robot.zip'), config);
return {
...snapshot(),
retiredModelDeleted: previous?.model.isDeleted() ?? true,
retiredDataDeleted: previous?.data.isDeleted() ?? true,
};
},
async invalidCandidate(url: string) {
const config = structuredClone(MOBILE_ROBOTS[1]);
const wrong = {
...config,
armActuators: ['missing_actuator', ...config.armActuators.slice(1)],
};
await manager.loadZip(new File([await (await fetch(url)).blob()], 'robot.zip'), wrong);
},
step(n: number, action?: number[], manual = false) {
const env = manager.env!;
if (action) env.action.set(action);
const started = performance.now();
for (let i = 0; i < n; i++) env.step(env.action, manual);
return { ...snapshot(), wallMs: performance.now() - started };
},
reset() {
manager.reset();
return snapshot();
},
move(entity: 'object' | 'goal', position: number[]) {
manager.move(entity, position);
return snapshot();
},
state: snapshot,
workspaces() {
const module = manager.adapter.session!.module as unknown as {
FS: { readdir(p: string): string[] };
};
return module.FS.readdir('/workspace').filter((s) => s !== '.' && s !== '..');
},
async download() {
const bytes = await manager.exportTrainingBundle();
const href = URL.createObjectURL(new Blob([bytes.slice().buffer]));
const link = document.createElement('a');
link.href = href;
link.download = 'training.zip';
link.click();
setTimeout(() => URL.revokeObjectURL(href), 1000);
},
async policy(modelUrl: string, metadata: MobilePolicyMetadata) {
await manager.loadPolicy(new Uint8Array(await (await fetch(modelUrl)).arrayBuffer()), metadata);
manager.setMode('policy');
manager.setRunning(true);
},
tick(now: number) {
return {
...manager.advance(now),
metrics: manager.policy?.metrics,
time: Number(manager.env!.session.data.time),
};
},
show() {
const host = document.createElement('div');
host.style.cssText = 'width:1000px;height:650px;position:relative';
document.body.append(host);
const noop = () => {};
viewer = new MuJoCoViewer(host, {
advance: (now) => manager.advance(now),
onSelection: noop,
onFrame: noop,
onError: (e) => {
throw e;
},
onMapEditorSelect: noop,
onMapEditorPreviewSelect: noop,
onParametricMapSelect: noop,
onParametricMapTransform: noop,
onMapEditorTransform: noop,
});
viewer.attach(manager.adapter.session);
drag = new TaskDragController(viewer, manager);
},
screen(entity: 'object' | 'goal') {
const s = manager.env!.state,
offset = entity === 'object' ? 37 : 47;
const p = new Vector3(s[offset], s[offset + 1], s[offset + 2]).project(viewer!.camera);
const r = viewer!.renderer.domElement.getBoundingClientRect();
return { x: r.left + (p.x + 1) * 0.5 * r.width, y: r.top + (1 - p.y) * 0.5 * r.height };
},
async dispose() {
drag?.dispose();
await manager.dispose();
viewer?.dispose();
},
};
declare global {
interface Window {
mobilePhysics: typeof api;
}
}
window.mobilePhysics = api;
+87
View File
@@ -1,6 +1,9 @@
import { MainThreadPhysicsAdapter } from '../src/simulation/PhysicsAdapter';
import { SimulationSession } from '../src/simulation/SimulationSession';
import { ModelBindings } from '../src/simulation/ModelBindings';
import { mergeCachedFiles } from '../src/project/cachedFiles';
import { compiledProjectPackage } from '../src/project/exportPackage';
import { importBrowserFiles } from '../src/project/importer';
import { LEKIWI_PROFILE as p, bodyToWheels } from '../src/robot/profiles/lekiwi';
import type { ProjectManifest } from '../src/project/types';
import { ExternalControlClient } from '../src/robot/ExternalControlClient';
@@ -13,6 +16,7 @@ import { surfaceDistance, type Surface } from './meshDistance';
const adapter = new MainThreadPhysicsAdapter();
let bindings: ModelBindings;
let currentManifest: ProjectManifest;
let roundtripPackage: ReturnType<typeof compiledProjectPackage> | undefined;
let frame: number | undefined;
let bridge: ExternalControlClient | undefined;
function session() {
@@ -169,6 +173,7 @@ const harness = {
bridge?.sync();
},
async boot(assetBase: string) {
roundtripPackage = undefined;
const response = await fetch(`${assetBase}/source-manifest.json`);
if (!response.ok) throw new Error('Run examples/lekiwi/prepare_assets.py first');
const source = (await response.json()) as { files: Record<string, string> };
@@ -198,6 +203,8 @@ const harness = {
map: { kind: 'none' },
enhancements: { addActuators: false, addSensors: false, sensorType: 'camera' },
});
currentManifest = mergeCachedFiles(currentManifest, adapter.cachedSupportFiles());
adapter.releaseRetired();
bindings = new ModelBindings(session().model);
return { version: session().module.mj_versionString(), ...read() };
},
@@ -778,6 +785,86 @@ const harness = {
cancel: await cancel,
};
},
visualStats() {
const s = session(),
m = s.model,
d = s.data;
s.module.mj_forward(m, d);
const wheels = p.wheels.map((name) => {
const wheelBody = bindings.resolve(bindings.bodies, `__lekiwi_${name}`);
for (let id = 0; id < m.ngeom; id++) {
const geom = m.geom(id);
try {
if (
!geom.name.startsWith('4-Omni-Directional-Wheel_Single_Body') ||
Number(m.geom_group[id]) !== 1
)
continue;
let body = Number(m.geom_bodyid[id]);
while (body && body !== wheelBody) body = Number(m.body_parentid[body]);
if (body !== wheelBody) continue;
const meshId = Number(m.geom_dataid[id]);
const centerError = Math.hypot(
...[0, 1, 2].map((i) => d.geom_xpos[id * 3 + i] - d.xpos[wheelBody * 3 + i]),
);
return {
name,
faces: Number(m.mesh_facenum[meshId]),
vertices: Number(m.mesh_vertnum[meshId]),
scale: Array.from(m.mesh_scale.slice(meshId * 3, meshId * 3 + 3)),
centerError,
rotation: Array.from(d.geom_xmat.slice(id * 9, id * 9 + 9), Number),
collision: Boolean(m.geom_contype[id] || m.geom_conaffinity[id]),
mass: Number(m.body_mass[Number(m.geom_bodyid[id])]),
};
} finally {
geom.delete();
}
}
throw new Error(`missing CAD wheel ${name}`);
});
const fs = s.module as unknown as { FS: { readdir(path: string): string[] } };
return {
wheels,
heap: (d.qpos as Float64Array).buffer.byteLength,
projectBytes: currentManifest.files.reduce((sum, file) => sum + file.data.byteLength, 0),
xmlFiles: currentManifest.files
.filter((f) => f.path.endsWith('.xml'))
.map((f) => ({ path: f.path, bytes: f.size })),
workspaces: fs.FS.readdir('/workspace').filter((name) => name !== '.' && name !== '..'),
generatedMeshes: currentManifest.files
.filter((f) => /stl-obj-v1-.*\.obj$/.test(f.path))
.map((f) => ({ path: f.path, bytes: f.size })),
};
},
async packageRoundtrip() {
// Reload the SAME exported package. Re-exporting into the imported sources
// would intentionally append a new XML snapshot each round, not test leaks.
const packaged = (roundtripPackage ??= compiledProjectPackage(
currentManifest,
'URDF/LeKiwi.urdf',
adapter.exportMjcf(),
adapter.cachedSupportFiles(),
));
const imported = await importBrowserFiles([
new File([packaged.data.slice().buffer], 'complete.zip'),
]);
const old = session();
await adapter.load(imported, packaged.entryPath, {
robotProfileId: p.id,
urdfMode: 'mjcf',
baseMode: 'floating',
map: { kind: 'none' },
});
adapter.releaseRetired();
currentManifest = imported;
bindings = new ModelBindings(session().model);
return {
...harness.visualStats(),
oldModelDeleted: old.model.isDeleted(),
oldDataDeleted: old.data.isDeleted(),
};
},
async reimport() {
const data = adapter.exportMjcf();
const file = {
+14
View File
@@ -0,0 +1,14 @@
import { defineConfig } from '@playwright/test';
export default defineConfig({
testDir: './e2e',
testMatch: /lekiwi\.agent(?:\.(?:workspace|live|faults|lifecycle|subscription))?\.spec\.ts/,
outputDir: '../build/e2e/lekiwi-agent',
timeout: 300_000,
workers: 1,
use: { baseURL: 'http://127.0.0.1:4176' },
webServer: {
command: 'npm run dev --prefix .. -- --host 127.0.0.1 --port 4176 --strictPort',
url: 'http://127.0.0.1:4176',
reuseExistingServer: false,
},
});
+1 -1
View File
@@ -1,7 +1,7 @@
import { defineConfig } from '@playwright/test';
export default defineConfig({
testDir: './e2e',
testIgnore: /lekiwi.*\.spec\.ts/,
testIgnore: /(?:lekiwi.*|mobile|website.*)\.spec\.ts/,
timeout: 120_000,
use: { baseURL: 'http://127.0.0.1:4173' },
webServer: [
+1
View File
@@ -3,6 +3,7 @@ export default defineConfig({
testDir: './e2e',
outputDir: '../build/e2e/lekiwi',
testMatch: /lekiwi.*\.spec\.ts/,
testIgnore: /lekiwi\.agent.*\.spec\.ts/, // Dedicated clock/Origin fixture: playwright.agent.config.ts
timeout: 180_000,
workers: 1,
use: { baseURL: 'http://127.0.0.1:4174' },
+14
View File
@@ -0,0 +1,14 @@
import { defineConfig } from '@playwright/test';
export default defineConfig({
testDir: './e2e',
testMatch: /mobile\.spec\.ts/,
outputDir: '../build/e2e/mobile',
timeout: 240_000,
workers: 1,
use: { baseURL: 'http://127.0.0.1:4175' },
webServer: {
command: 'npm run dev --prefix .. -- --host 127.0.0.1 --port 4175 --strictPort',
url: 'http://127.0.0.1:4175',
reuseExistingServer: false,
},
});
@@ -0,0 +1,21 @@
import { defineConfig } from '@playwright/test';
export default defineConfig({
testDir: './e2e',
testMatch: 'website.production.spec.ts',
workers: 1,
timeout: 240_000,
use: {
baseURL: 'https://cadworld-sim.robotquan.com',
viewport: { width: 1440, height: 1000 },
ignoreHTTPSErrors: false,
trace: 'off',
screenshot: 'only-on-failure',
launchOptions: {
args: [
'--use-angle=swiftshader',
'--enable-unsafe-swiftshader',
'--host-resolver-rules=MAP cadworld-sim.robotquan.com 47.93.31.109',
],
},
},
});
+27
View File
@@ -0,0 +1,27 @@
import { defineConfig } from '@playwright/test';
export default defineConfig({
testDir: './e2e',
testMatch: 'website.spec.ts',
workers: 1,
timeout: 240_000,
use: {
baseURL: 'http://127.0.0.1:4180',
viewport: { width: 1440, height: 1000 },
launchOptions: { args: ['--use-angle=swiftshader', '--enable-unsafe-swiftshader'] },
screenshot: 'only-on-failure',
trace: 'retain-on-failure',
},
webServer: [
{
command: 'cd .. && CADWORLD_E2E=1 .venv/bin/python -m decision_server.tests.website_fixture',
url: 'http://127.0.0.1:8769/healthz',
reuseExistingServer: false,
},
{
command:
'cd .. && CADWORLD_DEV_API=http://127.0.0.1:8769 npm run preview -- --mode website --host 127.0.0.1 --port 4180 --strictPort',
url: 'http://127.0.0.1:4180',
reuseExistingServer: false,
},
],
});
+189 -11
View File
@@ -1,3 +1,11 @@
import {
mobileTrainingRobot,
mobileSceneSnapshot,
isMobileTrainingTask,
type MobileDeployment,
} from '../mobile/training';
import type { RobotConfig } from '../mobile/RobotDescriptor';
import { sha256 } from '../robot/registry';
import { useThemePreference } from './hooks/useThemePreference';
import { useExternalControl } from './hooks/useExternalControl';
import { ExternalControlPanel } from '../robot/ExternalControlPanel';
@@ -48,7 +56,13 @@ import {
type UrdfEnhancementOptions,
type UrdfLoadMode,
} from '../simulation/PhysicsAdapter';
import type { ActuatorParameters } from '../simulation/SimulationSession';
import type { ActuatorParameters, SimulationSession } from '../simulation/SimulationSession';
import { AgentTaskPanel } from '../mobile/agent/AgentTaskPanel';
import {
composePickPlaceScene,
PICK_PLACE_ROBOT,
type PickPlaceTaskConfig,
} from '../mobile/agent/PickPlaceScene';
import type { DataRecorderConfig } from '../telemetry/DataRecorder';
import type { ControllerCommand, ControllerStatus } from '../controller/types';
import type { RLCommand, RLPolicyStatus } from '../rl/types';
@@ -99,6 +113,7 @@ import { DiagnosticsDrawer } from './components/DiagnosticsDrawer';
import { ToolbarOverflowMenu } from './components/ToolbarOverflowMenu';
import { UrdfImportOptionsDialog } from './components/UrdfImportOptionsDialog';
import { compiledProjectPackage } from '../project/exportPackage';
import {
downloadBytes,
exportedFileName,
@@ -291,6 +306,9 @@ export function App() {
setRightOpen,
} = useSidebarLayout();
const robotProfilesRef = useRef(new Map<string, string>());
const agentSceneSession = useRef<SimulationSession | undefined>(undefined);
const lastAgentConfig = useRef<PickPlaceTaskConfig>({ seed: 0 });
const [agentScene, setAgentScene] = useState(false);
const sensorCameraFrame = useRef<HTMLDivElement>(null);
const orientationHost = useRef<HTMLDivElement>(null);
const [mapCommitState, setMapCommitState] = useState<'idle' | 'submitting' | 'failed'>('idle');
@@ -572,10 +590,18 @@ export function App() {
}, [showVisualMap, showMapCollision]);
useEffect(() => {
viewer.current?.setParametricMapAssets(
trainingDeployment ? [] : placedMapAssets,
trainingDeployment || agentScene ? [] : placedMapAssets,
mapSceneDraft.changedIds,
);
}, [placedMapAssets, mapSceneDraft.changedIds, trainingDeployment]);
}, [placedMapAssets, mapSceneDraft.changedIds, trainingDeployment, agentScene]);
useEffect(() => {
const session = adapter.current.session;
if (session?.snapshot().controlOwner === 'agent') {
session.stopAgent('地图草稿编辑使旧计划失效');
session.setPaused(true);
publishExternalSnapshot();
}
}, [placedMapAssets, publishExternalSnapshot]);
useEffect(() => {
viewer.current?.setShowPerceptionRays(showPerceptionRays);
}, [showPerceptionRays]);
@@ -597,6 +623,8 @@ export function App() {
requestedSceneAssets?: readonly PlacedMapAsset[],
requestedDeployment?: PolicyDeployment,
requestedPolicy?: { data: Uint8Array; path: string },
requestedMobile?: RobotConfig,
requestedTask?: PickPlaceTaskConfig,
) => {
if (!manifest.current || loadInFlight.current) return false;
// 普通模型重载始终使用上次成功应用的地图基线。只有场景提交入口可以显式
@@ -634,12 +662,30 @@ export function App() {
try {
const activeViewer = viewer.current ?? (await viewerReady.current);
if (!activeViewer) throw new Error('三维视口尚未就绪,请重试');
const selectedProfile = robotProfilesRef.current.get(path);
const mobileRobot =
(requestedTask ? PICK_PLACE_ROBOT : requestedMobile) ??
(selectedProfile && isMobileTrainingTask(selectedProfile)
? mobileTrainingRobot(selectedProfile)
: undefined);
const snapshot = await adapter.current.load(manifest.current, path, {
urdfMode: requestedMode ?? urdfModeRef.current,
baseMode: baseModeRef.current,
enhancements: urdfEnhancementsRef.current,
robotProfileId: robotProfilesRef.current.get(path),
mapAssets: sceneAssets,
baseMode: requestedTask ? 'floating' : baseModeRef.current,
enhancements: requestedTask
? { addActuators: true, addSensors: false, sensorType: 'camera' }
: urdfEnhancementsRef.current,
robotProfileId: mobileRobot
? mobileRobot.recipe === 'lekiwi-v1'
? 'lekiwi-v1'
: undefined
: selectedProfile,
mobileRobot,
configureRobotRuntime: requestedTask ? false : undefined,
sceneComposer: requestedTask
? (xml) => composePickPlaceScene(xml, requestedTask.seed, true, requestedTask.goal)
: undefined,
mapAssets: mobileRobot ? [] : sceneAssets,
map: requestedTask ? { kind: 'none' } : undefined,
trainingDeployment: requestedDeployment,
trainingPolicy: requestedPolicy,
onProgress: ({ value, label }) =>
@@ -691,7 +737,7 @@ export function App() {
let visualMapWarning: string | undefined;
try {
const assets =
manifest.current && !requestedDeployment
manifest.current && !requestedDeployment && !requestedTask
? visualMapAssets(manifest.current, placedMapAssetsRef.current)
: [];
await activeViewer.setVisualMaps(assets);
@@ -727,12 +773,17 @@ export function App() {
setAppliedMapAssets(committedMapAssets);
}
activeViewer.setParametricMapAssets(
requestedDeployment ? [] : placedMapAssetsRef.current,
requestedDeployment || requestedTask ? [] : placedMapAssetsRef.current,
summarizeMapSceneDraft(placedMapAssetsRef.current, appliedMapAssetsRef.current)
.changedIds,
);
adapter.current.releaseRetired();
setTrainingDeployment(requestedDeployment);
agentSceneSession.current = requestedTask
? (adapter.current.session ?? undefined)
: undefined;
if (requestedTask) lastAgentConfig.current = requestedTask;
setAgentScene(Boolean(requestedTask));
sessionSwapped = false;
return true;
} catch (error) {
@@ -1205,6 +1256,12 @@ export function App() {
const current = manifest.current;
const activeViewer = viewer.current;
if (!current || !activeViewer) return;
if (agentSceneSession.current && agentSceneSession.current === adapter.current.session) {
agentSceneSession.current.stopAgent('地图编辑使旧计划失效');
agentSceneSession.current.setPaused(true);
publishExternalSnapshot();
return;
}
try {
const visuals = visualMapAssets(current, assets);
if (reload)
@@ -1935,6 +1992,10 @@ export function App() {
adapter.current.setPaused(value);
};
const reset = () => {
if (agentSceneSession.current && agentSceneSession.current === adapter.current.session) {
void prepareAgentScene(lastAgentConfig.current).catch(() => undefined);
return;
}
adapter.current.setPaused(true);
adapter.current.reset();
state.setSnapshot(adapter.current.snapshot() ?? undefined);
@@ -1950,6 +2011,7 @@ export function App() {
};
const jointPoseEditingDisabled = Boolean(
state.snapshot?.robot ||
state.snapshot?.mobileRobotId ||
(state.snapshot?.controlOwner && state.snapshot.controlOwner !== 'manual'),
);
const mode = (value: InteractionMode) => {
@@ -2145,6 +2207,75 @@ export function App() {
setSelectedPolicyPath(path);
void loadPolicyBytes(file.data, path).catch(() => {});
};
const prepareAgentScene = async (config: PickPlaceTaskConfig): Promise<SimulationSession> => {
const entry = manifest.current?.entries.find(
(e) =>
e.format === 'urdf' &&
(robotProfilesRef.current.get(e.path) === 'lekiwi-v1' ||
/(^|\/)URDF\/LeKiwi\.urdf$/.test(e.path)),
);
if (!entry) throw new Error('请先导入 A / lekiwi-v1 ZIP;不会使用其他机器人替代');
if (
!(await loadEntry(
entry.path,
'mjcf',
undefined,
undefined,
undefined,
PICK_PLACE_ROBOT,
config,
))
)
throw new Error('A 任务场景加载失败,请查看编译诊断');
return adapter.current.session!;
};
const prepareMobileScene = async (taskId: string) => {
if (loadInFlight.current || policyLoadInFlight.current) throw new Error('模型/策略正在加载');
const config = mobileTrainingRobot(taskId);
const entry = useAppStore.getState().selectedEntry;
if (!entry || !manifest.current) throw new Error('请先导入对应机器人 URDF/ZIP');
if (
agentSceneSession.current === adapter.current.session ||
adapter.current.session?.mobile?.env.config.id !== config.id
) {
if (!(await loadEntry(entry, 'mjcf', undefined, undefined, undefined, config)))
throw new Error(useAppStore.getState().diagnostic?.detail ?? '移动操作场景加载失败');
}
return { config, entry };
};
const importMobilePolicy = async (file: File, deployment: MobileDeployment) => {
if (deployment.robotId !== mobileTrainingRobot(deployment.trainingTaskId).id)
throw new Error('部署变体不匹配');
await prepareMobileScene(deployment.trainingTaskId);
const session = adapter.current.session!;
policyLoadInFlight.current = true;
state.setLoading(true);
const previousPaused = session.paused;
session.setPaused(true);
try {
if (deployment.sceneSha256 !== (await sha256(adapter.current.exportMjcf())))
throw new Error('当前场景与训练场景 SHA-256 不匹配,请恢复训练时的模型');
if (file.size > 64 * 1024 ** 2) throw new Error('策略超过 64 MiB');
const data = new Uint8Array(await file.arrayBuffer());
if (session !== adapter.current.session) throw new Error('场景已切换');
await session.loadMobilePolicy(data, file.name, deployment);
if (session !== adapter.current.session) throw new Error('场景已切换');
session.setPaused(false);
state.setPaused(false);
setPolicyStatus(session.snapshot().rlPolicy);
state.setSnapshot(session.snapshot());
notify('移动操作策略已导入并运行', deployment.trainingTaskId);
} catch (error) {
if (session === adapter.current.session) {
session.setPaused(previousPaused);
state.setPaused(previousPaused);
}
throw error;
} finally {
policyLoadInFlight.current = false;
state.setLoading(false);
}
};
const importPolicy = async (file: File, expected?: PolicyDeployment) => {
try {
if (!/\.onnx$/i.test(file.name)) throw new Error('请选择 .onnx 文件');
@@ -2280,11 +2411,26 @@ export function App() {
try {
const data = adapter.current.exportMjcf();
downloadBytes(data, exportedFileName(manifest.current?.name ?? 'model', 'xml'));
notify('MJCF 已导出', '导出内容来自当前已编译模型');
notify('MJCF 已导出', '仅 XML;如需携带原始及派生网格,请使用“导出完整仿真工程 ZIP”');
} catch (error) {
state.setDiagnostic(diagnostic('模型编译', error, state.selectedEntry));
}
};
const exportProjectPackage = () => {
if (!manifest.current || !state.selectedEntry) return;
try {
const result = compiledProjectPackage(
manifest.current,
state.selectedEntry,
adapter.current.exportMjcf(),
adapter.current.cachedSupportFiles(),
);
downloadBytes(result.data, exportedFileName(manifest.current.name, 'zip'), 'application/zip');
notify('完整仿真工程已导出', `包含原始资源、派生网格及 ${result.entryPath}`);
} catch (error) {
state.setDiagnostic(diagnostic('文件系统', error, state.selectedEntry));
}
};
const toggleFullscreen = () => {
if (document.fullscreenElement) void document.exitFullscreen().catch(() => {});
else if (root.current) void root.current.requestFullscreen().catch(() => {});
@@ -2430,6 +2576,14 @@ export function App() {
disabled: !state.snapshot,
run: exportMjcf,
},
{
id: 'export-project-package',
label: '导出完整仿真工程 ZIP',
group: '工程',
icon: <Download className="h-4 w-4" />,
disabled: !state.snapshot || state.loading,
run: exportProjectPackage,
},
{
id: 'workspace-controls',
label: '在右侧打开控制台',
@@ -2500,6 +2654,21 @@ export function App() {
active={workspaceTool ?? 'controls'}
snapshot={state.snapshot}
loading={state.loading}
agentTask={
<AgentTaskPanel
loading={state.loading}
bridge={{
loadScene: prepareAgentScene,
getSession: () => adapter.current.session ?? undefined,
isTaskScene: () =>
Boolean(
agentSceneSession.current &&
agentSceneSession.current === adapter.current.session,
),
publish: publishExternalSnapshot,
}}
/>
}
externalControl={
state.snapshot && (
<ExternalControlPanel
@@ -2542,10 +2711,19 @@ export function App() {
onSelectPolicyPath={setSelectedPolicyPath}
onLoadPolicyPath={loadPolicyPath}
onImportPolicy={importPolicy}
mobileTraining={{
robotId: state.snapshot?.mobileRobotId ?? state.snapshot?.robot?.profileId,
prepare: async (taskId) => {
const { config, entry } = await prepareMobileScene(taskId);
return mobileSceneSnapshot(adapter.current, manifest.current!, entry, config);
},
importPolicy: importMobilePolicy,
}}
compileTrainingScene={(coordinates) => {
if (
mapSceneDirty ||
trainingDeployment ||
agentScene ||
useAppStore.getState().loading ||
loadInFlight.current
)
@@ -2553,7 +2731,7 @@ export function App() {
return adapter.current.exportTrainingTerrain(appliedMapAssets, coordinates);
}}
trainingSceneMaps={appliedMapAssets}
trainingSceneDirty={mapSceneDirty || Boolean(trainingDeployment)}
trainingSceneDirty={mapSceneDirty || Boolean(trainingDeployment) || agentScene}
onTogglePolicy={togglePolicy}
onPolicyCommand={setPolicyCommand}
navigationTargetMode={navigationTargetMode}
@@ -113,6 +113,10 @@ export function UrdfImportOptionsDialog({
onChange={(e) => setProfileId(e.target.value)}
>
<option value="">不启用(通用转换)</option>
<option value="MobileManipulator-LeKiwi-v1">LeKiwi v1 · 移动操作训练场景</option>
<option value="MobileManipulator-LeKiwi-Bundle">
LeKiwi Bundle · 移动操作训练场景
</option>
{ROBOT_PROFILES.map((p) => (
<option key={p.id} value={p.id}>
{p.label}
@@ -120,9 +124,15 @@ export function UrdfImportOptionsDialog({
))}
</Select>
</label>
{profileId && (
{profileId.startsWith('MobileManipulator-') && (
<p className="text-xs text-warning">
将校验固定模型版本,采用浮动基座、简化被动滚子/碰撞和九路伺服;不使用下方通用组件,不启用相机。
加载抓取物体与搬运目标;进入控制台 →
强化学习任务即可一键训练。移动操作模式不启用外部控制桥。
</p>
)}
{profileId && !profileId.startsWith('MobileManipulator-') && (
<p className="text-xs text-warning">
将校验模型语义与网格:允许换色、格式等非机械变体,拒绝关节拓扑、轴向、尺寸等变化。采用浮动基座、简化被动滚子/碰撞和九路伺服;不使用下方通用组件,不启用相机。
</p>
)}
<fieldset
@@ -1,6 +1,7 @@
import type { ChangeEvent, ReactNode } from 'react';
import { Boxes, Code2, FolderOpen, PanelLeft, PanelRight, Search, Upload } from 'lucide-react';
import { IconButton } from '../../components/ui';
import { WEBSITE_MODE } from '../websiteConfig';
const fileActionClass =
'cyber-action inline-flex h-8 cursor-pointer items-center gap-1.5 rounded-md border border-border bg-surface/80 px-2 text-xs font-medium text-text-primary shadow-sm transition-[background-color,border-color,transform] hover:-translate-y-px hover:border-border-strong hover:bg-element-hover focus-within:ring-2 focus-within:ring-accent/30';
const fileActionLabelClass = 'whitespace-nowrap';
@@ -39,7 +40,7 @@ export function WorkbenchHeader({
<Boxes aria-hidden="true" className="h-3.5 w-3.5" />
</span>
<h1 className="truncate text-sm font-semibold tracking-tight text-text-primary">
MuJoCo
{WEBSITE_MODE ? 'CADWorld Sim' : 'MuJoCo'}
</h1>
</div>
<label
@@ -103,6 +104,7 @@ export function WorkbenchHeader({
>
<PanelRight className="h-4 w-4" />
</IconButton>
{endActions}
{compactMenu}
<IconButton tooltip="命令面板(Ctrl+K)" aria-label="打开命令面板" onClick={onCommands}>
@@ -56,6 +56,17 @@ function props(): ComponentProps<typeof WorkspaceToolsPanel> {
}
describe('WorkspaceToolsPanel', () => {
it('旧描述符未声明 training 也可展开统一训练入口', () => {
const values = props();
values.snapshot = {
...snapshot,
robot: { capabilities: { training: false } },
} as SimulationSnapshot;
render(<WorkspaceToolsPanel {...values} />);
fireEvent.click(screen.getByRole('button', { name: '强化学习任务' }));
expect(screen.getByLabelText('训练服务访问令牌')).toBeVisible();
expect(screen.queryByText(/不支持训练/)).not.toBeInTheDocument();
});
it('首次展开后折叠训练组不丢连接凭据与状态', () => {
render(<WorkspaceToolsPanel {...props()} />);
expect(screen.queryByLabelText('训练服务访问令牌')).not.toBeInTheDocument();
@@ -1,3 +1,5 @@
import type { MobileTrainingBridge } from '../../mobile/training';
import { websiteCapabilities } from '../websiteConfig';
import { useState, type ReactNode } from 'react';
import type { TrainingSceneCompiler } from '../../map/trainingMap';
import type { PolicyDeployment } from '../../rl/deployment';
@@ -38,6 +40,7 @@ export function WorkspaceToolsPanel({
snapshot,
loading,
externalControl,
agentTask,
ignoreJointLimits,
jointAdvanced,
angleUnit,
@@ -65,6 +68,7 @@ export function WorkspaceToolsPanel({
onSelectPolicyPath,
onLoadPolicyPath,
onImportPolicy,
mobileTraining,
compileTrainingScene,
trainingSceneMaps,
trainingSceneDirty,
@@ -84,6 +88,7 @@ export function WorkspaceToolsPanel({
snapshot?: SimulationSnapshot;
loading: boolean;
externalControl?: ReactNode;
agentTask?: ReactNode;
ignoreJointLimits: boolean;
jointAdvanced: boolean;
angleUnit: 'rad' | 'deg';
@@ -114,6 +119,7 @@ export function WorkspaceToolsPanel({
onSelectPolicyPath: (path: string) => void;
onLoadPolicyPath: (path: string) => void;
onImportPolicy: (file: File, deployment?: PolicyDeployment) => void | Promise<void>;
mobileTraining?: MobileTrainingBridge;
compileTrainingScene?: TrainingSceneCompiler;
trainingSceneMaps?: readonly PlacedMapAsset[];
trainingSceneDirty?: boolean;
@@ -133,7 +139,12 @@ export function WorkspaceToolsPanel({
const resolvedPolicyStatus = policyStatus ?? snapshot?.rlPolicy;
const controls = snapshot ? (
<>
{externalControl && (
{agentTask && (
<CollapsibleSection title="LeKiwi 智能抓放" defaultOpen={false} keepMounted>
{agentTask}
</CollapsibleSection>
)}
{websiteCapabilities.externalControl && externalControl && (
<CollapsibleSection
title="开源项目 / 外部控制"
defaultOpen={Boolean(snapshot.robot)}
@@ -157,7 +168,7 @@ export function WorkspaceToolsPanel({
key={actuator.id}
showParameterNotice={false}
disabled={manualDisabled}
parametersDisabled={Boolean(snapshot.robot)}
parametersDisabled={Boolean(snapshot.robot || snapshot.mobileRobotId)}
actuator={actuator}
onControl={(value) => onActuator(actuator.id, value)}
onParameters={(parameters) => onActuatorParameters(actuator.id, parameters)}
@@ -174,13 +185,16 @@ export function WorkspaceToolsPanel({
badge={<ConsoleSectionBadges count={snapshot.joints.length} />}
>
<div className="mb-4 grid grid-cols-2 gap-2">
<Button onClick={onResetJoints} disabled={manualDisabled}>
<Button
onClick={onResetJoints}
disabled={manualDisabled || Boolean(snapshot.robot || snapshot.mobileRobotId)}
>
重置关节
</Button>
<Button
variant={ignoreJointLimits ? 'primary' : 'secondary'}
aria-pressed={ignoreJointLimits}
disabled={manualDisabled}
disabled={manualDisabled || Boolean(snapshot.robot || snapshot.mobileRobotId)}
onClick={onToggleJointLimits}
>
忽略关节限位
@@ -309,31 +323,28 @@ export function WorkspaceToolsPanel({
onRemove={onRemovePolicy}
/>
</CollapsibleSection>
<CollapsibleSection
title="强化学习任务"
defaultOpen={false}
keepMounted
badge={
<ConsoleSectionBadges
state={trainingSummary || undefined}
running={trainingSummary === '训练中'}
/>
}
>
{snapshot.robot && !snapshot.robot.capabilities.training ? (
<p className="text-xs text-text-secondary">
此机器人接口 V1 不支持训练。既有 Go2 训练任务不受影响,请使用对应模型。
</p>
) : (
{websiteCapabilities.training && (
<CollapsibleSection
title="强化学习任务"
defaultOpen={false}
keepMounted
badge={
<ConsoleSectionBadges
state={trainingSummary || undefined}
running={trainingSummary === '训练中'}
/>
}
>
<LocalTrainingPanel
mobileTraining={mobileTraining}
onStatusChange={setTrainingSummary}
onPolicyReady={onImportPolicy}
compileScene={compileTrainingScene}
sceneMaps={trainingSceneMaps}
sceneDirty={trainingSceneDirty}
/>
)}
</CollapsibleSection>
</CollapsibleSection>
)}
</>
) : (
<p className="p-4 text-sm text-text-tertiary">导入模型后显示控制工具</p>
+7
View File
@@ -0,0 +1,7 @@
/** Build-time product boundary; never contains credentials or upstream URLs. */
export const WEBSITE_MODE = import.meta.env.MODE === 'website';
export const websiteCapabilities = {
training: !WEBSITE_MODE,
externalControl: !WEBSITE_MODE,
subscription: true,
} as const;
@@ -0,0 +1,335 @@
import type { SimulationSession } from '../simulation/SimulationSession';
import { ModelBindings, type ScalarActuatorBinding } from '../simulation/ModelBindings';
import { MOBILE_TASK as T, validateRobotConfig, type RobotConfig } from './RobotDescriptor';
import {
clamp,
TaskKernel,
TRAINING_STAGES,
type TrainingStage,
type StepResult,
} from './TaskKernel';
import { SafeActionController } from './SafeActionController';
export class MobileManipulatorWasmEnv {
readonly kernel: TaskKernel;
readonly motion: SafeActionController;
readonly state = new Float64Array(T.stateSize);
readonly action = new Float32Array(T.actionSize);
readonly control: Float64Array;
readonly controlAddresses: Int32Array;
readonly objectBodyId: number;
readonly goalBodyId: number;
readonly frameSkip: number;
revision = 0;
private disposed = false;
private readonly arm: ScalarActuatorBinding[];
private readonly gripperQ: number;
private readonly grippers: ScalarActuatorBinding[];
private readonly baseQ: number;
private readonly baseV: number;
private readonly objectQ: number;
private readonly objectV: number;
private readonly eefBody: number;
private readonly eefSite: number;
private readonly goalMocap: number;
private qpos!: Float64Array;
private qvel!: Float64Array;
private ctrl!: Float64Array;
private xpos!: Float64Array;
private xquat!: Float64Array;
private sitePos!: Float64Array;
private siteMat!: Float64Array;
private mocapPos!: Float64Array;
private mocapQuat!: Float64Array;
constructor(
readonly session: SimulationSession,
readonly config: RobotConfig,
) {
validateRobotConfig(config);
const { model, module } = session,
b = new ModelBindings(model);
const free = (name: string): [number, number] => {
const id = b.resolve(b.joints, name);
if (Number(model.jnt_type[id]) !== 0) throw new Error(`${name} 必须是 freejoint`);
return [Number(model.jnt_qposadr[id]), Number(model.jnt_dofadr[id])];
};
[this.baseQ, this.baseV] = free(config.baseJointName);
if (
Number(model.jnt_bodyid[b.resolve(b.joints, config.baseJointName)]) !==
b.resolve(b.bodies, config.baseBodyName)
)
throw new Error('底座自由关节不属于配置的 baseBodyName');
[this.objectQ, this.objectV] = free('__mm_object_joint');
this.objectBodyId = b.resolve(b.bodies, '__mm_object');
this.goalBodyId = b.resolve(b.bodies, '__mm_goal');
this.goalMocap = Number(model.body_mocapid[this.goalBodyId]);
if (this.goalMocap < 0) throw new Error('目标必须是 mocap body');
this.eefBody = b.resolve(b.bodies, config.eefBodyName);
this.eefSite = config.eefSiteName ? module.mj_name2id(model, 6, config.eefSiteName) : -1;
if (config.eefSiteName && this.eefSite < 0) throw new Error('找不到 EEF site');
this.arm = config.armJoints.map((j, i) =>
b.scalarActuator(config.armActuators[i], j.name, j.mode),
);
this.gripperQ = b.scalarJoint(config.gripperJoint).qposAddress;
this.grippers = config.gripperActuators.map((g) =>
b.scalarActuator(g.name, g.joint ?? config.gripperJoint, 'position'),
);
const bindings = [
...config.baseActuators.map((n, i) => b.scalarActuator(n, config.baseJoints[i], 'velocity')),
...this.arm,
...this.grippers,
];
config.armJoints.forEach((spec, i) => {
const joint = model.jnt(this.arm[i].id);
try {
if (
!joint.limited ||
Math.abs(Number(joint.range[0]) - spec.min) > 1e-5 ||
Math.abs(Number(joint.range[1]) - spec.max) > 1e-5
)
throw new Error(`关节限位与 RobotConfig 不匹配:${spec.name}`);
} finally {
joint.delete();
}
});
this.controlAddresses = Int32Array.from(bindings.map((a) => a.controlAddress));
this.motion = new SafeActionController(config);
this.control = this.motion.control;
// Fail closed instead of silently clipping against a different compiled controller.
const ranges = [
...config.baseActuators.map(() => [-config.wheelLimit, config.wheelLimit]),
...config.armJoints.map((j) =>
j.mode === 'position' ? [j.min, j.max] : [-j.velocityLimit, j.velocityLimit],
),
...config.gripperActuators.map((g) => [
Math.min(g.closed, g.open),
Math.max(g.closed, g.open),
]),
];
bindings.forEach((binding, i) => {
// 3.11's bool-array memory_view binding is unavailable; use the scalar view.
const actuator = model.actuator(binding.actuatorId);
try {
if (
!actuator.ctrllimited ||
Math.abs(Number(actuator.ctrlrange[0]) - ranges[i][0]) > 1e-5 ||
Math.abs(Number(actuator.ctrlrange[1]) - ranges[i][1]) > 1e-5
)
throw new Error('RobotConfig 与编译模型 ctrlrange 不匹配');
} finally {
actuator.delete();
}
});
const opt = model.opt;
const dt = Number(opt.timestep);
opt.delete();
this.frameSkip = Math.round(T.controlDt / dt);
if (this.frameSkip < 1 || Math.abs(this.frameSkip * dt - T.controlDt) > 1e-9)
throw new Error('controlDt 必须是物理 dt 的整数倍');
this.kernel = new TaskKernel(config);
this.refreshViews();
this.reset();
}
/** Only at load/resume barriers: another model compilation may grow shared WASM memory. */
refreshViews(): void {
const d = this.session.data;
this.qpos = d.qpos;
this.qvel = d.qvel;
this.ctrl = d.ctrl;
this.xpos = d.xpos;
this.xquat = d.xquat;
this.sitePos = d.site_xpos;
this.siteMat = d.site_xmat;
this.mocapPos = d.mocap_pos;
this.mocapQuat = d.mocap_quat;
}
get wasmHeapBytes(): number {
return this.qpos.buffer.byteLength;
}
private check(): void {
if (this.disposed) throw new Error('环境已释放');
}
private forward(): void {
this.session.module.mj_forward(this.session.model, this.session.data);
}
setTrainingStage(stage: TrainingStage): void {
if (!TRAINING_STAGES.includes(stage)) throw new Error('未知训练阶段');
if (this.kernel.trainingStage !== stage) {
this.kernel.trainingStage = stage;
this.kernel.reset();
this.hold();
this.revision++;
}
}
reset(options?: { object?: readonly number[]; goal?: readonly number[] }): Float32Array {
this.check();
const { module, model, data } = this.session;
module.mj_resetData(model, data);
this.action.fill(0);
for (let i = 0; i < this.arm.length; i++) {
const spec = this.config.armJoints[i];
this.qpos[this.arm[i].qposAddress] = spec.neutral;
}
this.qpos[this.gripperQ] = this.config.gripperOpen;
for (let i = 0; i < this.grippers.length; i++)
this.qpos[this.grippers[i].qposAddress] = this.config.gripperActuators[i].open;
this.kernel.reset();
this.revision++;
this.forward();
this.hold(false);
if (options?.object) this.moveTaskEntity('object', options.object);
if (options?.goal) this.moveTaskEntity('goal', options.goal);
return this.observe();
}
/** Preserve safe servo targets (not gravity-sagged qpos); zero velocity commands.
* Re-anchoring to measured q on every pause would make a loaded arm progressively sag. */
hold(preserveTargets = true): void {
this.check();
this.action.fill(0);
this.capture();
this.motion.reset(
this.state,
preserveTargets ? Float64Array.from(this.controlAddresses, (i) => this.ctrl[i]) : undefined,
);
this.kernel.recordAction(this.motion.applied, this.motion.targets);
for (let i = 0; i < this.control.length; i++)
this.ctrl[this.controlAddresses[i]] = this.control[i];
}
applyAction(action: Float32Array, manual = false): void {
this.check();
this.motion.apply(
action,
this.state,
manual ? 'pick-place' : this.kernel.trainingStage,
manual || this.kernel.hasLifted,
);
this.kernel.recordAction(this.motion.applied, this.motion.targets);
for (let i = 0; i < this.control.length; i++)
this.ctrl[this.controlAddresses[i]] = this.control[i];
}
observe(): Float32Array {
this.check();
this.capture();
return this.kernel.observe(this.state);
}
step(action: Float32Array, manual = false): StepResult {
this.check();
if (this.kernel.result.terminated || this.kernel.result.truncated)
throw new Error('回合结束,请 reset');
this.applyAction(action, manual);
let peak = 0,
safety = '';
for (let i = 0; i < this.frameSkip; i++) {
this.session.module.mj_step(this.session.model, this.session.data);
for (const joint of this.arm) peak = Math.max(peak, Math.abs(this.qvel[joint.qvelAddress]));
for (const joint of this.grippers)
peak = Math.max(peak, Math.abs(this.qvel[joint.qvelAddress]));
if (peak > T.jointSpeedStop) {
safety = 'joint_velocity';
break;
}
}
// mj_step leaves position-derived fields at the integration boundary; both runtimes forward.
this.forward();
this.capture();
return this.kernel.evaluate(this.state, safety, peak);
}
private capture(): void {
const s = this.state;
for (let i = 0; i < 7; i++) {
s[i] = this.qpos[this.baseQ + i];
s[37 + i] = this.qpos[this.objectQ + i];
}
for (let i = 0; i < 3; i++) {
s[7 + i] = this.qvel[this.baseV + i];
s[44 + i] = this.qvel[this.objectV + i];
}
// MuJoCo freejoint angular velocity is local; expose world angular velocity in SI.
const w = s[3],
x = s[4],
y = s[5],
z = s[6];
const vx = this.qvel[this.baseV + 3],
vy = this.qvel[this.baseV + 4],
vz = this.qvel[this.baseV + 5];
const tx = 2 * (y * vz - z * vy),
ty = 2 * (z * vx - x * vz),
tz = 2 * (x * vy - y * vx);
s[10] = vx + w * tx + y * tz - z * ty;
s[11] = vy + w * ty + z * tx - x * tz;
s[12] = vz + w * tz + x * ty - y * tx;
for (let i = 0; i < this.arm.length; i++) {
s[13 + i] = this.qpos[this.arm[i].qposAddress];
s[21 + i] = this.qvel[this.arm[i].qvelAddress];
}
s[29] = clamp(
(this.qpos[this.gripperQ] - this.config.gripperClosed) /
(this.config.gripperOpen - this.config.gripperClosed),
0,
1,
);
for (let i = 0; i < 3; i++) {
s[30 + i] =
this.eefSite >= 0 ? this.sitePos[3 * this.eefSite + i] : this.xpos[3 * this.eefBody + i];
s[47 + i] = this.mocapPos[3 * this.goalMocap + i];
}
if (this.eefSite >= 0) matrixQuaternion(this.siteMat, 9 * this.eefSite, s, 33);
else for (let i = 0; i < 4; i++) s[33 + i] = this.xquat[4 * this.eefBody + i];
for (let i = 0; i < 4; i++) s[50 + i] = this.mocapQuat[4 * this.goalMocap + i];
for (let i = 0; i < s.length; i++)
if (!Number.isFinite(s[i])) throw new Error('仿真产生非有限状态,已停止');
}
/** World XY dragging on a fixed plane; editing begins a new task episode, not a robot reset. */
moveTaskEntity(entity: 'object' | 'goal', position: readonly number[]): void {
this.check();
if (
position.length !== 3 ||
position.some((v) => !Number.isFinite(v) || Math.abs(v) > T.positionScale)
)
throw new Error('拖动位置超出工作空间');
const target = entity === 'object' ? this.qpos : this.mocapPos;
const offset = entity === 'object' ? this.objectQ : this.goalMocap * 3;
for (let i = 0; i < 3; i++)
target[offset + i] = i === 2 ? Math.max(T.objectStart[2], position[i]) : position[i];
if (entity === 'object') {
for (let i = 0; i < 6; i++) this.qvel[this.objectV + i] = 0;
}
this.session.data.qacc_warmstart.fill(0);
this.kernel.reset();
this.revision++;
this.forward();
this.hold();
this.observe();
}
dispose(): void {
this.disposed = true;
this.revision++;
}
}
/** Matrix to unit quaternion, wxyz; no temporary array or native scratch allocation. */
function matrixQuaternion(m: Float64Array, a: number, out: Float64Array, b: number): void {
const trace = m[a] + m[a + 4] + m[a + 8];
if (trace > 0) {
const r = Math.sqrt(1 + trace) * 2;
out[b] = r / 4;
out[b + 1] = (m[a + 7] - m[a + 5]) / r;
out[b + 2] = (m[a + 2] - m[a + 6]) / r;
out[b + 3] = (m[a + 3] - m[a + 1]) / r;
} else {
const i = m[a] > m[a + 4] && m[a] > m[a + 8] ? 0 : m[a + 4] > m[a + 8] ? 1 : 2;
const j = (i + 1) % 3,
k = (i + 2) % 3;
const r = Math.sqrt(1 + m[a + i * 3 + i] - m[a + j * 3 + j] - m[a + k * 3 + k]) * 2;
out[b] = (m[a + k * 3 + j] - m[a + j * 3 + k]) / r;
out[b + 1 + i] = r / 4;
out[b + 1 + j] = (m[a + j * 3 + i] + m[a + i * 3 + j]) / r;
out[b + 1 + k] = (m[a + k * 3 + i] + m[a + i * 3 + k]) / r;
}
}
export { MobileManipulatorWasmEnv as InteractiveManipulatorEnv };
@@ -0,0 +1,120 @@
import { expect, it, vi } from 'vitest';
import { MOBILE_ROBOTS, MOBILE_TASK as T } from './RobotDescriptor';
import type { MobileDeployment } from './training';
import { SimulationSession } from '../simulation/SimulationSession';
import { ControlArbiter } from '../simulation/ControlArbiter';
const mocks = vi.hoisted(() => ({ load: vi.fn() }));
vi.mock('./MobileManipulatorWasmEnv', () => ({
MobileManipulatorWasmEnv: class {
config = MOBILE_ROBOTS[0];
reset = vi.fn();
setTrainingStage = vi.fn();
moveTaskEntity = vi.fn();
dispose = vi.fn();
kernel = { result: { terminated: false, truncated: false, info: { safety_stop: '' } } };
},
}));
vi.mock('./ONNXPolicyRunner', () => ({ ONNXPolicyRunner: { load: mocks.load } }));
import { MobilePolicyController } from './MobilePolicyController';
const deployment = {
trainingTaskId: 'MobileManipulator-LeKiwi-v1',
trainingStage: 'navigate',
resetOptions: { object: [0.25, 0, 0.019], goal: [0.55, 0.15, 0.019] },
input: { name: 'observation', dtype: 'float32', shape: [1, T.observationSize] },
output: { name: 'action', dtype: 'float32', shape: [1, 12] },
} as MobileDeployment;
const runner = () => ({
setEnabled: vi.fn(),
dispose: vi.fn().mockResolvedValue(undefined),
tick: vi.fn().mockReturnValue(true),
metrics: { inferenceCount: 1, lastInferenceMs: 1, error: '' },
});
it('主会话策略加载、暂停恢复、reset目标和销毁', async () => {
const r = runner();
mocks.load.mockResolvedValue(r);
const controller = new MobilePolicyController({} as SimulationSession, MOBILE_ROBOTS[0]);
await controller.load(new Uint8Array(), 'policy.onnx', deployment);
expect(controller.env.moveTaskEntity).toHaveBeenCalledWith('goal', deployment.resetOptions.goal);
controller.setEnabled(true);
expect(r.setEnabled).toHaveBeenLastCalledWith(false);
controller.setPaused(false);
expect(r.setEnabled).toHaveBeenLastCalledWith(true);
expect(controller.status()).toMatchObject({
enabled: true,
observationSize: T.observationSize,
actionSize: 12,
});
controller.setPaused(true);
expect(r.setEnabled).toHaveBeenLastCalledWith(false);
controller.dispose();
expect(r.dispose).toHaveBeenCalledOnce();
expect(controller.env.dispose).toHaveBeenCalledOnce();
});
it('移除/销毁期间迟到ORT加载不得重置旧场景', async () => {
let finish!: (value: ReturnType<typeof runner>) => void;
mocks.load.mockImplementation(
() =>
new Promise((resolve) => {
finish = resolve;
}),
);
const c = new MobilePolicyController({} as SimulationSession, MOBILE_ROBOTS[0]);
const loading = c.load(new Uint8Array(), 'policy.onnx', deployment);
await vi.waitFor(() => expect(finish).toBeDefined());
c.dispose();
const r = runner();
finish(r);
await expect(loading).rejects.toThrow('取消');
expect(r.dispose).toHaveBeenCalledOnce();
expect(c.env.reset).not.toHaveBeenCalled();
});
it('移动策略交出控制权时保持关节位置,不能把位置舵机 ctrl 清零', () => {
const ctrl = new Float64Array(9).fill(1);
const hold = vi.fn(() => {
ctrl.fill(0);
ctrl[3] = 0.6;
});
const session = Object.assign(Object.create(SimulationSession.prototype), {
mobile: { env: { hold } },
data: { ctrl },
}) as { clearControlTargets(): void };
session.clearControlTargets();
expect(hold).toHaveBeenCalledOnce();
expect(ctrl[3]).toBe(0.6);
});
it('主会话移动策略不再调用普通 mj_step,等待单飞输出;终止即暂停', () => {
const arbiter = new ControlArbiter();
arbiter.claim('policy');
const mobile = {
tick: vi.fn().mockReturnValue(false),
ended: false,
error: undefined,
setPaused: vi.fn(),
};
const mj_step = vi.fn();
const capture = vi.fn();
const s = Object.assign(Object.create(SimulationSession.prototype) as object, {
mobile,
controlArbiter: arbiter,
model: { opt: { timestep: 0.002 } },
data: { time: 0 },
module: { mj_step },
dataRecorder: { capture },
paused: false,
lastNow: 0,
accumulator: 0,
speed: 1,
maxCatchUpSteps: 10,
frameBudgetMs: 100,
}) as unknown as SimulationSession;
expect(s.advance(20).steps).toBe(0);
expect(mj_step).not.toHaveBeenCalled();
mobile.tick.mockReturnValue(true);
expect(s.advance(21).steps).toBe(1);
expect(capture).toHaveBeenCalledOnce();
expect(mj_step).not.toHaveBeenCalled();
mobile.ended = true;
s.advance(41);
expect(s.paused).toBe(true);
expect(mobile.setPaused).toHaveBeenCalledWith(true);
});
@@ -0,0 +1,118 @@
import type { SimulationSession } from '../simulation/SimulationSession';
import type { RLPolicyStatus } from '../rl/types';
import { MobileManipulatorWasmEnv } from './MobileManipulatorWasmEnv';
import { MOBILE_TASK as T, type RobotConfig } from './RobotDescriptor';
import type { ONNXPolicyRunner } from './ONNXPolicyRunner';
import type { MobileDeployment } from './training';
/** Owned by the main SimulationSession; the ONNX runner is the sole physics writer
* while policy control is active. In-flight results never advance retired sessions. */
export class MobilePolicyController {
readonly env: MobileManipulatorWasmEnv;
private runner?: ONNXPolicyRunner;
private generation = 0;
private disposed = false;
private enabled = false;
private paused = true;
private path = '';
private deployment?: MobileDeployment;
constructor(session: SimulationSession, config: RobotConfig) {
this.env = new MobileManipulatorWasmEnv(session, structuredClone(config));
}
async load(bytes: Uint8Array, path: string, deployment: MobileDeployment): Promise<void> {
const generation = ++this.generation;
for (const position of [deployment.resetOptions?.object, deployment.resetOptions?.goal]) {
if (
!Array.isArray(position) ||
position.length !== 3 ||
position.some((v) => !Number.isFinite(v) || Math.abs(v) > T.positionScale) ||
position[2] < T.objectStart[2]
)
throw new Error('策略初始目标坐标无效');
}
if (
deployment.input?.name !== 'observation' ||
deployment.output?.name !== 'action' ||
deployment.input.dtype !== 'float32' ||
deployment.output.dtype !== 'float32' ||
JSON.stringify(deployment.input.shape) !== JSON.stringify([1, T.observationSize]) ||
JSON.stringify(deployment.output.shape) !== '[1,12]'
)
throw new Error('移动操作部署张量契约不匹配');
const { ONNXPolicyRunner } = await import('./ONNXPolicyRunner');
const runner = await ONNXPolicyRunner.load(bytes, deployment, this.env);
if (generation !== this.generation || this.disposed) {
await runner.dispose();
throw new Error('移动操作策略加载已取消');
}
this.removeRunner();
this.runner = runner;
this.path = path;
this.deployment = structuredClone(deployment);
this.reset();
}
private removeRunner(): void {
this.runner?.setEnabled(false);
void this.runner?.dispose();
this.runner = undefined;
this.enabled = false;
}
remove(): void {
this.generation++;
this.removeRunner();
}
setEnabled(value: boolean): void {
this.enabled = value && !!this.runner;
this.runner?.setEnabled(this.enabled && !this.paused);
}
setPaused(value: boolean): void {
this.paused = value;
this.runner?.setEnabled(this.enabled && !value);
}
reset(): void {
if (this.deployment) this.env.setTrainingStage(this.deployment.trainingStage);
this.env.reset();
if (this.deployment) {
this.env.moveTaskEntity('object', this.deployment.resetOptions.object);
this.env.moveTaskEntity('goal', this.deployment.resetOptions.goal);
}
}
tick(): boolean {
return this.runner?.tick() ?? false;
}
get ended(): boolean {
return this.env.kernel.result.terminated || this.env.kernel.result.truncated;
}
get error(): string | undefined {
return (
this.runner?.metrics.error ||
(this.env.kernel.result.info.safety_stop
? `安全停止:${this.env.kernel.result.info.safety_stop},请重置并检查模型/策略`
: undefined)
);
}
status(): RLPolicyStatus | undefined {
if (!this.runner) return undefined;
return {
taskId: this.deployment!.trainingTaskId,
taskName: `${this.env.config.label} · ${{ navigate: '底盘接近', reach: '末端接近', 'pick-place': '抓取放置' }[this.deployment!.trainingStage]}`,
path: this.path,
loaded: true,
enabled: this.enabled && !this.error,
controlHz: 1 / T.controlDt,
observationSize: T.observationSize,
actionSize: T.actionSize,
inputName: 'observation',
outputName: 'action',
command: { linearX: 0, linearY: 0, angularZ: 0 },
inferenceCount: this.runner.metrics.inferenceCount,
lastInferenceMs: this.runner.metrics.lastInferenceMs,
error: this.error,
};
}
dispose(): void {
this.disposed = true;
this.remove();
this.env.dispose();
}
}
@@ -0,0 +1,135 @@
import { webcrypto } from 'node:crypto';
import { ONNXPolicyRunner, type MobilePolicyMetadata } from './ONNXPolicyRunner';
import { MOBILE_ROBOTS, MOBILE_TASK as T } from './RobotDescriptor';
import type { MobileManipulatorWasmEnv } from './MobileManipulatorWasmEnv';
import { sha256 } from '../robot/registry';
const ort = vi.hoisted(() => ({
create: vi.fn(),
tensors: [] as { dispose: ReturnType<typeof vi.fn> }[],
}));
vi.mock('onnxruntime-web/wasm', () => ({
env: { wasm: {} },
InferenceSession: { create: ort.create },
Tensor: class {
dispose = vi.fn();
constructor() {
ort.tensors.push(this);
}
},
}));
const flush = async () => {
for (let i = 0; i < 10; i++) await Promise.resolve();
};
async function setup() {
Object.defineProperty(globalThis, 'crypto', { configurable: true, value: webcrypto });
const session = {
inputNames: ['observation'],
outputNames: ['action'],
inputMetadata: [{ isTensor: true, type: 'float32', shape: [1, T.observationSize] }],
outputMetadata: [{ isTensor: true, type: 'float32', shape: [1, 12] }],
run: vi.fn(),
release: vi.fn(async () => {}),
};
ort.create.mockResolvedValue(session);
const env = {
config: MOBILE_ROBOTS[0],
revision: 0,
observe: vi.fn(() => new Float32Array(T.observationSize)),
setTrainingStage: vi.fn(),
step: vi.fn(),
hold: vi.fn(),
};
const bytes = new Uint8Array([1, 2, 3]);
const metadata: MobilePolicyMetadata = {
taskId: T.id,
robotId: env.config.id,
observationSize: T.observationSize,
actionSemantics: T.actionSemantics,
trainingStage: 'navigate',
actionSize: 12,
controlDt: 0.02,
normalized: true,
modelSha256: await sha256(bytes),
robotConfigSha256: await sha256(new TextEncoder().encode(JSON.stringify(env.config))),
};
const output = { type: 'float32', dims: [1, 12], data: new Float32Array(12), dispose: vi.fn() };
return {
session,
env,
bytes,
metadata,
output,
load: () => ONNXPolicyRunner.load(bytes, metadata, env as unknown as MobileManipulatorWasmEnv),
};
}
it('旧动作语义即使伪装新张量尺寸也拒绝加载', async () => {
const x = await setup();
ort.create.mockClear();
x.metadata.taskId = 'mobile-manipulator-v1';
await expect(x.load()).rejects.toThrow('重新训练');
x.metadata.taskId = T.id;
x.metadata.actionSemantics = 'absolute-position';
await expect(x.load()).rejects.toThrow('重新训练');
expect(ort.create).not.toHaveBeenCalled();
});
it('单飞推理;过期/reset/disable 结果不写入;释放等待 run 完成', async () => {
const x = await setup(),
runner = await x.load();
let finish!: (v: { action: typeof x.output }) => void;
x.session.run.mockImplementation(
() =>
new Promise((r) => {
finish = r;
}),
);
runner.setEnabled(true);
runner.tick();
runner.tick();
expect(x.session.run).toHaveBeenCalledTimes(1);
x.env.revision++;
finish({ action: x.output });
await flush();
expect(x.env.step).not.toHaveBeenCalled();
expect(x.output.dispose).toHaveBeenCalledOnce();
runner.tick();
runner.setEnabled(false);
finish({ action: x.output });
await flush();
runner.setEnabled(true);
runner.tick();
const releasing = runner.dispose();
expect(x.session.release).not.toHaveBeenCalled();
finish({ action: x.output });
await releasing;
expect(x.session.release).toHaveBeenCalledOnce();
expect(x.env.step).not.toHaveBeenCalled();
});
it('成功输出仅在下一 rAF 消费;无效输出安全停机', async () => {
const x = await setup(),
runner = await x.load();
x.session.run.mockResolvedValue({ action: x.output });
runner.setEnabled(true);
runner.tick();
await flush();
expect(x.env.step).not.toHaveBeenCalled();
expect(runner.tick()).toBe(true);
expect(x.env.step).toHaveBeenCalledOnce();
x.output.data[3] = NaN;
runner.tick();
await flush();
expect(runner.metrics.error).toContain('NaN');
expect(runner.tick()).toBe(false);
await runner.dispose();
});
it('拒绝错误维度、机器人、权重哈希并释放已创建的 ORT session', async () => {
const x = await setup();
x.session.inputMetadata[0].shape = [1, 67];
await expect(x.load()).rejects.toThrow('float32');
expect(x.session.release).toHaveBeenCalledOnce();
x.metadata.robotId = 'other';
await expect(x.load()).rejects.toThrow('元数据');
x.metadata.robotId = x.env.config.id;
x.metadata.modelSha256 = 'wrong';
await expect(x.load()).rejects.toThrow('SHA-256');
});
+190
View File
@@ -0,0 +1,190 @@
import * as ort from 'onnxruntime-web/wasm';
import { sha256 } from '../robot/registry';
import { MOBILE_TASK as T } from './RobotDescriptor';
import { TRAINING_STAGES, type TrainingStage } from './TaskKernel';
import type { MobileManipulatorWasmEnv } from './MobileManipulatorWasmEnv';
ort.env.wasm.numThreads = 1;
ort.env.wasm.proxy = false;
export interface MobilePolicyMetadata {
taskId: string;
actionSemantics: string;
trainingStage: TrainingStage;
robotId: string;
observationSize: number;
actionSize: number;
controlDt: number;
normalized: boolean;
modelSha256: string;
robotConfigSha256: string;
}
/** Single-flight, lockstep inference. Rendering continues while inference runs; physics
* waits, so training and deployment never differ by an implicit action delay. */
export class ONNXPolicyRunner {
readonly action = new Float32Array(T.actionSize);
readonly input = new Float32Array(T.observationSize);
readonly metrics = { inferenceCount: 0, lastInferenceMs: 0, maxInferenceMs: 0, error: '' };
private readonly tensor: ort.Tensor;
private readonly feeds: Record<string, ort.Tensor>;
private inFlight?: Promise<void>;
private epoch = 0;
private ready = false;
private enabled = false;
private disposed = false;
private revision: number;
private releasePromise?: Promise<void>;
private constructor(
private readonly session: ort.InferenceSession,
readonly env: MobileManipulatorWasmEnv,
private readonly stage: TrainingStage,
) {
this.tensor = new ort.Tensor('float32', this.input, [1, T.observationSize]);
this.feeds = { observation: this.tensor };
this.revision = env.revision;
}
static async load(
model: Uint8Array,
metadata: MobilePolicyMetadata,
env: MobileManipulatorWasmEnv,
): Promise<ONNXPolicyRunner> {
if (
metadata.taskId !== T.id ||
metadata.actionSemantics !== T.actionSemantics ||
!TRAINING_STAGES.includes(metadata.trainingStage) ||
metadata.robotId !== env.config.id ||
metadata.observationSize !== T.observationSize ||
metadata.actionSize !== T.actionSize ||
metadata.controlDt !== T.controlDt ||
metadata.normalized !== true
)
throw new Error('策略元数据与 v2 安全控制契约/机器人不匹配;旧全行程位置策略必须重新训练');
if (
metadata.modelSha256 !== (await sha256(model)) ||
metadata.robotConfigSha256 !==
(await sha256(new TextEncoder().encode(JSON.stringify(env.config))))
)
throw new Error('策略权重/机器人配置 SHA-256 不匹配');
const session = await ort.InferenceSession.create(model.slice(), {
executionProviders: ['wasm'],
graphOptimizationLevel: 'all',
});
try {
if (
session.inputNames.length !== 1 ||
session.outputNames.length !== 1 ||
session.inputNames[0] !== 'observation' ||
session.outputNames[0] !== 'action'
)
throw new Error('ONNX 必须为 observation → action');
for (const [m, size] of [
[session.inputMetadata[0], T.observationSize],
[session.outputMetadata[0], T.actionSize],
] as const)
if (
!m?.isTensor ||
m.type !== 'float32' ||
m.shape.length !== 2 ||
m.shape[0] !== 1 ||
m.shape[1] !== size
)
throw new Error('ONNX 必须声明固定 float32 [1,92] → [1,12]');
return new ONNXPolicyRunner(session, env, metadata.trainingStage);
} catch (error) {
await session.release();
throw error;
}
}
setEnabled(enabled: boolean): void {
if (this.disposed) return;
const wasEnabled = this.enabled;
this.enabled = enabled;
this.epoch++;
this.ready = false;
if (enabled) {
this.env.setTrainingStage(this.stage);
this.metrics.error = '';
}
this.revision = this.env.revision;
if (enabled || wasEnabled) this.env.hold();
}
/** Called from the existing viewer's rAF. Returns true only after exactly one env step. */
tick(): boolean {
if (!this.enabled || this.disposed) return false;
if (this.revision !== this.env.revision) {
this.epoch++;
this.ready = false;
this.revision = this.env.revision;
}
if (this.ready) {
this.ready = false;
this.env.step(this.action);
return true;
}
if (this.inFlight) return false;
this.input.set(this.env.observe());
const epoch = this.epoch,
revision = this.env.revision,
started = performance.now();
this.inFlight = this.session
.run(this.feeds)
.then((outputs) => {
try {
const output = outputs.action;
if (
!output ||
output.type !== 'float32' ||
output.dims.length !== 2 ||
output.dims[0] !== 1 ||
output.dims[1] !== T.actionSize
)
throw new Error('策略输出形状错误');
const values = output.data as Float32Array;
for (let i = 0; i < values.length; i++)
if (!Number.isFinite(values[i])) throw new Error('策略输出包含 NaN/Infinity');
if (
this.disposed ||
!this.enabled ||
epoch !== this.epoch ||
revision !== this.env.revision
)
return;
this.action.set(values);
this.ready = true;
this.metrics.inferenceCount++;
this.metrics.lastInferenceMs = performance.now() - started;
this.metrics.maxInferenceMs = Math.max(
this.metrics.maxInferenceMs,
this.metrics.lastInferenceMs,
);
} finally {
for (const name in outputs) outputs[name].dispose();
}
})
.catch((error: unknown) => {
if (this.disposed || epoch !== this.epoch || revision !== this.env.revision) return;
this.metrics.error = error instanceof Error ? error.message : String(error);
this.enabled = false;
this.ready = false;
this.env.hold();
})
.finally(() => {
this.inFlight = undefined;
});
return false;
}
async dispose(): Promise<void> {
if (this.releasePromise) return this.releasePromise;
this.disposed = true;
this.enabled = false;
this.epoch++;
this.ready = false;
// Never release ORT while run() still owns its input; never touch a retired env on completion.
this.releasePromise = (this.inFlight ?? Promise.resolve()).finally(async () => {
this.tensor.dispose();
await this.session.release();
});
return this.releasePromise;
}
}
@@ -0,0 +1,99 @@
import robots from '../../../contracts/mobile-robots-v1.json';
import task from '../../../contracts/mobile-manipulator-v2.json';
export interface ArmJointConfig {
name: string;
min: number;
max: number;
neutral: number;
mode: 'position' | 'velocity';
velocityLimit: number;
}
/** Names, not file-order indices. Resolved once against the compiled model. */
export interface RobotConfig {
id: string;
label: string;
recipe: 'lekiwi-v1' | 'lekiwi-bundle' | 'mjcf';
baseBodyName: string;
baseJointName: string;
baseJoints: readonly string[];
baseActuators: readonly string[];
/** wheel rad/s = matrix * body-frame [vx m/s, vy m/s, wz rad/s]. */
baseMix: readonly (readonly number[])[];
baseLimits: readonly number[];
wheelLimit: number;
armJoints: readonly ArmJointConfig[];
armActuators: readonly string[];
gripperJoint: string;
gripperActuators: readonly { name: string; joint?: string; closed: number; open: number }[];
gripperClosed: number;
gripperOpen: number;
eefBodyName: string;
eefSiteName?: string;
eefOffset: readonly number[];
}
export const MOBILE_TASK = Object.freeze(task);
export const MOBILE_ROBOTS: readonly RobotConfig[] = robots as RobotConfig[];
export function validateRobotConfig(c: RobotConfig): void {
if (
!c.id ||
!c.baseBodyName ||
!c.baseJointName ||
!c.eefBodyName ||
!['lekiwi-v1', 'lekiwi-bundle', 'mjcf'].includes(c.recipe) ||
!c.armJoints.length ||
c.armJoints.length > task.maxArmJoints ||
c.armJoints.length !== c.armActuators.length ||
!c.baseJoints.length ||
c.baseJoints.length !== c.baseActuators.length ||
c.baseMix.length !== c.baseJoints.length ||
c.baseMix.some((row) => row.length !== 3 || row.some((v) => !Number.isFinite(v))) ||
c.baseLimits.length !== 3 ||
c.baseLimits.some((v) => !Number.isFinite(v) || v <= 0) ||
!Number.isFinite(c.wheelLimit) ||
c.wheelLimit <= 0 ||
c.eefOffset.length !== 3 ||
c.eefOffset.some((v) => !Number.isFinite(v)) ||
!Number.isFinite(c.gripperClosed) ||
!Number.isFinite(c.gripperOpen) ||
c.gripperClosed === c.gripperOpen ||
!c.gripperActuators.length ||
c.gripperActuators.some(
(v) =>
!v.name || !Number.isFinite(v.closed) || !Number.isFinite(v.open) || v.closed === v.open,
) ||
c.armJoints.some(
(j) =>
!j.name ||
!Number.isFinite(j.min) ||
!Number.isFinite(j.max) ||
j.min >= j.max ||
!Number.isFinite(j.neutral) ||
j.neutral < j.min ||
j.neutral > j.max ||
!['position', 'velocity'].includes(j.mode) ||
!Number.isFinite(j.velocityLimit) ||
j.velocityLimit <= 0,
)
)
throw new Error('RobotConfig 无效或超过 mobile-manipulator-v2 的固定槽位');
const primary = c.gripperActuators.filter((g) => (g.joint ?? c.gripperJoint) === c.gripperJoint);
if (
!primary.length ||
primary.some((g) => g.closed !== c.gripperClosed || g.open !== c.gripperOpen)
)
throw new Error('夹爪观测关节与执行器行程不一致');
const joints = [
...c.baseJoints,
...c.armJoints.map((j) => j.name),
...new Set([c.gripperJoint, ...c.gripperActuators.map((g) => g.joint ?? c.gripperJoint)]),
];
const actuators = [
...c.baseActuators,
...c.armActuators,
...c.gripperActuators.map((a) => a.name),
];
if (new Set(joints).size !== joints.length || new Set(actuators).size !== actuators.length)
throw new Error('RobotConfig 存在重复关节/执行器');
}
@@ -0,0 +1,145 @@
import { RobotManager } from './RobotManager';
import { MOBILE_ROBOTS } from './RobotDescriptor';
const fakes = vi.hoisted(() => ({
imports: vi.fn(),
gate: undefined as Promise<void> | undefined,
loads: [] as string[],
deleted: [] as string[],
}));
vi.mock('../project/importer', () => ({ importBrowserFiles: fakes.imports }));
vi.mock('./ONNXPolicyRunner', () => ({ ONNXPolicyRunner: { load: vi.fn() } }));
vi.mock('../simulation/PhysicsAdapter', () => ({
MainThreadPhysicsAdapter: class {
session: { name: string } | null = null;
workspace: { root: string } | null = null;
retired: { name: string } | null = null;
oldWorkspace: { root: string } | null = null;
async load(manifest: { name: string }, _entry: string, options: { workspaceSlot: string }) {
fakes.loads.push(manifest.name);
await fakes.gate;
this.retired = this.session;
this.oldWorkspace = this.workspace;
this.session = { name: manifest.name };
this.workspace = { root: `/workspace/${options.workspaceSlot}` };
}
releaseRetired() {
if (this.retired) fakes.deleted.push(this.retired.name);
this.retired = null;
this.oldWorkspace = null;
}
rollbackRetired() {
if (this.session) fakes.deleted.push(this.session.name);
this.session = this.retired;
this.workspace = this.oldWorkspace;
this.retired = null;
}
dispose() {
this.releaseRetired();
if (this.session) fakes.deleted.push(this.session.name);
this.session = this.workspace = null;
}
},
}));
vi.mock('./MobileManipulatorWasmEnv', () => ({
MobileManipulatorWasmEnv: class {
refreshViews = vi.fn();
hold = vi.fn();
reset = vi.fn();
dispose = vi.fn();
constructor(
readonly session: unknown,
readonly config: { id: string },
) {
if (config.id === 'bad') throw new Error('binding mismatch');
}
},
}));
function manifest(name: string) {
return { id: name, name, entries: [{ path: 'scene.xml', format: 'mjcf' }], files: [] };
}
const file = (name: string) => new File([], name);
const config = { ...MOBILE_ROBOTS[0], recipe: 'mjcf' as const };
const flush = async () => {
for (let i = 0; i < 5; i++) await Promise.resolve();
};
beforeEach(() => {
fakes.loads.length = fakes.deleted.length = 0;
fakes.gate = undefined;
fakes.imports.mockReset().mockImplementation(async ([f]: File[]) => manifest(f.name));
});
it('使用空闲双槽,先 viewer attach 后销毁旧模型;绑定失败恢复旧环境', async () => {
const attach = vi.fn(),
manager = new RobotManager({ attach });
await manager.loadZip(file('first'), config);
const old = manager.env,
slot = manager.adapter.workspace!.root;
await expect(manager.loadZip(file('bad'), { ...config, id: 'bad' })).rejects.toThrow(
'binding mismatch',
);
expect(manager.env).toBe(old);
expect(manager.adapter.workspace!.root).toBe(slot);
expect(fakes.deleted).toEqual(['bad']);
expect(old!.refreshViews).toHaveBeenCalled();
await manager.loadZip(file('next'), config);
expect(manager.adapter.workspace!.root).not.toBe(slot);
expect(old!.dispose).toHaveBeenCalledOnce();
expect(fakes.deleted).toEqual(['bad', 'first']);
expect(attach).toHaveBeenLastCalledWith(manager.adapter.session);
await manager.dispose();
expect(fakes.deleted).toEqual(['bad', 'first', 'next']);
});
it('编译屏障期间的失焦/暂停不会访问可能失效的旧 WASM views', async () => {
const manager = new RobotManager({ attach: vi.fn() });
await manager.loadZip(file('first'), config);
const old = manager.env!;
let release!: () => void;
fakes.gate = new Promise<void>((r) => {
release = r;
});
const loading = manager.loadZip(file('next'), config);
const calls = vi.mocked(old.hold).mock.calls.length;
manager.setRunning(false);
expect(vi.mocked(old.hold).mock.calls.length).toBe(calls);
expect(() => manager.reset()).toThrow('正在切换');
expect(() => manager.setMode('teleop')).toThrow('正在切换');
await expect(manager.exportTrainingBundle()).rejects.toThrow();
release();
await loading;
await manager.dispose();
});
it('最新切换优先,取消的解包不进入 native compile', async () => {
let resolve!: (value: ReturnType<typeof manifest>) => void;
fakes.imports.mockImplementationOnce(
() =>
new Promise((r) => {
resolve = r;
}),
);
const manager = new RobotManager({ attach: vi.fn() });
const first = manager.loadZip(file('first'), config).catch((e: Error) => e.message);
await flush();
const next = manager.loadZip(file('next'), config);
resolve(manifest('first'));
await next;
expect(await first).toContain('取消');
expect(fakes.loads).toEqual(['next']);
expect(manager.loading).toBe(false);
await manager.dispose();
});
it('加载途中 dispose 不提交迟到模型,所有候选被释放', async () => {
let release!: () => void;
fakes.gate = new Promise<void>((r) => {
release = r;
});
const manager = new RobotManager({ attach: vi.fn() });
const loading = manager.loadZip(file('late'), config).catch((e: Error) => e.message);
await flush();
expect(() => manager.setRunning(true)).toThrow();
const closing = manager.dispose();
release();
await closing;
expect(await loading).toContain('取消');
expect(fakes.deleted).toEqual(['late']);
expect(manager.env).toBeUndefined();
});
+224
View File
@@ -0,0 +1,224 @@
import { importBrowserFiles } from '../project/importer';
import type { ProjectManifest } from '../project/types';
import { MainThreadPhysicsAdapter } from '../simulation/PhysicsAdapter';
import type { FrameResult, SimulationSession } from '../simulation/SimulationSession';
import { MobileManipulatorWasmEnv } from './MobileManipulatorWasmEnv';
import { ONNXPolicyRunner, type MobilePolicyMetadata } from './ONNXPolicyRunner';
import { composeMobileScene, prepareBundle } from './SceneComposer';
import { MOBILE_TASK as T, validateRobotConfig, type RobotConfig } from './RobotDescriptor';
/** One physics owner. Viewer GPU objects detach before native model/data are deleted. */
export class RobotManager {
readonly adapter = new MainThreadPhysicsAdapter();
env?: MobileManipulatorWasmEnv;
policy?: ONNXPolicyRunner;
mode: 'teleop' | 'policy' = 'teleop';
running = false;
loading = false;
private readonly workspacePrefix = `mobile_${crypto.randomUUID().replaceAll('-', '_')}`;
private generation = 0;
private policyGeneration = 0;
private disposed = false;
private queue: Promise<unknown> = Promise.resolve();
private project?: ProjectManifest;
private entryPath = '';
private lastNow?: number;
private accumulator = 0;
private readonly frame: FrameResult = { steps: 0, stepMs: 0, overBudget: false };
constructor(private readonly viewer: { attach(session: SimulationSession | null): void }) {}
async loadZip(file: File, config: RobotConfig, entryPath?: string): Promise<void> {
if (this.disposed) throw new Error('RobotManager 已释放');
validateRobotConfig(config);
// Clone at the boundary: changing a caller's config cannot mutate a running contract.
const descriptor = structuredClone(config);
const generation = ++this.generation;
++this.policyGeneration;
this.setRunning(false);
this.loading = true;
const load = this.queue
.catch(() => {})
.then(async () => {
if (generation !== this.generation || this.disposed) throw new Error('机器人切换已取消');
let manifest = await importBrowserFiles([file]);
const entry =
entryPath ??
(manifest.entries.length === 1 ? manifest.entries[0].path : manifest.selectedEntry);
if (!entry || !manifest.entries.some((e) => e.path === entry))
throw new Error('请显式指定 ZIP 中的模型入口');
if (descriptor.recipe === 'lekiwi-bundle') {
if (!entry.toLowerCase().endsWith('.urdf')) throw new Error('bundle 适配器要求原始 URDF');
manifest = prepareBundle(manifest, entry, descriptor);
}
if (generation !== this.generation || this.disposed) throw new Error('机器人切换已取消');
const oldEnv = this.env;
let candidate: MobileManipulatorWasmEnv | undefined;
let staged = false;
try {
await this.adapter.load(manifest, entry, {
robotProfileId: descriptor.recipe === 'lekiwi-v1' ? 'lekiwi-v1' : undefined,
configureRobotRuntime: false,
// Unique per manager, but bounded to two paths. MuJoCo caches meshes by
// resource path; monotonically new mount roots grow the native asset cache.
workspaceSlot: `${this.workspacePrefix}_${this.adapter.workspace?.root.endsWith('_0') ? 1 : 0}`,
map: { kind: 'none' },
sceneComposer: (xml) => composeMobileScene(xml, descriptor),
});
staged = true;
oldEnv?.refreshViews();
if (generation !== this.generation || this.disposed) throw new Error('机器人切换已取消');
candidate = new MobileManipulatorWasmEnv(this.adapter.session!, descriptor);
this.viewer.attach(this.adapter.session);
const oldPolicy = this.policy;
this.policy = undefined;
await oldPolicy?.dispose();
if (generation !== this.generation || this.disposed) throw new Error('机器人切换已取消');
this.env = candidate;
this.mode = 'teleop';
this.project = manifest;
this.entryPath = entry;
oldEnv?.dispose();
this.adapter.releaseRetired();
} catch (error) {
candidate?.dispose();
if (staged) {
// A failed viewer attachment may own candidate mjvScene objects.
this.viewer.attach(null);
this.adapter.rollbackRetired();
}
oldEnv?.refreshViews();
if (!this.disposed) this.viewer.attach(this.adapter.session);
throw error;
}
});
this.queue = load;
return load.finally(() => {
if (generation === this.generation) this.loading = false;
});
}
async loadPolicy(bytes: Uint8Array, metadata: MobilePolicyMetadata): Promise<void> {
const env = this.env,
generation = ++this.policyGeneration;
if (!env || this.disposed || this.loading) throw new Error('请先完成机器人加载');
this.setRunning(false);
const runner = await ONNXPolicyRunner.load(bytes, metadata, env);
if (generation !== this.policyGeneration || env !== this.env || this.disposed) {
await runner.dispose();
throw new Error('策略加载已取消');
}
const previous = this.policy;
this.policy = runner;
await previous?.dispose();
}
setMode(mode: 'teleop' | 'policy'): void {
if (this.loading) throw new Error('正在切换机器人');
if (mode === 'policy' && !this.policy) throw new Error('请先加载匹配策略及元数据');
this.setRunning(false);
this.mode = mode;
if (mode === 'teleop') this.env?.setTrainingStage('pick-place');
this.env?.hold();
}
setRunning(value: boolean): void {
if (value && (!this.env || this.disposed || this.loading))
throw new Error('没有可运行的环境,或正在切换模型');
this.running = value;
this.accumulator = 0;
this.lastNow = undefined;
this.policy?.setEnabled(value && this.mode === 'policy');
// During compile an old WASM view can be detached. It was already held before
// loading began; blur/visibility callbacks must not touch it until the barrier.
if (!value && !this.loading) this.env?.hold();
}
reset(): void {
if (this.loading) throw new Error('正在切换机器人');
this.setRunning(false);
this.env?.reset();
}
move(entity: 'object' | 'goal', position: readonly number[]): void {
if (this.loading) return;
this.setRunning(false);
this.env?.moveTaskEntity(entity, position);
}
/** Passed into MuJoCoViewer; never also call SimulationSession.advance. */
advance(now: number): FrameResult {
const f = this.frame;
f.steps = 0;
f.stepMs = 0;
f.overBudget = false;
const elapsed =
this.lastNow === undefined ? 0 : Math.min(0.1, Math.max(0, (now - this.lastNow) / 1000));
this.lastNow = now;
if (!this.running || !this.env) return f;
this.accumulator = Math.min(0.1, this.accumulator + elapsed);
const started = performance.now();
try {
while (
this.accumulator + 1e-9 >= T.controlDt &&
f.steps < 4 &&
performance.now() - started < 8
) {
if (this.mode === 'policy') {
if (!this.policy?.tick()) {
if (this.policy?.metrics.error) this.setRunning(false);
break;
}
} else this.env.step(this.env.action, true); // Teleop bypasses phase masks, never speed limits.
this.accumulator -= T.controlDt;
f.steps++;
if (this.env.kernel.result.terminated || this.env.kernel.result.truncated) {
this.setRunning(false);
break;
}
}
} catch (error) {
this.setRunning(false);
throw error;
}
f.stepMs = performance.now() - started;
f.overBudget = this.accumulator >= T.controlDt;
return f;
}
/** Same composed MJCF and assets for native training; no second URDF converter. */
async exportTrainingBundle(): Promise<Uint8Array> {
if (!this.env || !this.project || !this.adapter.workspace || this.loading)
throw new Error('尚未完成机器人加载');
const generation = this.generation;
const { zipSync } = await import('fflate');
if (this.disposed || this.loading || generation !== this.generation)
throw new Error('导出已取消');
const encoder = new TextEncoder();
const scene = `${this.entryPath}.training.xml`;
const files: Record<string, Uint8Array> = {};
for (const f of [...this.project.files, ...this.adapter.cachedSupportFiles()])
files[f.path] = f.data;
files[scene] = this.adapter.exportMjcf();
files['robot.json'] = encoder.encode(JSON.stringify(this.env.config));
files['task.json'] = encoder.encode(JSON.stringify(T));
files['environment.json'] = encoder.encode(
JSON.stringify({
scene,
mujoco: this.adapter.session!.module.mj_versionString(),
taskId: T.id,
robotId: this.env.config.id,
}),
);
return zipSync(files, { level: 1 });
}
async dispose(): Promise<void> {
if (this.disposed) return;
this.setRunning(false);
this.disposed = true;
this.generation++;
this.policyGeneration++;
await this.queue.catch(() => {});
await this.policy?.dispose();
this.policy = undefined;
this.viewer.attach(null);
this.env?.dispose();
this.env = undefined;
this.adapter.dispose();
this.project = undefined;
}
}
@@ -0,0 +1,100 @@
import golden from '../../../contracts/fixtures/mobile-motion-v2-golden.json';
import { MOBILE_ROBOTS, MOBILE_TASK as T } from './RobotDescriptor';
import { SafeActionController } from './SafeActionController';
import { TaskKernel, type TrainingStage } from './TaskKernel';
it('两个变体、三个阶段多步与 Python 控制/滤波状态一致;目标速度有界', () => {
for (const sample of golden) {
const config = MOBILE_ROBOTS.find((r) => r.id === sample.robotId)!;
const controller = new SafeActionController(config);
const state = Float64Array.from(sample.state);
controller.reset(state);
for (const frame of sample.frames) {
const previous = controller.control.slice();
controller.apply(
Float32Array.from(frame.action),
state,
sample.stage as TrainingStage,
false,
);
frame.control.forEach((v, i) => expect(controller.control[i]).toBeCloseTo(v, 6));
frame.applied.forEach((v, i) => expect(controller.applied[i]).toBeCloseTo(v, 6));
frame.targets.forEach((v, i) => expect(controller.targets[i]).toBeCloseTo(v, 6));
config.armJoints.forEach((joint, i) => {
expect(Math.abs(controller.control[3 + i] - previous[3 + i])).toBeLessThanOrEqual(
Math.min(joint.velocityLimit, T.armSpeedLimit) * T.controlDt + 1e-12,
);
});
if (sample.stage === 'navigate')
expect(Array.from(controller.applied.slice(3))).toEqual(Array(9).fill(0));
}
const previous = controller.control.slice();
expect(() => controller.apply(new Float32Array(12).fill(NaN), state, 'reach', false)).toThrow();
expect(controller.control).toEqual(previous);
const kernel = new TaskKernel(config);
kernel.recordAction(controller.applied, controller.targets);
expect(Array.from(kernel.observe(state).slice(68, 80))).toEqual(Array.from(controller.applied));
expect(Array.from(kernel.observe(state).slice(80))).toEqual(Array.from(controller.targets));
}
});
it('底盘加速度、速度模式和夹爪开度速率受限;复位清除滤波历史', () => {
const c = {
...MOBILE_ROBOTS[0],
armJoints: MOBILE_ROBOTS[0].armJoints.map((j, i) =>
i === 0 ? { ...j, mode: 'velocity' as const } : j,
),
};
const controller = new SafeActionController(c);
const state = new Float64Array(T.stateSize);
state[3] = 1;
state[37] = 1;
state[29] = 1;
controller.reset(state);
for (let i = 0; i < 50; i++) {
const previous = controller.applied.slice();
const opening = controller.targets[11];
controller.apply(new Float32Array(12).fill(i < 20 ? -1 : 1), state, 'pick-place', true);
for (let j = 0; j < 3; j++)
expect(
Math.abs(controller.applied[j] - previous[j]) * T.baseSpeedLimits[j],
).toBeLessThanOrEqual(T.baseAccelerationLimits[j] * T.controlDt + 1e-7);
expect(Math.abs(controller.control[3])).toBeLessThanOrEqual(T.armSpeedLimit);
expect(Math.abs(controller.targets[11] - opening)).toBeLessThanOrEqual(
2 * T.gripperOpeningRate * T.controlDt + 1e-7,
);
}
controller.reset(state);
expect(controller.applied).toEqual(new Float32Array(12));
});
it('暂停保持既有安全目标,不反复追随重力下垂后的 qpos', () => {
const controller = new SafeActionController(MOBILE_ROBOTS[0]);
const state = new Float64Array(T.stateSize);
state[29] = 1;
controller.reset(state);
state[13] = -0.1;
for (let i = 0; i < 10; i++) {
controller.reset(state, controller.control);
expect(controller.control[3]).toBe(0);
expect(controller.applied).toEqual(new Float32Array(12));
}
});
it('导航需要接近位、朝向、停车和稳定驻留;超速不能算成功', () => {
const kernel = new TaskKernel(MOBILE_ROBOTS[0], 'navigate');
const s = new Float64Array(T.stateSize);
s[3] = 1;
s[37] = T.navigationOffset[0];
s[7] = 0.2;
for (let i = 0; i < T.settleSteps; i++) expect(kernel.evaluate(s).info.is_success).toBe(false);
s[7] = 0;
for (let i = 0; i < T.settleSteps - 1; i++) expect(kernel.evaluate(s).terminated).toBe(false);
expect(kernel.evaluate(s).info.is_success).toBe(true);
kernel.reset();
s[21] = T.jointSpeedStop + 0.1;
const result = kernel.evaluate(s);
expect(result.terminated).toBe(true);
expect(result.info.safety_stop).toBe('joint_velocity');
expect(result.info.is_success).toBe(false);
expect(result.info.reward_components.joint_velocity).toBeLessThan(0);
});
@@ -0,0 +1,115 @@
import { MOBILE_TASK as T, type RobotConfig } from './RobotDescriptor';
import { clamp, navigationError, type TrainingStage } from './TaskKernel';
/** v2: integrated position setpoints, bounded speeds/accelerations, no full-range jumps.
* Mirrored by training_server/mobile_manipulator/motion.py. All filter state is observed. */
export class SafeActionController {
readonly applied = new Float32Array(T.actionSize);
readonly targets = new Float32Array(T.actionSize);
readonly control: Float64Array;
constructor(readonly config: RobotConfig) {
this.control = new Float64Array(
config.baseJoints.length + config.armJoints.length + config.gripperActuators.length,
);
}
reset(state: Float64Array, previousControl?: Float64Array): void {
previousControl = previousControl?.slice();
this.applied.fill(0);
this.targets.fill(0);
this.control.fill(0);
const n = this.config.baseJoints.length;
this.config.armJoints.forEach((j, i) => {
if (j.mode === 'position') {
let q = state[13 + i];
if (previousControl && Number.isFinite(previousControl[n + i]))
q = clamp(previousControl[n + i], q - T.armTrackingError, q + T.armTrackingError);
q = clamp(q, j.min, j.max);
this.control[n + i] = q;
this.targets[3 + i] = (2 * (q - j.min)) / (j.max - j.min) - 1;
}
});
let opening = state[29];
for (let i = 0; i < this.config.gripperActuators.length; i++) {
const g = this.config.gripperActuators[i],
index = n + this.config.armJoints.length + i;
if (
previousControl &&
(g.joint ?? this.config.gripperJoint) === this.config.gripperJoint &&
Number.isFinite(previousControl[index])
) {
opening = clamp((previousControl[index] - g.closed) / (g.open - g.closed), 0, 1);
break;
}
}
this.targets[11] = 2 * opening - 1;
this.gripper(opening);
}
private gripper(opening: number): void {
const n = this.config.baseJoints.length + this.config.armJoints.length;
this.config.gripperActuators.forEach((g, i) => {
this.control[n + i] = g.closed + opening * (g.open - g.closed);
});
}
apply(action: Float32Array, state: Float64Array, stage: TrainingStage, lifted: boolean): void {
if (action.length !== T.actionSize || action.some((v) => !Number.isFinite(v)))
throw new Error('动作必须是有限 [12]');
const { distance, yaw } = navigationError(state);
const near = distance < T.navigationTolerance && Math.abs(yaw) < T.navigationYawTolerance;
const manipulate = stage !== 'navigate' && (near || lifted);
const c = this.config,
dt = T.controlDt;
for (let i = 0; i < 3; i++) {
const limit = Math.min(c.baseLimits[i], T.baseSpeedLimits[i]);
const desired = near && !lifted ? 0 : clamp(action[i]);
const delta = (T.baseAccelerationLimits[i] * dt) / limit;
this.applied[i] = clamp(desired, this.applied[i] - delta, this.applied[i] + delta);
}
let largest = c.wheelLimit;
c.baseMix.forEach((row, i) => {
this.control[i] = 0;
for (let j = 0; j < 3; j++)
this.control[i] +=
row[j] * this.applied[j] * Math.min(c.baseLimits[j], T.baseSpeedLimits[j]);
largest = Math.max(largest, Math.abs(this.control[i]));
});
const n = c.baseJoints.length;
for (let i = 0; i < n; i++) this.control[i] *= c.wheelLimit / largest;
c.armJoints.forEach((j, i) => {
const k = 3 + i;
const speed = Math.min(j.velocityLimit, T.armSpeedLimit);
const delta = (T.armAccelerationLimit * dt) / speed;
const a = clamp(
manipulate ? clamp(action[k]) : 0,
this.applied[k] - delta,
this.applied[k] + delta,
);
if (j.mode === 'position') {
const old = this.control[n + i];
const desired = clamp(
old + a * speed * dt,
state[13 + i] - T.armTrackingError,
state[13 + i] + T.armTrackingError,
);
const target = manipulate
? clamp(clamp(desired, old - speed * dt, old + speed * dt), j.min, j.max)
: old;
this.control[n + i] = target;
this.applied[k] = (target - old) / (speed * dt);
this.targets[k] = (2 * (target - j.min)) / (j.max - j.min) - 1;
} else {
this.applied[k] = manipulate ? a : 0;
this.control[n + i] = this.applied[k] * speed;
}
});
const opening = (this.targets[11] + 1) / 2;
const target = clamp(
opening +
(manipulate && stage === 'pick-place' ? clamp(action[11]) : 0) * T.gripperOpeningRate * dt,
0,
1,
);
this.applied[11] = (target - opening) / (T.gripperOpeningRate * dt);
this.targets[11] = 2 * target - 1;
this.gripper(target);
}
}
@@ -0,0 +1,58 @@
import { composeMobileScene, prepareBundle } from './SceneComposer';
import { MOBILE_ROBOTS } from './RobotDescriptor';
import type { ProjectManifest } from '../project/types';
const encode = (xml: string) => new TextEncoder().encode(xml);
const xml =
'<mujoco><compiler angle="radian"/><worldbody><geom type="plane" size="1 1 .1"/><body name="Wrist_Roll_08c-v1"/></worldbody><keyframe><key/></keyframe></mujoco>';
const config = { ...MOBILE_ROBOTS[0], recipe: 'mjcf' as const };
it('组合现有模型,任务视觉不被查看器 collision-group 过滤;重复组合拒绝', () => {
const output = composeMobileScene(encode(xml), config);
const doc = new DOMParser().parseFromString(new TextDecoder().decode(output), 'application/xml');
expect(doc.querySelectorAll('geom[type="plane"]')).toHaveLength(1);
expect(doc.querySelector('geom[name="__mm_floor"]')?.getAttribute('group')).toBe('2');
expect(doc.querySelector('geom[name="__mm_object_geom"]')?.getAttribute('group')).toBe('1');
expect(doc.querySelector('body[name="__mm_goal"]')?.getAttribute('mocap')).toBe('true');
expect(doc.querySelector('site[name="__mm_eef"]')).not.toBeNull();
expect(doc.querySelector('keyframe')).toBeNull();
expect(() => composeMobileScene(output, config)).toThrow('命名空间');
});
it('bundle 不把原始 LeKiwi 五轴名称按顺序误认成 Link1..4', () => {
const data = encode(
'<robot><joint name="base_left_wheel"/><joint name="base_back_wheel"/><joint name="base_right_wheel"/><joint name="arm_shoulder_pan"/></robot>',
);
const manifest = {
files: [{ path: 'robot.urdf', data, size: data.length }],
entries: [{ path: 'robot.urdf', format: 'urdf' }],
} as ProjectManifest;
expect(() => prepareBundle(manifest, 'robot.urdf', MOBILE_ROBOTS[1])).toThrow('Link1');
});
it('Bundle 关节级限力不能覆盖仿真舵机限力,否则零位保持也会坍塌', () => {
const c = MOBILE_ROBOTS[1];
const joints = [...c.armJoints.map((j) => j.name), c.gripperJoint]
.map(
(name) =>
`<joint name="${name}" axis="0 0 1" actuatorfrclimited="true" actuatorfrcrange="-1 1"/>`,
)
.join('');
const wheels = c.baseJoints.map((name) => `<body><joint name="${name}"/></body>`).join('');
const input = `<mujoco><worldbody><body name="${c.baseBodyName}">${wheels}<body name="base_link">${joints}<body name="${c.eefBodyName}"/></body></body></worldbody></mujoco>`;
const output = new DOMParser().parseFromString(
new TextDecoder().decode(composeMobileScene(encode(input), c)),
'application/xml',
);
for (const j of c.armJoints)
expect(output.querySelector(`joint[name="${j.name}"]`)?.getAttribute('actuatorfrcrange')).toBe(
'-8 8',
);
expect(
output.querySelector(`joint[name="${c.gripperJoint}"]`)?.getAttribute('actuatorfrcrange'),
).toBe('-2 2');
});
it('拒绝无效 XML/DOCTYPE/未知 eefBodyName,不输出无法绑定的任务', () => {
expect(() => composeMobileScene(encode('<broken>'), config)).toThrow();
expect(() => composeMobileScene(encode('<!DOCTYPE mujoco><mujoco/>'), config)).toThrow();
expect(() => composeMobileScene(encode(xml), { ...config, eefBodyName: 'missing' })).toThrow(
'missing',
);
});
+289
View File
@@ -0,0 +1,289 @@
import type { ProjectManifest } from '../project/types';
import { MOBILE_TASK as T, type RobotConfig } from './RobotDescriptor';
const encoder = new TextEncoder();
function parse(data: Uint8Array): Document {
const doc = new DOMParser().parseFromString(new TextDecoder().decode(data), 'application/xml');
if (doc.querySelector('parsererror') || doc.doctype) throw new Error('模型 XML 无效');
return doc;
}
function node(
doc: Document,
tag: string,
attributes: Record<string, string | number> = {},
): Element {
const n = doc.createElement(tag);
for (const [key, value] of Object.entries(attributes)) n.setAttribute(key, String(value));
return n;
}
function named(doc: Document, tag: string, name: string): Element {
const result = Array.from(doc.getElementsByTagName(tag)).filter(
(e) => e.getAttribute('name') === name,
);
if (result.length !== 1) throw new Error(`模型缺少/重复 ${tag}: ${name}`);
return result[0];
}
/** The new bundle has a different arm. Never route it through the old source/FK gate. */
export function prepareBundle(
manifest: ProjectManifest,
entryPath: string,
c: RobotConfig,
): ProjectManifest {
const source = manifest.files.find((f) => f.path === entryPath);
if (!source) throw new Error('缺少 bundle URDF');
const doc = parse(source.data);
for (const name of [...c.baseJoints, ...c.armJoints.map((j) => j.name), c.gripperJoint])
named(doc, 'joint', name);
for (const mesh of doc.querySelectorAll('mesh[filename]')) {
if (mesh.getAttribute('filename')!.includes('4-Omni-Directional-Wheel_Single_Body'))
mesh.replaceWith(node(doc, 'sphere', { radius: 0.05 }));
}
// Exporter duplicated collisions. Keep one representation per visual, not overlapping copies.
for (const link of doc.querySelectorAll('link')) {
const seen = new Set<string>();
for (const collision of link.querySelectorAll(':scope > collision')) {
const mesh = collision.querySelector('mesh');
if (!mesh) continue;
const origin = collision.querySelector('origin');
const transform = [
mesh.getAttribute('scale') ?? '1 1 1',
origin?.getAttribute('xyz') ?? '0 0 0',
origin?.getAttribute('rpy') ?? '0 0 0',
]
.map((text) =>
text
.trim()
.split(/\s+/)
.map((v) => Math.round(Number(v) * 1e9) / 1e9)
.join(' '),
)
.join('|');
const key = `${mesh.getAttribute('filename')}|${transform}`;
if (seen.has(key)) collision.remove();
else seen.add(key);
}
}
const data = encoder.encode(new XMLSerializer().serializeToString(doc));
return {
...manifest,
files: manifest.files.map((f) => (f === source ? { ...f, data, size: data.length } : f)),
};
}
/** Simulation-only ideal omni base; preserves Link1..4 arm frames/inertias/limits.
* Native arm mesh contacts are convex approximations, not the old CAD hull recipe.
*/
function adaptBundle(doc: Document, c: RobotConfig): void {
const root = doc.documentElement,
cad = named(doc, 'body', c.baseBodyName);
for (const name of c.baseJoints) named(doc, 'joint', name).parentElement!.remove();
for (const j of cad.querySelectorAll(':scope > joint, :scope > freejoint')) j.remove();
cad.setAttribute('name', '__mm_bundle_cad');
cad.setAttribute('pos', '0 0 -0.01786');
cad.setAttribute('quat', '0.7071067811865476 0 0 -0.7071067811865476');
const base = node(doc, 'body', { name: c.baseBodyName, pos: '0 0 0.0515' });
base.append(node(doc, 'freejoint', { name: c.baseJointName }));
base.append(node(doc, 'inertial', { pos: '0 0 .04', mass: 2.2, diaginertia: '.014 .014 .022' }));
cad.replaceWith(base);
base.append(cad);
// Keep collision on the arm, simplify chassis to avoid convex CAD covering wheel wells.
const arm = named(doc, 'body', 'base_link');
for (const geom of cad.querySelectorAll('geom')) {
if (!arm.contains(geom)) {
geom.setAttribute('contype', '0');
geom.setAttribute('conaffinity', '0');
} else {
geom.setAttribute('friction', '.8 .005 .0001');
}
}
base.append(
node(doc, 'geom', { type: 'cylinder', size: '.112 .025', pos: '0 0 .04', group: 3, mass: 0 }),
);
root.querySelector('actuator')?.remove();
const actuators = node(doc, 'actuator');
root.append(actuators);
const specs = [
...c.armJoints,
{ name: c.gripperJoint, min: c.gripperClosed, max: c.gripperOpen, mode: 'position' },
];
for (const spec of specs) {
const joint = named(doc, 'joint', spec.name);
joint.setAttribute('limited', 'true');
joint.setAttribute('range', `${spec.min} ${spec.max}`);
joint.setAttribute('damping', '.02');
joint.setAttribute('armature', '.002');
// URDF conversion leaves a joint-level ±1 N·m clamp. It otherwise overrides
// the simulation servos below and lets the loaded arm collapse under gravity.
joint.setAttribute('actuatorfrclimited', 'true');
joint.setAttribute('actuatorfrcrange', spec.name === c.gripperJoint ? '-2 2' : '-8 8');
if (spec.name === c.gripperJoint)
joint.setAttribute(
'axis',
joint
.getAttribute('axis')!
.split(/\s+/)
.map((v) => -Number(v))
.join(' '),
);
actuators.append(
node(doc, 'position', {
name: `${spec.name}_servo`,
joint: spec.name,
kp: spec.name === c.gripperJoint ? 15 : 40,
kv: 2,
ctrllimited: 'true',
ctrlrange: `${spec.min} ${spec.max}`,
forcelimited: 'true',
forcerange: spec.name === c.gripperJoint ? '-2 2' : '-8 8',
}),
);
}
for (let i = 0; i < c.baseJoints.length; i++) {
const name = c.baseJoints[i],
tx = c.baseMix[i][0] * 0.05,
ty = c.baseMix[i][1] * 0.05;
const axle = `${-ty} ${tx} 0`;
const wheel = node(doc, 'body', {
name: `__mm_${name}`,
pos: `${0.125 * ty} ${-0.125 * tx} 0`,
});
wheel.append(
node(doc, 'joint', {
name,
axis: axle,
limited: 'false',
damping: '.0002',
armature: '.00005',
}),
);
wheel.append(
node(doc, 'inertial', { pos: '0 0 0', mass: '.06', diaginertia: '.00004 .00004 .00004' }),
);
wheel.append(
node(doc, 'geom', {
type: 'cylinder',
size: '.033 .012',
zaxis: axle,
contype: 0,
conaffinity: 0,
mass: 0,
group: 1,
rgba: '.15 .18 .2 1',
}),
);
for (let j = 0; j < 12; j++) {
const angle = (2 * Math.PI * j) / 12,
s = Math.sin(angle),
k = Math.cos(angle);
const axis = `${k * tx} ${k * ty} ${-s}`;
const roller = node(doc, 'body', { pos: `${0.041 * s * tx} ${0.041 * s * ty} ${0.041 * k}` });
roller.append(
node(doc, 'joint', { axis, limited: 'false', damping: '.000005', armature: '.0000001' }),
);
roller.append(
node(doc, 'geom', {
type: 'capsule',
size: '.009 .006',
zaxis: axis,
mass: '.003',
contype: 4,
conaffinity: 1,
group: 1,
friction: '1 .001 .0001',
condim: 3,
solref: '.008 1',
rgba: '.3 .3 .3 1',
}),
);
wheel.append(roller);
}
base.append(wheel);
actuators.append(
node(doc, 'velocity', {
name: c.baseActuators[i],
joint: name,
kv: '.5',
ctrllimited: 'true',
ctrlrange: `${-c.wheelLimit} ${c.wheelLimit}`,
forcelimited: 'true',
forcerange: '-2 2',
}),
);
}
}
export function composeMobileScene(data: Uint8Array, config: RobotConfig): Uint8Array {
const doc = parse(data),
root = doc.documentElement;
if (root.tagName !== 'mujoco') throw new Error('场景组合需要已转换的 MJCF');
if (
Array.from(doc.querySelectorAll('[name]')).some((e) =>
e.getAttribute('name')!.startsWith('__mm_'),
)
)
throw new Error('任务命名空间 __mm_ 冲突(请勿重复组合)');
if (config.recipe === 'lekiwi-bundle') adaptBundle(doc, config);
const world = root.querySelector('worldbody');
if (!world) throw new Error('缺少 worldbody');
// A single task floor, no inherited robot-scene floors.
for (const geom of world.querySelectorAll(':scope > geom[type="plane"]')) geom.remove();
world.append(
node(doc, 'geom', {
name: '__mm_floor',
type: 'plane',
group: 2,
size: '3 3 .1',
friction: '.8 .005 .0001',
contype: 1,
conaffinity: 7,
rgba: '.25 .3 .35 1',
}),
);
const object = node(doc, 'body', { name: '__mm_object', pos: T.objectStart.join(' ') });
object.append(node(doc, 'freejoint', { name: '__mm_object_joint' }));
object.append(
node(doc, 'geom', {
name: '__mm_object_geom',
type: 'box',
group: 1,
size: `${T.objectHalfSize} ${T.objectHalfSize} ${T.objectHalfSize}`,
mass: '.04',
friction: '1 .005 .0001',
condim: 4,
contype: 1,
conaffinity: 7,
rgba: '1 .35 .08 1',
}),
);
world.append(object);
const goal = node(doc, 'body', { name: '__mm_goal', mocap: 'true', pos: T.goalStart.join(' ') });
goal.append(
node(doc, 'geom', {
name: '__mm_goal_geom',
type: 'cylinder',
group: 1,
size: '.05 .003',
contype: 0,
conaffinity: 0,
rgba: '.1 1 .35 .45',
}),
);
world.append(goal);
if (config.eefSiteName)
named(doc, 'body', config.eefBodyName).append(
node(doc, 'site', {
name: config.eefSiteName,
pos: config.eefOffset.join(' '),
size: '.007',
rgba: '.2 .8 1 1',
}),
);
const option = root.querySelector('option') ?? node(doc, 'option');
if (!option.parentElement) root.prepend(option);
// Preserve reviewed source physics (the full LeKiwi collision recipe uses 1 ms).
if (!option.hasAttribute('timestep')) option.setAttribute('timestep', '.002');
if (!option.hasAttribute('integrator')) option.setAttribute('integrator', 'implicitfast');
root.querySelector('keyframe')?.remove(); // nq grew: old keys are no longer valid.
return encoder.encode(new XMLSerializer().serializeToString(doc));
}
@@ -0,0 +1,107 @@
import * as THREE from 'three';
import type { MuJoCoViewer } from '../viewer/MuJoCoViewer';
import type { RobotManager } from './RobotManager';
/** Shift + primary drag uses the viewer's actual rendered meshes and world XY plane. */
export class TaskDragController {
private readonly ray = new THREE.Raycaster();
private readonly pointer = new THREE.Vector2();
private readonly plane = new THREE.Plane(new THREE.Vector3(0, 0, 1));
private readonly point = new THREE.Vector3();
private readonly offset = new THREE.Vector3();
private readonly position = [0, 0, 0];
private entity?: 'object' | 'goal';
private pointerId?: number;
private generation?: number;
private dragEnv?: RobotManager['env'];
private readonly canvas: HTMLCanvasElement;
constructor(
private readonly viewer: MuJoCoViewer,
private readonly manager: RobotManager,
) {
this.canvas = viewer.renderer.domElement;
this.canvas.addEventListener('pointerdown', this.down, true);
this.canvas.addEventListener('pointermove', this.move, true);
this.canvas.addEventListener('pointerup', this.up, true);
this.canvas.addEventListener('pointercancel', this.up, true);
this.canvas.addEventListener('lostpointercapture', this.up, true);
window.addEventListener('blur', this.up);
}
private setRay(e: PointerEvent): void {
const r = this.canvas.getBoundingClientRect();
this.pointer.set(
(2 * (e.clientX - r.left)) / r.width - 1,
1 - (2 * (e.clientY - r.top)) / r.height,
);
this.ray.setFromCamera(this.pointer, this.viewer.camera);
}
private down = (e: PointerEvent): void => {
const env = this.manager.env;
if (!env || this.manager.loading || e.button !== 0 || !e.shiftKey) return;
this.setRay(e);
const hit = this.ray
.intersectObjects(this.viewer.scene.children, true)
.find(
(h) =>
h.object.visible &&
(h.object.userData.bodyId === env.objectBodyId ||
h.object.userData.bodyId === env.goalBodyId),
);
if (!hit) return;
this.entity = hit.object.userData.bodyId === env.objectBodyId ? 'object' : 'goal';
this.manager.setRunning(false);
env.observe();
const index = this.entity === 'object' ? 37 : 47;
const z = env.state[index + 2];
this.plane.constant = -z;
if (!this.ray.ray.intersectPlane(this.plane, this.point)) {
this.entity = undefined;
return;
}
this.offset.set(env.state[index], env.state[index + 1], z).sub(this.point);
this.generation = env.revision;
this.dragEnv = env;
this.pointerId = e.pointerId;
this.viewer.controls.enabled = false;
this.canvas.setPointerCapture(e.pointerId);
e.preventDefault();
e.stopImmediatePropagation();
};
private move = (e: PointerEvent): void => {
if (!this.entity || e.pointerId !== this.pointerId) return;
const env = this.manager.env;
if (!env || this.manager.loading || env !== this.dragEnv || env.revision !== this.generation) {
this.up();
return;
}
this.setRay(e);
if (this.ray.ray.intersectPlane(this.plane, this.point)) {
this.point.add(this.offset);
this.position[0] = Math.max(-1.9, Math.min(1.9, this.point.x));
this.position[1] = Math.max(-1.9, Math.min(1.9, this.point.y));
this.position[2] = Math.max(0.019, Math.min(1.9, this.point.z));
this.manager.move(this.entity, this.position);
this.generation = env.revision;
}
e.preventDefault();
e.stopImmediatePropagation();
};
private up = (): void => {
const id = this.pointerId;
this.entity = undefined;
this.pointerId = undefined;
this.dragEnv = undefined;
this.viewer.controls.enabled = true;
if (id !== undefined && this.canvas.hasPointerCapture(id))
this.canvas.releasePointerCapture(id);
};
dispose(): void {
this.up();
this.canvas.removeEventListener('pointerdown', this.down, true);
this.canvas.removeEventListener('pointermove', this.move, true);
this.canvas.removeEventListener('pointerup', this.up, true);
this.canvas.removeEventListener('pointercancel', this.up, true);
this.canvas.removeEventListener('lostpointercapture', this.up, true);
window.removeEventListener('blur', this.up);
}
}
@@ -0,0 +1,95 @@
import golden from '../../../contracts/fixtures/mobile-golden.json';
import { MOBILE_ROBOTS, MOBILE_TASK as T, validateRobotConfig } from './RobotDescriptor';
import { decodeAction, TaskKernel } from './TaskKernel';
function state() {
const s = new Float64Array(T.stateSize);
for (const i of [3, 33, 40, 50]) s[i] = 1;
s[39] = s[49] = T.objectStart[2];
return s;
}
describe('mobile-manipulator-v2 数学契约', () => {
it('两个描述符有效,名称/维度错误拒绝', () => {
for (const c of MOBILE_ROBOTS) validateRobotConfig(c);
expect(() => validateRobotConfig({ ...MOBILE_ROBOTS[0], baseMix: [[1]] })).toThrow();
expect(() =>
validateRobotConfig({
...MOBILE_ROBOTS[0],
armJoints: Array(9).fill(MOBILE_ROBOTS[0].armJoints[0]),
}),
).toThrow();
});
it('与 Python 40 组独立 golden 状态、旋转、控制及奖励一致', () => {
for (const sample of golden) {
const config = MOBILE_ROBOTS.find((r) => r.id === sample.robotId)!;
const kernel = new TaskKernel(config);
kernel.hasLifted = sample.lifted;
const result = kernel.evaluate(Float64Array.from(sample.state));
sample.observation.forEach((v, i) => expect(result.observation[i]).toBeCloseTo(v, 6));
expect(result.reward).toBeCloseTo(sample.reward, 12);
expect(result.info.stage).toBe(sample.info.stage);
expect(result.terminated).toBe(sample.terminated);
expect(result.truncated).toBe(sample.truncated);
const control = new Float64Array(9);
decodeAction(config, Float32Array.from(sample.action), control);
sample.control.forEach((v, i) => expect(control[i]).toBeCloseTo(v, 10));
}
});
it('退化四元数采用与 Python 相同的单位姿态回退', () => {
const k = new TaskKernel(MOBILE_ROBOTS[0]);
const o = k.observe(new Float64Array(T.stateSize));
for (const start of [3, 41, 48, 55, 62])
expect(Array.from(o.slice(start, start + 4))).toEqual([1, 0, 0, 0]);
});
it('成功必须经过抓取抬升、张开夹爪、末端撤离、连续稳定放置', () => {
expect(
T.controlDt * T.maxSteps * (T.reachWeight + T.liftWeight + T.transportWeight),
).toBeLessThan(T.successBonus);
const k = new TaskKernel(MOBILE_ROBOTS[0]),
s = state();
s[29] = 1;
s[30] = 1;
for (let i = 0; i < 20; i++) expect(k.evaluate(s).terminated).toBe(false);
s[29] = 0;
s[30] = 0;
s[39] = s[32] = 0.12;
expect(k.evaluate(s).info.stage).toBe('transport');
s[29] = 1;
s[30] = 1;
s[39] = T.objectStart[2];
for (let i = 0; i < T.settleSteps - 1; i++) expect(k.evaluate(s).terminated).toBe(false);
expect(k.evaluate(s).info.reward_components.success).toBe(T.successBonus);
expect(() => k.evaluate(s)).toThrow('回合');
k.reset();
expect(k.hasLifted).toBe(false);
expect(k.result.terminated).toBe(false);
k.steps = T.maxSteps - 1;
expect(k.evaluate(s).truncated).toBe(true);
});
it('复用所有输出缓冲区,位置/速度控制,拒绝 NaN 不部分写入', () => {
const c = structuredClone(MOBILE_ROBOTS[1]);
const config = {
...c,
armJoints: c.armJoints.map((j, i) => (i === 0 ? { ...j, mode: 'velocity' as const } : j)),
};
const action = new Float32Array(12),
ctrl = new Float64Array(9);
action[3] = 0.5;
action[11] = 1;
decodeAction(config, action, ctrl);
expect(ctrl[3]).toBe(1);
expect(ctrl[8]).toBeCloseTo(0.9, 12);
const old = ctrl.slice();
action[11] = NaN;
expect(() => decodeAction(config, action, ctrl)).toThrow();
expect(ctrl).toEqual(old);
const k = new TaskKernel(config),
s = state(),
first = k.evaluate(s);
for (let i = 0; i < 100; i++) {
expect(k.evaluate(s)).toBe(first);
expect(k.observe(s)).toBe(first.observation);
expect(k.observation.slice(34, 37)).toEqual(new Float32Array(3));
}
});
});
+301
View File
@@ -0,0 +1,301 @@
import { MOBILE_TASK as T, type RobotConfig } from './RobotDescriptor';
export const clamp = (v: number, lo = -1, hi = 1): number => Math.max(lo, Math.min(hi, v));
export const TRAINING_STAGES = ['navigate', 'reach', 'pick-place'] as const;
export type TrainingStage = (typeof TRAINING_STAGES)[number];
export type TaskStage = 'navigate' | 'reach' | 'lift' | 'transport' | 'success' | 'safety-stop';
export function navigationError(s: Float64Array): { distance: number; yaw: number } {
return {
distance: Math.hypot(
s[37] - T.navigationOffset[0] - s[0],
s[38] - T.navigationOffset[1] - s[1],
),
yaw: Math.atan2(2 * (s[3] * s[6] + s[4] * s[5]), 1 - 2 * (s[5] ** 2 + s[6] ** 2)),
};
}
export interface TaskInfo {
reward_components: {
reach: number;
lift: number;
transport: number;
success: number;
navigation: number;
action_rate: number;
joint_velocity: number;
safety: number;
};
safety_stop: string;
navigation_distance: number;
max_joint_velocity: number;
is_success: boolean;
stage: TaskStage;
}
export interface StepResult {
observation: Float32Array;
reward: number;
terminated: boolean;
truncated: boolean;
info: TaskInfo;
}
/** v1 raw state layout (SI, world frame, quaternions wxyz):
* base pose 0:7, world linear/angular velocity 7:13, arm q 13:21, qd 21:29,
* opening [0,1] 29, EEF pose 30:37, object pose 37:44, object world velocity 44:47,
* goal pose 47:54. Padding slots are zero. See docs/mobile-manipulator.md.
*/
export class TaskKernel {
readonly observation = new Float32Array(T.observationSize);
readonly lastAction = new Float32Array(T.actionSize);
readonly targets = new Float32Array(T.actionSize);
private actionRate = 0;
private previousDistance?: number;
readonly result: StepResult = {
observation: this.observation,
reward: 0,
terminated: false,
truncated: false,
info: {
reward_components: {
reach: 0,
lift: 0,
transport: 0,
success: 0,
navigation: 0,
action_rate: 0,
joint_velocity: 0,
safety: 0,
},
safety_stop: '',
navigation_distance: 0,
max_joint_velocity: 0,
is_success: false,
stage: 'reach',
},
};
steps = 0;
hasLifted = false;
settle = 0;
constructor(
readonly config: RobotConfig,
public trainingStage: TrainingStage = 'pick-place',
) {}
reset(): void {
this.steps = this.settle = 0;
this.hasLifted = false;
this.result.reward = 0;
this.result.terminated = this.result.truncated = this.result.info.is_success = false;
this.result.info.stage = this.trainingStage === 'navigate' ? 'navigate' : 'reach';
this.result.info.safety_stop = '';
this.result.info.max_joint_velocity = this.result.info.navigation_distance = 0;
this.lastAction.fill(0);
this.targets.fill(0);
this.actionRate = 0;
this.previousDistance = undefined;
const r = this.result.info.reward_components;
r.reach =
r.lift =
r.transport =
r.success =
r.navigation =
r.action_rate =
r.joint_velocity =
r.safety =
0;
}
recordAction(applied: Float32Array, targets: Float32Array): void {
this.actionRate = 0;
for (let i = 0; i < T.actionSize; i++)
this.actionRate += (applied[i] - this.lastAction[i]) ** 2;
this.lastAction.set(applied);
this.targets.set(targets);
}
observe(s: Float64Array): Float32Array {
const o = this.observation;
o.fill(0);
for (let i = 0; i < 3; i++) o[i] = clamp(s[i] / T.positionScale);
this.quat(s, 3, 3);
for (let i = 0; i < 6; i++)
o[7 + i] = clamp(s[7 + i] / (i < 3 ? T.linearVelocityScale : T.angularVelocityScale));
for (let i = 0; i < this.config.armJoints.length; i++) {
const j = this.config.armJoints[i];
o[13 + i] = clamp((2 * (s[13 + i] - j.min)) / (j.max - j.min) - 1);
o[21 + i] = clamp(s[21 + i] / T.jointVelocityScale);
o[29 + i] = 1;
}
o[37] = clamp(2 * s[29] - 1);
for (let i = 0; i < 3; i++) {
o[38 + i] = clamp(s[30 + i] / T.positionScale);
o[59 + i] = clamp(s[47 + i] / T.positionScale);
}
this.quat(s, 33, 41);
this.relative(s, 30, 37, 45);
this.relative(s, 47, 37, 52);
this.quat(s, 50, 62);
o[66] = Number(this.hasLifted);
o[67] = this.settle / T.settleSteps;
o.set(this.lastAction, 68);
o.set(this.targets, 80);
return o;
}
private quat(s: Float64Array, start: number, target: number): void {
const n = Math.hypot(s[start], s[start + 1], s[start + 2], s[start + 3]);
const sign = s[start] < 0 ? -1 : 1;
for (let i = 0; i < 4; i++)
this.observation[target + i] = n > 1e-12 ? (sign * s[start + i]) / n : Number(i === 0);
}
/** inverse(parent) * child, including position rotated into parent frame. */
private relative(s: Float64Array, parent: number, child: number, out: number): void {
const w = s[parent + 3],
x = -s[parent + 4],
y = -s[parent + 5],
z = -s[parent + 6];
const vx = s[child] - s[parent],
vy = s[child + 1] - s[parent + 1],
vz = s[child + 2] - s[parent + 2];
const tx = 2 * (y * vz - z * vy),
ty = 2 * (z * vx - x * vz),
tz = 2 * (x * vy - y * vx);
const o = this.observation;
o[out] = clamp((vx + w * tx + y * tz - z * ty) / T.positionScale);
o[out + 1] = clamp((vy + w * ty + z * tx - x * tz) / T.positionScale);
o[out + 2] = clamp((vz + w * tz + x * ty - y * tx) / T.positionScale);
const a = s[child + 3],
b = s[child + 4],
c = s[child + 5],
d = s[child + 6];
const rw = w * a - x * b - y * c - z * d;
const rx = w * b + x * a + y * d - z * c;
const ry = w * c - x * d + y * a + z * b;
const rz = w * d + x * c - y * b + z * a;
const n = Math.hypot(rw, rx, ry, rz),
k = n > 1e-12 ? (rw < 0 ? -1 : 1) / n : 0;
o[out + 3] = n > 1e-12 ? rw * k : 1;
o[out + 4] = rx * k;
o[out + 5] = ry * k;
o[out + 6] = rz * k;
}
evaluate(s: Float64Array, safetyStop = '', peakVelocity = 0): StepResult {
if (this.result.terminated || this.result.truncated) throw new Error('回合已结束,请 reset');
const reach = Math.hypot(s[37] - s[30], s[38] - s[31], s[39] - s[32]);
const goal = Math.hypot(s[37] - s[47], s[38] - s[48], s[39] - s[49]);
const lift = clamp((s[39] - T.objectStart[2]) / T.liftHeight, 0, 1);
if (this.trainingStage === 'pick-place' && lift >= 1 && reach < T.graspDistance && s[29] < 0.4)
this.hasLifted = true;
const { distance, yaw } = navigationError(s);
const near = distance < T.navigationTolerance && Math.abs(yaw) < T.navigationYawTolerance;
const stopped =
Math.hypot(s[7], s[8], s[9]) < T.navigationSpeedTolerance &&
Math.hypot(s[10], s[11], s[12]) < 0.1;
let settled =
this.hasLifted &&
goal < T.goalTolerance &&
s[29] > T.releaseOpening &&
reach > T.graspDistance &&
Math.hypot(s[44], s[45], s[46]) < T.settleSpeed;
if (this.trainingStage === 'navigate') settled = near && stopped;
else if (this.trainingStage === 'reach')
settled =
near && stopped && reach < T.graspDistance && Math.hypot(...s.subarray(21, 29)) < 0.15;
this.settle = settled ? Math.min(T.settleSteps, this.settle + 1) : 0;
let success = this.settle === T.settleSteps;
const r = this.result.info.reward_components;
// Dense rates are integrated over time; action-rate cost acts on control changes.
r.reach = T.controlDt * T.reachWeight * Math.exp(-T.reachGain * reach);
r.lift = T.controlDt * T.liftWeight * lift;
r.transport = this.hasLifted
? T.controlDt * T.transportWeight * Math.exp(-T.transportGain * goal)
: 0;
if (this.trainingStage === 'navigate') r.reach = r.lift = r.transport = 0;
else if (this.trainingStage === 'reach') r.lift = r.transport = 0;
const progress = this.previousDistance === undefined ? 0 : this.previousDistance - distance;
this.previousDistance = distance;
r.navigation = this.hasLifted
? 0
: T.navigationProgressWeight * progress - T.controlDt * (distance + 0.1 * Math.abs(yaw));
r.action_rate = -T.actionRateWeight * this.actionRate;
r.joint_velocity = 0;
for (let i = 21; i < 29; i++) {
r.joint_velocity -= T.controlDt * T.jointVelocityWeight * s[i] ** 2;
peakVelocity = Math.max(peakVelocity, Math.abs(s[i]));
}
if (!safetyStop && peakVelocity > T.jointSpeedStop) safetyStop = 'joint_velocity';
if (
!safetyStop &&
(1 - 2 * (s[4] ** 2 + s[5] ** 2) < 0.5 ||
Math.max(Math.abs(s[0]), Math.abs(s[1])) > T.positionScale)
)
safetyStop = 'base_pose';
if (safetyStop) success = false;
r.success = success ? T.successBonus : 0;
r.safety = safetyStop ? -T.safetyPenalty : 0;
this.result.reward =
r.reach +
r.lift +
r.transport +
r.success +
r.navigation +
r.action_rate +
r.joint_velocity +
r.safety;
this.result.terminated = success || !!safetyStop;
this.result.info.safety_stop = safetyStop;
this.result.info.navigation_distance = distance;
this.result.info.max_joint_velocity = Math.max(
this.result.info.max_joint_velocity,
peakVelocity,
);
this.result.truncated = ++this.steps >= T.maxSteps && !success;
this.result.info.is_success = success;
this.result.info.stage = safetyStop
? 'safety-stop'
: success
? 'success'
: this.trainingStage === 'navigate' || (!near && !this.hasLifted)
? 'navigate'
: this.hasLifted
? 'transport'
: reach < T.graspDistance
? 'lift'
: 'reach';
this.observe(s);
return this.result;
}
}
/** @deprecated v1 math oracle retained ONLY for historical regression fixtures.
* Production v2 environments use SafeActionController, never this full-range mapper.
* Output order: base actuators, arm actuators, gripper actuators. */
export function decodeAction(
config: RobotConfig,
action: Float32Array,
output: Float64Array,
): void {
if (action.length !== T.actionSize) throw new Error('动作必须是 [12]');
for (let i = 0; i < action.length; i++)
if (!Number.isFinite(action[i])) throw new Error('动作包含非有限数');
let largest = config.wheelLimit;
for (let i = 0; i < config.baseMix.length; i++) {
const row = config.baseMix[i];
output[i] =
row[0] * clamp(action[0]) * config.baseLimits[0] +
row[1] * clamp(action[1]) * config.baseLimits[1] +
row[2] * clamp(action[2]) * config.baseLimits[2];
largest = Math.max(largest, Math.abs(output[i]));
}
for (let i = 0; i < config.baseMix.length; i++) output[i] *= config.wheelLimit / largest;
let offset = config.baseMix.length;
for (let i = 0; i < config.armJoints.length; i++) {
const j = config.armJoints[i],
a = clamp(action[3 + i]);
output[offset++] =
j.mode === 'position' ? j.min + (a + 1) * 0.5 * (j.max - j.min) : a * j.velocityLimit;
}
const opening = (clamp(action[11]) + 1) * 0.5;
for (const g of config.gripperActuators)
output[offset++] = g.closed + opening * (g.open - g.closed);
}
@@ -0,0 +1,336 @@
import { describe, expect, it, vi } from 'vitest';
import { AgentTaskController, type SkillExecutor } from './AgentTaskController';
import { MockDecisionProvider } from './MockDecisionProvider';
import { PickPlaceEvaluator, type PhysicalSample } from './PhysicalEvidence';
import { SKILLS, type Skill } from './protocol';
class FakeSkills implements SkillExecutor {
phase: Skill = 'open';
waiting = true;
status: 'running' | 'succeeded' | 'failed' = 'running';
error = '';
completed = false;
fault = '';
goal = [0.257, 0.615, 0.128];
evaluator = new PickPlaceEvaluator([0.257, 0.015, 0.128], this.goal);
sample: PhysicalSample = {
time: 0,
base: [0, 0, 0.05],
baseSpeed: 0,
yaw: 0,
tilt: 0,
joints: [0, 0, 0, 0, 0],
peakJointSpeed: 0,
opening: 1,
tcp: [0.257, 0.015, 0.2],
object: [0.257, 0.015, 0.128],
objectSpeed: 0,
fingerForces: [0, 0],
supported: true,
onGoalSupport: false,
};
authorized: Skill[] = [];
hold = vi.fn();
dispose = vi.fn();
beforeStep = vi.fn();
get candidate(): Skill {
return this.completed ? (SKILLS[SKILLS.indexOf(this.phase) + 1] ?? 'settle') : this.phase;
}
authorize(skill: Skill) {
if (!this.waiting || skill !== this.candidate) throw new Error('bad switch');
if (skill === 'carry' && !this.evaluator.secure) throw new Error('unsafe');
this.authorized.push(skill);
this.phase = skill;
this.completed = false;
this.waiting = false;
}
afterStep(dt: number) {
this.sample.time += dt;
if (this.fault && this.phase === 'close') {
this.status = 'failed';
this.error = this.fault;
return;
}
if (this.phase === 'verify') this.evaluator.secure = true;
if (this.phase === 'carry') this.evaluator.transported = 0.6;
if (this.phase === 'release') this.evaluator.secure = false;
this.completed = true;
this.waiting = true;
if (this.phase === 'settle') {
this.evaluator.succeeded = true;
this.status = 'succeeded';
}
}
canRecover() {
return this.status === 'failed' && ['empty_grasp', 'alignment_timeout'].includes(this.error);
}
recover() {
if (!this.canRecover()) throw new Error('unsafe recovery');
this.phase = 'open';
this.status = 'running';
this.error = '';
this.waiting = true;
this.completed = false;
}
}
async function flush() {
for (let i = 0; i < 12; i++) await Promise.resolve();
}
async function run(driver: AgentTaskController) {
for (let i = 0; i < 300 && !driver.terminal; i++) {
if (driver.beforeStep()) driver.afterStep(0.001);
await flush();
}
expect(driver.terminal).toBe(true);
}
describe('模型闭环调度(技能为 mock,物理另有真实 WASM 验收)', () => {
it('LLM 一次,Jev 每个边界一次;异步回调不授权,独立证据才成功', async () => {
const skills = new FakeSkills();
const driver = new AgentTaskController(skills, new MockDecisionProvider(), '搬运方块');
expect(driver.beforeStep()).toBe(false);
await flush();
expect(skills.authorized).toEqual([]);
expect(skills.sample.time).toBe(0);
await run(driver);
expect(driver.state).toBe('succeeded');
expect(driver.llmCalls).toBe(1);
expect(driver.jevCalls).toBe(11);
expect(skills.authorized).toEqual(SKILLS);
expect(driver.results).toHaveLength(11);
driver.dispose();
});
it('支持可恢复空抓,但跨重规划也不能重置每技能 2 次上限', async () => {
const skills = new FakeSkills();
skills.fault = 'empty_grasp';
const driver = new AgentTaskController(skills, new MockDecisionProvider(), '搬运方块');
await run(driver);
expect(driver.state).toBe('failed');
expect(driver.recoveries.close).toBe(2);
expect(driver.failures).toHaveLength(3);
expect(driver.jevCalls).toBeLessThanOrEqual(60);
expect(skills.authorized).not.toContain('carry');
driver.dispose();
});
it('关节卡住/失抓/不可达等硬失败立即停,不询问模型放行', async () => {
for (const fault of ['joint_velocity', 'lost_grasp', 'unreachable', 'navigation_timeout']) {
const skills = new FakeSkills();
skills.fault = fault;
const driver = new AgentTaskController(skills, new MockDecisionProvider(), '搬运方块');
await run(driver);
expect(driver.error).toBe(fault);
expect(driver.recoveries).toEqual({});
expect(driver.jevCalls).toBe(4);
driver.dispose();
}
});
it('物理成功不能由模型的 secure 声明替代', async () => {
const skills = new FakeSkills(),
provider = new MockDecisionProvider();
const normal = provider.decide.bind(provider);
provider.decide = async (...args) => {
const value = await normal(...args);
value.value.grasp = 'secure';
return value;
};
const driver = new AgentTaskController(skills, provider, '搬运方块');
await run(driver);
expect(driver.state).toBe('failed');
expect(driver.error).toContain('矛盾');
expect(skills.authorized).toEqual([]);
driver.dispose();
});
it('未知候选/循环计划拒绝;真实 API 模式错误不会降级 mock', async () => {
const skills = new FakeSkills(),
provider = new MockDecisionProvider();
const normal = provider.plan.bind(provider);
provider.plan = async (...args) => {
const value = await normal(...args);
value.value.steps.reverse();
return value;
};
const driver = new AgentTaskController(skills, provider, '搬运方块');
await run(driver);
expect(driver.state).toBe('failed');
expect(driver.jevCalls).toBe(0);
driver.dispose();
const api = {
mode: 'api' as const,
plan: vi.fn().mockRejectedValue(new Error('upstream_http_401')),
decide: vi.fn(),
};
const failed = new AgentTaskController(new FakeSkills(), api, '搬运方块');
await run(failed);
expect(failed.error).toBe('upstream_http_401');
expect(api.decide).not.toHaveBeenCalled();
failed.dispose();
});
it('正常边界可请求重规划但最多 2 次,不暗中继续动作', async () => {
const skills = new FakeSkills(),
provider = new MockDecisionProvider();
const normal = provider.decide.bind(provider);
provider.decide = async (...args) => {
const value = await normal(...args);
value.value.recovery = 'replan';
return value;
};
const driver = new AgentTaskController(skills, provider, '搬运方块');
await run(driver);
expect(driver.replans).toBe(2);
expect(driver.llmCalls).toBe(3);
expect(driver.error).toContain('重规划预算');
expect(skills.authorized).toEqual([]);
driver.dispose();
});
it('LLM 要求重规划时,Jev 的重试建议不能绕过该分支', async () => {
const skills = new FakeSkills();
skills.fault = 'empty_grasp';
const provider = new MockDecisionProvider(),
normal = provider.plan.bind(provider);
provider.plan = async (...args) => {
const result = await normal(...args);
result.value.steps.forEach((step) => {
step.onFailure = 'replan';
});
return result;
};
const driver = new AgentTaskController(skills, provider, '搬运');
await run(driver);
expect(driver.replans).toBe(2);
expect(driver.llmCalls).toBe(3);
expect(driver.recoveries.close).toBe(2);
driver.dispose();
});
it('恢复中的重规划也受技能次数限制', async () => {
const skills = new FakeSkills();
skills.fault = 'empty_grasp';
const provider = new MockDecisionProvider(),
normal = provider.decide.bind(provider);
provider.decide = async (...args) => {
const value = await normal(...args);
if (args[2] !== 'none') value.value.recovery = 'replan';
return value;
};
const driver = new AgentTaskController(skills, provider, '搬运方块');
await run(driver);
expect(driver.replans).toBe(2);
expect(driver.recoveries.close).toBe(2);
expect(driver.state).toBe('failed');
driver.dispose();
});
it('请求量和墙钟期限同时有限;模型等待不推进技能/物理时间', async () => {
const skills = new FakeSkills();
let now = 0;
const driver = new AgentTaskController(skills, new MockDecisionProvider(), '搬运方块', {
limits: { maxJev: 1 },
now: () => now,
});
await run(driver);
expect(driver.error).toContain('Jev 请求预算');
expect(driver.jevCalls).toBe(1);
driver.dispose();
const limited = new AgentTaskController(
new FakeSkills(),
new MockDecisionProvider(),
'搬运方块',
{ limits: { wallMs: 10 }, now: () => now },
);
limited.beforeStep();
now = 11;
limited.beforeStep();
expect(limited.error).toContain('墙钟期限');
limited.dispose();
});
it('任务暂停取消旧响应;继续只能处理新请求,支持单技能步进', async () => {
const skills = new FakeSkills(),
provider = new MockDecisionProvider();
const normal = provider.plan.bind(provider);
let resolve!: () => void;
provider.plan = async (...args) => {
await new Promise<void>((r) => {
resolve = r;
});
return normal(...args);
};
const driver = new AgentTaskController(skills, provider, '搬运方块');
driver.beforeStep();
driver.pause();
resolve();
await flush();
expect(driver.state).toBe('paused');
expect(skills.sample.time).toBe(0);
provider.plan = normal;
driver.resume(true);
for (let i = 0; i < 30 && driver.state !== 'paused'; i++) {
if (driver.beforeStep()) driver.afterStep(0.001);
await flush();
}
expect(driver.state).toBe('paused');
expect(skills.authorized).toEqual(['open']);
const time = skills.sample.time;
driver.beforeStep();
expect(skills.sample.time).toBe(time);
driver.resume();
await run(driver);
expect(driver.state).toBe('succeeded');
driver.dispose();
});
it('模型超时/错误候选均冻结并失败,禁止生成额外物理步', async () => {
const skills = new FakeSkills(),
provider = new MockDecisionProvider();
provider.plan = () => new Promise(() => undefined);
const driver = new AgentTaskController(skills, provider, '搬运', { limits: { requestMs: 1 } });
driver.beforeStep();
await new Promise((r) => setTimeout(r, 5));
await flush();
driver.beforeStep();
expect(driver.error).toContain('超时');
expect(skills.sample.time).toBe(0);
driver.dispose();
const bad = new MockDecisionProvider(),
normal = bad.decide.bind(bad);
bad.decide = async (...args) => {
const value = await normal(...args);
value.value.choice = 'carry';
return value;
};
const invalid = new AgentTaskController(new FakeSkills(), bad, '搬运');
await run(invalid);
expect(invalid.error).toContain('候选');
invalid.dispose();
});
it('技能自报成功但缺少独立评估证据仍然失败', async () => {
const skills = new FakeSkills(),
normal = skills.afterStep.bind(skills);
skills.afterStep = (dt) => {
normal(dt);
if (skills.status === 'succeeded') skills.evaluator.succeeded = false;
};
const driver = new AgentTaskController(skills, new MockDecisionProvider(), '搬运');
await run(driver);
expect(driver.state).toBe('failed');
expect(driver.results.at(-1)?.code).toBe('placement_unverified');
driver.dispose();
});
it('取消/销毁后迟到结果不能授权;模型忽略 abort 也不能越过门禁', async () => {
const skills = new FakeSkills(),
provider = new MockDecisionProvider();
const normal = provider.plan.bind(provider);
let resolve!: () => void;
provider.plan = async (obs, instruction, remaining) => {
await new Promise<void>((r) => {
resolve = r;
});
return normal(obs, instruction, remaining, new AbortController().signal);
};
const driver = new AgentTaskController(skills, provider, '搬运方块');
driver.beforeStep();
driver.cancel('scene changed');
driver.dispose();
resolve();
await flush();
expect(driver.beforeStep()).toBe(false);
expect(skills.authorized).toEqual([]);
expect(skills.dispose).toHaveBeenCalledOnce();
expect(driver.state).toBe('cancelled');
});
});
@@ -0,0 +1,412 @@
import type { AgentDriver } from '../../simulation/AgentDriver';
import type { DeterministicBaseline } from './DeterministicBaseline';
import type { DecisionProvider } from './DecisionClient';
import { RequestGate, type RequestStamp } from './RequestGate';
import {
AGENT_VERSION,
SKILLS,
observation,
validateJevDecision,
validatePlan,
skillErrorCode,
type JevDecision,
type Plan,
type Skill,
type SkillResult,
} from './protocol';
export type SkillExecutor = Pick<
DeterministicBaseline,
| 'sample'
| 'evaluator'
| 'phase'
| 'candidate'
| 'goal'
| 'status'
| 'error'
| 'waiting'
| 'beforeStep'
| 'afterStep'
| 'authorize'
| 'canRecover'
| 'recover'
| 'hold'
| 'dispose'
>;
export type AgentState =
'idle' | 'planning' | 'deciding' | 'executing' | 'paused' | 'succeeded' | 'failed' | 'cancelled';
export const AGENT_LIMITS = {
maxRecoveries: 2,
maxReplans: 2,
maxJev: 60,
maxLlm: 3,
wallMs: 1_200_000,
requestMs: 65_000,
} as const;
type Limits = { [K in keyof typeof AGENT_LIMITS]: number };
type Outcome =
| { kind: 'plan'; value: unknown; remaining: Skill[] }
| { kind: 'jev'; value: unknown; candidates: (Skill | 'stop')[] }
| { kind: 'error'; error: string };
/** The sole driver of already-validated local skills. Promises only populate a mailbox;
* authorize/recover/servo/hold operations happen in the session's synchronous write window. */
export class AgentTaskController implements AgentDriver {
state: AgentState = 'idle';
error = '';
plan?: Plan;
lastDecision?: JevDecision;
readonly results: SkillResult[] = [];
readonly failures: { skill: Skill; code: string; sequence: number }[] = [];
readonly recoveries: Partial<Record<Skill, number>> = {};
llmCalls = 0;
jevCalls = 0;
replans = 0;
private readonly gate = new RequestGate();
private readonly limits: Limits;
private readonly now: () => number;
private readonly began: number;
private finishedAt?: number;
private readonly runId: string;
private readonly sceneRevision: number;
private sequence = 0;
private requestId = 0;
private planRevision = 0;
private epoch = 0;
private inflight = false;
private outcome?: Outcome;
private needPlan = true;
private recoveryPlan = false;
private paused = false;
private single = false;
private disposed = false;
private failureRecorded = false;
constructor(
readonly skills: SkillExecutor,
readonly provider: DecisionProvider,
private readonly instruction: string,
options: {
runId?: string;
sceneRevision?: number;
now?: () => number;
limits?: Partial<Limits>;
} = {},
) {
this.runId = options.runId ?? crypto.randomUUID();
this.sceneRevision = options.sceneRevision ?? 0;
this.now = options.now ?? (() => performance.now());
this.began = this.now();
this.limits = { ...AGENT_LIMITS, ...options.limits };
for (const [key, value] of Object.entries(this.limits))
if (
!Number.isInteger(value) ||
value < (['maxRecoveries', 'maxReplans'].includes(key) ? 0 : 1) ||
value > AGENT_LIMITS[key as keyof Limits]
)
throw new Error('智能任务预算非法或超过硬上限');
if (!instruction.trim() || Array.from(instruction).length > 2000)
throw new Error('任务指令为空或过长');
}
get terminal(): boolean {
return ['succeeded', 'failed', 'cancelled'].includes(this.state);
}
snapshot() {
return {
version: AGENT_VERSION,
goal: [...this.skills.goal],
mode: this.provider.mode,
wallMs: (this.finishedAt ?? this.now()) - this.began,
state: this.state,
error: this.error,
phase: this.skills.phase,
candidate: this.skills.candidate,
plan: this.plan,
decision: this.lastDecision,
llmCalls: this.llmCalls,
jevCalls: this.jevCalls,
replans: this.replans,
recoveries: { ...this.recoveries },
sample: this.skills.sample,
secure: this.skills.evaluator.secure,
transported: this.skills.evaluator.transported,
succeeded: this.skills.evaluator.succeeded,
runId: this.runId,
sceneRevision: this.sceneRevision,
sequence: this.sequence,
};
}
/** Task pause keeps the owner and freezes physics. Global session pause cancels the task. */
pause(): void {
if (this.terminal || this.disposed) return;
this.paused = true;
this.invalidate();
this.state = 'paused';
}
resume(singleSkill = false): void {
if (this.terminal || this.disposed) return;
this.paused = false;
this.single = singleSkill;
this.state = 'idle';
}
private invalidate(): void {
this.epoch++;
this.gate.invalidate('任务版本已失效');
this.inflight = false;
this.outcome = undefined;
}
private stamp(): RequestStamp {
return {
runId: this.runId,
sceneRevision: this.sceneRevision,
sequence: this.sequence,
planRevision: this.planRevision,
requestId: `${this.runId}:${++this.requestId}`,
};
}
private finish(state: 'failed' | 'succeeded', code = ''): void {
this.invalidate();
this.state = state;
this.finishedAt = this.now();
this.error = code;
this.skills.hold();
}
private recordFailure(): void {
if (this.failureRecorded) return;
this.failureRecorded = true;
this.failures.push({
skill: this.skills.phase,
code: this.skills.error,
sequence: this.sequence,
});
this.results.push({
version: AGENT_VERSION,
skill: this.skills.phase,
status: 'failed',
code: skillErrorCode(this.skills.error),
evidenceSequence: this.sequence,
});
}
private recoveryAllowed(): boolean {
return (
this.skills.canRecover() &&
(this.recoveries[this.skills.phase] ?? 0) < this.limits.maxRecoveries
);
}
private recover(): void {
if (!this.recoveryAllowed()) throw new Error('恢复不安全或次数已耗尽');
const phase = this.skills.phase;
this.skills.recover();
this.recoveries[phase] = (this.recoveries[phase] ?? 0) + 1;
this.failureRecorded = false;
}
private replan(recover: boolean): void {
if (this.replans >= this.limits.maxReplans) throw new Error('重规划预算耗尽');
if (recover && !this.recoveryAllowed()) throw new Error('恢复不安全或次数已耗尽');
this.replans++;
this.recoveryPlan = recover;
this.needPlan = true;
}
private requestPlan(): void {
if (this.llmCalls >= this.limits.maxLlm) throw new Error('LLM 请求预算耗尽');
this.llmCalls++;
this.planRevision++;
const remaining = this.recoveryPlan
? [...SKILLS]
: SKILLS.slice(SKILLS.indexOf(this.skills.candidate));
const stamp = this.stamp();
const obs = observation(
this.skills.sample,
this.skills.evaluator,
this.skills.goal,
stamp,
this.skills.phase,
);
const epoch = this.epoch;
this.state = 'planning';
this.inflight = true;
void this.gate
.request(stamp, this.limits.requestMs, (signal) =>
this.provider.plan(obs, this.instruction, remaining, signal),
)
.then((value) => this.deliver(epoch, { kind: 'plan', value, remaining }))
.catch((error: unknown) =>
this.deliver(epoch, {
kind: 'error',
error: error instanceof Error ? error.message : '模型规划失败',
}),
);
}
private requestJev(): void {
if (this.jevCalls >= this.limits.maxJev) throw new Error('Jev 请求预算耗尽');
this.jevCalls++;
const failed = this.skills.status === 'failed';
const candidates: (Skill | 'stop')[] = [failed ? 'open' : this.skills.candidate, 'stop'];
const stamp = this.stamp();
const obs = observation(
this.skills.sample,
this.skills.evaluator,
this.skills.goal,
stamp,
this.skills.phase,
);
const epoch = this.epoch;
this.state = 'deciding';
this.inflight = true;
void this.gate
.request(stamp, this.limits.requestMs, (signal) =>
this.provider.decide(obs, candidates, failed ? this.skills.error : 'none', signal),
)
.then((value) => this.deliver(epoch, { kind: 'jev', value, candidates }))
.catch((error: unknown) =>
this.deliver(epoch, {
kind: 'error',
error: error instanceof Error ? error.message : 'Jev 判定失败',
}),
);
}
private deliver(epoch: number, outcome: Outcome): void {
if (this.epoch !== epoch || this.terminal || this.disposed || this.paused) return;
this.inflight = false;
this.outcome = outcome; // No writes, physical steps or re-entrant ownership changes here.
}
private apply(outcome: Outcome): void {
if (outcome.kind === 'error') throw new Error(outcome.error);
if (outcome.kind === 'plan') {
this.plan = validatePlan(outcome.value, outcome.remaining);
this.needPlan = false;
if (this.recoveryPlan) {
this.recover();
this.recoveryPlan = false;
}
return;
}
const value = validateJevDecision(outcome.value, outcome.candidates);
this.lastDecision = value;
if (value.choice === 'stop' || value.recovery === 'stop') throw new Error('Jev 请求安全停止');
if (value.grasp === 'secure' && !this.skills.evaluator.secure)
throw new Error('Jev 抓持判断与本地物理证据矛盾');
const failed = this.skills.status === 'failed';
if (value.recovery === 'replan') {
this.replan(failed);
return;
}
if (failed) {
if (value.recovery !== 'retry' || value.choice !== 'open')
throw new Error('故障没有合法恢复选择');
const policy = this.plan?.steps.find((s) => s.skill === this.skills.phase)?.onFailure;
if (policy === 'replan') {
this.replan(true);
return;
}
if (policy !== 'retry') throw new Error('当前计划不允许直接重试');
this.recover();
} else if (value.recovery !== 'continue') throw new Error('无故障时不能重复技能');
if (
['carry', 'stop-base'].includes(value.choice) &&
(value.grasp !== 'secure' || !this.skills.evaluator.secure)
)
throw new Error('搬运前未取得一致抓持证据');
this.skills.authorize(value.choice);
this.state = 'executing';
}
beforeStep(): boolean {
if (this.disposed) return false;
try {
if (this.terminal) {
this.skills.hold();
return false;
}
if (this.now() - this.began > this.limits.wallMs) throw new Error('任务墙钟期限耗尽');
if (this.paused) {
this.skills.hold();
return false;
}
if (this.outcome) {
const outcome = this.outcome;
this.outcome = undefined;
this.apply(outcome);
}
if (this.needPlan && !this.inflight) this.requestPlan();
if (this.inflight) {
this.skills.hold();
return false;
}
if (this.skills.status === 'failed') {
this.recordFailure();
if (
!this.recoveryAllowed() ||
this.plan?.steps.find((s) => s.skill === this.skills.phase)?.onFailure === 'stop'
)
throw new Error(this.skills.error || '技能失败');
this.requestJev();
this.skills.hold();
return false;
}
if (this.skills.status === 'succeeded') {
if (!this.skills.evaluator.succeeded) throw new Error('缺少独立物理成功证据');
this.finish('succeeded');
return false;
}
if (this.skills.waiting) {
this.requestJev();
this.skills.hold();
return false;
}
this.state = 'executing';
this.skills.beforeStep();
return this.skills.status === 'running';
} catch (error) {
this.finish('failed', error instanceof Error ? error.message : '智能任务执行失败');
return false;
}
}
afterStep(dt: number): void {
if (this.disposed || this.terminal || this.paused) return;
this.sequence++;
this.skills.afterStep(dt);
if (this.skills.status === 'failed') this.recordFailure();
else if (this.skills.waiting || this.skills.status === 'succeeded') {
if (this.skills.status === 'succeeded' && !this.skills.evaluator.succeeded) {
this.results.push({
version: AGENT_VERSION,
skill: this.skills.phase,
status: 'failed',
code: 'placement_unverified',
evidenceSequence: this.sequence,
});
this.finish('failed', '缺少独立物理成功证据');
return;
}
this.results.push({
version: AGENT_VERSION,
skill: this.skills.phase,
status: 'completed',
code: 'none',
evidenceSequence: this.sequence,
});
if (this.skills.status === 'succeeded') {
this.finish(
this.skills.evaluator.succeeded ? 'succeeded' : 'failed',
this.skills.evaluator.succeeded ? '' : '缺少独立物理成功证据',
);
} else if (this.single) this.pause();
}
}
hold(): void {
if (!this.disposed) this.skills.hold();
}
cancel(reason: string): void {
this.invalidate();
if (!this.terminal) {
this.state = 'cancelled';
this.finishedAt = this.now();
this.error = reason;
}
}
dispose(): void {
if (this.disposed) return;
this.cancel('任务已释放');
this.disposed = true;
this.skills.dispose();
}
}
@@ -0,0 +1,418 @@
import { useEffect, useRef, useState } from 'react';
import { Button } from '../../components/ui';
import { WEBSITE_MODE } from '../../app/websiteConfig';
import { downloadBytes } from '../../project/cachedFiles';
import type { SimulationSession } from '../../simulation/SimulationSession';
import { AgentTaskController } from './AgentTaskController';
import { DecisionClient } from './DecisionClient';
import { DeterministicBaseline } from './DeterministicBaseline';
import { MockDecisionProvider } from './MockDecisionProvider';
import { ModelConnectionPanel } from './ModelConnectionPanel';
import {
PICK_PLACE,
TASK_GOAL_LIMITS,
seedPositions,
type PickPlaceTaskConfig,
} from './PickPlaceScene';
export interface AgentWorkbenchBridge {
loadScene(config: PickPlaceTaskConfig): Promise<SimulationSession>;
getSession(): SimulationSession | undefined;
isTaskScene(): boolean;
publish(): void;
}
const states = {
idle: '准备',
planning: '等待 LLM 规划',
deciding: '等待 Jev 判定',
executing: '执行技能',
paused: '任务已暂停',
succeeded: '物理验收成功',
failed: '失败 / 安全停止',
cancelled: '已取消',
};
export function AgentTaskPanel({
bridge,
loading,
}: {
bridge: AgentWorkbenchBridge;
loading: boolean;
}) {
const liveBridge = useRef(bridge);
useEffect(() => {
liveBridge.current = bridge;
}, [bridge]);
const task = useRef<AgentTaskController | undefined>(undefined);
const ownSession = useRef<SimulationSession | undefined>(undefined);
const [snapshot, setSnapshot] = useState<ReturnType<AgentTaskController['snapshot']>>();
const [client, setClient] = useState<DecisionClient>();
const [settingsOpen, setSettingsOpen] = useState(false);
const [mode, setMode] = useState<'mock' | 'api'>('mock');
const [instruction, setInstruction] = useState(
'把方块抓起,使用底盘搬运到指定支撑台,放下、松开并撤离。',
);
const [goal, setGoal] = useState<number[]>([...PICK_PLACE.goal.position]);
const [busy, setBusy] = useState(false);
const [message, setMessage] = useState('需要 A / lekiwi-v1 资产;不会自动下载或替换机器人。');
const epoch = useRef(0);
const sceneDirty = useRef(true);
const alive = useRef(true);
function publish() {
const current = task.current;
if (current) setSnapshot(current.snapshot());
liveBridge.current.publish();
}
function stop(reason = '用户停止任务') {
epoch.current++;
const current = task.current,
session = ownSession.current;
if (current) current.cancel(reason);
if (
session &&
session === liveBridge.current.getSession() &&
session.snapshot().controlOwner === 'agent'
) {
session.stopAgent(reason);
session.setPaused(true);
}
if (alive.current) publish();
}
useEffect(() => {
alive.current = true;
const timer = setInterval(() => {
const current = task.current,
session = ownSession.current;
if (!current) return;
setSnapshot(current.snapshot());
if (
current.terminal &&
session === liveBridge.current.getSession() &&
session?.snapshot().controlOwner === 'agent'
) {
session.stopAgent('任务已结束');
session.setPaused(true);
liveBridge.current.publish();
}
}, 250);
const invalidatePending = () => {
epoch.current++;
};
return () => {
alive.current = false;
invalidatePending();
clearInterval(timer);
const session = ownSession.current;
if (
session === liveBridge.current.getSession() &&
session?.snapshot().controlOwner === 'agent'
) {
session.stopAgent('任务面板已卸载');
session.setPaused(true);
}
task.current?.dispose();
};
}, []);
async function load() {
stop('任务场景重新加载');
const generation = epoch.current;
setBusy(true);
try {
seedPositions(0, goal);
const session = await liveBridge.current.loadScene({ seed: 0, goal: [...goal] });
if (generation !== epoch.current || !alive.current) return;
ownSession.current = session;
task.current = undefined;
sceneDirty.current = false;
setSnapshot(undefined);
setMessage('A 抓放场景已加载并暂停;用户地图草稿保留,但不用于本次空旷平地任务。');
} catch (error) {
if (generation === epoch.current && alive.current)
setMessage(error instanceof Error ? error.message : '加载失败');
} finally {
if (alive.current) setBusy(false);
}
}
async function start(single = false) {
if (busy || loading) return;
if (mode === 'api' && !client) {
setMessage(
WEBSITE_MODE
? '请在模型设置中保存 LLM 与 Jev 连接;不会自动转为演示。'
: '真实模式必须先连接本机模型服务;不会转为 mock。',
);
return;
}
if (task.current) {
stop('开始新回合');
task.current.dispose();
task.current = undefined;
sceneDirty.current = true;
}
setSnapshot(undefined);
const generation = ++epoch.current;
setBusy(true);
try {
seedPositions(0, goal);
let session = liveBridge.current.getSession();
if (sceneDirty.current || task.current || !liveBridge.current.isTaskScene())
session = await liveBridge.current.loadScene({ seed: 0, goal: [...goal] });
if (generation !== epoch.current || !alive.current) return;
if (!session) throw new Error('尚未加载任务场景');
const provider = mode === 'api' ? client! : new MockDecisionProvider();
session.startAgent((write) => {
const skills = new DeterministicBaseline(session!, 0, write, true, goal);
try {
const controller = new AgentTaskController(skills, provider, instruction);
if (single) controller.resume(true);
task.current = controller;
return controller;
} catch (error) {
skills.dispose();
throw error;
}
});
ownSession.current = session;
sceneDirty.current = false;
session.setPaused(false);
setMessage(
mode === 'mock'
? '显式 mock 决策;执行与验收为真实 MuJoCo 物理。'
: '真实模型模式,可能计费;失败不降级规则。',
);
publish();
} catch (error) {
if (generation === epoch.current && alive.current)
setMessage(error instanceof Error ? error.message : '启动失败');
} finally {
if (alive.current) setBusy(false);
}
}
function resume(single = false) {
const current = task.current;
if (current && !current.terminal && ownSession.current === liveBridge.current.getSession()) {
current.resume(single);
publish();
} else void start(single);
}
function changeGoal(axis: number, value: number) {
stop('目标编辑使旧计划失效');
sceneDirty.current = true;
setGoal((previous) => previous.map((v, i) => (i === axis ? value : v)));
// Any edit requires a new scene before starting, even if values are subsequently restored.
task.current?.dispose();
setMessage('目标已编辑;开始时会重新加载场景与方块,旧计划不可复用。');
}
const running = Boolean(
snapshot && !['succeeded', 'failed', 'cancelled', 'paused'].includes(snapshot.state),
);
const locked = busy || loading;
const receipts = client?.receipts.filter((r) => r.stamp.runId === snapshot?.runId) ?? [];
const costs = receipts.filter((r) => r.usage.cost !== undefined);
const tokens = receipts.filter((r) =>
['total_tokens', 'input_tokens', 'prompt_tokens'].some((key) => r.usage[key] !== undefined),
);
return (
<div className="space-y-3 text-xs" aria-label="LeKiwi 智能抓放任务">
<p>
仅仿真 · MuJoCo 结构化真值(非视觉) · A /
lekiwi-v1。空旷平地单方块,不承诺障碍导航或任意目标泛化。
</p>
<p>任务模式:{mode === 'mock' ? 'MOCK(不调用模型 API)' : '真实 LLM + Jev(可能计费)'}</p>
<label className="block">
决策模式
<select
aria-label="智能任务决策模式"
disabled={locked}
value={mode}
onChange={(e) => {
stop('决策模式改变');
setMode(e.target.value as 'mock' | 'api');
}}
className="w-full bg-surface p-1"
>
<option value="mock">显式 mock LLM + Jev</option>
<option value="api">真实模型服务(无自动回退)</option>
</select>
</label>
<label className="block">
中文任务指令
<textarea
className="w-full rounded border border-border bg-surface p-2"
rows={3}
maxLength={2000}
value={instruction}
disabled={locked || running}
onChange={(e) => {
stop('任务指令修改');
setInstruction(e.target.value);
}}
/>
</label>
<details open={!WEBSITE_MODE}>
<summary>高级:目标位置与实验范围</summary>
<p>语言计划必须遵循安全技能顺序;放置位置以以下场景坐标为准。</p>
<fieldset disabled={locked || running} className="space-y-1">
<legend>目标位置(米,暂停后编辑;重新载入回合)</legend>
{[0, 1].map((axis) => (
<label key={axis} className="block">
目标 {axis ? 'Y' : 'X'}
<input
className="ml-2 w-24 bg-surface p-1"
type="number"
min={TASK_GOAL_LIMITS.min[axis]}
max={TASK_GOAL_LIMITS.max[axis]}
step="0.001"
value={Number.isNaN(goal[axis]) ? '' : goal[axis]}
onChange={(e) => changeGoal(axis, e.target.valueAsNumber)}
/>
</label>
))}
<p>Z 固定 {goal[2]};坐标编辑为受限实验,默认冻结场景已验收。</p>
</fieldset>
</details>
<div className="flex flex-wrap gap-2">
<Button disabled={locked || running} onClick={() => void start()}>
开始任务({mode === 'api' ? '真实 API' : 'mock'})
</Button>
<Button
disabled={locked || !running}
onClick={() => {
task.current?.pause();
publish();
}}
>
暂停任务
</Button>
<Button disabled={locked || snapshot?.state !== 'paused'} onClick={() => resume()}>
继续任务
</Button>
<Button onClick={() => stop()}>停止任务</Button>
</div>
<details open={!WEBSITE_MODE}>
<summary>高级操作与证据导出</summary>
<div className="mt-2 flex flex-wrap gap-2">
<Button disabled={locked || running} onClick={() => void load()}>
加载 A 任务场景
</Button>
<Button disabled={locked || running} onClick={() => resume(true)}>
单技能步进
</Button>
<Button disabled={locked} onClick={() => void load()}>
重置任务场景
</Button>
<Button
disabled={!snapshot}
onClick={() => {
const current = task.current;
if (!current) return;
downloadBytes(
new TextEncoder().encode(
JSON.stringify(
{
...current.snapshot(),
results: current.results,
failures: current.failures,
trace: (current.skills as DeterministicBaseline).trace,
receipts,
},
null,
2,
),
),
`lekiwi-agent-${current.snapshot().runId}.json`,
);
}}
>
导出任务证据
</Button>
</div>
</details>
<p role="status">{message}</p>
{snapshot && (
<div data-testid="agent-task-state" className="space-y-1">
<p>
状态:{states[snapshot.state]} · 当前技能:{snapshot.phase}
</p>
{snapshot.error && (
<p role="alert" className="text-warning">
{snapshot.error}
</p>
)}
<details open={!WEBSITE_MODE}>
<summary>运行详情与物理证据</summary>
<p>
规划 {snapshot.llmCalls}/3 · Jev {snapshot.jevCalls}/60 · 重规划 {snapshot.replans}/2
</p>
<p>
墙钟 {(snapshot.wallMs / 1000).toFixed(1)} s · 仿真 {snapshot.sample.time.toFixed(2)}{' '}
s · 安全抓持 {snapshot.secure ? '是' : '否'} · 有效搬运{' '}
{snapshot.transported.toFixed(3)} m
</p>
<p>
双指力 {snapshot.sample.fingerForces.map((v) => v.toFixed(2)).join(' / ')} N ·
目标支撑 {snapshot.sample.onGoalSupport ? '是' : '否'}
</p>
<p>
Jev:
{snapshot.decision
? `${snapshot.decision.choice} / ${snapshot.decision.grasp} / ${snapshot.decision.diagnosis} / ${snapshot.decision.recovery}`
: '尚未判定'}
</p>
<p>
已返回费用:
{costs.length
? `$${costs.reduce((sum, r) => sum + r.usage.cost, 0).toFixed(6)}(部分服务报告;非完整账单)`
: '未报告,不估算为零'}
</p>
<p>
已报告 Token:
{tokens.length
? tokens.reduce(
(sum, r) =>
sum +
(r.usage.total_tokens ??
(r.usage.input_tokens ?? r.usage.prompt_tokens ?? 0) +
(r.usage.output_tokens ?? r.usage.completion_tokens ?? 0)),
0,
)
: '未报告'}{' '}
· 最近调用耗时{' '}
{receipts.at(-1)?.elapsedMs === undefined
? '未报告'
: `${receipts.at(-1)!.elapsedMs!.toFixed(0)} ms`}
</p>
<p>最近模型:{receipts.at(-1)?.model ?? (mode === 'mock' ? 'mock' : '尚无有效响应')}</p>
</details>
{snapshot.plan && (
<details open>
<summary>LLM 受约束计划</summary>
<p>{snapshot.plan.summary}</p>
<ol className="list-inside list-decimal">
{snapshot.plan.steps.map((s) => (
<li key={s.skill}>
{s.skill} · 失败时 {s.onFailure}
</li>
))}
</ol>
</details>
)}
</div>
)}
<p>
全局暂停/隐藏/重载会取消任务;任务级暂停可继续。重置前请导出当前证据。
{!WEBSITE_MODE && 'RL/外控与任务互斥,后台训练作业不会被停止。'}
</p>
<details onToggle={(event) => setSettingsOpen(event.currentTarget.open)}>
<summary>
{WEBSITE_MODE ? '模型设置 · API_KEY / ChatGPT 订阅' : '模型连接与官方订阅登录'}
</summary>
{(!WEBSITE_MODE || settingsOpen) && (
<ModelConnectionPanel
onClient={setClient}
onInvalidate={() => stop('模型配置或账号操作使任务失效')}
/>
)}
</details>
</div>
);
}
@@ -0,0 +1,116 @@
import { describe, expect, it, vi } from 'vitest';
import { DecisionClient } from './DecisionClient';
import { MockDecisionProvider } from './MockDecisionProvider';
import { PickPlaceEvaluator, type PhysicalSample } from './PhysicalEvidence';
import { observation, SKILLS } from './protocol';
const sample: PhysicalSample = {
time: 0,
base: [0, 0, 0.05],
baseSpeed: 0,
yaw: 0,
tilt: 0,
joints: [0, 0, 0, 0, 0],
peakJointSpeed: 0,
opening: 1,
tcp: [0.257, 0.015, 0.2],
object: [0.257, 0.015, 0.128],
objectSpeed: 0,
fingerForces: [0, 0],
supported: true,
onGoalSupport: false,
};
const stamp = {
runId: 'run',
sceneRevision: 1,
sequence: 0,
planRevision: 1,
requestId: 'request',
};
const obs = observation(
sample,
new PickPlaceEvaluator(sample.object, [0.257, 0.615, 0.128]),
[0.257, 0.615, 0.128],
stamp,
'open',
);
describe('DecisionClient', () => {
it('只连接本机桥;令牌仅 Header,usage 仅数值白名单,不持久化', async () => {
const result = await new MockDecisionProvider().plan(
obs,
'搬运',
SKILLS,
new AbortController().signal,
);
const fetcher = vi.fn<typeof fetch>(async () =>
Response.json({
...result,
model: 'fixture',
provider: 'responses',
usage: { input_tokens: 10, cost: 0.01, raw: 'discard' },
}),
);
const persist = vi.spyOn(Storage.prototype, 'setItem');
const client = new DecisionClient('http://127.0.0.1:8768', 'service-token-fixture', fetcher);
expect((await client.plan(obs, '搬运', SKILLS, new AbortController().signal)).stamp).toEqual(
stamp,
);
const [url, init] = fetcher.mock.calls[0];
expect(url).toContain('/api/decision/v1/plan');
expect(init?.headers).toMatchObject({ Authorization: 'Bearer service-token-fixture' });
expect(init?.body).not.toContain('service-token-fixture');
expect(init?.redirect).toBe('error');
expect(client.receipts[0].usage).toEqual({ input_tokens: 10, cost: 0.01 });
expect(persist).not.toHaveBeenCalled();
persist.mockRestore();
});
it('浏览器原生 fetch 必须使用全局 receiver,不能绑定到客户端实例', async () => {
const fetcher: typeof fetch = async function (this: unknown) {
if (this !== globalThis) throw new TypeError('Illegal invocation');
return Response.json({ ok: true });
};
const client = new DecisionClient('http://localhost:8768', 'fixture', fetcher);
expect(await client.call('/status')).toEqual({ ok: true });
});
it('拒绝非本机地址、用户名、路径、query 等令牌外泄入口', () => {
for (const url of [
'https://example.com',
'http://localhost.evil.test',
'http://x:y@localhost',
'http://localhost/path',
'http://localhost?secret=value',
])
expect(() => new DecisionClient(url, 'fixture')).toThrow();
});
it('abort 同时取消 HTTP 和服务端指定请求', async () => {
const fetcher = vi.fn<typeof fetch>(async (url, init) => {
if (String(url).endsWith('/cancel')) return Response.json({ cancelled: true });
return await new Promise<Response>((_, reject) =>
init?.signal?.addEventListener('abort', () => reject(new Error('aborted'))),
);
});
const client = new DecisionClient('http://localhost:8768', 'fixture', fetcher);
const controller = new AbortController();
const pending = client.plan(obs, '搬运', SKILLS, controller.signal);
controller.abort();
await expect(pending).rejects.toThrow('aborted');
expect(fetcher.mock.calls).toHaveLength(2);
expect(JSON.parse(String(fetcher.mock.calls[1][1]?.body))).toEqual({
runId: 'run',
requestId: 'request',
});
});
it('错误原文/非法 JSON/超长响应均不能成为决策或泄露到错误消息', async () => {
for (const response of [
new Response('not-json-private', { status: 502 }),
Response.json({ error: 'sensitive raw error sk-example' }, { status: 401 }),
new Response('x'.repeat(131073)),
]) {
const client = new DecisionClient('http://localhost:8768', 'fixture', async () => response);
await expect(
client.plan(obs, '搬运', SKILLS, new AbortController().signal),
).rejects.not.toThrow(/private|sk-example/);
expect(client.receipts).toHaveLength(0);
}
});
});
@@ -0,0 +1,227 @@
import type { RequestStamp, Stamped } from './RequestGate';
import {
validateContract,
validateJevDecision,
validatePlan,
type JevDecision,
type Observation,
type Plan,
type Skill,
} from './protocol';
export interface DecisionProvider {
readonly mode: 'api' | 'mock';
plan(
obs: Observation,
instruction: string,
remaining: readonly Skill[],
signal: AbortSignal,
): Promise<Stamped<Plan>>;
decide(
obs: Observation,
candidates: readonly (Skill | 'stop')[],
failure: string,
signal: AbortSignal,
): Promise<Stamped<JevDecision>>;
}
export interface DecisionReceipt {
kind: 'plan' | 'decide';
stamp: RequestStamp;
model: string;
elapsedMs?: number;
provider: string;
usage: Record<string, number>;
}
const PREFIX = '/api/decision/v1';
/** Local bridge only. Neither the service token nor API keys enter browser storage. */
export class DecisionClient implements DecisionProvider {
readonly mode = 'api' as const;
readonly receipts: DecisionReceipt[] = [];
readonly origin: string;
readonly websiteMode: boolean;
private csrfToken = '';
private configVersion = 0;
static async website(signal?: AbortSignal, fetcher: typeof fetch = fetch) {
const client = new DecisionClient(location.origin, '', fetcher, { website: true });
const value = (await client.call('/session', 'POST', {}, signal)) as {
csrfToken?: unknown;
configVersion?: unknown;
};
if (typeof value.csrfToken !== 'string' || !Number.isSafeInteger(value.configVersion))
throw new Error('网站会话响应无效');
client.csrfToken = value.csrfToken;
client.configVersion = value.configVersion as number;
return client;
}
constructor(
address: string,
private readonly token: string,
private readonly fetcher: typeof fetch = fetch,
options: { website?: boolean } = {},
) {
const url = new URL(address);
this.websiteMode = options.website === true;
if (
!['http:', 'https:'].includes(url.protocol) ||
(this.websiteMode
? url.origin !== location.origin
: !['localhost', '127.0.0.1', '[::1]'].includes(url.hostname)) ||
url.username ||
url.password ||
url.search ||
url.hash ||
!['', '/'].includes(url.pathname)
)
throw new Error('模型服务必须是本机地址,不可包含凭据或路径');
this.origin = url.origin;
}
async call(path: string, method = 'GET', body?: unknown, signal?: AbortSignal): Promise<unknown> {
const response = await this.fetcher.call(globalThis, this.origin + PREFIX + path, {
method,
headers: {
...(this.websiteMode
? { 'X-CSRF-Token': this.csrfToken, 'X-Config-Version': String(this.configVersion) }
: { Authorization: `Bearer ${this.token}` }),
...(body === undefined ? {} : { 'Content-Type': 'application/json' }),
},
body: body === undefined ? undefined : JSON.stringify(body),
signal: signal ?? AbortSignal.timeout(65000),
redirect: 'error',
cache: 'no-store',
credentials: this.websiteMode ? 'same-origin' : 'omit',
});
if (
this.websiteMode &&
method !== 'GET' &&
(path === '/configuration' || path.startsWith('/codex/'))
) {
const version = response.headers.get('X-Config-Version');
if (version && /^\d+$/.test(version)) this.configVersion = Number(version);
}
const reader = response.body?.getReader();
if (!reader) throw new Error('模型服务没有响应正文');
const chunks: Uint8Array[] = [];
let size = 0;
try {
for (;;) {
const { done, value } = await reader.read();
if (done) break;
size += value.byteLength;
if (size > 131072) throw new Error('模型服务响应过大');
chunks.push(value);
}
} finally {
await reader.cancel();
reader.releaseLock();
}
const bytes = new Uint8Array(size);
let offset = 0;
for (const chunk of chunks) {
bytes.set(chunk, offset);
offset += chunk.length;
}
let value: unknown;
try {
value = JSON.parse(new TextDecoder('utf-8', { fatal: true }).decode(bytes));
} catch {
throw new Error('模型服务 JSON 非法');
}
if (!response.ok) {
const code = (value as { error?: unknown })?.error;
throw new Error(
typeof code === 'string' && /^[a-z0-9_]{1,100}$/.test(code)
? code
: `模型服务 HTTP ${response.status}`,
);
}
return value;
}
private async request(
kind: 'plan' | 'decide',
obs: Observation,
body: object,
signal: AbortSignal,
) {
const cancel = () => {
void this.call(
'/cancel',
'POST',
{ runId: obs.stamp.runId, requestId: obs.stamp.requestId },
AbortSignal.timeout(2000),
).catch(() => undefined);
};
if (signal.aborted) throw new Error('模型请求已取消');
signal.addEventListener('abort', cancel, { once: true });
try {
const result = (await this.call(
'/' + kind,
'POST',
{ observation: obs, ...body },
signal,
)) as {
stamp: unknown;
value: unknown;
model?: unknown;
provider?: unknown;
usage?: unknown;
elapsedMs?: unknown;
};
const stamp = validateContract<RequestStamp>('Stamp', result.stamp);
const usage: Record<string, number> = {};
if (result.usage && typeof result.usage === 'object') {
for (const [key, value] of Object.entries(result.usage))
if (
[
'input_tokens',
'output_tokens',
'total_tokens',
'prompt_tokens',
'completion_tokens',
'cost',
].includes(key) &&
typeof value === 'number' &&
Number.isFinite(value) &&
value >= 0
)
usage[key] = value;
}
this.receipts.push({
kind,
stamp,
usage,
elapsedMs:
typeof result.elapsedMs === 'number' &&
Number.isFinite(result.elapsedMs) &&
result.elapsedMs >= 0
? result.elapsedMs
: undefined,
model: typeof result.model === 'string' ? result.model.slice(0, 128) : 'unknown',
provider: typeof result.provider === 'string' ? result.provider.slice(0, 64) : 'unknown',
});
if (this.receipts.length > 128) this.receipts.shift();
return { stamp, value: result.value };
} finally {
signal.removeEventListener('abort', cancel);
}
}
async plan(
obs: Observation,
instruction: string,
remaining: readonly Skill[],
signal: AbortSignal,
): Promise<Stamped<Plan>> {
const result = await this.request('plan', obs, { instruction, remaining }, signal);
return { stamp: result.stamp, value: validatePlan(result.value, remaining) };
}
async decide(
obs: Observation,
candidates: readonly (Skill | 'stop')[],
failure: string,
signal: AbortSignal,
): Promise<Stamped<JevDecision>> {
const result = await this.request('decide', obs, { candidates, failure }, signal);
return { stamp: result.stamp, value: validateJevDecision(result.value, candidates) };
}
}
@@ -0,0 +1,350 @@
import type { SimulationSession } from '../../simulation/SimulationSession';
import { ModelBindings } from '../../simulation/ModelBindings';
import { LeKiwiIK } from './LeKiwiIK';
import { PhysicalSampler, PickPlaceEvaluator, type PhysicalSample } from './PhysicalEvidence';
import { PICK_PLACE as P, PICK_PLACE_ROBOT as R, seedPositions } from './PickPlaceScene';
const phases = [
'open',
'pregrasp',
'descend',
'close',
'verify',
'carry',
'stop-base',
'place',
'release',
'retreat',
'settle',
] as const;
export type BaselinePhase = (typeof phases)[number];
const durations = [2, 6, 3, 3, 4, 22, 1, 4, 3, 4, 1];
const clamp = (v: number, lo: number, hi: number) => Math.max(lo, Math.min(hi, v));
/** No network, RL reward, object attachment or independent physics loop.
* The caller exclusively owns the existing simulation clock and validates writes.
* Ungated mode is the frozen Stage-5 reference. Gated mode waits at each verified
* skill boundary; only a current synchronous owner callback may authorize/recover. */
export class DeterministicBaseline {
readonly evaluator: PickPlaceEvaluator;
readonly trace: (PhysicalSample & {
phase: BaselinePhase;
secure: boolean;
transported: number;
})[] = [];
readonly ik: LeKiwiIK;
private readonly sampler: PhysicalSampler;
private readonly arm;
private readonly gripper;
private readonly wheels;
private readonly positions;
private graspSource: number[];
private readonly baseStart: number[];
private readonly servo: number[];
private readonly velocities = Array<number>(5).fill(0);
private readonly baseVelocity = [0, 0, 0];
private readonly translation: readonly number[];
private opening: number;
private phaseIndex = 0;
private phaseStart: number;
private readonly start: number;
private nextControl: number;
private lostGraspTime = 0;
private readonly armStallTime = Array<number>(5).fill(0);
private disposed = false;
private completedPhase = false;
waiting = false;
sample: PhysicalSample;
status: 'running' | 'succeeded' | 'failed' = 'running';
error = '';
maxJointSpeed = 0;
constructor(
private readonly session: SimulationSession,
seed: number,
private readonly writeServo: (actuator: number, value: number) => void,
private readonly gated = false,
goal?: readonly number[],
) {
const b = new ModelBindings(session.model);
this.positions = seedPositions(seed, goal);
this.translation = goal
? this.positions.goal.slice(0, 2).map((v, i) => v - this.positions.object[i])
: P.baseTranslation;
this.graspSource = [...this.positions.object];
this.waiting = gated;
this.arm = R.armJoints.map((spec, i) =>
b.scalarActuator(R.armActuators[i], spec.name, 'position'),
);
this.gripper = b.scalarActuator(R.gripperActuators[0].name, R.gripperJoint, 'position');
this.wheels = R.baseActuators.map((name, i) =>
b.scalarActuator(name, R.baseJoints[i], 'velocity'),
);
this.sampler = new PhysicalSampler(session);
let ik: LeKiwiIK | undefined;
try {
this.ik = ik = new LeKiwiIK(session);
this.sample = this.sampler.sample();
} catch (error) {
ik?.dispose();
this.sampler.dispose();
throw error;
}
this.baseStart = [...this.sample.base];
this.servo = [...this.sample.joints];
this.opening = this.sample.opening;
this.start = this.phaseStart = this.nextControl = this.sample.time;
this.evaluator = new PickPlaceEvaluator(this.positions.object, this.positions.goal);
}
get phase(): BaselinePhase {
return phases[this.phaseIndex];
}
get goal(): readonly number[] {
return this.positions.goal;
}
get candidate(): BaselinePhase {
return phases[this.phaseIndex + (this.completedPhase ? 1 : 0)] ?? 'settle';
}
/** Re-check facts even if both models recommend continuation. No model can grant this gate. */
authorize(skill: BaselinePhase): void {
if (!this.gated || !this.waiting || this.status !== 'running' || skill !== this.candidate)
throw new Error('非法技能切换');
const s = this.sample;
if (skill === 'pregrasp' && s.baseSpeed > 0.01) throw new Error('base_not_stopped');
if (skill === 'descend' && s.tcp[2] < s.object[2] + 0.04) throw new Error('tcp_not_above');
if (skill === 'close' && Math.hypot(...s.tcp.map((v, i) => v - s.object[i])) > 0.012)
throw new Error('alignment_timeout');
if (skill === 'verify' && s.fingerForces.some((v) => v < P.fingerForceMin))
throw new Error('empty_grasp');
if (['carry', 'stop-base'].includes(skill) && !this.evaluator.secure)
throw new Error('grasp_unverified');
if (skill === 'place' && (this.evaluator.transported < P.transportMin || s.baseSpeed > 0.01))
throw new Error('transport_unverified');
if (skill === 'release' && !s.onGoalSupport) throw new Error('placement_unsupported');
if (
skill === 'retreat' &&
(s.opening < 0.65 || s.fingerForces.some((v) => v >= P.fingerForceMin))
)
throw new Error('release_blocked');
if (this.completedPhase) this.phaseIndex++;
this.completedPhase = false;
this.waiting = false;
this.phaseStart = this.nextControl = s.time;
}
canRecover(): boolean {
const s = this.sample;
return (
this.gated &&
this.status === 'failed' &&
this.phaseIndex <= phases.indexOf('verify') &&
['alignment_timeout', 'empty_grasp', 'grasp_unverified'].includes(this.error) &&
s.supported &&
s.objectSpeed < P.placementSpeedMax &&
s.baseSpeed < 0.01 &&
this.evaluator.transported === 0 &&
Math.hypot(...s.object.map((v, i) => v - this.positions.object[i])) < 0.012
);
}
/** Only a supported, stationary object still at the source may be realigned/regrasped. */
recover(): void {
if (!this.canRecover()) throw new Error('unsafe_recovery');
this.hold();
// An empty/blocked attempt may not have reached its requested gripper position.
// Re-seed targets from measured pose so a released fault cannot cause a servo jump.
this.opening = this.sample.opening;
this.servo.splice(0, this.servo.length, ...this.sample.joints);
this.armStallTime.fill(0);
this.graspSource = [...this.sample.object];
this.phaseIndex = 0;
this.phaseStart = this.nextControl = this.sample.time;
this.completedPhase = false;
this.waiting = true;
this.status = 'running';
this.error = '';
// Do not reset the episode deadline or fabricate/reset the evaluator's transport history.
}
private target(): number[] | undefined {
const source = this.graspSource,
goal = this.positions.goal;
switch (this.phase) {
case 'pregrasp':
return [source[0], source[1], P.pregraspHeight];
case 'descend':
case 'close':
return [...source];
case 'verify':
return [source[0], source[1], P.carryHeight];
case 'place':
case 'release':
return [...goal];
case 'retreat':
case 'settle':
return [goal[0], goal[1], P.retreatHeight];
default:
return undefined;
}
}
beforeStep(): void {
if (
this.disposed ||
this.waiting ||
this.status !== 'running' ||
this.sample.time + 1e-9 < this.nextControl
)
return;
this.nextControl = this.sample.time + P.controlDt;
try {
const target = this.target();
if (target) {
const solution = this.ik.solve(target);
if (solution.residual > 0.004) throw new Error('unreachable');
this.arm.forEach((joint, i) => {
const desired = clamp(5 * (solution.targets[i] - this.servo[i]), -P.armSpeed, P.armSpeed);
this.velocities[i] += clamp(
desired - this.velocities[i],
-P.armAcceleration * P.controlDt,
P.armAcceleration * P.controlDt,
);
const old = this.servo[i],
spec = R.armJoints[i];
const next = clamp(
clamp(
old + this.velocities[i] * P.controlDt,
this.sample.joints[i] - P.armTrackingError,
this.sample.joints[i] + P.armTrackingError,
),
old - P.armSpeed * P.controlDt,
old + P.armSpeed * P.controlDt,
);
this.servo[i] = clamp(next, spec.min, spec.max);
this.velocities[i] = (this.servo[i] - old) / P.controlDt;
});
}
this.arm.forEach((joint, i) => this.writeServo(joint.actuatorId, this.servo[i]));
const close = ['close', 'verify', 'carry', 'stop-base', 'place'].includes(this.phase);
this.opening += clamp(
(close ? 0 : 1) - this.opening,
-P.gripperOpeningRate * P.controlDt,
P.gripperOpeningRate * P.controlDt,
);
this.writeServo(
this.gripper.actuatorId,
R.gripperClosed + this.opening * (R.gripperOpen - R.gripperClosed),
);
const desired = [0, 0, 0];
if (this.phase === 'carry') {
const dx = this.baseStart[0] + this.translation[0] - this.sample.base[0];
const dy = this.baseStart[1] + this.translation[1] - this.sample.base[1];
const scale = Math.min(1, P.baseSpeed / Math.max(1e-12, Math.hypot(dx, dy)));
const c = Math.cos(this.sample.yaw),
s = Math.sin(this.sample.yaw);
desired[0] = scale * (c * dx + s * dy);
desired[1] = scale * (-s * dx + c * dy);
desired[2] = clamp(-1.5 * this.sample.yaw, -P.yawSpeed, P.yawSpeed);
}
for (let i = 0; i < 3; i++) {
const bound = (i === 2 ? P.yawAcceleration : P.baseAcceleration) * P.controlDt;
this.baseVelocity[i] += clamp(desired[i] - this.baseVelocity[i], -bound, bound);
}
const wheelSpeeds = R.baseMix.map((row) =>
row.reduce((sum, v, i) => sum + v * this.baseVelocity[i], 0),
);
const wheelScale = R.wheelLimit / Math.max(R.wheelLimit, ...wheelSpeeds.map(Math.abs));
this.wheels.forEach((wheel, i) =>
this.writeServo(wheel.actuatorId, wheelSpeeds[i] * wheelScale),
);
this.trace.push({
...this.sample,
phase: this.phase,
secure: this.evaluator.secure,
transported: this.evaluator.transported,
});
} catch (error) {
this.fail(error instanceof Error ? error.message : String(error));
}
}
afterStep(dt: number): void {
if (this.disposed || this.status !== 'running') return;
try {
const previous = this.sample;
this.sample = this.sampler.sample();
this.evaluator.update(this.sample, dt);
// Only arm joints: gripper contact intentionally blocks a closing servo.
// Frozen model waits do not call afterStep and cannot accumulate stall time.
for (let i = 0; i < this.arm.length; i++) {
const speed = Math.abs(this.sample.joints[i] - previous.joints[i]) / dt;
const blocked =
Math.abs(this.servo[i] - this.sample.joints[i]) >= P.armStallError &&
speed < P.armStallSpeed;
this.armStallTime[i] = blocked ? this.armStallTime[i] + dt : 0;
if (this.armStallTime[i] >= P.armStallDuration) throw new Error('joint_stall');
}
this.maxJointSpeed = Math.max(this.maxJointSpeed, this.sample.peakJointSpeed);
if (this.sample.peakJointSpeed > P.jointSpeedStop) throw new Error('joint_velocity');
if (this.sample.base[2] < 0.03 || this.sample.base[2] > 0.1 || this.sample.tilt > 0.35)
throw new Error('base_unstable');
if (this.sample.time - this.start > P.episodeTimeout) throw new Error('episode_timeout');
if (this.phase === 'carry' || this.phase === 'stop-base') {
this.lostGraspTime = this.evaluator.secure ? 0 : this.lostGraspTime + dt;
if (this.lostGraspTime > 0.2) throw new Error('lost_grasp');
}
let ready = this.sample.time - this.phaseStart >= durations[this.phaseIndex];
if (this.phase === 'carry') {
const distance = Math.hypot(
...this.sample.base
.slice(0, 2)
.map((v, i) => v - this.baseStart[i] - this.translation[i]),
);
if (distance < 0.007) ready = true;
else if (ready) throw new Error('navigation_timeout');
}
if (!ready) return;
const target = this.target();
if (target && Math.hypot(...target.map((v, i) => v - this.sample.tcp[i])) > 0.012)
throw new Error('alignment_timeout');
if (this.phase === 'close' && this.sample.fingerForces.some((f) => f < P.fingerForceMin))
throw new Error('empty_grasp');
if (this.phase === 'verify' && !this.evaluator.secure) throw new Error('grasp_unverified');
if (this.phase === 'carry' && this.evaluator.transported < P.transportMin)
throw new Error('transport_unverified');
if (this.phase === 'place' && !this.sample.onGoalSupport)
throw new Error('placement_unsupported');
if (
this.phase === 'release' &&
(this.sample.opening < 0.65 || this.sample.fingerForces.some((f) => f >= P.fingerForceMin))
)
throw new Error('release_blocked');
if (this.phase === 'settle') {
if (!this.evaluator.succeeded) throw new Error('placement_unverified');
this.status = 'succeeded';
this.hold();
} else if (this.gated) {
this.completedPhase = true;
this.waiting = true;
this.hold();
} else {
this.phaseIndex++;
this.phaseStart = this.sample.time;
}
} catch (error) {
this.fail(error instanceof Error ? error.message : String(error));
}
}
/** Called only through the session's current owner write capability. */
hold(): void {
this.baseVelocity.fill(0);
this.velocities.fill(0);
this.wheels.forEach((wheel) => this.writeServo(wheel.actuatorId, 0));
// Keep arm/gripper's LAST safe servo targets; never zero loaded position servos.
}
private fail(reason: string): void {
this.status = 'failed';
this.error = reason;
this.hold();
}
dispose(): void {
if (this.disposed) return;
this.disposed = true;
this.sampler.dispose();
this.ik.dispose();
}
}
+134
View File
@@ -0,0 +1,134 @@
import type { DoubleBuffer, MjData } from '@mujoco/mujoco';
import type { SimulationSession } from '../../simulation/SimulationSession';
import { ModelBindings } from '../../simulation/ModelBindings';
import { PICK_PLACE, PICK_PLACE_ROBOT } from './PickPlaceScene';
/** Small pivoted solve; DLS adds a positive diagonal before calling it. */
export function solveLinear(matrix: number[][], rhs: number[]): number[] {
const n = rhs.length;
const a = matrix.map((row, i) => [...row, rhs[i]]);
for (let k = 0; k < n; k++) {
let pivot = k;
for (let r = k + 1; r < n; r++) if (Math.abs(a[r][k]) > Math.abs(a[pivot][k])) pivot = r;
if (!Number.isFinite(a[pivot][k]) || Math.abs(a[pivot][k]) < 1e-12)
throw new Error('IK 矩阵不可解');
[a[k], a[pivot]] = [a[pivot], a[k]];
const divisor = a[k][k];
for (let c = k; c <= n; c++) a[k][c] /= divisor;
for (let r = 0; r < n; r++)
if (r !== k) {
const f = a[r][k];
for (let c = k; c <= n; c++) a[r][c] -= f * a[k][c];
}
}
return a.map((row) => row[n]);
}
/** Five-axis position + upright approach IK, not a seven-axis/full-pose solver.
* Kinematics run exclusively in scratch MjData; never teleport the live robot. */
export class LeKiwiIK {
private readonly scratch: MjData;
private readonly positionJacobian: DoubleBuffer;
private readonly rotationJacobian: DoubleBuffer;
private readonly joints;
readonly site: number;
private disposed = false;
constructor(private readonly session: SimulationSession) {
const { module, model } = session,
b = new ModelBindings(model);
this.joints = PICK_PLACE_ROBOT.armJoints.map((j) => b.scalarJoint(j.name));
this.site = module.mj_name2id(model, 6, PICK_PLACE.tcpSite);
if (this.site < 0 || this.joints.length !== 5) throw new Error('抓放 IK 缺少 A 模型绑定');
this.scratch = new module.MjData(model);
let positionBuffer: DoubleBuffer | undefined;
try {
this.positionJacobian = positionBuffer = new module.DoubleBuffer(3 * model.nv);
this.rotationJacobian = new module.DoubleBuffer(3 * model.nv);
} catch (error) {
positionBuffer?.delete();
this.scratch.delete();
throw error;
}
}
solve(target: readonly number[]): { targets: number[]; residual: number; iterations: number } {
if (this.disposed) throw new Error('IK 已释放');
if (target.length !== 3 || target.some((v) => !Number.isFinite(v) || Math.abs(v) > 2))
throw new Error('IK 目标超出标定范围');
const { module, model, data } = this.session;
// Refresh views at each call; compiling another candidate can grow WASM memory.
const qpos: Float64Array = this.scratch.qpos;
qpos.set(data.qpos);
const q = this.joints.map((j) => qpos[j.qposAddress]);
const jp: Float64Array = this.positionJacobian.GetView(),
jr: Float64Array = this.rotationJacobian.GetView();
let residual = Infinity,
iterations = 0;
for (; iterations < 80; iterations++) {
this.joints.forEach((j, i) => {
qpos[j.qposAddress] = q[i];
});
module.mj_kinematics(model, this.scratch);
module.mj_comPos(model, this.scratch);
const pos: Float64Array = this.scratch.site_xpos,
mat: Float64Array = this.scratch.site_xmat;
const error = [
...target.map((v, i) => v - pos[this.site * 3 + i]),
0.07 * mat[this.site * 9 + 5],
-0.07 * mat[this.site * 9 + 2],
];
residual = Math.hypot(...error);
if (residual < 0.0001) break;
module.mj_jacSite(
model,
this.scratch,
this.positionJacobian,
this.rotationJacobian,
this.site,
);
const jac = Array.from({ length: 5 }, (_, row) =>
this.joints.map((joint) =>
row < 3
? jp[row * model.nv + joint.qvelAddress]
: 0.07 * jr[(row - 3) * model.nv + joint.qvelAddress],
),
);
const gram = jac.map((row, i) =>
jac.map(
(other, k) => row.reduce((sum, v, j) => sum + v * other[j], 0) + (i === k ? 0.00002 : 0),
),
);
const inverseError = solveLinear(gram, error);
for (let j = 0; j < q.length; j++) {
const update = jac.reduce((sum, row, i) => sum + row[j] * inverseError[i], 0);
const spec = PICK_PLACE_ROBOT.armJoints[j];
q[j] = Math.max(
spec.min + 0.015,
Math.min(spec.max - 0.015, q[j] + Math.max(-0.07, Math.min(0.07, update))),
);
}
}
if (iterations === 80) {
this.joints.forEach((j, i) => {
qpos[j.qposAddress] = q[i];
});
module.mj_kinematics(model, this.scratch);
const pos: Float64Array = this.scratch.site_xpos,
mat: Float64Array = this.scratch.site_xmat;
residual = Math.hypot(
...target.map((v, i) => v - pos[this.site * 3 + i]),
0.07 * mat[this.site * 9 + 5],
-0.07 * mat[this.site * 9 + 2],
);
}
if (!Number.isFinite(residual) || q.some((v) => !Number.isFinite(v)))
throw new Error('IK 非有限结果');
return { targets: q, residual, iterations };
}
dispose(): void {
if (this.disposed) return;
this.disposed = true;
this.rotationJacobian.delete();
this.positionJacobian.delete();
this.scratch.delete();
}
}
@@ -0,0 +1,59 @@
import type { DecisionProvider } from './DecisionClient';
import type { Stamped } from './RequestGate';
import {
AGENT_VERSION,
PRECONDITIONS,
type Observation,
type Plan,
type Skill,
type JevDecision,
} from './protocol';
/** Explicit no-network fixture. Never selected as a fallback by DecisionClient. */
export class MockDecisionProvider implements DecisionProvider {
readonly mode = 'mock' as const;
async plan(
obs: Observation,
_instruction: string,
remaining: readonly Skill[],
signal: AbortSignal,
): Promise<Stamped<Plan>> {
signal.throwIfAborted();
return {
stamp: { ...obs.stamp },
value: {
version: AGENT_VERSION,
objectId: 'block',
goalId: 'placement',
summary: 'MOCK:固定规范顺序,成功仅由物理评估决定',
steps: remaining.map((skill) => ({
skill,
precondition: PRECONDITIONS[skill],
onFailure: 'retry',
})),
},
};
}
async decide(
obs: Observation,
candidates: readonly (Skill | 'stop')[],
failure: string,
signal: AbortSignal,
): Promise<Stamped<JevDecision>> {
signal.throwIfAborted();
return {
stamp: { ...obs.stamp },
value: {
version: AGENT_VERSION,
choice: candidates[0],
grasp: obs.evidence.secure
? 'secure'
: obs.evidence.fingerForces.some((v) => v > 0.2)
? 'uncertain'
: 'empty',
diagnosis: failure === 'none' ? 'none' : failure === 'empty_grasp' ? 'empty' : 'misaligned',
recovery: failure === 'none' ? 'continue' : 'retry',
},
};
}
}
@@ -0,0 +1,152 @@
import { cleanup, fireEvent, render, screen, waitFor } from '@testing-library/react';
import { afterEach, describe, expect, it, vi } from 'vitest';
import { ModelConnectionPanel } from './ModelConnectionPanel';
afterEach(() => {
cleanup();
vi.unstubAllGlobals();
vi.restoreAllMocks();
});
const status = {
version: 'lekiwi-agent-v1',
connections: {
llm: {
protocol: 'responses',
baseUrl: 'https://api.deepseek.com',
model: 'deepseek-flash',
hasKey: true,
},
jev: {
protocol: 'openrouter-decisions',
baseUrl: 'https://openrouter.ai/api/alpha/decisions',
model: 'typesafe/jev-1.13',
hasKey: true,
},
},
};
async function connect() {
fireEvent.change(screen.getByLabelText('本机服务令牌'), {
target: { value: 'fixture-service-token' },
});
fireEvent.click(screen.getByRole('button', { name: '连接本机模型服务' }));
await screen.findByText(/本机桥已连接/);
}
describe('模型连接面板(HTTP 为 mock,不调用云端)', () => {
it('读取脱敏状态不推理、不持久化;换地址清空草稿 key,保存后也清空', async () => {
const fetcher = vi.fn<typeof fetch>(async () => Response.json(status));
vi.stubGlobal('fetch', fetcher);
const storage = vi.spyOn(Storage.prototype, 'setItem');
render(<ModelConnectionPanel onClient={vi.fn()} onInvalidate={vi.fn()} />);
await connect();
expect(fetcher).toHaveBeenCalledOnce();
expect(screen.getByLabelText('LLM 型号')).toHaveValue('deepseek-flash');
fireEvent.change(screen.getByLabelText('LLM API Key'), {
target: { value: 'test-key-not-real' },
});
fireEvent.change(screen.getByLabelText('LLM Base URL'), {
target: { value: 'https://example.test/v1' },
});
expect(screen.getByLabelText('LLM API Key')).toHaveValue('');
fireEvent.change(screen.getByLabelText('LLM API Key'), {
target: { value: 'test-key-not-real' },
});
fireEvent.click(screen.getByRole('button', { name: '保存 LLM 连接' }));
await screen.findByText(/LLM 配置已保存/);
expect(screen.getByLabelText('LLM API Key')).toHaveValue('');
expect(storage).not.toHaveBeenCalled();
expect(fetcher.mock.calls.some(([url]) => /\/(plan|decide)$/.test(String(url)))).toBe(false);
});
it('未保存的型号草稿禁止测试/任务,不能仍用旧付费配置', async () => {
const fetcher = vi
.fn<typeof fetch>()
.mockResolvedValueOnce(Response.json(status))
.mockResolvedValueOnce(Response.json({ ok: true }))
.mockResolvedValueOnce(
Response.json({
...status,
connections: {
...status.connections,
llm: { ...status.connections.llm, model: 'new-model' },
},
}),
);
vi.stubGlobal('fetch', fetcher);
const onClient = vi.fn();
render(<ModelConnectionPanel onClient={onClient} onInvalidate={vi.fn()} />);
await connect();
fireEvent.change(screen.getByLabelText('LLM 型号'), { target: { value: 'new-model' } });
expect(onClient).toHaveBeenLastCalledWith(undefined);
expect(screen.getByRole('button', { name: '测试 LLM(可能计费)' })).toBeDisabled();
fireEvent.click(screen.getByRole('button', { name: '测试 LLM(可能计费)' }));
expect(fetcher).toHaveBeenCalledOnce();
fireEvent.change(screen.getByLabelText('LLM API Key'), { target: { value: 'fixture-key' } });
fireEvent.click(screen.getByRole('button', { name: '保存 LLM 连接' }));
await screen.findByText(/LLM 配置已保存/);
expect(screen.getByLabelText('LLM 型号')).toHaveValue('new-model');
expect(screen.getByRole('button', { name: '测试 LLM(可能计费)' })).toBeEnabled();
expect(onClient.mock.lastCall?.[0]).toBeDefined();
});
it('断开后迟到的连接状态不会重新启用客户端', async () => {
let resolve!: (response: Response) => void;
vi.stubGlobal(
'fetch',
vi.fn(
() =>
new Promise<Response>((r) => {
resolve = r;
}),
),
);
const onClient = vi.fn();
render(<ModelConnectionPanel onClient={onClient} onInvalidate={vi.fn()} />);
fireEvent.change(screen.getByLabelText('本机服务令牌'), { target: { value: 'fixture-token' } });
fireEvent.click(screen.getByRole('button', { name: '连接本机模型服务' }));
fireEvent.click(screen.getByRole('button', { name: '断开模型服务' }));
resolve(Response.json(status));
await waitFor(() => expect(screen.getByText(/本机桥已断开/)).toBeInTheDocument());
expect(screen.queryByLabelText('LLM 协议')).not.toBeInTheDocument();
expect(onClient.mock.calls.every(([client]) => client === undefined)).toBe(true);
});
it('账号只读查询不取消物理任务;拒绝伪官方登录地址', async () => {
vi.stubGlobal(
'fetch',
vi.fn<typeof fetch>(async (url) => {
if (String(url).endsWith('/codex/status')) return Response.json({ loggedIn: false });
if (String(url).endsWith('/login'))
return Response.json({ authUrl: 'https://auth.openai.com.attacker.test/' });
return Response.json(status);
}),
);
const invalidate = vi.fn();
render(<ModelConnectionPanel onClient={vi.fn()} onInvalidate={invalidate} />);
await connect();
invalidate.mockClear();
fireEvent.click(screen.getByText(/ChatGPT 订阅 · 官方/));
fireEvent.click(screen.getByRole('button', { name: '刷新账号' }));
await screen.findByText('ChatGPT 未登录');
expect(invalidate).not.toHaveBeenCalled();
fireEvent.click(screen.getByRole('button', { name: '发起官方登录' }));
await screen.findByText('官方登录地址校验失败');
expect(screen.queryByRole('link', { name: '打开 OpenAI 官方登录页' })).not.toBeInTheDocument();
});
it('官方登录只展示 OpenAI 链接,取消清理;不保存/导出完整地址', async () => {
vi.stubGlobal(
'fetch',
vi.fn<typeof fetch>(async (url) => {
if (String(url).endsWith('/login'))
return Response.json({ authUrl: 'https://auth.openai.com/oauth?state=fixture-only' });
if (String(url).endsWith('/cancel')) return Response.json({ cancelled: true });
return Response.json(status);
}),
);
render(<ModelConnectionPanel onClient={vi.fn()} onInvalidate={vi.fn()} />);
await connect();
fireEvent.click(screen.getByText(/ChatGPT 订阅 · 官方/));
fireEvent.click(screen.getByRole('button', { name: '发起官方登录' }));
const link = await screen.findByRole('link', { name: '打开 OpenAI 官方登录页' });
expect(link).toHaveAttribute('rel', 'noopener noreferrer');
fireEvent.click(screen.getByRole('button', { name: '取消官方登录' }));
await screen.findByText('登录已取消');
expect(screen.queryByRole('link', { name: '打开 OpenAI 官方登录页' })).not.toBeInTheDocument();
});
});
@@ -0,0 +1,385 @@
import { useEffect, useRef, useState, type ComponentProps } from 'react';
import { WEBSITE_MODE } from '../../app/websiteConfig';
import { WebsiteModelSettings } from './WebsiteModelSettings';
import { Button } from '../../components/ui';
import { DecisionClient } from './DecisionClient';
type Role = 'llm' | 'jev';
interface Connection {
protocol: string;
baseUrl: string;
model: string;
hasKey?: boolean;
apiKey: string;
}
const defaults: Record<Role, Connection> = {
llm: { protocol: 'responses', baseUrl: 'https://api.openai.com/v1', model: '', apiKey: '' },
jev: {
protocol: 'typesafe',
baseUrl: 'https://api.typesafe.ai/v1/systemone',
model: 'jev-latest',
apiKey: '',
},
};
function protocolDefaults(value: string): Partial<Connection> {
const next =
value === 'codex'
? { baseUrl: '', model: '' }
: value === 'openrouter-decisions'
? { baseUrl: 'https://openrouter.ai/api/alpha/decisions', model: 'typesafe/jev-1.13' }
: value === 'typesafe'
? { baseUrl: defaults.jev.baseUrl, model: defaults.jev.model }
: { baseUrl: defaults.llm.baseUrl, model: '' };
return { protocol: value, ...next };
}
const inputClass = 'w-full rounded border border-border bg-surface px-2 py-1 text-xs';
export function ModelConnectionPanel(props: ComponentProps<typeof LocalModelConnectionPanel>) {
return WEBSITE_MODE ? (
<WebsiteModelSettings {...props} />
) : (
<LocalModelConnectionPanel {...props} />
);
}
function LocalModelConnectionPanel({
onClient,
onInvalidate,
}: {
onClient(client: DecisionClient | undefined): void;
onInvalidate(): void;
}) {
const [address, setAddress] = useState('http://127.0.0.1:8768');
const [token, setToken] = useState('');
const [client, setClient] = useState<DecisionClient>();
const [drafts, setDrafts] = useState(defaults);
const [busy, setBusy] = useState(false);
const [dirty, setDirty] = useState(false);
const [message, setMessage] = useState(
'启动 npm run decision-server,再输入终端显示的本机令牌。',
);
const [authUrl, setAuthUrl] = useState('');
const [account, setAccount] = useState('订阅尚未检查');
const [models, setModels] = useState<{ id: string; name: string }[]>([]);
const [limits, setLimits] = useState('尚未读取官方配额');
const epoch = useRef(0),
pending = useRef<AbortController | undefined>(undefined);
useEffect(
() => () => {
epoch.current++;
pending.current?.abort();
},
[],
);
function invalidate(stopTask = true) {
epoch.current++;
pending.current?.abort();
pending.current = undefined;
setBusy(false);
if (stopTask) onInvalidate();
}
function disconnect() {
invalidate();
if (client)
void client
.call('/codex/cancel', 'POST', {}, AbortSignal.timeout(3000))
.catch(() => undefined);
setAuthUrl('');
setAccount('订阅尚未检查');
setModels([]);
setLimits('尚未读取官方配额');
setDirty(false);
setClient(undefined);
onClient(undefined);
setToken('');
setDrafts(defaults);
setMessage('本机桥已断开;未修改任何全局 Codex 登录。');
}
async function run(
operation: (signal: AbortSignal, generation: number) => Promise<void>,
stopTask = true,
) {
invalidate(stopTask);
const generation = epoch.current,
abort = new AbortController();
pending.current = abort;
setBusy(true);
try {
await operation(abort.signal, generation);
} catch (error) {
if (generation === epoch.current)
setMessage(error instanceof Error ? error.message : '连接失败');
} finally {
if (generation === epoch.current) {
setBusy(false);
pending.current = undefined;
}
}
}
async function refresh(c: DecisionClient, signal: AbortSignal, generation: number) {
const status = (await c.call('/status', 'GET', undefined, signal)) as {
version?: string;
connections?: Partial<Record<Role, Connection>>;
};
if (status.version !== 'lekiwi-agent-v1') throw new Error('模型服务版本不匹配');
if (generation !== epoch.current) return;
setDrafts({
llm: { ...defaults.llm, ...status.connections?.llm, apiKey: '' },
jev: { ...defaults.jev, ...status.connections?.jev, apiKey: '' },
});
setDirty(false);
setClient(c);
onClient(c);
}
function markDirty() {
invalidate();
setDirty(true);
onClient(undefined); // Do not run against the old paid connection behind an unsaved draft.
setMessage('配置草稿尚未保存;真实任务/测试已禁用。保存会刷新两角色的服务配置。');
}
function patch(role: Role, value: Partial<Connection>) {
markDirty();
setDrafts((previous) => ({
...previous,
[role]: {
...previous[role],
...value,
...('baseUrl' in value || 'protocol' in value ? { apiKey: '', hasKey: false } : {}),
},
}));
}
async function saveConnection(role: Role) {
if (!client) return;
await run(async (signal, generation) => {
const { protocol, baseUrl, model, apiKey } = drafts[role];
await client.call('/connections', 'PUT', { role, protocol, baseUrl, model, apiKey }, signal);
await refresh(client, signal, generation);
if (generation === epoch.current)
setMessage(`${role.toUpperCase()} 配置已保存;密钥不落盘。`);
});
}
async function testConnection(role: Role) {
if (!client || dirty) return;
await run(async (signal, generation) => {
const result = (await client.call('/test', 'POST', { role }, signal)) as { usage?: unknown };
if (generation === epoch.current)
setMessage(
drafts[role].protocol === 'codex'
? 'Codex 离线禁工具门禁通过;不代表订阅真实推理通过。'
: `连接测试通过(真实请求);服务报告 usage:${JSON.stringify(result.usage ?? {})}`,
);
});
}
async function codex(operation: 'status' | 'models' | 'limits' | 'login' | 'cancel' | 'logout') {
if (!client) return;
await run(async (signal, generation) => {
const post = ['login', 'cancel', 'logout'].includes(operation);
const value = (await client.call(
'/codex/' + operation,
post ? 'POST' : 'GET',
post ? {} : undefined,
signal,
)) as Record<string, unknown>;
if (generation !== epoch.current) return;
if (operation === 'login') {
const url = new URL(String(value.authUrl));
if (
url.protocol !== 'https:' ||
url.hostname !== 'auth.openai.com' ||
url.username ||
url.password ||
(url.port && url.port !== '443')
)
throw new Error('官方登录地址校验失败');
setAuthUrl(url.href);
setMessage('请打开官方登录页;完成后点击刷新账号。认证地址不进入任务导出。');
} else if (operation === 'models') setModels(value.models as { id: string; name: string }[]);
else if (operation === 'limits') setLimits(JSON.stringify(value));
else if (operation === 'status') {
setAccount(value.loggedIn ? 'ChatGPT 已登录(仅本次服务会话)' : 'ChatGPT 未登录');
if (value.loggedIn) setAuthUrl('');
} else {
setAuthUrl('');
setAccount(operation === 'logout' ? '已退出本次隔离会话' : '登录已取消');
}
}, !['status', 'models', 'limits'].includes(operation));
}
return (
<div className="space-y-3 text-xs" aria-label="模型连接管理">
<p>密钥仅服务内存;本机令牌仅页面内存。无隐式协议、型号或付费来源切换。</p>
<label className="block">
本机模型服务
<input
className={inputClass}
value={address}
onChange={(e) => {
disconnect();
setAddress(e.target.value);
}}
/>
</label>
<label className="block">
本机服务令牌
<input
className={inputClass}
type="password"
autoComplete="off"
value={token}
onChange={(e) => {
disconnect();
setToken(e.target.value);
}}
/>
</label>
<div className="flex flex-wrap gap-2">
<Button
disabled={busy || !token.trim()}
onClick={() =>
void run(async (signal, generation) => {
const c = new DecisionClient(address, token);
await refresh(c, signal, generation);
if (generation === epoch.current)
setMessage('本机桥已连接;未自动发送任何模型推理。');
})
}
>
连接本机模型服务
</Button>
<Button onClick={disconnect}>断开模型服务</Button>
</div>
{client &&
(['llm', 'jev'] as const).map((role) => {
const draft = drafts[role];
return (
<fieldset
key={role}
disabled={busy}
className="space-y-2 rounded border border-border p-2"
>
<legend>{role === 'llm' ? 'LLM 规划' : 'Jev 决策'}</legend>
<label className="block">
{role.toUpperCase()} 协议
<select
className={inputClass}
value={draft.protocol}
aria-label={`${role.toUpperCase()} 协议`}
onChange={(e) => {
markDirty();
const config = protocolDefaults(e.target.value);
setDrafts((previous) => ({
...previous,
[role]: { ...previous[role], ...config, apiKey: '', hasKey: false },
}));
}}
>
{(role === 'llm'
? ['responses', 'chat-completions', 'codex']
: ['typesafe', 'openrouter-decisions']
).map((p) => (
<option key={p} value={p}>
{p}
</option>
))}
</select>
</label>
{draft.protocol !== 'codex' && (
<>
<label className="block">
{role.toUpperCase()} Base URL
<input
className={inputClass}
value={draft.baseUrl}
onChange={(e) => patch(role, { baseUrl: e.target.value })}
/>
</label>
<label className="block">
{role.toUpperCase()} API Key
<input
className={inputClass}
type="password"
autoComplete="off"
value={draft.apiKey}
onChange={(e) => patch(role, { apiKey: e.target.value })}
/>
</label>
<p>
{draft.hasKey ? '服务已有内存密钥;修改配置必须重新输入。' : '尚未载入密钥。'}
</p>
</>
)}
<label className="block">
{role.toUpperCase()} 型号
{draft.protocol === 'codex' ? (
<select
className={inputClass}
value={draft.model}
aria-label={`${role.toUpperCase()} 型号`}
onChange={(e) => patch(role, { model: e.target.value })}
>
<option value="">先查询并选择官方型号</option>
{models.map((m) => (
<option key={m.id} value={m.id}>
{m.name} · {m.id}
</option>
))}
</select>
) : (
<input
className={inputClass}
value={draft.model}
onChange={(e) => patch(role, { model: e.target.value })}
/>
)}
</label>
<div className="flex flex-wrap gap-2">
<Button onClick={() => void saveConnection(role)}>
保存 {role.toUpperCase()} 连接
</Button>
<Button disabled={dirty} onClick={() => void testConnection(role)}>
测试 {role.toUpperCase()}(可能计费)
</Button>
</div>
</fieldset>
);
})}
{client && (
<details>
<summary>ChatGPT 订阅 · 官方 Codex 0.147.0(实验性)</summary>
<p>
隔离临时 HOME,仅会话登录,不读取现有全局登录。只做规划,禁止工具。CLI
缺失/版本不匹配不会转用 API。
</p>
<div className="flex flex-wrap gap-2">
<Button disabled={busy} onClick={() => void codex('login')}>
发起官方登录
</Button>
<Button disabled={busy} onClick={() => void codex('cancel')}>
取消官方登录
</Button>
<Button disabled={busy} onClick={() => void codex('logout')}>
退出隔离账号
</Button>
<Button disabled={busy} onClick={() => void codex('status')}>
刷新账号
</Button>
<Button disabled={busy} onClick={() => void codex('models')}>
查询官方型号
</Button>
<Button disabled={busy} onClick={() => void codex('limits')}>
读取官方配额
</Button>
</div>
{authUrl && (
<a href={authUrl} target="_blank" rel="noopener noreferrer" className="underline">
打开 OpenAI 官方登录页
</a>
)}
<p>{account}</p>
<p className="break-all">{limits}</p>
</details>
)}
<p role="status" className="break-words">
{busy ? '请求进行中…' : message}
</p>
</div>
);
}
@@ -0,0 +1,115 @@
import { PickPlaceEvaluator, type PhysicalSample } from './PhysicalEvidence';
import { PICK_PLACE as P } from './PickPlaceScene';
function fixture() {
const evaluator = new PickPlaceEvaluator(P.object.position, P.goal.position);
let time = 0;
const sample: PhysicalSample = {
time: 0,
base: [0, 0, 0.05],
baseSpeed: 0,
yaw: 0,
tilt: 0,
joints: [0, 0, 0, 0, 0],
peakJointSpeed: 0,
opening: 0.4,
tcp: [0.257, 0.015, 0.19],
object: [0.257, 0.015, 0.19],
objectSpeed: 0,
fingerForces: [1, 1],
supported: false,
onGoalSupport: false,
};
const tick = (count: number) => {
for (let i = 0; i < count; i++) {
sample.time = time += 0.02;
evaluator.update(sample, 0.02);
}
};
const carry = () => {
tick(10);
sample.base[1] = 0.6;
sample.object[1] += 0.6;
sample.tcp[1] += 0.6;
tick(2);
};
const release = () => {
Object.assign(sample, {
opening: 1,
object: [...P.goal.position],
tcp: [P.goal.position[0], P.goal.position[1], P.goal.position[2] + 0.1],
fingerForces: [0, 0],
supported: true,
onGoalSupport: true,
});
};
return { evaluator, sample, tick, carry, release };
}
it('requires actual lift, BOTH fingers, held translation, release, retreat and continuous supported stability', () => {
const f = fixture();
f.carry();
expect(f.evaluator.transported).toBeCloseTo(0.6);
f.release();
f.tick(24);
expect(f.evaluator.succeeded).toBe(false);
f.tick(2);
expect(f.evaluator.succeeded).toBe(true);
});
it.each([
[10, 0],
[0, 10],
[0, 0],
])('many contacts on one side/empty cannot establish grasp: %j', (a, b) => {
const f = fixture();
f.sample.fingerForces = [a, b];
f.carry();
expect(f.evaluator.hasGrasped).toBe(false);
f.release();
f.tick(100);
expect(f.evaluator.succeeded).toBe(false);
});
it('an object initially at the goal has no transport history', () => {
const f = fixture();
f.release();
f.tick(100);
expect(f.evaluator.succeeded).toBe(false);
});
it('contact on a support is not verified lifting', () => {
const f = fixture();
f.sample.supported = true;
f.carry();
expect(f.evaluator.secure).toBe(false);
});
it('slip invalidates history; empty base travel is not credited after a regrasp', () => {
const f = fixture();
f.carry();
f.sample.fingerForces = [0, 0];
f.tick(10);
expect(f.evaluator.transported).toBe(0);
f.sample.base[1] = 1.2;
f.sample.fingerForces = [1, 1];
f.tick(20);
expect(f.evaluator.secure).toBe(true);
expect(f.evaluator.transported).toBe(0);
});
it('duplicate/frozen observations never accumulate stability and time jumps fail closed', () => {
const f = fixture();
f.carry();
f.release();
f.tick(1);
for (let i = 0; i < 100; i++) f.evaluator.update(f.sample, 0.02);
expect(f.evaluator.succeeded).toBe(false);
expect(() => f.evaluator.update({ ...f.sample, time: f.sample.time + 1 }, 0.02)).toThrow(/时钟/);
});
it('ongoing grasp, high velocity or missing retreat cannot produce placement success', () => {
for (const fault of ['grasp', 'speed', 'retreat'] as const) {
const f = fixture();
f.carry();
f.release();
if (fault === 'grasp') f.sample.fingerForces = [1, 1];
if (fault === 'speed') f.sample.objectSpeed = 0.1;
if (fault === 'retreat') f.sample.tcp = [...f.sample.object];
f.tick(100);
expect(f.evaluator.succeeded).toBe(false);
}
});
@@ -0,0 +1,206 @@
import type { DoubleBuffer } from '@mujoco/mujoco';
import type { SimulationSession } from '../../simulation/SimulationSession';
import { ModelBindings } from '../../simulation/ModelBindings';
import { PICK_PLACE as P, PICK_PLACE_ROBOT as R } from './PickPlaceScene';
export interface PhysicalSample {
time: number;
base: number[];
baseSpeed: number;
yaw: number;
tilt: number;
joints: number[];
peakJointSpeed: number;
opening: number;
tcp: number[];
object: number[];
objectSpeed: number;
fingerForces: [number, number];
supported: boolean;
onGoalSupport: boolean;
}
/** Geom-to-finger aggregation: many hull contacts on ONE finger never count as two. */
export class PhysicalSampler {
private readonly force: DoubleBuffer;
private readonly arm;
private readonly gripper;
private readonly objectQ: number;
private readonly objectV: number;
private readonly baseQ: number;
private readonly baseV: number;
private readonly objectBody: number;
private readonly fingerBodies: number[];
private readonly goalSupport: number;
private readonly site: number;
private disposed = false;
constructor(private readonly session: SimulationSession) {
const { model, module } = session,
b = new ModelBindings(model);
this.arm = R.armJoints.map((joint) => b.scalarJoint(joint.name));
this.gripper = b.scalarJoint(R.gripperJoint);
const objectJoint = b.resolve(b.joints, '__mm_object_joint');
this.objectQ = Number(model.jnt_qposadr[objectJoint]);
this.objectV = Number(model.jnt_dofadr[objectJoint]);
const baseJoint = b.resolve(b.joints, R.baseJointName);
this.baseQ = Number(model.jnt_qposadr[baseJoint]);
this.baseV = Number(model.jnt_dofadr[baseJoint]);
this.objectBody = b.resolve(b.bodies, '__mm_object');
this.fingerBodies = ['Wrist_Roll_08c-v1', 'Moving_Jaw_08d-v1'].map((name) =>
Number(model.body_weldid[b.resolve(b.bodies, name)]),
);
this.goalSupport = module.mj_name2id(model, 5, '__agent_support_1');
this.site = module.mj_name2id(model, 6, P.tcpSite);
if (this.site < 0 || this.goalSupport < 0) throw new Error('缺少抓放场景');
this.force = new module.DoubleBuffer(6);
}
sample(): PhysicalSample {
if (this.disposed) throw new Error('物理采样器已释放');
const { model: m, data: d, module } = this.session;
const forces: [number, number] = [0, 0];
let supported = false,
onGoalSupport = false;
const contacts = d.contact;
try {
for (let i = 0; i < d.ncon; i++) {
const contact = contacts.get(i)!;
try {
const a = Number(m.geom_bodyid[contact.geom1]),
b = Number(m.geom_bodyid[contact.geom2]);
if (a !== this.objectBody && b !== this.objectBody) continue;
const other = a === this.objectBody ? b : a;
const geom = a === this.objectBody ? contact.geom2 : contact.geom1;
module.mj_contactForce(m, d, i, this.force);
const normal = Math.max(0, Number(this.force.GetView()[0]));
if (normal <= 0) continue;
if (Number(m.body_weldid[other]) === 0) supported = true;
if (geom === this.goalSupport) onGoalSupport = true;
const finger = this.fingerBodies.indexOf(Number(m.body_weldid[other]));
if (finger >= 0) forces[finger] += normal;
} finally {
contact.delete();
}
}
} finally {
contacts.delete();
}
const q: Float64Array = d.qpos,
v: Float64Array = d.qvel;
const [w, x, y, z] = Array.from(q.subarray(this.baseQ + 3, this.baseQ + 7));
const result: PhysicalSample = {
time: Number(d.time),
base: Array.from(q.subarray(this.baseQ, this.baseQ + 3)),
baseSpeed: Math.hypot(...v.subarray(this.baseV, this.baseV + 3)),
yaw: Math.atan2(2 * (w * z + x * y), 1 - 2 * (y * y + z * z)),
tilt: Math.acos(Math.max(-1, Math.min(1, 1 - 2 * (x * x + y * y)))),
joints: this.arm.map((joint) => q[joint.qposAddress]),
peakJointSpeed: Math.max(
...[...this.arm, this.gripper].map((joint) => Math.abs(v[joint.qvelAddress])),
),
opening: Math.max(
0,
Math.min(
1,
(q[this.gripper.qposAddress] - R.gripperClosed) / (R.gripperOpen - R.gripperClosed),
),
),
tcp: Array.from(d.site_xpos.slice(this.site * 3, this.site * 3 + 3), Number),
object: Array.from(q.subarray(this.objectQ, this.objectQ + 3)),
objectSpeed: Math.hypot(...v.subarray(this.objectV, this.objectV + 3)),
fingerForces: forces,
supported,
onGoalSupport,
};
if (
[
result.time,
result.baseSpeed,
result.yaw,
result.tilt,
result.opening,
result.peakJointSpeed,
result.objectSpeed,
...result.base,
...result.joints,
...result.tcp,
...result.object,
...forces,
].some((value) => !Number.isFinite(value))
)
throw new Error('物理状态非有限');
return result;
}
dispose(): void {
if (!this.disposed) {
this.disposed = true;
this.force.delete();
}
}
}
export class PickPlaceEvaluator {
private graspTime = 0;
private stableTime = 0;
private reference?: number[];
private lastSecureBase?: number[];
private readonly transportVector = [0, 0];
private lostTime = 0;
private lastTime?: number;
secure = false;
hasGrasped = false;
transported = 0;
succeeded = false;
constructor(
readonly initialObject: readonly number[],
readonly goal: readonly number[],
) {}
update(sample: PhysicalSample, dt: number): void {
if (!Number.isFinite(dt) || dt <= 0 || dt > P.controlDt) throw new Error('非法物理采样步长');
if (this.lastTime !== undefined) {
const elapsed = sample.time - this.lastTime;
if (Math.abs(elapsed) < 1e-10) return; // Frozen/repeated observation is not stability evidence.
if (Math.abs(elapsed - dt) > 1e-6) throw new Error('物理证据时钟不连续');
}
this.lastTime = sample.time;
const offset = sample.object.map((v, i) => v - sample.tcp[i]);
const fingers = sample.fingerForces.every((force) => force >= P.fingerForceMin);
if (!fingers) this.reference = undefined;
else this.reference ??= offset;
const drift = this.reference
? Math.hypot(...offset.map((v, i) => v - this.reference![i]))
: Infinity;
const lifted =
fingers &&
!sample.supported &&
sample.object[2] - this.initialObject[2] >= P.liftMin &&
drift <= P.relativeDriftMax;
this.graspTime = lifted ? this.graspTime + dt : 0;
this.secure = this.graspTime >= P.graspStableTime;
if (this.secure) {
this.hasGrasped = true;
if (this.lastSecureBase)
for (let i = 0; i < 2; i++)
this.transportVector[i] += sample.base[i] - this.lastSecureBase[i];
this.transported = Math.max(this.transported, Math.hypot(...this.transportVector));
}
this.lastSecureBase = this.secure ? [...sample.base] : undefined;
this.lostTime = !fingers && !sample.onGoalSupport ? this.lostTime + dt : 0;
if (this.lostTime >= 0.1) {
this.hasGrasped = false;
this.transported = 0;
this.transportVector.fill(0);
}
const released =
sample.opening >= 0.65 && sample.fingerForces.every((force) => force < P.fingerForceMin);
const placed =
this.hasGrasped &&
this.transported >= P.transportMin &&
released &&
sample.onGoalSupport &&
Math.hypot(...sample.object.map((v, i) => v - this.goal[i])) < P.placementTolerance &&
sample.objectSpeed < P.placementSpeedMax &&
Math.hypot(...sample.object.map((v, i) => v - sample.tcp[i])) >= P.retreatMin;
this.stableTime = placed ? this.stableTime + dt : 0;
this.succeeded = this.stableTime >= P.placementStableTime;
}
}
@@ -0,0 +1,37 @@
import { describe, expect, it } from 'vitest';
import { composePickPlaceScene, PICK_PLACE_ROBOT, seedPositions } from './PickPlaceScene';
describe('受限任务场景与可见支撑台', () => {
it('目标编辑保留源位置,拒绝范围外/非有限/高度改动', () => {
expect(seedPositions(0, [0.237, 0.565, 0.128])).toEqual({
object: [0.257, 0.015, 0.128],
goal: [0.237, 0.565, 0.128],
});
for (const goal of [
[NaN, 0.615, 0.128],
[0.8, 0.6, 0.128],
[0.257, 0.1, 0.128],
[0.257, 0.615, 0.13],
[0.257, 0.615],
])
expect(() => seedPositions(0, goal)).toThrow('目标');
});
it('两个实体支撑台进入视觉组,仍保留物理碰撞;不覆盖用户模型', () => {
const input = new TextEncoder().encode(
`<mujoco><worldbody><body name="__mm_object"><geom type="box" size=".018 .018 .018"/></body><body name="__mm_goal"/><body name="${PICK_PLACE_ROBOT.eefBodyName}"/></worldbody></mujoco>`,
);
const result = composePickPlaceScene(input, 0, true, [0.257, 0.617, 0.128]);
const doc = new DOMParser().parseFromString(
new TextDecoder().decode(result),
'application/xml',
);
expect(doc.querySelector('geom[name="__agent_support_1"]')?.getAttribute('group')).toBe('1');
expect(doc.querySelector('geom[name="__agent_support_1"]')?.getAttribute('pos')).toBe(
'0.257 0.617 0.055',
);
expect(doc.querySelector('geom[name="__agent_support_1"]')?.getAttribute('contype')).not.toBe(
'0',
);
expect(new TextDecoder().decode(input)).not.toContain('__agent_');
expect(() => composePickPlaceScene(result, 0, true)).toThrow('命名空间');
});
});
@@ -0,0 +1,73 @@
import preset from '../../../../contracts/lekiwi-pick-place-v1.json';
import { MOBILE_ROBOTS } from '../RobotDescriptor';
import { composeMobileScene } from '../SceneComposer';
export const PICK_PLACE = preset;
export const PICK_PLACE_ROBOT = MOBILE_ROBOTS.find((robot) => robot.id === preset.robotId)!;
export const TASK_GOAL_LIMITS = { min: [0.237, 0.565, 0.128], max: [0.277, 0.665, 0.128] } as const;
export interface PickPlaceTaskConfig {
seed: number;
goal?: readonly number[];
}
export function seedPositions(
seed: number,
goal?: readonly number[],
): { object: number[]; goal: number[] } {
if (
goal &&
(goal.length !== 3 ||
goal.some(
(v, i) => !Number.isFinite(v) || v < TASK_GOAL_LIMITS.min[i] || v > TASK_GOAL_LIMITS.max[i],
))
)
throw new Error('目标超出首版受限区域;支撑高度不可更改');
const item = preset.frozenSeeds.find((candidate) => candidate.id === seed);
if (!item) throw new Error('未知抓放标定种子');
const offset = [...item.offset, 0];
return {
object: preset.object.position.map((v, i) => v + offset[i]),
goal: goal ? [...goal] : preset.goal.position.map((v, i) => v + offset[i]),
};
}
/** Task-only scene, never rewrites a user's map or the RL scene contract. */
export function composePickPlaceScene(
source: Uint8Array,
seed = 0,
alreadyMobile = false,
goal?: readonly number[],
): Uint8Array {
const doc = new DOMParser().parseFromString(
new TextDecoder().decode(alreadyMobile ? source : composeMobileScene(source, PICK_PLACE_ROBOT)),
'application/xml',
);
if (doc.querySelector('parsererror')) throw new Error('抓放场景 XML 非法');
if (doc.querySelector('[name="__agent_tcp"], [name="__agent_support_0"]'))
throw new Error('抓放任务命名空间冲突');
const positions = seedPositions(seed, goal);
const object = doc.querySelector('body[name="__mm_object"]')!;
object.setAttribute('pos', positions.object.join(' '));
object.querySelector('geom')!.setAttribute('mass', String(preset.object.mass));
doc.querySelector('body[name="__mm_goal"]')!.setAttribute('pos', positions.goal.join(' '));
const world = doc.querySelector('mujoco > worldbody')!;
for (const [i, position] of [positions.object, positions.goal].entries()) {
const geom = doc.createElement('geom');
for (const [name, value] of Object.entries({
name: `__agent_support_${i}`,
type: 'box',
group: '1', // Simple support solids are both visible and collidable.
pos: `${position[0]} ${position[1]} ${preset.supportHeight / 2}`,
size: preset.supportHalfSize.join(' '),
friction: '1 .005 .0001',
rgba: '.25 .35 .45 1',
}))
geom.setAttribute(name, value);
world.append(geom);
}
const site = doc.createElement('site');
site.setAttribute('name', preset.tcpSite);
site.setAttribute('pos', preset.tcpOffset.join(' '));
site.setAttribute('size', '.002');
doc.querySelector(`body[name="${PICK_PLACE_ROBOT.eefBodyName}"]`)!.append(site);
return new TextEncoder().encode(new XMLSerializer().serializeToString(doc));
}
@@ -0,0 +1,67 @@
import { RequestGate, type RequestStamp, type Stamped } from './RequestGate';
const stamp: RequestStamp = {
runId: 'run',
sceneRevision: 1,
sequence: 1,
planRevision: 1,
requestId: 'request',
};
function deferred<T>() {
let resolve!: (v: T) => void;
const promise = new Promise<T>((r) => {
resolve = r;
});
return { promise, resolve };
}
it('accepts only the matching request identity', async () => {
const gate = new RequestGate();
await expect(gate.request(stamp, 1000, async () => ({ stamp, value: 1 }))).resolves.toBe(1);
for (const key of Object.keys(stamp) as (keyof RequestStamp)[])
await expect(
gate.request(stamp, 1000, async () => ({
stamp: { ...stamp, [key]: 'different' },
value: 1,
})),
).rejects.toThrow(/过期/);
});
it('cancellation rejects immediately even when the provider ignores AbortSignal', async () => {
const gate = new RequestGate(),
work = deferred<Stamped<number>>();
const result = gate.request(stamp, 1000, () => work.promise);
const rejected = expect(result).rejects.toThrow('pause');
gate.invalidate('pause');
await rejected;
work.resolve({ stamp, value: 99 });
await Promise.resolve();
});
it('replacement isolates a delayed old result from a new run', async () => {
const gate = new RequestGate(),
work = deferred<Stamped<number>>();
const old = gate.request(stamp, 1000, () => work.promise);
const rejected = expect(old).rejects.toThrow('替换');
const next = await gate.request({ ...stamp, requestId: 'new' }, 1000, async () => ({
stamp: { ...stamp, requestId: 'new' },
value: 2,
}));
await rejected;
work.resolve({ stamp, value: 1 });
expect(next).toBe(2);
});
it('timeout aborts the operation, does not fall back, and permits a new request', async () => {
vi.useFakeTimers();
try {
const gate = new RequestGate();
let signal: AbortSignal | undefined;
const result = gate.request(stamp, 10, async (s) => {
signal = s;
return new Promise<never>(() => {});
});
const rejected = expect(result).rejects.toThrow('超时');
await vi.advanceTimersByTimeAsync(11);
await rejected;
expect(signal?.aborted).toBe(true);
await expect(gate.request(stamp, 10, async () => ({ stamp, value: 3 }))).resolves.toBe(3);
} finally {
vi.useRealTimers();
}
});
@@ -0,0 +1,57 @@
export interface RequestStamp {
runId: string;
sceneRevision: number;
sequence: number;
planRevision: number;
requestId: string;
}
export interface Stamped<T> {
stamp: RequestStamp;
value: T;
}
const sameStamp = (a: RequestStamp, b: RequestStamp) =>
a.runId === b.runId &&
a.sceneRevision === b.sceneRevision &&
a.sequence === b.sequence &&
a.planRevision === b.planRevision &&
a.requestId === b.requestId;
/** One outstanding decision, explicit cancellation, and rejection of providers
* that ignore AbortSignal or reply against a different run/scene/observation. */
export class RequestGate {
private epoch = 0;
private controller?: AbortController;
invalidate(reason = '请求已取消'): void {
this.epoch++;
this.controller?.abort(reason);
this.controller = undefined;
}
async request<T>(
stamp: RequestStamp,
timeoutMs: number,
operation: (signal: AbortSignal) => Promise<Stamped<T>>,
): Promise<T> {
this.invalidate('请求已替换');
const epoch = this.epoch,
expected = { ...stamp },
controller = new AbortController();
this.controller = controller;
let rejectAbort!: (error: Error) => void;
const aborted = new Promise<never>((_, reject) => {
rejectAbort = reject;
});
const listener = () => rejectAbort(new Error(String(controller.signal.reason ?? '请求已取消')));
controller.signal.addEventListener('abort', listener, { once: true });
const timer = setTimeout(() => controller.abort('模型请求超时'), timeoutMs);
try {
const response = await Promise.race([operation(controller.signal), aborted]);
if (epoch !== this.epoch || controller.signal.aborted || !sameStamp(expected, response.stamp))
throw new Error('过期或不匹配的模型响应');
return response.value;
} finally {
clearTimeout(timer);
controller.signal.removeEventListener('abort', listener);
if (epoch === this.epoch) this.controller = undefined;
}
}
}
@@ -0,0 +1,95 @@
import { cleanup, fireEvent, render, screen, waitFor } from '@testing-library/react';
import { afterEach, expect, it, vi } from 'vitest';
import { WebsiteModelSettings } from './WebsiteModelSettings';
import { DecisionClient } from './DecisionClient';
afterEach(() => {
cleanup();
vi.restoreAllMocks();
});
const status = {
ready: false,
configuration: {
llm: { provider: 'deepseek', model: 'deepseek-flash', hasKey: false },
jev: { hasKey: false },
},
};
function fixture() {
const call = vi.fn(async (path: string) => {
if (path === '/models')
return {
openrouterAvailable: true,
models: [
{ provider: 'deepseek', id: 'deepseek-flash', name: 'DeepSeek' },
{ provider: 'openrouter', id: 'vendor/model', name: 'Router Model' },
],
};
if (path === '/configuration') return { ...status, ready: true };
if (path === '/codex/login') throw new Error('codex_rpc_timeout');
return status;
});
const client = { call, receipts: [], mode: 'api' } as unknown as DecisionClient;
vi.spyOn(DecisionClient, 'website').mockResolvedValue(client);
return { call, client };
}
it('只有模型与两角色密钥,原子保存,不持久化/自动测试', async () => {
const { call, client } = fixture();
const onClient = vi.fn();
const storage = vi.spyOn(Storage.prototype, 'setItem');
render(<WebsiteModelSettings onClient={onClient} onInvalidate={vi.fn()} />);
await waitFor(() => expect(screen.getByRole('button', { name: '保存设置' })).not.toBeDisabled());
expect(screen.queryByLabelText('本机服务令牌')).not.toBeInTheDocument();
expect(screen.queryByLabelText('LLM Base URL')).not.toBeInTheDocument();
fireEvent.change(screen.getByLabelText('LLM API_KEY'), { target: { value: 'fixture-llm' } });
fireEvent.change(screen.getByLabelText('Jev API_KEY(OpenRouter)'), {
target: { value: 'fixture-jev' },
});
fireEvent.click(screen.getByRole('button', { name: '保存设置' }));
await screen.findByText(/设置已保存/);
expect(call).toHaveBeenCalledWith(
'/configuration',
'PUT',
{
llm: { provider: 'deepseek', model: 'deepseek-flash', apiKey: 'fixture-llm' },
jev: { apiKey: 'fixture-jev' },
},
expect.any(AbortSignal),
);
expect(screen.getByLabelText('LLM API_KEY')).toHaveValue('');
expect(screen.getByLabelText('Jev API_KEY(OpenRouter)')).toHaveValue('');
expect(onClient).toHaveBeenLastCalledWith(client);
expect(storage).not.toHaveBeenCalled();
expect(call.mock.calls.some(([path]) => ['/plan', '/test', '/decide'].includes(path))).toBe(
false,
);
});
it('换提供方清空密钥,保留订阅入口并明确网络不可用', async () => {
const { call } = fixture();
render(<WebsiteModelSettings onClient={vi.fn()} onInvalidate={vi.fn()} />);
await waitFor(() => expect(screen.getByRole('button', { name: '保存设置' })).not.toBeDisabled());
fireEvent.change(screen.getByLabelText('LLM API_KEY'), { target: { value: 'fixture-old' } });
fireEvent.change(screen.getByLabelText('模型选择'), {
target: { value: 'openrouter:vendor/model' },
});
expect(screen.getByLabelText('LLM API_KEY')).toHaveValue('');
fireEvent.click(screen.getByRole('button', { name: 'ChatGPT 订阅' }));
fireEvent.click(screen.getByRole('button', { name: '登录 ChatGPT 订阅' }));
await screen.findByText(/官方订阅服务连接超时/);
expect(call.mock.calls.filter(([path]) => path === '/configuration')).toHaveLength(0);
});
it('同源客户端自动携带 Cookie/CSRF,但不给原本机模式放开任意远端', async () => {
const fetcher = vi.fn<typeof fetch>(async (_url, opts) => {
if (String(_url).endsWith('/session'))
return Response.json({ csrfToken: 'csrf-fixture', configVersion: 2 });
expect(opts?.credentials).toBe('same-origin');
expect(opts?.headers).toMatchObject({
'X-CSRF-Token': 'csrf-fixture',
'X-Config-Version': '2',
});
expect(opts?.headers).not.toHaveProperty('Authorization');
return Response.json({});
});
const c = await DecisionClient.website(undefined, fetcher);
await c.call('/configuration', 'PUT', {});
expect(() => new DecisionClient('https://evil.test', 'token')).toThrow();
});
@@ -0,0 +1,436 @@
import { useEffect, useRef, useState } from 'react';
import { Button } from '../../components/ui';
import { DecisionClient } from './DecisionClient';
interface Props {
onClient(client: DecisionClient | undefined): void;
onInvalidate(): void;
}
interface Configuration {
llm: { provider: string; model: string; hasKey: boolean };
jev: { hasKey: boolean };
}
interface Status {
ready: boolean;
configuration: Configuration;
}
interface Model {
provider: string;
id: string;
name: string;
}
const inputClass = 'w-full rounded border border-border bg-surface px-2 py-1.5 text-xs';
const errors: Record<string, string> = {
session_expired: '本次会话已过期,请重新打开模型设置并填写凭据。',
configuration_changed: '配置已在其他页面变更,请重新打开设置,不会使用旧配置继续任务。',
api_key_required: '请填写对应的 API_KEY。切换提供方后需要重新填写。',
server_busy: '服务器当前繁忙,请稍后手动重试。',
ip_rate_limit: '请求次数已达上限,请稍后再试。',
subscription_capacity: '订阅登录名额暂满,请稍后再试;不会自动切换付费 API。',
codex_chatgpt_login_required: '请先完成官方订阅登录。',
codex_rpc_failed: '当前订阅服务不可用,可能是网络或账号限制;不会切换付费 API。',
codex_rpc_timeout: '官方订阅服务连接超时;API 模式仍可手动使用。',
codex_not_installed: '当前服务器未启用订阅服务。',
model_unavailable: '当前模型不可用,请重新选择;不会自动替换。',
csrf_required: '会话验证失效,请重新打开模型设置。',
};
function message(error: unknown) {
const code = error instanceof Error ? error.message : '请求失败';
return errors[code] ?? `请求未完成:${code}`;
}
export function WebsiteModelSettings({ onClient, onInvalidate }: Props) {
const callbacks = useRef({ onClient, onInvalidate });
useEffect(() => {
callbacks.current = { onClient, onInvalidate };
}, [onClient, onInvalidate]);
const [client, setClient] = useState<DecisionClient>();
const [config, setConfig] = useState<Configuration>();
const [mode, setMode] = useState<'api' | 'codex'>('api');
const [selection, setSelection] = useState('deepseek:deepseek-flash');
const [apiKey, setApiKey] = useState('');
const [jevKey, setJevKey] = useState('');
const [models, setModels] = useState<Model[]>([]);
const [search, setSearch] = useState('');
const [codexModels, setCodexModels] = useState<{ id: string; name: string }[]>([]);
const [codexModel, setCodexModel] = useState('');
const [loggedIn, setLoggedIn] = useState(false);
const [device, setDevice] = useState<{ verificationUrl: string; userCode: string }>();
const [busy, setBusy] = useState(true);
const [dirty, setDirty] = useState(false);
const [notice, setNotice] = useState('正在连接网站模型服务…');
const generation = useRef(0);
const pending = useRef<AbortController | undefined>(undefined);
useEffect(() => {
const abort = new AbortController();
pending.current = abort;
let alive = true;
void (async () => {
try {
const c = await DecisionClient.website(abort.signal);
const status = (await c.call('/status', 'GET', undefined, abort.signal)) as Status;
if (!alive) return;
setClient(c);
setConfig(status.configuration);
const llm = status.configuration.llm;
setMode(llm.provider === 'codex' ? 'codex' : 'api');
if (llm.provider === 'codex') setCodexModel(llm.model);
else setSelection(`${llm.provider}:${llm.model}`);
callbacks.current.onClient(status.ready ? c : undefined);
setNotice('会话已就绪。保存不会调用模型;开始真实任务或测试连接可能计费。');
const list = (await c.call('/models', 'GET', undefined, abort.signal)) as {
models: Model[];
openrouterAvailable: boolean;
};
if (!alive) return;
setModels(list.models);
if (!list.openrouterAvailable)
setNotice('OpenRouter 模型目录暂不可用;已保存的型号不会自动替换。');
} catch (error) {
if (alive) setNotice(message(error));
} finally {
if (alive) setBusy(false);
}
})();
const cancelPending = () => {
generation.current++;
pending.current?.abort();
};
return () => {
alive = false;
cancelPending();
abort.abort();
};
}, []);
useEffect(() => {
if (!device || !client) return;
const abort = new AbortController();
const deadline = Date.now() + 600_000;
let timer: ReturnType<typeof setTimeout>;
const poll = async () => {
if (Date.now() >= deadline) {
setDevice(undefined);
setNotice('设备码已过期,请重新发起登录。');
return;
}
try {
const status = (await client.call('/codex/status', 'GET', undefined, abort.signal)) as {
loggedIn?: boolean;
};
if (abort.signal.aborted) return;
if (status.loggedIn) {
const list = (await client.call('/codex/models', 'GET', undefined, abort.signal)) as {
models: { id: string; name: string }[];
};
if (abort.signal.aborted) return;
setLoggedIn(true);
setCodexModels(list.models);
setDevice(undefined);
setNotice('官方登录完成;请选择订阅模型并保存。');
return;
}
} catch (error) {
if (abort.signal.aborted) return;
setDevice(undefined);
setNotice(message(error));
return;
}
timer = setTimeout(() => void poll(), 3000);
};
timer = setTimeout(() => void poll(), 3000);
return () => {
clearTimeout(timer);
abort.abort();
};
}, [device, client]);
function invalidate() {
callbacks.current.onInvalidate();
callbacks.current.onClient(undefined);
setDirty(true);
}
async function run(operation: (c: DecisionClient, signal: AbortSignal) => Promise<void>) {
if (!client || busy) return;
const current = ++generation.current;
const abort = new AbortController();
pending.current = abort;
setBusy(true);
try {
await operation(client, abort.signal);
} catch (error) {
if (current === generation.current) {
setNotice(message(error));
callbacks.current.onClient(undefined);
}
} finally {
if (current === generation.current) setBusy(false);
}
}
async function save() {
await run(async (c, signal) => {
invalidate();
const split = selection.indexOf(':');
const provider = mode === 'codex' ? 'codex' : selection.slice(0, split);
const model = mode === 'codex' ? codexModel : selection.slice(split + 1);
const result = (await c.call(
'/configuration',
'PUT',
{
llm: { provider, model, ...(mode === 'api' && apiKey ? { apiKey } : {}) },
jev: { ...(jevKey ? { apiKey: jevKey } : {}) },
},
signal,
)) as Status;
if (signal.aborted) return;
setConfig(result.configuration);
setApiKey('');
setJevKey('');
setDirty(false);
callbacks.current.onClient(result.ready ? c : undefined);
setNotice('设置已保存;输入框已清空。密钥仅在本次服务会话内存中。');
});
}
async function account(operation: 'login' | 'status' | 'cancel' | 'logout') {
await run(async (c, signal) => {
const post = operation !== 'status';
if (post) invalidate();
const result = (await c.call(
`/codex/${operation}`,
post ? 'POST' : 'GET',
post ? {} : undefined,
signal,
)) as {
loggedIn?: boolean;
verificationUrl?: string;
userCode?: string;
};
if (signal.aborted) return;
if (operation === 'login') {
const url = new URL(result.verificationUrl ?? '');
if (
url.protocol !== 'https:' ||
url.hostname !== 'auth.openai.com' ||
url.username ||
url.password ||
(url.port && url.port !== '443')
)
throw new Error('官方登录地址校验失败');
setDevice({ verificationUrl: url.href, userCode: result.userCode ?? '' });
setNotice('请在官方页面输入一次性码。账户可能需要先开启设备码登录;完成后刷新登录状态。');
} else {
setLoggedIn(Boolean(result.loggedIn));
if (operation !== 'status' || result.loggedIn) setDevice(undefined);
if (result.loggedIn) {
const list = (await c.call('/codex/models', 'GET', undefined, signal)) as {
models: { id: string; name: string }[];
};
if (signal.aborted) return;
setCodexModels(list.models);
setNotice('官方登录已完成;选择模型并保存。登录不代表已通过真实推理。');
} else {
setCodexModels([]);
setNotice(
operation === 'status' ? '尚未完成官方登录;当前网络可能不可用。' : '订阅凭据已清除。',
);
}
}
});
}
const matching = models.filter((m) =>
`${m.provider} ${m.id} ${m.name}`.toLowerCase().includes(search.toLowerCase()),
);
return (
<div className="space-y-3 text-xs" aria-label="网站模型设置">
<p className="text-text-secondary">
凭据经 HTTPS 发送到网站后端,仅存会话内存;空闲 30 分钟过期。模型文件仍在浏览器内。
</p>
<div className="flex gap-2" role="group" aria-label="LLM 连接方式">
{(['api', 'codex'] as const).map((value) => (
<Button
key={value}
disabled={busy}
aria-pressed={mode === value}
onClick={() => {
if (mode !== value) {
invalidate();
setMode(value);
setApiKey('');
setNotice('连接方式已修改,请完成设置并保存。');
}
}}
>
{value === 'api' ? 'API_KEY' : 'ChatGPT 订阅'}
</Button>
))}
</div>
<fieldset disabled={busy || !client} className="space-y-2">
{mode === 'api' ? (
<>
<label className="block">
LLM API_KEY
<input
className={inputClass}
type="password"
autoComplete="off"
value={apiKey}
placeholder={
config?.llm.hasKey ? '已设置;留空保持同提供方密钥' : '填写自己的 API_KEY'
}
onChange={(e) => {
invalidate();
setApiKey(e.target.value);
}}
/>
</label>
<label className="block">
搜索模型
<input
className={inputClass}
type="search"
value={search}
onChange={(e) => setSearch(e.target.value)}
placeholder="DeepSeek / OpenRouter"
/>
</label>
<label className="block">
模型选择
<select
className={inputClass}
value={selection}
onChange={(e) => {
invalidate();
if (e.target.value.split(':')[0] !== selection.split(':')[0]) setApiKey('');
setSelection(e.target.value);
}}
>
{!matching.some((m) => `${m.provider}:${m.id}` === selection) && (
<option value={selection}>{selection}</option>
)}
{['deepseek', 'openrouter'].map((provider) => (
<optgroup
key={provider}
label={provider === 'deepseek' ? 'DeepSeek' : 'OpenRouter'}
>
{matching
.filter((m) => m.provider === provider)
.map((m) => (
<option key={m.id} value={`${m.provider}:${m.id}`}>
{m.name} · {m.id}
</option>
))}
</optgroup>
))}
</select>
</label>
</>
) : (
<div className="space-y-2">
<p>官方订阅登录(实验性)。当前服务器可能无法连接官方服务;不会自动转付费 API。</p>
<div className="flex flex-wrap gap-2">
<Button onClick={() => void account('login')}>登录 ChatGPT 订阅</Button>
<Button onClick={() => void account('status')}>刷新登录状态</Button>
<Button onClick={() => void account('cancel')}>取消登录</Button>
<Button onClick={() => void account('logout')}>退出订阅</Button>
</div>
{device && (
<div className="space-y-1">
<a
className="underline"
href={device.verificationUrl}
target="_blank"
rel="noopener noreferrer"
>
打开 OpenAI 官方验证页
</a>
<p>
一次性码:<strong>{device.userCode}</strong>(10 分钟内完成,过期请重新登录)
</p>
</div>
)}
<p>{loggedIn ? '订阅已登录' : '订阅尚未确认登录'}</p>
<label className="block">
订阅模型
<select
className={inputClass}
value={codexModel}
onChange={(e) => {
invalidate();
setCodexModel(e.target.value);
}}
>
<option value="">登录后选择官方可用模型</option>
{codexModels.map((m) => (
<option key={m.id} value={m.id}>
{m.name}
</option>
))}
</select>
</label>
</div>
)}
<label className="block">
Jev API_KEY(OpenRouter)
<input
className={inputClass}
type="password"
autoComplete="off"
value={jevKey}
placeholder={config?.jev.hasKey ? '已设置;留空保持' : '填写 Jev 的 OpenRouter API_KEY'}
onChange={(e) => {
invalidate();
setJevKey(e.target.value);
}}
/>
</label>
<Button onClick={() => void save()}>保存设置</Button>
<details>
<summary className="cursor-pointer">连接测试与凭据管理</summary>
<div className="mt-2 flex flex-wrap gap-2">
{(['llm', 'jev'] as const).map((role) => (
<Button
key={role}
disabled={dirty || !config?.jev.hasKey}
onClick={() =>
void run(async (c, signal) => {
callbacks.current.onInvalidate();
await c.call('/test', 'POST', { role }, signal);
if (!signal.aborted)
setNotice(
`${role.toUpperCase()} 连接测试通过;真实 API 可能计费,订阅测试仅验证工具门禁。`,
);
})
}
>
测试 {role.toUpperCase()}(可能计费)
</Button>
))}
<Button
onClick={() =>
void run(async (c, signal) => {
invalidate();
await c.call('/session', 'DELETE', {}, signal);
if (signal.aborted) return;
setClient(undefined);
setConfig(undefined);
setDevice(undefined);
setApiKey('');
setJevKey('');
setLoggedIn(false);
setNotice('本次凭据已清除。关闭并重新打开模型设置可创建新会话。');
})
}
>
清除本次凭据
</Button>
</div>
</details>
</fieldset>
<p role="status" className="break-words">
{busy ? '正在连接…' : notice}
</p>
{dirty && <p className="text-warning">设置未保存,真实任务暂不可用。</p>}
</div>
);
}
@@ -0,0 +1,93 @@
import schema from '../../../../contracts/lekiwi-agent-v1.schema.json';
import {
AGENT_VERSION,
PRECONDITIONS,
SKILLS,
validatePlan,
validateJevDecision,
validateContract,
observation,
} from './protocol';
import { PickPlaceEvaluator, type PhysicalSample } from './PhysicalEvidence';
import { PICK_PLACE as P } from './PickPlaceScene';
const plan = () => ({
version: AGENT_VERSION,
objectId: 'block',
goalId: 'placement',
summary: '抓取、搬运和放下',
steps: SKILLS.map((skill) => ({ skill, precondition: PRECONDITIONS[skill], onFailure: 'stop' })),
});
it('shares skill vocabulary with the wire schema and validates a bounded plan without executing text', () => {
expect(schema.$defs.Plan.properties.steps.items.properties.skill.enum).toEqual(SKILLS);
const input = plan();
input.summary = 'rm -rf / 不会被作为命令执行';
expect(validatePlan(input).summary).toBe(input.summary);
});
it.each(['object', 'skill', 'loop', 'order', 'precondition', 'extra', 'version', 'long'])(
'rejects invalid planning: %s',
(fault) => {
const input = plan();
if (fault === 'object') input.objectId = 'unknown';
if (fault === 'skill') Object.assign(input.steps[0], { skill: 'teleport' });
if (fault === 'loop') input.steps.push(input.steps[0]);
if (fault === 'order') input.steps.reverse();
if (fault === 'precondition') input.steps[0].precondition = 'verified-grasp';
if (fault === 'extra') Object.assign(input, { command: 'execute' });
if (fault === 'version') Object.assign(input, { version: 'other' });
if (fault === 'long') input.summary = 'x'.repeat(513);
expect(() => validatePlan(input)).toThrow();
},
);
it('accepts only the local remaining sequence on replan, never skipping verification', () => {
const input = plan();
input.steps = input.steps.slice(4);
expect(validatePlan(input, SKILLS.slice(4)).steps).toHaveLength(7);
input.steps.shift();
expect(() => validatePlan(input, SKILLS.slice(4))).toThrow();
});
it('Jev classification remains advisory; unknown or out-of-context candidates are rejected', () => {
const value = {
version: AGENT_VERSION,
choice: 'carry',
grasp: 'secure',
diagnosis: 'none',
recovery: 'continue',
};
expect(validateJevDecision(value, ['carry', 'stop']).grasp).toBe('secure');
expect(() => validateJevDecision(value, ['close', 'stop'])).toThrow(/候选/);
expect(() => validateJevDecision({ ...value, recovery: 'shell' }, ['carry', 'stop'])).toThrow();
});
it('named ground-truth observations reject nonfinite values, unknown fields/units and invalid identities', () => {
const sample: PhysicalSample = {
time: 0,
base: [0, 0, 0.05],
baseSpeed: 0,
yaw: 0,
tilt: 0,
joints: [0, 0, 0, 0, 0],
opening: 1,
tcp: [0.2, 0, 0.2],
object: [...P.object.position],
objectSpeed: 0,
peakJointSpeed: 0,
fingerForces: [0, 0],
supported: true,
onGoalSupport: false,
};
const value = observation(
sample,
new PickPlaceEvaluator(P.object.position, P.goal.position),
P.goal.position,
{ runId: 'x', requestId: 'r', sceneRevision: 1, sequence: 0, planRevision: 0 },
'open',
);
expect(value.source).toBe('mujoco-ground-truth');
for (const invalid of [
{ ...value, time: NaN },
{ ...value, units: 'mm' },
{ ...value, command: 'x' },
{ ...value, stamp: { ...value.stamp, sequence: -1 } },
{ ...value, tcp: [3, 0, 0] },
])
expect(() => validateContract('Observation', invalid)).toThrow();
});
+196
View File
@@ -0,0 +1,196 @@
import schema from '../../../../contracts/lekiwi-agent-v1.schema.json';
import type { BaselinePhase } from './DeterministicBaseline';
import type { PhysicalSample, PickPlaceEvaluator } from './PhysicalEvidence';
import type { RequestStamp } from './RequestGate';
export const AGENT_VERSION = 'lekiwi-agent-v1' as const;
export type Skill = BaselinePhase;
export const PRECONDITIONS = {
open: 'scene-ready',
pregrasp: 'base-stopped',
descend: 'tcp-above',
close: 'aligned',
verify: 'dual-contact',
carry: 'verified-grasp',
'stop-base': 'verified-grasp',
place: 'transported',
release: 'supported',
retreat: 'released',
settle: 'retreat',
} as const satisfies Record<Skill, string>;
export const SKILLS = Object.keys(PRECONDITIONS) as Skill[];
export interface Plan {
version: typeof AGENT_VERSION;
objectId: 'block';
goalId: 'placement';
summary: string;
steps: { skill: Skill; precondition: string; onFailure: 'retry' | 'replan' | 'stop' }[];
}
export interface JevDecision {
version: typeof AGENT_VERSION;
choice: Skill | 'stop';
grasp: 'secure' | 'empty' | 'slipping' | 'uncertain';
diagnosis: 'none' | 'empty' | 'slipping' | 'misaligned' | 'unreachable' | 'stalled' | 'uncertain';
recovery: 'continue' | 'retry' | 'replan' | 'stop';
}
export interface Observation {
version: typeof AGENT_VERSION;
stamp: RequestStamp;
source: 'mujoco-ground-truth';
units: 'SI';
frame: 'world-z-up';
time: number;
phase: Skill;
base: { position: number[]; yaw: number };
joints: number[];
opening: number;
tcp: number[];
object: { id: 'block'; position: number[]; speed: number };
goal: { id: 'placement'; position: number[] };
evidence: {
fingerForces: number[];
supported: boolean;
onGoalSupport: boolean;
secure: boolean;
transported: number;
};
safety: string[];
}
export interface SkillResult {
version: typeof AGENT_VERSION;
skill: Skill;
status: 'running' | 'completed' | 'failed';
code: string;
evidenceSequence: number;
}
type Node = {
$ref?: string;
type?: string;
const?: unknown;
enum?: unknown[];
properties?: Record<string, Node>;
required?: string[];
additionalProperties?: boolean;
items?: Node;
minItems?: number;
maxItems?: number;
uniqueItems?: boolean;
minimum?: number;
maximum?: number;
minLength?: number;
maxLength?: number;
};
type ContractName = keyof typeof schema.$defs;
const definitions = schema.$defs as unknown as Record<ContractName, Node>;
/** The intentionally small JSON Schema subset used by this version; fail closed
* on data, not an eval/template engine. Python validates this same schema. */
function check(node: Node, value: unknown, path: string): void {
const fail = () => {
throw new Error(`${path}: 智能任务契约不匹配`);
};
if (node.$ref) {
const name = node.$ref.replace('#/$defs/', '') as ContractName;
if (!Object.hasOwn(definitions, name)) fail();
return check(definitions[name], value, path);
}
if (Object.hasOwn(node, 'const') && value !== node.const) fail();
if (node.enum && !node.enum.includes(value)) fail();
if (node.type === 'number' || node.type === 'integer') {
if (
typeof value !== 'number' ||
!Number.isFinite(value) ||
(node.type === 'integer' && !Number.isInteger(value))
)
fail();
if (
(value as number) < (node.minimum ?? -Infinity) ||
(value as number) > (node.maximum ?? Infinity)
)
fail();
} else if (node.type === 'string') {
if (typeof value !== 'string') fail();
const length = Array.from(value as string).length;
if (length < (node.minLength ?? 0) || length > (node.maxLength ?? Infinity)) fail();
} else if (node.type === 'boolean') {
if (typeof value !== 'boolean') fail();
} else if (node.type === 'array') {
if (!Array.isArray(value)) fail();
const values = value as unknown[];
if (values.length < (node.minItems ?? 0) || values.length > (node.maxItems ?? Infinity)) fail();
if (node.uniqueItems && new Set(values.map((v) => JSON.stringify(v))).size !== values.length)
fail();
if (node.items) values.forEach((v, i) => check(node.items!, v, `${path}[${i}]`));
} else if (node.type === 'object') {
if (value === null || typeof value !== 'object' || Array.isArray(value)) fail();
const obj = value as Record<string, unknown>;
for (const key of node.required ?? []) if (!Object.hasOwn(obj, key)) fail();
for (const key of Object.keys(obj)) {
if (!Object.hasOwn(node.properties ?? {}, key)) {
if (node.additionalProperties === false) fail();
} else check(node.properties![key], obj[key], `${path}.${key}`);
}
}
}
export function validateContract<T>(name: ContractName, value: unknown): T {
if ((JSON.stringify(value)?.length ?? 0) > 65536) throw new Error('智能任务消息超长');
check(definitions[name], value, name);
return structuredClone(value) as T;
}
export function validatePlan(value: unknown, remaining: readonly Skill[] = SKILLS): Plan {
const plan = validateContract<Plan>('Plan', value);
if (
plan.steps.length !== remaining.length ||
plan.steps.some(
(step, i) => step.skill !== remaining[i] || step.precondition !== PRECONDITIONS[step.skill],
)
)
throw new Error('计划缺少物理前置阶段、顺序非法或存在循环');
return plan;
}
export function validateJevDecision(
value: unknown,
candidates: readonly (Skill | 'stop')[],
): JevDecision {
const decision = validateContract<JevDecision>('JevDecision', value);
if (!candidates.includes(decision.choice)) throw new Error('Jev 选择不在当前候选中');
return decision;
}
export function observation(
sample: PhysicalSample,
evaluator: PickPlaceEvaluator,
goal: readonly number[],
stamp: RequestStamp,
phase: Skill,
safety: string[] = [],
): Observation {
return validateContract<Observation>('Observation', {
version: AGENT_VERSION,
stamp,
source: 'mujoco-ground-truth',
units: 'SI',
frame: 'world-z-up',
time: sample.time,
phase,
base: { position: sample.base, yaw: sample.yaw },
joints: sample.joints,
opening: sample.opening,
tcp: sample.tcp,
object: { id: 'block', position: sample.object, speed: sample.objectSpeed },
goal: { id: 'placement', position: [...goal] },
evidence: {
fingerForces: sample.fingerForces,
supported: sample.supported,
onGoalSupport: sample.onGoalSupport,
secure: evaluator.secure,
transported: evaluator.transported,
},
safety,
});
}
export const PLAN_OUTPUT_SCHEMA = schema.$defs.Plan;
export function skillErrorCode(error: string): string {
return (schema.$defs.SkillResult.properties.code.enum as readonly string[]).includes(error)
? error
: 'controller_error';
}
+87
View File
@@ -0,0 +1,87 @@
import type { ProjectManifest } from '../project/types';
import type { MainThreadPhysicsAdapter } from '../simulation/PhysicsAdapter';
import { MOBILE_ROBOTS, MOBILE_TASK, type RobotConfig } from './RobotDescriptor';
import type { TrainingStage } from './TaskKernel';
import type { MobilePolicyMetadata } from './ONNXPolicyRunner';
export const MOBILE_TRAINING_TASKS: Record<string, string> = {
'MobileManipulator-LeKiwi-v1': 'lekiwi-v1',
'MobileManipulator-LeKiwi-Bundle': 'lekiwi-bundle',
};
export const isMobileTrainingTask = (id: string) => Object.hasOwn(MOBILE_TRAINING_TASKS, id);
export function mobileTrainingRobot(taskId: string): RobotConfig {
const robot = MOBILE_ROBOTS.find((r) => r.id === MOBILE_TRAINING_TASKS[taskId]);
if (!robot) throw new Error('未知移动操作任务');
return robot;
}
export interface MobileTrainingParams {
rolloutSteps: number;
stage?: TrainingStage;
sourceJobId?: string;
positionJitter?: number;
evaluationEpisodes?: number;
navigationBootstrapSteps?: number;
objectPosition: number[];
goalPosition: number[];
}
export interface MobileDeployment extends MobilePolicyMetadata {
version: 1;
evaluation?: {
episodes: number;
successRate: number;
safetyStops: number;
maxJointVelocity: number;
meanNavigationDistance: number;
seed: number;
};
trainedTimesteps?: number;
trainingParams?: MobileTrainingParams;
initialization?: string;
browserCompatible: boolean;
trainingTaskId: string;
sceneSha256: string;
resetOptions: { object: number[]; goal: number[] };
input: { name: string; dtype: string; shape: number[] };
output: { name: string; dtype: string; shape: number[] };
terrain?: never;
}
export interface MobileTrainingBridge {
robotId?: string;
prepare(taskId: string): Promise<File>;
importPolicy(file: File, deployment: MobileDeployment): Promise<void>;
}
/** Internal transport only: the user never downloads or unpacks a training ZIP. */
export async function mobileSceneSnapshot(
adapter: MainThreadPhysicsAdapter,
project: ProjectManifest,
entryPath: string,
config: RobotConfig,
): Promise<File> {
const session = adapter.session;
if (!session) throw new Error('请先加载机器人');
const { zipSync } = await import('fflate');
if (session !== adapter.session) throw new Error('模型已切换,场景同步取消');
const scene = `${entryPath}.training.xml`;
const files: Record<string, Uint8Array> = {};
for (const f of [...project.files, ...adapter.cachedSupportFiles()]) files[f.path] = f.data;
const encoder = new TextEncoder();
files[scene] = adapter.exportMjcf();
files['robot.json'] = encoder.encode(JSON.stringify(config));
files['task.json'] = encoder.encode(JSON.stringify(MOBILE_TASK));
files['environment.json'] = encoder.encode(
JSON.stringify({
scene,
mujoco: session.module.mj_versionString(),
taskId: MOBILE_TASK.id,
robotId: config.id,
}),
);
return new File(
[zipSync(files, { level: 1, mtime: new Date(1980, 0, 1) }).slice().buffer],
'scene.zip',
{
type: 'application/zip',
},
);
}
+1 -1
View File
@@ -98,7 +98,7 @@ export function downloadBytes(
setTimeout(() => URL.revokeObjectURL(url), 0);
}
export function exportedFileName(projectName: string, extension: 'urdf' | 'xml'): string {
export function exportedFileName(projectName: string, extension: 'urdf' | 'xml' | 'zip'): string {
const stem =
projectName.replace(/\.(?:zip|xml|urdf)$/i, '').replace(/[^\p{L}\p{N}._-]+/gu, '_') || 'model';
return `${stem}.${extension}`;
@@ -0,0 +1,31 @@
import { unzipSync } from 'fflate';
import { compiledProjectPackage } from './exportPackage';
import type { ProjectManifest } from './types';
it('exports original and generated resources next to a unique compiled entry', () => {
const data = new TextEncoder().encode('original');
const manifest: ProjectManifest = {
id: 'x',
name: 'x',
maps: [],
entries: [],
totalBytes: data.length,
files: [
{ path: 'robot/compiled-scene.xml', data, size: data.length, source: 'file', mimeType: '' },
],
};
const xml = new TextEncoder().encode('<mujoco/>');
const resource = {
...manifest.files[0],
path: 'robot/full.obj',
data: new Uint8Array([1, 2, 3]),
size: 3,
};
const result = compiledProjectPackage(manifest, 'robot/original.urdf', xml, [resource]);
expect(result.entryPath).toBe('robot/compiled-scene-2.xml');
const files = unzipSync(result.data);
expect(files['robot/full.obj']).toEqual(resource.data);
expect(Array.from(files['robot/compiled-scene.xml'])).toEqual(Array.from(data));
expect(Array.from(files[result.entryPath])).toEqual(Array.from(xml));
expect(() => compiledProjectPackage(manifest, '../evil.xml', xml, [])).toThrow(/越界/);
});
+23
View File
@@ -0,0 +1,23 @@
import { zipSync } from 'fflate';
import { normalizeProjectPath } from './importer';
import type { ProjectFile, ProjectManifest } from './types';
/** Portable companion to XML-only export: preserve original sources/provenance
* and include every derived mesh, without overwriting any user's entry file. */
export function compiledProjectPackage(
manifest: ProjectManifest,
sourceEntry: string,
xml: Uint8Array,
supportFiles: readonly ProjectFile[],
): { data: Uint8Array; entryPath: string } {
const files: Record<string, Uint8Array> = Object.create(null);
for (const file of [...manifest.files, ...supportFiles])
files[normalizeProjectPath(file.path)] = file.data;
const entry = normalizeProjectPath(sourceEntry);
const directory = entry.slice(0, entry.lastIndexOf('/') + 1);
let entryPath = `${directory}compiled-scene.xml`;
for (let i = 2; Object.hasOwn(files, entryPath); i++)
entryPath = `${directory}compiled-scene-${i}.xml`;
files[entryPath] = xml;
return { data: zipSync(files, { level: 1, mtime: new Date(1980, 0, 1) }), entryPath };
}
+58
View File
@@ -158,6 +158,64 @@ describe('project importer', () => {
expect(text).toContain('fusestatic="false"');
expect(prepared.warnings).toHaveLength(2);
});
it.each(['package://机械臂/meshes/Link1.STL', 'meshes/package://机械臂/meshes/Link1.STL'])(
'修复重新打包的 ROS URI,保留原始工程:%s',
async (filename) => {
const urdf = projectFile(
'bundle/robot.urdf',
`<robot><link name="Link1"><collision><geometry><mesh filename="${filename}"/></geometry></collision></link></robot>`,
);
const original = new TextDecoder().decode(urdf.data);
const manifest = {
id: 'bundle',
name: 'bundle',
maps: [],
totalBytes: urdf.size,
entries: [{ path: urdf.path, format: 'urdf' as const, label: 'robot' }],
files: [urdf, projectFile('bundle/meshes/Link1.STL', 'mesh')],
};
const result = await prepareProjectForMujoco(manifest, urdf.path);
expect(new TextDecoder().decode(result.manifest.files[0].data)).toContain(
'filename="meshes/Link1.STL"',
);
expect(new TextDecoder().decode(result.manifest.files[0].data)).toContain(
'strippath="false"',
);
expect(new TextDecoder().decode(urdf.data)).toBe(original);
expect(result.warnings.join(' ')).toContain('1 个 package://');
},
);
it('不按同名 STL 猜测资源,不自动选择歧义包,也不放行路径穿越', async () => {
const prepare = (filename: string, paths: string[]) => {
const urdf = projectFile(
'robot.urdf',
`<robot><link><visual><geometry><mesh filename="${filename}"/></geometry></visual></link></robot>`,
);
return prepareProjectForMujoco(
{
id: 'x',
name: 'x',
maps: [],
totalBytes: urdf.size,
entries: [{ path: urdf.path, format: 'urdf', label: 'x' }],
files: [urdf, ...paths.map((path) => projectFile(path, 'mesh'))],
},
urdf.path,
);
};
await expect(
prepare('package://arm/meshes/a.stl', ['a/arm/meshes/a.stl', 'b/arm/meshes/a.stl']),
).rejects.toThrow(/不唯一/);
await expect(
prepare('meshes/package://arm/meshes/a.stl', ['a/meshes/a.stl', 'b/meshes/a.stl']),
).rejects.toThrow(/不唯一/);
await expect(prepare('meshes/package://arm/../a.stl', ['a.stl'])).rejects.toThrow(/越界/);
const missing = await prepare('meshes/package://arm/meshes/a.stl', ['other/a.stl']);
expect(missing.warnings.join(' ')).toContain('未在工程中找到');
expect(new TextDecoder().decode(missing.manifest.files[0].data)).toContain(
'meshes/package://arm/meshes/a.stl',
);
});
it('将 DAE mesh 转换为 MuJoCo 可读取的 OBJ,并复用于 visual/collision', async () => {
const urdf = projectFile(
'robot/robot.urdf',
+21 -7
View File
@@ -154,6 +154,7 @@ export async function prepareProjectForMujoco(
}
compiler.setAttribute('discardvisual', 'false');
compiler.setAttribute('fusestatic', 'false');
compiler.setAttribute('strippath', 'false');
let removedMaterials = 0;
for (const visual of Array.from(document.querySelectorAll('visual'))) {
@@ -173,16 +174,29 @@ export async function prepareProjectForMujoco(
const unresolved: string[] = [];
for (const element of Array.from(document.querySelectorAll('[filename]'))) {
const value = element.getAttribute('filename');
if (!value?.startsWith('package://')) continue;
const packagePath = normalizeProjectPath(value.slice('package://'.length));
const target =
paths.find((path) => path === packagePath) ??
paths.find((path) => path.endsWith(`/${packagePath}`));
if (!target) {
// Some CAD/URDF editors bundle a ROS resource but accidentally prepend
// "meshes/" to its package URI. Only this observed spelling is repaired;
// do not guess by basename or resolve URLs/files outside the imported project.
if (!value || !/^(?:meshes\/)?package:\/\//.test(value)) continue;
const packagePath = normalizeProjectPath(value.replace(/^(?:meshes\/)?package:\/\//, ''));
let targets = paths.filter((path) => path === packagePath || path.endsWith(`/${packagePath}`));
if (!targets.length) {
const slash = packagePath.indexOf('/');
const bundledPath = slash < 0 ? '' : packagePath.slice(slash + 1);
if (bundledPath) {
const local = resolveProjectReference(entryPath, bundledPath);
targets = paths.includes(local ?? '')
? [local!]
: paths.filter((path) => path === bundledPath || path.endsWith(`/${bundledPath}`));
}
}
if (targets.length > 1)
throw new ProjectImportError(`package:// 资源路径不唯一,不能自动选择:${value}`, entryPath);
if (!targets.length) {
unresolved.push(value);
continue;
}
element.setAttribute('filename', relativeProjectPath(entryPath, target));
element.setAttribute('filename', relativeProjectPath(entryPath, targets[0]));
rewrittenUris += 1;
}
if (rewrittenUris)
@@ -0,0 +1,146 @@
import { webcrypto } from 'node:crypto';
import { prepareLargeMeshes } from './meshCompatibility';
import { binaryStl } from '../test/stlFixture';
import type { ProjectManifest } from './types';
const encoder = new TextEncoder(),
decoder = new TextDecoder();
function project(
entries: Record<string, string | Uint8Array>,
entry = 'robot/main.xml',
): ProjectManifest {
const files = Object.entries(entries).map(([path, value]) => {
const data = typeof value === 'string' ? encoder.encode(value) : value;
return { path, data, size: data.length, source: 'file' as const, mimeType: '' };
});
return {
id: 'test',
name: 'test',
files,
entries: [{ path: entry, format: entry.endsWith('urdf') ? 'urdf' : 'mjcf', label: 'test' }],
maps: [],
totalBytes: files.reduce((sum, f) => sum + f.size, 0),
};
}
beforeEach(() => vi.stubGlobal('crypto', webcrypto));
afterEach(() => vi.unstubAllGlobals());
it('converts shared URDF resources once, preserving scale, origin and input bytes', async () => {
const source =
'<robot><link name="x"><visual><origin xyz="1 2 3"/><geometry><mesh filename="meshes/w.stl" scale=".001 .001 .001"/></geometry></visual><collision><geometry><mesh filename="meshes/w.stl"/></geometry></collision></link></robot>';
const p = project(
{ 'robot/main.urdf': source, 'robot/meshes/w.stl': binaryStl(200001) },
'robot/main.urdf',
);
const first = await prepareLargeMeshes(p, 'robot/main.urdf');
expect(first.manifest.files.filter((f) => f.path.endsWith('.obj'))).toHaveLength(1);
const xml = decoder.decode(first.manifest.files[0].data);
expect(xml).toContain('xyz="1 2 3"');
expect(xml).toContain('scale=".001 .001 .001"');
expect(xml.match(/filename="stl-obj-v1-/g)).toHaveLength(2);
expect(decoder.decode(p.files[0].data)).toBe(source);
expect(first.warnings).toHaveLength(1);
const again = await prepareLargeMeshes(p, 'robot/main.urdf');
expect(again.manifest.files.at(-1)!.data).toBe(first.manifest.files.at(-1)!.data);
const cooked = await prepareLargeMeshes(first.manifest, 'robot/main.urdf');
expect(cooked.manifest).toBe(first.manifest);
});
it('honors meshdir/content_type and include fallback; does not touch unrelated models', async () => {
const p = project({
'robot/main.xml':
'<mujoco><compiler meshdir="meshes"/><include file="sub/assets.xml"/><worldbody/></mujoco>',
'robot/sub/assets.xml':
'<mujocoinclude><asset><mesh name="wheel" content_type="model/stl" file="wheel.stl" scale=".001 .001 .001"/></asset></mujocoinclude>',
'robot/sub/meshes/wheel.stl': binaryStl(200001),
'other.xml': '<mujoco><asset><mesh file="not-existing.stl"/></asset></mujoco>',
});
const result = await prepareLargeMeshes(p, 'robot/main.xml');
const generated = result.manifest.files.find((f) => f.path.endsWith('.obj'))!;
expect(generated.path).toMatch(/^robot\/meshes\/stl-obj-v1-/);
const xml = decoder.decode(
result.manifest.files.find((f) => f.path.endsWith('assets.xml'))!.data,
);
expect(xml).toContain('content_type="model/obj"');
expect(xml).toContain('scale=".001 .001 .001"');
expect(decoder.decode(result.manifest.files.find((f) => f.path === 'other.xml')!.data)).toContain(
'not-existing.stl',
);
});
it('uses main-directory asset before include-directory asset, as MuJoCo does', async () => {
const p = project({
'robot/main.xml': '<mujoco><include file="sub/assets.xml"/></mujoco>',
'robot/sub/assets.xml': '<mujocoinclude><asset><mesh file="w.stl"/></asset></mujocoinclude>',
'robot/w.stl': binaryStl(1),
'robot/sub/w.stl': binaryStl(200001),
});
const result = await prepareLargeMeshes(p, 'robot/main.xml');
expect(result.manifest).toBe(p);
});
it('honors assetdir/strippath and never overwrites a user-supplied derived filename', async () => {
const p = project({
'robot/main.xml':
'<mujoco><compiler assetdir="meshes" strippath="true"/><asset><mesh file="discarded/path/w.stl"/></asset></mujoco>',
'robot/meshes/w.stl': binaryStl(200001),
});
const first = await prepareLargeMeshes(p, 'robot/main.xml');
const generated = first.manifest.files.at(-1)!;
const fake = { ...generated, data: encoder.encode('user content'), size: 12 };
const result = await prepareLargeMeshes({ ...p, files: [...p.files, fake] }, 'robot/main.xml');
expect(decoder.decode(result.manifest.files.find((f) => f.path === fake.path)!.data)).toBe(
'user content',
);
expect(result.manifest.files.at(-1)!.path).toMatch(/-2\.obj$/);
});
it('new source bytes invalidate the conversion cache and preserve old resources', async () => {
const data = binaryStl(200001);
const p = project({
'robot/main.xml': '<mujoco><asset><mesh file="w.stl"/></asset></mujoco>',
'robot/w.stl': data,
});
const before = await prepareLargeMeshes(p, 'robot/main.xml');
const changed = data.slice();
new DataView(changed.buffer).setFloat32(96, 0.25, true);
const result = await prepareLargeMeshes(
project({ 'robot/main.xml': decoder.decode(p.files[0].data), 'robot/w.stl': changed }),
'robot/main.xml',
);
expect(result.manifest.files.at(-1)!.path).not.toBe(before.manifest.files.at(-1)!.path);
});
it.each(['../../escape.stl', 'https://evil.test/w.stl', '/host/w.stl'])(
'rejects unsafe paths: %s',
async (path) => {
await expect(
prepareLargeMeshes(
project({ 'robot/main.xml': `<mujoco><asset><mesh file="${path}"/></asset></mujoco>` }),
'robot/main.xml',
),
).rejects.toThrow(/路径/);
},
);
it('rejects cyclic includes and aborts cooperatively', async () => {
await expect(
prepareLargeMeshes(
project({ 'robot/main.xml': '<mujoco><include file="main.xml"/></mujoco>' }),
'robot/main.xml',
),
).rejects.toThrow(/循环/);
const p = project({
'robot/main.xml': '<mujoco><asset><mesh file="w.stl"/></asset></mujoco>',
'robot/w.stl': binaryStl(200001),
});
let cancelled = false;
await expect(
prepareLargeMeshes(p, 'robot/main.xml', {
shouldCancel: () => cancelled,
onMeshProgress: () => {
cancelled = true;
},
}),
).rejects.toThrow(/取消/);
expect(p.files).toHaveLength(2);
});
@@ -0,0 +1,166 @@
import { relativeAssetPath, resolveProjectAssetPath } from '../map/mapPaths';
import { sha256 } from '../robot/registry';
import { DEFAULT_IMPORT_LIMITS, type ProjectFile, type ProjectManifest } from './types';
import {
binaryStlFaceCount,
convertStlToObj,
MUJOCO_STL_FACE_LIMIT,
STL_CONVERTER_VERSION,
type MeshConversionOptions,
} from './stlConverter';
const decoder = new TextDecoder(),
encoder = new TextEncoder();
// Successful conversions only; weak values never pin meshes from a closed project.
// Content keys also share the three byte-identical LeKiwi wheel source files.
const cache = new Map<string, WeakRef<Uint8Array>>();
export interface MeshPreparationOptions extends MeshConversionOptions {
onMeshProgress?: (path: string, completed: number, total: number) => void;
}
/** Prepare only the selected entry/include graph. Source assets remain untouched.
* MuJoCo resolves assets in the main XML directory first, then the including file's
* directory. Do not use basename guessing: two same-named CAD parts can differ.
*/
export async function prepareLargeMeshes(
manifest: ProjectManifest,
entryPath: string,
options: MeshPreparationOptions = {},
): Promise<{ manifest: ProjectManifest; warnings: string[] }> {
const files = new Map(manifest.files.map((file) => [file.path, file]));
const documents = new Map<string, Document>();
const active = new Set<string>();
const compiler = new Map<string, string>();
const check = () => {
if (options.shouldCancel?.()) throw new Error('网格准备已取消');
};
const resolveExisting = (from: string, reference: string) => {
const main = resolveProjectAssetPath(entryPath, reference);
if (files.has(main)) return main;
return resolveProjectAssetPath(from, reference);
};
const visit = (path: string, depth = 0): void => {
check();
if (active.has(path) || depth > 32) throw new Error(`模型 include 循环或嵌套过深:${path}`);
if (documents.has(path)) return;
const file = files.get(path);
if (!file) throw new Error(`模型引用的 XML 不存在:${path}`);
const doc = new DOMParser().parseFromString(decoder.decode(file.data), 'application/xml');
if (doc.querySelector('parsererror') || doc.doctype) throw new Error(`模型 XML 无效:${path}`);
documents.set(path, doc);
active.add(path);
for (const node of doc.querySelectorAll('compiler, include[file]')) {
if (node.tagName === 'compiler') {
for (const attribute of Array.from(node.attributes))
compiler.set(attribute.name, attribute.value);
} else {
visit(resolveExisting(path, node.getAttribute('file')!), depth + 1);
}
}
active.delete(path);
};
visit(entryPath);
const meshdir = compiler.get('meshdir') ?? compiler.get('assetdir') ?? '';
const strip = compiler.get('strippath') === 'true';
const assetAnchor = resolveProjectAssetPath(
entryPath,
`${meshdir ? meshdir + '/' : ''}.__asset__`,
);
const warnings: string[] = [];
const generated = new Map<string, ProjectFile>();
const changed = new Set<string>();
let totalBytes = manifest.files.reduce((sum, file) => sum + file.data.byteLength, 0);
for (const [path, doc] of documents) {
const urdf = doc.documentElement.tagName === 'robot';
for (const mesh of doc.querySelectorAll(urdf ? 'mesh[filename]' : 'mesh[file]')) {
check();
const attr = urdf ? 'filename' : 'file';
let reference = mesh.getAttribute(attr)!;
const type = mesh.getAttribute('content_type');
if (type && type !== 'model/stl') continue;
if (type !== 'model/stl' && !/\.stl$/i.test(reference)) continue;
// Validate before strippath, too: imported resources never grant host access.
resolveProjectAssetPath(path, reference);
if (strip) reference = reference.replaceAll('\\', '/').split('/').at(-1)!;
const sourcePath = resolveExisting(path, `${meshdir ? meshdir + '/' : ''}${reference}`);
const source = files.get(sourcePath);
if (!source) throw new Error(`STL 网格不存在:${sourcePath}`);
const faces = binaryStlFaceCount(source.data);
if (faces <= MUJOCO_STL_FACE_LIMIT) continue;
let file = generated.get(sourcePath);
if (!file) {
const hash = await sha256(source.data);
check();
let output = cache.get(hash)?.deref();
if (!output) {
const result = await convertStlToObj(source.data, {
...options,
maxOutputBytes: Math.min(
options.maxOutputBytes ?? DEFAULT_IMPORT_LIMITS.maxFileBytes,
DEFAULT_IMPORT_LIMITS.maxFileBytes,
DEFAULT_IMPORT_LIMITS.maxTotalBytes - totalBytes,
),
onProgress: (done, count) => options.onMeshProgress?.(sourcePath, done, count),
});
output = result.data;
if (cache.size >= 64) cache.delete(cache.keys().next().value!);
cache.set(hash, new WeakRef(output));
}
check();
if (output.byteLength > (options.maxOutputBytes ?? DEFAULT_IMPORT_LIMITS.maxFileBytes))
throw new Error('派生 OBJ 超过文件预算');
options.onMeshProgress?.(sourcePath, faces, faces);
const converted = { data: output };
const stem = `${STL_CONVERTER_VERSION}-${hash}`;
let name = `${stem}.obj`,
suffix = 1;
let target = resolveProjectAssetPath(assetAnchor, name);
// Reuse only byte-identical generated data. Never trust a user's filename.
while (files.has(target) && !sameBytes(files.get(target)!.data, converted.data)) {
name = `${stem}-${++suffix}.obj`;
target = resolveProjectAssetPath(assetAnchor, name);
}
if (!files.has(target)) {
totalBytes += converted.data.byteLength;
if (
converted.data.byteLength > DEFAULT_IMPORT_LIMITS.maxFileBytes ||
totalBytes > DEFAULT_IMPORT_LIMITS.maxTotalBytes ||
files.size >= DEFAULT_IMPORT_LIMITS.maxFiles
)
throw new Error('高面数网格派生资源超过工程导入预算');
file = {
...source,
path: target,
data: converted.data,
size: converted.data.byteLength,
mimeType: 'text/plain',
};
files.set(target, file);
} else file = files.get(target)!;
generated.set(sourcePath, file);
warnings.push(
`${sourcePath}:${faces.toLocaleString()} 面已无减面转换为 OBJ;原始 STL 保留`,
);
}
// Generated data always lives under the main compiler's meshdir. Thus this
// reference is unambiguous even when the original was in a nested include.
mesh.setAttribute(attr, relativeAssetPath(assetAnchor, file.path));
if (type) mesh.setAttribute('content_type', 'model/obj');
changed.add(path);
}
}
check();
if (!changed.size) return { manifest, warnings };
for (const path of changed) {
const file = files.get(path)!;
const data = encoder.encode(new XMLSerializer().serializeToString(documents.get(path)!));
totalBytes += data.byteLength - file.data.byteLength;
files.set(path, { ...file, data, size: data.byteLength });
}
if (totalBytes > DEFAULT_IMPORT_LIMITS.maxTotalBytes) throw new Error('派生工程超过总大小预算');
return { manifest: { ...manifest, files: [...files.values()], totalBytes }, warnings };
}
function sameBytes(a: Uint8Array, b: Uint8Array): boolean {
return a === b || (a.length === b.length && a.every((value, i) => value === b[i]));
}
@@ -2,20 +2,26 @@ import type { ProjectManifest } from '../types';
import { LEKIWI_PROFILE } from '../../robot/profiles/lekiwi';
import { sha256, validateLeKiwiSource } from '../../robot/registry';
import { RobotError } from '../../robot/types';
import sourceMeshes from '../../../../robot_profiles/lekiwi-source-meshes.json';
import {
LEKIWI_VISUAL_REVISION,
LEKIWI_COLLISION_REVISION,
LEKIWI_COLLISION_RECIPE_HASH,
LEKIWI_COLLISION_SOURCE_HASHES,
prepareLeKiwiUrdf,
} from './lekiwi';
async function validateCollisionSources(manifest: ProjectManifest, entryPath: string) {
async function validateMeshSources(
manifest: ProjectManifest,
entryPath: string,
hashes: Record<string, string>,
) {
const directory = entryPath.slice(0, entryPath.lastIndexOf('/') + 1);
await Promise.all(
Object.entries(LEKIWI_COLLISION_SOURCE_HASHES).map(async ([path, expected]) => {
Object.entries(hashes).map(async ([path, expected]) => {
const file = manifest.files.find((candidate) => candidate.path === directory + path);
if (!file || (await sha256(file.data)) !== expected)
throw new RobotError('INCOMPATIBLE_MODEL', `碰撞体源网格缺失或 SHA-256 不匹配:${path}`);
throw new RobotError('INCOMPATIBLE_MODEL', `模型源网格缺失或 SHA-256 不匹配:${path}`);
}),
);
}
@@ -58,12 +64,24 @@ export async function prepareRobotProject(
'INCOMPATIBLE_MODEL',
'LeKiwi 碰撞配方已更新;请从原始 URDF 重新转换,旧 MJCF 不会自动升级',
);
if (
xml
.querySelector('custom text[name="platform_lekiwi_visual_revision"]')
?.getAttribute('data') !== String(LEKIWI_VISUAL_REVISION)
)
throw new RobotError(
'INCOMPATIBLE_MODEL',
'LeKiwi 完整轮视觉配方已更新;请从原始 URDF 重新转换,不能复用旧缺轮 MJCF',
);
// Never enhance twice. Compiled topology/actuators are checked by the backend.
await validateCollisionSources(manifest, entryPath);
await validateMeshSources(manifest, entryPath, LEKIWI_COLLISION_SOURCE_HASHES);
await validateMeshSources(manifest, entryPath, sourceMeshes);
return manifest;
}
await validateLeKiwiSource(source.data);
await validateCollisionSources(manifest, entryPath);
// Equivalent URDF text is not permission to silently replace its mechanical
// assets. The fixed recipe also simplifies the chassis and all three wheels.
await validateMeshSources(manifest, entryPath, sourceMeshes);
const data = prepareLeKiwiUrdf(source.data);
const files = manifest.files.map((f) =>
f === source ? { ...f, data, size: data.byteLength } : f,
@@ -1,6 +1,7 @@
import { webcrypto } from 'node:crypto';
import {
enhanceLeKiwiMjcf,
LEKIWI_VISUAL_REVISION,
LEKIWI_COLLISION_REVISION,
LEKIWI_COLLISION_RECIPE_HASH,
type LeKiwiJointGeometry,
@@ -33,8 +34,20 @@ function source() {
<inertial mass="1" pos="0 0 0" diaginertia="1 1 1"/>
<joint name="${p.arm[i].joint}" axis="${i === 5 ? '0 -0.906308 -0.422618' : '1 0 0'}"/>
${visuals(p.arm[i].joint)}${arm(i + 1)}</body>`;
geometry.push(
...p.wheels.map((name) => ({
name,
body: name,
position: [0, 0, 0],
wheelCenter: [0, 0, 0],
rotation: [1, 0, 0, 0, 1, 0, 0, 0, 1],
})),
);
const wheels = p.wheels
.map((name) => `<body name="${name}"><joint name="${name}"/></body>`)
.map(
(name) =>
`<body name="${name}"><joint name="${name}"/><geom name="${name}_visual" type="mesh" mesh="wheel" group="1"/></body>`,
)
.join('');
return {
data: encode(`<mujoco><asset/><worldbody><body name="${p.baseBody}">
@@ -152,6 +165,7 @@ function manifest(revision?: number, hash = LEKIWI_COLLISION_RECIPE_HASH): Proje
const data = encode(`<mujoco><custom>
<text name="platform_robot_profile" data="${p.id}"/>
<text name="platform_robot_profile_version" data="${p.version}"/>
<text name="platform_lekiwi_visual_revision" data="${LEKIWI_VISUAL_REVISION}"/>
<text name="platform_robot_source_sha256" data="${p.source.urdfSha256}"/>
<text name="platform_lekiwi_collision_recipe_sha256" data="${hash}"/>
${revision === undefined ? '' : `<text name="platform_lekiwi_collision_revision" data="${revision}"/>`}
@@ -5,16 +5,16 @@ import type { ProjectManifest } from '../types';
const encode = (xml: string) => new TextEncoder().encode(xml);
describe('explicit LeKiwi profile preparation', () => {
it('replaces only oversized wheels and preserves source bytes', () => {
it('preserves complete wheel visuals and source bytes', () => {
const xml =
'<robot><link name="wheel"><visual><geometry><mesh filename="meshes/4-Omni-Directional-Wheel_Single_Body-v1.stl"/></geometry></visual></link><link name="arm"><visual><geometry><mesh filename="meshes/arm.stl"/></geometry></visual></link></robot>';
const data = encode(xml),
result = new TextDecoder().decode(prepareLeKiwiUrdf(data));
expect(new TextDecoder().decode(data)).toBe(xml);
expect(result).toContain('strippath="false"');
expect(result).toContain('<sphere');
expect(result).not.toContain('<sphere');
expect(result).toContain('meshes/arm.stl');
expect(result).not.toContain('Omni-Directional');
expect(result).toContain('4-Omni-Directional-Wheel_Single_Body-v1.stl');
});
it('rejects invalid topology and repeated enhancement', () => {
expect(() => enhanceLeKiwiMjcf(encode('<mujoco><worldbody/></mujoco>'), [])).toThrow(/基座/);
@@ -43,7 +43,7 @@ describe('explicit LeKiwi profile preparation', () => {
/未注册/,
);
await expect(prepareRobotProject(manifest, 'LeKiwi.urdf', 'lekiwi-v1')).rejects.toThrow(
/版本不符/,
/控制拓扑不匹配/,
);
} finally {
vi.unstubAllGlobals();
@@ -1,7 +1,9 @@
import { LEKIWI_PROFILE as p, WHEEL_TANGENTS } from '../../robot/profiles/lekiwi';
import { RobotError } from '../../robot/types';
import collision from '../../../../robot_profiles/lekiwi-full-collision.json';
import { Matrix4, Quaternion, Vector3 } from 'three';
export const LEKIWI_VISUAL_REVISION = 1;
export const LEKIWI_COLLISION_REVISION = collision.revision;
export const LEKIWI_COLLISION_RECIPE_HASH = collision.recipeSha256;
export const LEKIWI_COLLISION_SOURCE_HASHES = collision.source.meshes;
@@ -13,6 +15,8 @@ export interface LeKiwiJointGeometry {
body: string;
position: number[];
rotation: number[];
/** World center of the CAD wheel visual, from MuJoCo's compiled mesh frame. */
wheelCenter?: number[];
}
function parse(data: Uint8Array): Document {
const doc = new DOMParser().parseFromString(decoder.decode(data), 'application/xml');
@@ -28,9 +32,9 @@ function element(doc: Document, tag: string, attributes: Record<string, string |
for (const [name, value] of Object.entries(attributes)) node.setAttribute(name, String(value));
return node;
}
/** The CAD wheel STL has >300k faces, above MuJoCo's STL limit. This explicit,
* source-hash-gated preprocessing removes only the oversized wheel visual/collision.
* The original source files remain intact; the simulation recipe replaces wheels.
/** Preserve full CAD visuals. The generic mesh compatibility pass converts large
* STL references to lossless OBJ after source validation. Only temporary wheel
* collision is simplified; the final recipe still installs passive rollers.
*/
export function prepareLeKiwiUrdf(data: Uint8Array): Uint8Array {
const doc = parse(data),
@@ -46,7 +50,7 @@ export function prepareLeKiwiUrdf(data: Uint8Array): Uint8Array {
mujoco.append(compiler);
}
compiler.setAttribute('strippath', 'false');
for (const mesh of doc.querySelectorAll('mesh[filename]')) {
for (const mesh of doc.querySelectorAll('collision mesh[filename]')) {
if (mesh.getAttribute('filename')!.includes('4-Omni-Directional-Wheel_Single_Body'))
mesh.replaceWith(element(doc, 'sphere', { radius: p.wheelRadius }));
}
@@ -143,6 +147,7 @@ function addArmCollisions(doc: Document, cad: Element, contact: Element): void {
export function enhanceLeKiwiMjcf(
data: Uint8Array,
geometry: readonly LeKiwiJointGeometry[],
inputSha256 = p.source.urdfSha256,
): Uint8Array {
const doc = parse(data),
root = doc.documentElement;
@@ -176,7 +181,17 @@ export function enhanceLeKiwiMjcf(
);
for (const inertial of cad.querySelectorAll('inertial')) inertial.remove();
for (const joint of cad.querySelectorAll(':scope > freejoint, :scope > joint')) joint.remove();
for (const wheel of p.wheels) joints.get(wheel)!.parentElement!.remove();
const wheelVisuals = p.wheels.map((name) => {
const joint = joints.get(name)!;
const body = joint.parentElement!;
const frame = geometry.find((g) => g.name === name && g.body === body.getAttribute('name'));
if (!frame?.wheelCenter || !body.querySelector('geom[group="1"][type="mesh"]'))
throw new RobotError('INCOMPATIBLE_MODEL', `缺少完整轮视觉/运动学:${name}`);
const axis = (joint.getAttribute('axis') ?? '0 0 1').split(/\s+/).map(Number);
joint.remove();
body.remove();
return { body, frame, axis };
});
root.querySelector('actuator')?.remove();
const actuators = element(doc, 'actuator', {});
root.append(actuators);
@@ -307,6 +322,36 @@ export function enhanceLeKiwiMjcf(
diaginertia: '.00004 .00004 .00004',
}),
);
// Map the original moving CAD subtree to the ideal wheel's axle. Keep every
// visual vertex/scale and fixed child transform; change neither inertia nor
// collision. The wheel center (not motor-mount origin) sits on the new axle.
const { body: visual, frame, axis } = wheelVisuals[i];
const r = frame.rotation;
const sourceRotation = new Quaternion().setFromRotationMatrix(
new Matrix4().set(r[0], r[1], r[2], 0, r[3], r[4], r[5], 0, r[6], r[7], r[8], 0, 0, 0, 0, 1),
);
const canonicalRotation = new Quaternion().setFromAxisAngle(
new Vector3(0, 0, 1),
p.mapping.cadYawRad,
);
const rotation = canonicalRotation.multiply(sourceRotation);
const sourceAxis = new Vector3(...(axis as [number, number, number]))
.normalize()
.applyQuaternion(rotation);
rotation.premultiply(
new Quaternion().setFromUnitVectors(
sourceAxis,
new Vector3(...(axle as [number, number, number])).normalize(),
),
);
const center = new Vector3(...(frame.wheelCenter! as [number, number, number])).sub(
new Vector3(...(frame.position as [number, number, number])),
);
// SourceRotation was not mutated by canonicalRotation.multiply().
center.applyQuaternion(sourceRotation.clone().invert()).applyQuaternion(rotation).negate();
visual.setAttribute('pos', center.toArray().join(' '));
visual.setAttribute('quat', [rotation.w, rotation.x, rotation.y, rotation.z].join(' '));
wheel.append(visual);
wheel.append(
element(doc, 'geom', {
name: `__lekiwi_${name}_hub`,
@@ -315,7 +360,8 @@ export function enhanceLeKiwiMjcf(
zaxis: axle.join(' '),
contype: 0,
conaffinity: 0,
group: 1,
// Diagnostic proxy only; full CAD supplies ordinary wheel rendering.
group: 3,
mass: 0,
rgba: '.12 .15 .18 1',
}),
@@ -346,7 +392,7 @@ export function enhanceLeKiwiMjcf(
size: '.009 .006',
zaxis: axis.join(' '),
mass: p.simulation.rollerMass,
group: 0,
group: 3,
contype: 4,
conaffinity: 1,
friction: '1 .001 .0001',
@@ -374,6 +420,12 @@ export function enhanceLeKiwiMjcf(
const custom = root.querySelector('custom') ?? element(doc, 'custom', {});
if (!custom.parentElement) root.append(custom);
custom.append(element(doc, 'text', { name: 'platform_robot_profile', data: p.id }));
custom.append(
element(doc, 'text', {
name: 'platform_lekiwi_visual_revision',
data: LEKIWI_VISUAL_REVISION,
}),
);
custom.append(
element(doc, 'text', {
name: 'platform_lekiwi_collision_revision',
@@ -389,8 +441,10 @@ export function enhanceLeKiwiMjcf(
custom.append(
element(doc, 'text', { name: 'platform_robot_profile_version', data: String(p.version) }),
);
// Keep the reviewed recipe's source separate from the actual imported bytes.
custom.append(
element(doc, 'text', { name: 'platform_robot_source_sha256', data: p.source.urdfSha256 }),
element(doc, 'text', { name: 'platform_robot_input_sha256', data: inputSha256 }),
);
return serialize(doc);
}
@@ -0,0 +1,53 @@
import { binaryStlFaceCount, convertStlToObj, MUJOCO_STL_FACE_LIMIT } from './stlConverter';
import { binaryStl } from '../test/stlFixture';
it.each([200_000, 200_001])('preserves all %i faces and original bytes', async (count) => {
const input = binaryStl(count);
const before = input.slice();
const result = await convertStlToObj(input);
const text = new TextDecoder().decode(result.data);
expect(binaryStlFaceCount(input)).toBe(count);
expect(result.faces).toBe(count);
expect(result.vertices).toBe(3);
expect(text.match(/^f /gm)).toHaveLength(count);
expect(text.split('\n').slice(0, 4)).toEqual(['v 0 0 0', 'v 1 0 0', 'v 0 1 0', 'f 1 2 3']);
expect(Buffer.from(input).equals(Buffer.from(before))).toBe(true);
expect(MUJOCO_STL_FACE_LIMIT).toBe(200_000);
});
it('does not quantize almost-identical float32 vertices or reverse winding', async () => {
const data = binaryStl(2);
const v = new DataView(data.buffer);
v.setFloat32(84 + 50 + 12 + 12, 1.0000001192092896, true);
const result = await convertStlToObj(data);
expect(result.vertices).toBe(4);
expect(new TextDecoder().decode(result.data)).toContain('v 1.0000001192092896 0 0');
expect(new TextDecoder().decode(result.data)).toContain('f 1 4 3');
});
it('rejects truncated/invalid declarations, non-finite vertices and bounded budgets', async () => {
expect(() => binaryStlFaceCount(new Uint8Array(80))).toThrow(/头/);
expect(() => binaryStlFaceCount(binaryStl(1).slice(0, -1))).toThrow(/长度/);
expect(() => binaryStlFaceCount(binaryStl(0))).toThrow(/面数/);
const bad = binaryStl(1);
new DataView(bad.buffer).setFloat32(96, NaN, true);
await expect(convertStlToObj(bad)).rejects.toThrow(/非法/);
await expect(convertStlToObj(binaryStl(1), { maxOutputBytes: 10 })).rejects.toThrow(/预算/);
await expect(convertStlToObj(binaryStl(1), { maxWorkingBytes: 10 })).rejects.toThrow(/预算/);
});
it('supports byte offsets and cooperatively cancels without returning partial data', async () => {
const data = binaryStl(5000);
const buffer = new Uint8Array(data.length + 100);
buffer.set(data, 50);
let cancelled = false;
await expect(
convertStlToObj(buffer.subarray(50, 50 + data.length), {
shouldCancel: () => cancelled,
onProgress: () => {
cancelled = true;
},
}),
).rejects.toThrow(/取消/);
});
+87
View File
@@ -0,0 +1,87 @@
/** MuJoCo 3.11's STL decoder has a format-specific 200,000 face limit.
* Convert geometry, not fidelity: exact float32 vertices, original winding/all faces.
* No quantization, welding tolerance, decimation, filesystem or network access.
*/
export const MUJOCO_STL_FACE_LIMIT = 200_000;
export const STL_CONVERTER_VERSION = 'stl-obj-v1';
const encoder = new TextEncoder();
const BATCH_FACES = 4096;
export interface MeshConversionOptions {
maxOutputBytes?: number;
maxWorkingBytes?: number;
shouldCancel?: () => boolean;
onProgress?: (completed: number, total: number) => void;
}
export function binaryStlFaceCount(data: Uint8Array): number {
if (data.byteLength < 84) throw new Error('STL 二进制头不完整(不支持 ASCII STL)');
const count = new DataView(data.buffer, data.byteOffset, data.byteLength).getUint32(80, true);
if (!count || 84 + count * 50 !== data.byteLength)
throw new Error('STL 声明面数与文件长度不符(需要有效二进制 STL)');
return count;
}
export async function convertStlToObj(
data: Uint8Array,
options: MeshConversionOptions = {},
): Promise<{ data: Uint8Array; faces: number; vertices: number }> {
const faces = binaryStlFaceCount(data);
// Bound worst-case JS map/string/typed-array intermediates as well as output.
// This is an admission estimate, not a promise about a browser's heap use.
if (faces * 600 + data.byteLength > (options.maxWorkingBytes ?? 512 * 1024 * 1024))
throw new Error('STL 转换峰值内存估算超过预算,请拆分网格或显式降低视觉精度');
const limit = options.maxOutputBytes ?? 128 * 1024 * 1024;
const view = new DataView(data.buffer, data.byteOffset, data.byteLength);
const seen = new Map<string, number>();
const vertexChunks: Uint8Array[] = [],
faceChunks: Uint8Array[] = [];
let bytes = 0;
const append = (target: Uint8Array[], lines: string[]) => {
const chunk = encoder.encode(lines.join('\n') + '\n');
bytes += chunk.length;
if (bytes > limit) throw new Error('派生 OBJ 超过文件或工程总大小预算');
target.push(chunk);
};
const check = () => {
if (options.shouldCancel?.()) throw new Error('网格转换已取消');
};
check();
for (let first = 0; first < faces; first += BATCH_FACES) {
check();
const vertices: string[] = [],
triangles: string[] = [];
const end = Math.min(faces, first + BATCH_FACES);
for (let i = first; i < end; i++) {
const ids: number[] = [];
for (let j = 0; j < 3; j++) {
const offset = 84 + 50 * i + 12 + j * 12;
const xyz = [0, 4, 8].map((k) => view.getFloat32(offset + k, true));
if (xyz.some((value) => !Number.isFinite(value) || Math.abs(value) > 2 ** 30))
throw new Error(`STL 第 ${i + 1} 面含非法/超范围顶点`);
const key = [0, 4, 8].map((k) => view.getUint32(offset + k, true)).join(',');
let id = seen.get(key);
if (id === undefined) {
id = seen.size + 1;
seen.set(key, id);
vertices.push(`v ${xyz.map((n) => (Object.is(n, -0) ? '-0' : String(n))).join(' ')}`);
}
ids.push(id);
}
triangles.push(`f ${ids.join(' ')}`);
}
if (vertices.length) append(vertexChunks, vertices);
append(faceChunks, triangles);
options.onProgress?.(end, faces);
// Allow rendering, cancellation and disposal between bounded batches.
if (end < faces) await new Promise<void>((resolve) => setTimeout(resolve, 0));
}
check();
const output = new Uint8Array(bytes);
let offset = 0;
for (const chunk of [...vertexChunks, ...faceChunks]) {
output.set(chunk, offset);
offset += chunk.length;
}
return { data: output, faces, vertices: seen.size };
}
@@ -70,4 +70,13 @@ it('仅避障策略显示实时导航状态,支持暂停时设定/复位', ()
/>,
);
expect(screen.queryByText('复位目标点')).not.toBeInTheDocument();
view.rerender(
<RLPolicyPanel
{...props}
status={{ ...status, taskId: 'MobileManipulator-LeKiwi-v1', observationSize: 68 }}
/>,
);
expect(screen.getByText('分阶段移动操作')).toBeVisible();
expect(screen.queryByText('速度指令')).not.toBeInTheDocument();
expect(screen.getByText('68 / 12')).toBeVisible();
});
+9 -1
View File
@@ -1,3 +1,4 @@
import { isMobileTrainingTask } from '../mobile/training';
import { useRef, type ChangeEvent } from 'react';
import { BrainCircuit, FileUp, Power, RotateCw, Trash2 } from 'lucide-react';
import type { RLCommand, RLPolicyStatus } from './types';
@@ -147,7 +148,14 @@ export function RLPolicyPanel({
</div>
</div>
)}
{status.observationSize === 81 || status.observationSize === 97 ? (
{isMobileTrainingTask(status.taskId) ? (
<div className="mt-2">
<Badge tone="accent">分阶段移动操作</Badge>
<p className="mt-1 text-xs text-warning">
按部署阶段运行;底盘接近阶段保持机械臂。成功、超时或安全终止后暂停,请重置再继续。策略及元数据请从强化学习任务面板一起导入。
</p>
</div>
) : status.observationSize === 81 || status.observationSize === 97 ? (
<div
className="mt-2"
title="评测在超时、跌倒或越界时停止;水平射线存在矮障碍与跌落盲区"
@@ -31,9 +31,13 @@ export function ExternalControlPanel({
const [profileId, setProfileId] = useState(snapshot.robot?.profileId ?? '');
const external = snapshot.externalControl,
ready = bridge.status.phase === 'ready';
const owner = { manual: '手动', python: 'Python', policy: 'ONNX', external: '外部' }[
snapshot.controlOwner ?? 'manual'
];
const owner = {
manual: '手动',
python: 'Python',
policy: 'ONNX',
external: '外部',
agent: '智能任务',
}[snapshot.controlOwner ?? 'manual'];
return (
<section aria-label="开源机器人控制" className="space-y-3 text-xs">
<label className="block">
+1 -1
View File
@@ -69,6 +69,6 @@ export function lekiwiDescriptor(modelFingerprint: string): RobotDescriptor {
: 'm',
})),
],
capabilities: { reset: true, lockstep: false, cameras: false, training: false },
capabilities: { reset: true, lockstep: false, cameras: false, training: true },
};
}
@@ -0,0 +1,125 @@
import { webcrypto } from 'node:crypto';
import reference from '../../../../robot_profiles/lekiwi-reference.urdf?raw';
import meshHashes from '../../../../robot_profiles/lekiwi-source-meshes.json';
import collision from '../../../../robot_profiles/lekiwi-full-collision.json';
import { lekiwiSourceMismatch } from './lekiwiSource';
import { LEKIWI_PROFILE as p } from './lekiwi';
import { sha256, validateLeKiwiSource } from '../registry';
const encode = (xml: string) => new TextEncoder().encode(xml);
function changed(edit: (doc: Document) => void): Uint8Array {
const doc = new DOMParser().parseFromString(reference, 'application/xml');
edit(doc);
return encode(new XMLSerializer().serializeToString(doc));
}
const joint = (doc: Document, name = 'arm_elbow_flex') =>
doc.querySelector(`joint[name="${name}"]`)!;
beforeEach(() => vi.stubGlobal('crypto', webcrypto));
afterEach(() => vi.unstubAllGlobals());
it('binds the checked-in reference and mesh inventory to the reviewed collision/profile source', async () => {
expect(await sha256(encode(reference))).toBe(p.source.urdfSha256);
expect(lekiwiSourceMismatch(encode(reference))).toBeUndefined();
const doc = new DOMParser().parseFromString(reference, 'application/xml');
expect(Object.keys(meshHashes).sort()).toEqual(
[
...new Set(Array.from(doc.querySelectorAll('mesh')).map((m) => m.getAttribute('filename'))),
].sort(),
);
for (const [path, hash] of Object.entries(collision.source.meshes))
expect(meshHashes[path as keyof typeof meshHashes]).toBe(hash);
});
it('accepts display name, colors, whitespace, attribute/root order and numeric spellings only', async () => {
const data = changed((doc) => {
doc.documentElement.setAttribute('name', 'LeKiwi 换色版');
doc.documentElement.append(doc.createComment('local export'));
const visual = doc.querySelector('visual')!;
const material = doc.createElement('material');
material.setAttribute('name', 'red');
const color = doc.createElement('color');
color.setAttribute('rgba', '1 0 0 1');
material.append(color);
visual.append(material);
const axis = joint(doc).querySelector('axis')!;
axis.setAttribute('xyz', '1e0 -0.000 0');
// Attribute order does not matter, but joint and geometry names still do.
const node = joint(doc);
node.removeAttribute('name');
node.setAttribute('name', 'arm_elbow_flex');
doc.documentElement.prepend(node);
});
expect(await sha256(data)).not.toBe(p.source.urdfSha256);
await expect(validateLeKiwiSource(data)).resolves.toBeUndefined();
});
it('compares equivalent Euler rotations as matrices, including gimbal-lock re-exports', () => {
const data = changed((doc) => {
joint(doc, 'base_back_wheel')
.querySelector('origin')!
.setAttribute('rpy', `0 ${Math.PI / 2} ${Math.PI / 2}`);
doc.querySelector('visual origin')!.removeAttribute('rpy');
doc.querySelector('collision origin')!.remove();
});
expect(lekiwiSourceMismatch(data)).toBeUndefined();
});
it.each([
['axis', (doc: Document) => joint(doc).querySelector('axis')!.setAttribute('xyz', '-1 0 0')],
['origin', (doc: Document) => joint(doc).querySelector('origin')!.setAttribute('xyz', '0 .01 0')],
[
'mesh scale',
(doc: Document) => doc.querySelector('mesh')!.setAttribute('scale', '.002 .001 .001'),
],
[
'mesh path',
(doc: Document) => doc.querySelector('mesh')!.setAttribute('filename', 'other.stl'),
],
['inertia', (doc: Document) => doc.querySelector('inertia')!.setAttribute('ixx', '.1')],
['NaN', (doc: Document) => doc.querySelector('mass')!.setAttribute('value', 'NaN')],
['mimic', (doc: Document) => joint(doc).append(doc.createElement('mimic'))],
['limits', (doc: Document) => joint(doc).append(doc.createElement('limit'))],
['compiler', (doc: Document) => doc.documentElement.append(doc.createElement('mujoco'))],
['visual name', (doc: Document) => doc.querySelector('visual')!.setAttribute('name', 'new')],
])('rejects same-topology physical/recipe changes: %s', async (_label, edit) => {
await expect(validateLeKiwiSource(changed(edit))).rejects.toThrow(/不能自动复用/);
});
it('diagnoses changed parents and joint types even with exactly the expected nine names', () => {
expect(
lekiwiSourceMismatch(
changed((doc) => joint(doc).querySelector('parent')!.setAttribute('link', p.baseBody)),
),
).toContain('运动链连接变化');
expect(
lekiwiSourceMismatch(changed((doc) => joint(doc).setAttribute('type', 'prismatic'))),
).toContain('关节类型变化');
expect(
lekiwiSourceMismatch(changed((doc) => doc.documentElement.append(joint(doc).cloneNode(true)))),
).toContain('实际 10 个');
});
it('does not infer Link1…Link7 as five arm channels and a normalized gripper', async () => {
const data = changed((doc) => {
p.arm.forEach((spec, index) => {
const node = joint(doc, spec.joint);
node.setAttribute('name', `Link${index + 1}`);
node.setAttribute('type', 'revolute');
});
const seventh = doc.createElement('joint');
seventh.setAttribute('name', 'Link7');
seventh.setAttribute('type', 'revolute');
doc.documentElement.append(seventh);
});
await expect(validateLeKiwiSource(data)).rejects.toThrow(/需要 9 个活动关节,实际 10 个/);
expect(lekiwiSourceMismatch(data)).toContain('arm_gripper');
expect(lekiwiSourceMismatch(data)).toContain('Link7');
});
it.each(['<robot/>', '<mujoco/>', '<robot>', '<!DOCTYPE robot><robot/>'])(
'fails closed on incomplete/invalid inputs: %s',
async (xml) => {
await expect(validateLeKiwiSource(encode(xml))).rejects.toThrow(/不兼容/);
},
);
@@ -0,0 +1,142 @@
import reference from '../../../../robot_profiles/lekiwi-reference.urdf?raw';
/** A deliberately narrow compatibility gate, NOT a generic robot/FK mapper.
* Ignore presentation only. Geometry, inertia, limits, extensions and the entire
* fixed/moving tree must still match the reviewed source. Mesh bytes are checked
* separately by the project loader, including meshes outside the arm recipe.
*/
const numeric: Record<string, readonly string[]> = {
origin: ['xyz', 'rpy'],
axis: ['xyz'],
mass: ['value'],
inertia: ['ixx', 'iyy', 'izz', 'ixy', 'ixz', 'iyz'],
mesh: ['scale'],
limit: ['lower', 'upper', 'effort', 'velocity'],
dynamics: ['damping', 'friction'],
box: ['size'],
sphere: ['radius'],
cylinder: ['radius', 'length'],
};
const parse = (xml: string) => new DOMParser().parseFromString(xml, 'application/xml');
function children(node: Element): Element[] {
const result = Array.from(node.children).filter(
(child) => !(child.tagName === 'material' && ['robot', 'visual'].includes(node.tagName)),
);
// An omitted origin is the identity, not a topology change. Compare a virtual
// default without mutating either the imported XML or the reference.
if (
['joint', 'inertial', 'visual', 'collision'].includes(node.tagName) &&
!result.some((child) => child.tagName === 'origin')
)
result.push(node.ownerDocument.createElement('origin'));
return result.sort((a, b) =>
`${a.tagName}:${a.getAttribute('name') ?? ''}`.localeCompare(
`${b.tagName}:${b.getAttribute('name') ?? ''}`,
),
);
}
const vector = (text: string) => (text.trim() ? text.trim().split(/\s+/).map(Number) : [NaN]);
// Compare Rz(yaw) Ry(pitch) Rx(roll), not ambiguous Euler angle spellings.
function rotation(values: number[]): number[] {
if (values.length !== 3) return [NaN];
const [r, p, y] = values,
[sr, sp, sy] = values.map(Math.sin),
[cr, cp, cy] = [r, p, y].map(Math.cos);
return [
cy * cp,
cy * sp * sr - sy * cr,
cy * sp * cr + sy * sr,
sy * cp,
sy * sp * sr + cy * cr,
sy * sp * cr - cy * sr,
-sp,
cp * sr,
cp * cr,
];
}
function attributes(node: Element): Map<string, string> {
const result = new Map(Array.from(node.attributes).map((a) => [a.name, a.value]));
if (node.tagName === 'robot') result.delete('name');
if (node.tagName === 'origin') {
if (!result.has('xyz')) result.set('xyz', '0 0 0');
if (!result.has('rpy')) result.set('rpy', '0 0 0');
}
if (node.tagName === 'mesh' && !result.has('scale')) result.set('scale', '1 1 1');
return result;
}
function difference(actual: Element, expected: Element, parent = ''): string | undefined {
const path = `${parent}/${expected.tagName}${expected.hasAttribute('name') ? `[${expected.getAttribute('name')}]` : ''}`;
if (actual.tagName !== expected.tagName) return `${path} 元素类型变化`;
// URDF has no character data in physical elements. Do not ignore extensions
// or injected text while ignoring ordinary whitespace/comments.
if (
Array.from(actual.childNodes).some(
(node) => (node.nodeType === 3 || node.nodeType === 4) && node.textContent?.trim(),
)
)
return `${path} 包含不支持的文本`;
const a = attributes(actual),
b = attributes(expected);
for (const key of new Set([...a.keys(), ...b.keys()])) {
if (!a.has(key) || !b.has(key)) return `${path}/@${key} 属性变化`;
if (numeric[expected.tagName]?.includes(key)) {
let av = vector(a.get(key)!),
bv = vector(b.get(key)!);
if (expected.tagName === 'origin' && key === 'rpy') {
av = rotation(av);
bv = rotation(bv);
}
if (
av.length !== bv.length ||
av.some(
(v, i) => !Number.isFinite(v) || Math.abs(v - bv[i]) > 1e-12 + 1e-9 * Math.abs(bv[i]),
)
)
return `${path}/@${key} 数值变化`;
} else if (a.get(key) !== b.get(key)) return `${path}/@${key} 变化`;
}
const ac = children(actual),
bc = children(expected);
if (ac.length !== bc.length) return `${path} 子元素数量变化`;
for (let i = 0; i < bc.length; i++) {
const reason = difference(ac[i], bc[i], path);
if (reason) return reason;
}
}
export function lekiwiSourceMismatch(data: Uint8Array): string | undefined {
const doc = parse(new TextDecoder().decode(data));
if (doc.querySelector('parsererror') || doc.documentElement.tagName !== 'robot' || doc.doctype)
return 'URDF XML 无效或包含不支持的 DOCTYPE';
const expected = parse(reference).documentElement,
actual = doc.documentElement;
const joints = (root: Element) =>
Array.from(root.children).filter((node) => node.tagName === 'joint');
const movable = (root: Element) =>
joints(root).filter((node) => node.getAttribute('type') !== 'fixed');
const oldNames = movable(expected).map((node) => node.getAttribute('name')!),
newNames = movable(actual).map((node) => node.getAttribute('name')!);
const missing = oldNames.filter((name) => !newNames.includes(name)),
extra = newNames.filter((name) => !oldNames.includes(name));
if (missing.length || extra.length || oldNames.length !== newNames.length)
return `控制拓扑不匹配:需要 ${oldNames.length} 个活动关节,实际 ${newNames.length} 个;缺少 [${missing.join(', ')}];新增 [${extra.join(', ')}]。不能按顺序重命名或把新关节当作原夹爪`;
for (const joint of joints(expected)) {
const candidates = joints(actual).filter(
(node) => node.getAttribute('name') === joint.getAttribute('name'),
);
if (candidates.length !== 1) return `运动链缺少或重复关节:${joint.getAttribute('name')}`;
const incoming = candidates[0];
for (const tag of ['parent', 'child'])
if (
incoming.querySelector(`:scope > ${tag}`)?.getAttribute('link') !==
joint.querySelector(`:scope > ${tag}`)?.getAttribute('link')
)
return `运动链连接变化:${joint.getAttribute('name')}/${tag}`;
if (incoming.getAttribute('type') !== joint.getAttribute('type'))
return `关节类型变化:${joint.getAttribute('name')}`;
}
const reason = difference(actual, expected);
return reason
? `${reason};现有碰撞/控制配方不能自动复用,需重新审核模型参数与碰撞数据`
: undefined;
}
+5 -2
View File
@@ -17,9 +17,12 @@ export async function sha256(data: Uint8Array): Promise<string> {
);
}
export async function validateLeKiwiSource(data: Uint8Array): Promise<void> {
if ((await sha256(data)) !== LEKIWI_PROFILE.source.urdfSha256)
if ((await sha256(data)) === LEKIWI_PROFILE.source.urdfSha256) return;
const { lekiwiSourceMismatch } = await import('./profiles/lekiwiSource');
const reason = lekiwiSourceMismatch(data);
if (reason)
throw new RobotError(
'INCOMPATIBLE_MODEL',
`LeKiwi URDF 与受支持的 ${LEKIWI_PROFILE.source.revision.slice(0, 8)} 版本不符;请使用原始 URDF,不能仅凭关节数量套用 profile`,
`LeKiwi URDF 不兼容:${reason}。参见 docs/lekiwi-urdf-compatibility.md`,
);
}
+1 -1
View File
@@ -17,7 +17,7 @@ export interface RobotDescriptor {
frame: 'x-forward-y-left-z-up';
actionChannels: RobotChannel[];
observationChannels: RobotChannel[];
capabilities: { reset: boolean; lockstep: false; cameras: false; training: false };
capabilities: { reset: boolean; lockstep: false; cameras: false; training: boolean };
}
export interface RobotIdentity {
sessionId: string;
+3 -3
View File
@@ -81,9 +81,9 @@ export function validateDescriptor(value: unknown): RobotDescriptor {
typeof caps.reset !== 'boolean' ||
caps.lockstep !== false ||
caps.cameras !== false ||
caps.training !== false
typeof caps.training !== 'boolean'
)
throw new RobotError('UNSUPPORTED', 'V1 不支持相机、锁步或训练');
throw new RobotError('UNSUPPORTED', 'V1 不支持相机或锁步;training 必须是布尔值');
return {
protocolVersion: 1,
profileId: identifier(d.profileId),
@@ -92,7 +92,7 @@ export function validateDescriptor(value: unknown): RobotDescriptor {
frame: 'x-forward-y-left-z-up',
actionChannels: channels(d.actionChannels),
observationChannels: channels(d.observationChannels),
capabilities: { reset: caps.reset, lockstep: false, cameras: false, training: false },
capabilities: { reset: caps.reset, lockstep: false, cameras: false, training: caps.training },
};
}
/** Complete channel set; reject non-numeric input rather than coercing strings/bools. */
@@ -0,0 +1,10 @@
/** Only the session physics clock may call beforeStep/afterStep. Network handlers
* must enqueue validated decisions, never invoke the supplied write capability. */
export interface AgentDriver {
beforeStep(): boolean;
afterStep(dt: number): void;
hold(): void;
cancel(reason: string): void;
dispose(): void;
}
export type AgentFactory = (writeServo: (actuatorId: number, value: number) => void) => AgentDriver;
@@ -1,4 +1,4 @@
export type ControlOwner = 'manual' | 'python' | 'policy' | 'external';
export type ControlOwner = 'manual' | 'python' | 'policy' | 'external' | 'agent';
export interface ControlTicket {
readonly owner: ControlOwner;
readonly generation: number;
+136 -7
View File
@@ -1,3 +1,5 @@
import type { RobotConfig } from '../mobile/RobotDescriptor';
import { composeMobileScene, prepareBundle } from '../mobile/SceneComposer';
import {
composeTrainingMap,
trainingTerrainFromCompiledScene,
@@ -8,6 +10,7 @@ import type { PolicyDeployment, TrainingTerrain } from '../rl/deployment';
import type { MainModule } from '@mujoco/mujoco';
import type { ProjectFile, ProjectManifest } from '../project/types';
import { prepareProjectForMujoco } from '../project/importer';
import { prepareLargeMeshes } from '../project/meshCompatibility';
import { prepareRobotProject } from '../project/robotProfiles';
import { sha256 } from '../robot/registry';
import type { ExternalControlStatus } from '../robot/RobotRuntime';
@@ -56,6 +59,13 @@ export interface PhysicsLoadProgress {
export interface PhysicsLoadOptions {
/** Explicit built-in profile; never inferred from actuator count. */
robotProfileId?: string;
/** false: reuse model recipe without installing the external-control runtime. */
configureRobotRuntime?: boolean;
/** Optional caller-owned free VFS slot; bounds native asset-cache path identities. */
workspaceSlot?: string;
/** Trusted application hook, after URDF/map conversion and include expansion. */
sceneComposer?: (mjcf: Uint8Array) => Uint8Array;
mobileRobot?: RobotConfig;
trainingDeployment?: PolicyDeployment;
/** Candidate policy is initialized/validated before replacing the active session. */
trainingPolicy?: { data: Uint8Array; path: string };
@@ -158,6 +168,8 @@ export class MainThreadPhysicsAdapter implements PhysicsAdapter {
}
>();
private loadGeneration = 0;
private readonly workspacePrefix = `adapter_${crypto.randomUUID().replaceAll('-', '_')}`;
private readonly candidateRoots = new Set<string>();
private disposed = false;
private retiredSession: SimulationSession | null = null;
private retiredWorkspace: MemfsWorkspace | null = null;
@@ -169,6 +181,7 @@ export class MainThreadPhysicsAdapter implements PhysicsAdapter {
options: PhysicsLoadOptions = {},
): Promise<SimulationSnapshot> {
if (this.disposed) throw new Error('物理适配器已释放');
this.session?.stopAgent?.('模型或地图正在重载');
if (this.session?.describeRobot?.())
this.session.stopExternalControl('模型正在重载,请重新授权');
const generation = ++this.loadGeneration;
@@ -190,19 +203,76 @@ export class MainThreadPhysicsAdapter implements PhysicsAdapter {
const module = await getMujocoModule();
if (this.disposed || generation !== this.loadGeneration) throw new Error('模型加载已取消');
report(0.32, '校验并准备模型资源');
const workspace = new MemfsWorkspace(module, `${manifest.id}_stage_${generation}`);
if (
options.robotProfileId &&
(urdfMode !== 'mjcf' || baseMode !== 'floating' || options.trainingDeployment)
)
throw new Error('机器人 profile 需要浮动基座 URDF 转换,不能与 Go2 部署混用');
const mobileRobot = options.mobileRobot;
if (
mobileRobot &&
(options.trainingDeployment || urdfMode !== 'mjcf' || baseMode !== 'floating')
)
throw new Error('移动操作需要浮动基座 MJCF,不能与 Go2 部署混用');
const sourceManifest =
mobileRobot?.recipe === 'lekiwi-bundle'
? prepareBundle(manifest, entryPath, mobileRobot)
: manifest;
const profileManifest = options.robotProfileId
? await prepareRobotProject(manifest, entryPath, options.robotProfileId)
: manifest;
const prepared = await prepareProjectForMujoco(profileManifest, entryPath);
? await prepareRobotProject(sourceManifest, entryPath, options.robotProfileId)
: sourceManifest;
const normalized = await prepareProjectForMujoco(profileManifest, entryPath);
const prepared = await prepareLargeMeshes(normalized.manifest, entryPath, {
shouldCancel: () => this.disposed || generation !== this.loadGeneration,
onMeshProgress: (path, done, total) =>
report(
0.34 + (0.1 * done) / total,
`兼容高面数网格 ${path}(${done}/${total} 面,保留全部三角形)`,
),
});
prepared.warnings.unshift(...normalized.warnings);
// Map files may not be in the robot's include graph. Prepare the selected
// physics assets before MapComposer rebases them into the resulting scene.
const mapPaths = new Set<string>();
for (const selection of placedMapAssets?.map((asset) => asset.selection) ?? [mapSelection]) {
if (selection.kind === 'project') {
const map = resolveProjectMap(prepared.manifest, selection.descriptorPath);
if (map.physicsPath) mapPaths.add(map.physicsPath);
}
}
for (const path of mapPaths) {
const next = await prepareLargeMeshes(prepared.manifest, path, {
shouldCancel: () => this.disposed || generation !== this.loadGeneration,
});
prepared.manifest = next.manifest;
prepared.warnings.push(...next.warnings);
}
if (this.disposed || generation !== this.loadGeneration) throw new Error('模型加载已取消');
const supportFiles = prepared.manifest.files.filter(
(file) => !manifest.files.some((original) => original.path === file.path),
);
// Bounded path identities stop the native mesh cache accumulating a new
// copy on every reload. Three slots permit active + rollback + candidate;
// callers with explicit two-slot viewer transactions retain their contract.
const freeSlot = [0, 1, 2]
.map((i) => `${this.workspacePrefix}_${i}`)
.find((slot) => {
const root = `/workspace/${slot}`;
return (
root !== this.workspace?.root &&
root !== this.retiredWorkspace?.root &&
!this.candidateRoots.has(root)
);
});
if (!options.workspaceSlot && !freeSlot) throw new Error('候选 VFS 槽已满,请等待当前加载结束');
const workspace = new MemfsWorkspace(module, options.workspaceSlot ?? freeSlot!);
if (
workspace.root === this.workspace?.root ||
workspace.root === this.retiredWorkspace?.root ||
this.candidateRoots.has(workspace.root)
)
throw new Error('候选 VFS 槽仍被活动/待释放模型占用');
this.candidateRoots.add(workspace.root);
let nextSession: SimulationSession | null = null;
try {
report(0.46, '写入浏览器内存文件系统');
@@ -237,8 +307,33 @@ export class MainThreadPhysicsAdapter implements PhysicsAdapter {
const bodyId = Number(intermediate.model.jnt_bodyid[id]);
const body = intermediate.model.body(bodyId);
try {
let wheelCenter: number[] | undefined;
if (joint.name.startsWith('base_') && joint.name.endsWith('_wheel')) {
for (let geomId = 0; geomId < intermediate.model.ngeom; geomId++) {
const geom = intermediate.model.geom(geomId);
try {
if (
!geom.name.startsWith('4-Omni-Directional-Wheel_Single_Body') ||
Number(intermediate.model.geom_group[geomId]) !== 1
)
continue;
let ancestor = Number(intermediate.model.geom_bodyid[geomId]);
while (ancestor && ancestor !== bodyId)
ancestor = Number(intermediate.model.body_parentid[ancestor]);
if (ancestor === bodyId) {
wheelCenter = Array.from(
intermediate.data.geom_xpos.slice(geomId * 3, geomId * 3 + 3),
);
break;
}
} finally {
geom.delete();
}
}
}
geometry.push({
name: joint.name,
wheelCenter,
body: body.name,
position: Array.from(
intermediate.data.xpos.slice(bodyId * 3, bodyId * 3 + 3),
@@ -254,9 +349,12 @@ export class MainThreadPhysicsAdapter implements PhysicsAdapter {
body.delete();
}
}
convertedData = enhanceLeKiwiMjcf(convertedData, geometry);
const inputHash = await sha256(
manifest.files.find((file) => file.path === entryPath)!.data,
);
convertedData = enhanceLeKiwiMjcf(convertedData, geometry, inputHash);
warnings.unshift(
'LeKiwi v1:仿真估计参数,简化被动滚子/碰撞与九路伺服;超大轮 STL 已替换为简化视觉,不代表实机标定',
'LeKiwi v1:完整 CAD 轮视觉;碰撞仍使用被动滚子/整臂凸分解与九路伺服,轮距/动力学为仿真估计,不代表实机标定',
);
}
workspace.writeGenerated(convertedPath, convertedData);
@@ -342,10 +440,36 @@ export class MainThreadPhysicsAdapter implements PhysicsAdapter {
}
warnings.push('已用策略配套训练布局替换场景地形;编辑器地图未更改。Go2-W动力学不等同Go2。');
}
const sceneComposer = mobileRobot
? (xml: Uint8Array) => {
const mobileScene = composeMobileScene(xml, mobileRobot);
return options.sceneComposer ? options.sceneComposer(mobileScene) : mobileScene;
}
: options.sceneComposer;
if (sceneComposer) {
if (entry?.format === 'urdf' && urdfMode === 'native')
throw new Error('任务场景组合需要 MJCF 转换模式');
const intermediate = new SimulationSession(module, modelPath);
try {
const scenePath = `${modelRelativePath}.task.xml`;
// Register before native save so failed composition is cleaned up too.
workspace.writeGenerated(scenePath, new Uint8Array());
if (!module.mj_saveLastXML(workspace.path(scenePath), intermediate.model))
throw new Error('无法展开任务场景');
workspace.writeGenerated(
scenePath,
sceneComposer(new TextEncoder().encode(workspace.readText(scenePath))),
);
modelRelativePath = scenePath;
modelPath = workspace.path(scenePath);
} finally {
intermediate.dispose();
}
}
report(0.84, '编译模型与物理数据');
console.info('[MuJoCo] 编译模型', modelPath);
nextSession = new SimulationSession(module, modelPath, warnings);
if (options.robotProfileId) {
if (options.robotProfileId && options.configureRobotRuntime !== false && !mobileRobot) {
const fingerprint = await sha256(
new TextEncoder().encode(workspace.readText(modelRelativePath)),
);
@@ -411,6 +535,7 @@ export class MainThreadPhysicsAdapter implements PhysicsAdapter {
});
}
}
if (mobileRobot) nextSession.configureMobile(mobileRobot);
if (options.trainingPolicy) {
if (!options.trainingDeployment?.terrain) throw new Error('事务策略加载需要配套训练地图');
report(0.9, '校验候选场景的 ONNX 策略');
@@ -445,6 +570,10 @@ export class MainThreadPhysicsAdapter implements PhysicsAdapter {
`模型编译失败(${entryPath}):${error instanceof Error ? error.message : String(error)}`,
{ cause: error },
);
} finally {
this.candidateRoots.delete(workspace.root);
this.session?.mobile?.env.refreshViews();
this.retiredSession?.mobile?.env.refreshViews();
}
}
exportTrainingTerrain(
@@ -0,0 +1,165 @@
import { SimulationSession } from './SimulationSession';
import { ControlArbiter } from './ControlArbiter';
import type { AgentDriver } from './AgentDriver';
function fixture() {
const data = {
ctrl: new Float64Array([0.4]),
qpos: new Float64Array([0.3]),
xfrc_applied: new Float64Array(6),
time: 0,
};
const arbiter = new ControlArbiter();
const session = Object.assign(Object.create(SimulationSession.prototype), {
disposed: false,
paused: false,
speed: 1,
forceBody: -1,
perturb: { active: 0 },
frameBudgetMs: 8,
maxCatchUpSteps: 100,
accumulator: 0,
controlArbiter: arbiter,
data,
model: {
nactuator: 1,
nu: 1,
actuator_ctrladr: [0],
opt: { timestep: 0.001, delete: vi.fn() },
actuator: () => ({ ctrllimited: true, ctrlrange: [-1, 1], delete: vi.fn() }),
},
robotAdapter: { safeStop: vi.fn() },
robotRuntime: {
revoke: vi.fn(),
capture: vi.fn(),
reset: vi.fn(),
afterStep: vi.fn(),
authorize: vi.fn(),
},
module: {
mj_step: () => {
data.time += 0.001;
},
mj_forward: vi.fn(),
mj_resetData: vi.fn(),
},
dataRecorder: { capture: vi.fn(), simulationReset: vi.fn() },
startExternalClock: vi.fn(),
}) as SimulationSession;
return { session, arbiter, data };
}
function driver(overrides: Partial<AgentDriver> = {}): AgentDriver {
return {
beforeStep: () => true,
afterStep: vi.fn(),
hold: vi.fn(),
cancel: vi.fn(),
dispose: vi.fn(),
...overrides,
};
}
it('external-force interaction revokes an agent and pauses before any further step', () => {
const f = fixture(),
agent = driver();
let write!: (id: number, value: number) => void;
f.session.startAgent((cap) => {
write = cap;
return agent;
});
f.session.setExternalForce(1, [1, 0, 0]);
expect(f.session.controlOwner).toBe('manual');
expect(f.session.paused).toBe(true);
expect(agent.cancel).toHaveBeenCalledWith('外力交互使旧计划失效');
expect(() => write(0, 0.8)).toThrow();
});
it('only synchronous physics hooks can use agent writes; manual and asynchronous writes are rejected', () => {
const f = fixture();
let write!: (id: number, value: number) => void;
const agent = driver({
beforeStep: () => {
write(0, 0.7);
return true;
},
});
f.session.startAgent((cap) => {
write = cap;
return agent;
});
expect(f.arbiter.current().owner).toBe('agent');
expect(() => write(0, 0.9)).toThrow(/物理步/);
expect(() => f.session.setActuator(0, 0.9)).toThrow(/停止当前/);
f.session.singleStep();
expect(f.data.ctrl[0]).toBe(0.7);
expect(agent.afterStep).toHaveBeenCalledWith(0.001);
});
it.each(['pause', 'reset', 'external'] as const)(
'%s revokes the old ticket, cancels the request, and preserves arm hold',
(kind) => {
const f = fixture();
let write!: (id: number, value: number) => void;
const agent = driver({ hold: () => write(0, 0.4) });
f.session.startAgent((cap) => {
write = cap;
return agent;
});
if (kind === 'pause') f.session.setPaused(true);
if (kind === 'reset') f.session.reset();
if (kind === 'external') f.session.setExternalControlEnabled(true);
expect(agent.cancel).toHaveBeenCalled();
expect(() => write(0, 0.9)).toThrow(/票据/);
if (kind !== 'reset') expect(f.data.ctrl[0]).toBe(0.4);
expect(f.data.qpos[0]).toBe(0.3);
},
);
it('a replaced callback cannot piggyback on the next agent physics write window', () => {
const f = fixture();
let old!: (id: number, value: number) => void;
const first = driver();
f.session.startAgent((cap) => {
old = cap;
return first;
});
f.session.startAgent(() =>
driver({
beforeStep: () => {
expect(() => old(0, 0.9)).toThrow(/票据/);
return true;
},
}),
);
f.session.singleStep();
expect(first.dispose).toHaveBeenCalledOnce();
expect(f.data.ctrl[0]).toBe(0.4);
});
it('model waits freeze the one clock, and resumed frames never catch up the wait', () => {
const f = fixture();
let ready = false;
f.session.startAgent(() => driver({ beforeStep: () => ready }));
f.session.advance(0);
f.session.advance(1000);
expect(f.data.time).toBe(0);
ready = true;
f.session.advance(200000);
expect(f.data.time).toBe(0);
f.session.advance(200001);
expect(f.data.time).toBeCloseTo(0.001);
});
it('stopping from a post-step hook prevents remaining frame substeps', () => {
const f = fixture();
f.session.startAgent(() => driver({ afterStep: () => f.session.stopAgent('done') }));
f.session.advance(0);
f.session.advance(50);
expect(f.data.time).toBeCloseTo(0.001);
expect(f.session.paused).toBe(true);
expect(f.arbiter.current().owner).toBe('manual');
});
it('factory failure fails paused/manual instead of retaining an orphan agent owner', () => {
const f = fixture();
expect(() =>
f.session.startAgent(() => {
throw new Error('binding');
}),
).toThrow('binding');
expect(f.session.paused).toBe(true);
expect(f.arbiter.current().owner).toBe('manual');
});
+178 -14
View File
@@ -1,3 +1,6 @@
import { MobilePolicyController } from '../mobile/MobilePolicyController';
import { MOBILE_TASK, type RobotConfig } from '../mobile/RobotDescriptor';
import type { MobileDeployment } from '../mobile/training';
import { Go2ObstacleAvoidanceBindings } from '../rl/runtime/Go2ObstacleAvoidanceBindings';
import {
OBSTACLE_TASK_ID,
@@ -10,6 +13,7 @@ import { meshIdFromSceneDataId } from './geometry';
import { PythonControllerRuntime } from '../controller/PythonControllerRuntime';
import { ModelBindings } from './ModelBindings';
import { ControlArbiter, type ControlOwner, type ControlTicket } from './ControlArbiter';
import type { AgentDriver, AgentFactory } from './AgentDriver';
import { MuJoCoRobotAdapter } from './MuJoCoRobotAdapter';
import { RobotRuntime, type ExternalControlStatus } from '../robot/RobotRuntime';
import {
@@ -96,6 +100,7 @@ export interface SimulationSnapshot {
warnings: string[];
controller?: ControllerStatus;
rlPolicy?: RLPolicyStatus;
mobileRobotId?: string;
controlOwner?: ControlOwner;
robot?: RobotDescriptor;
externalControl?: ExternalControlStatus;
@@ -135,12 +140,30 @@ export class SimulationSession {
private pythonController?: PythonControllerRuntime;
private controllerLoadGeneration = 0;
private rlPolicy?: OnnxPolicyRuntime;
mobile?: MobilePolicyController;
configureMobile(config: RobotConfig): void {
this.mobile = new MobilePolicyController(this, config);
}
async loadMobilePolicy(
model: Uint8Array,
path: string,
deployment: MobileDeployment,
): Promise<RLPolicyStatus> {
if (!this.mobile) throw new Error('请先加载移动操作任务场景');
this.setPaused(true);
await this.mobile.load(model, path, deployment);
this.setRLPolicyEnabled(true);
return this.mobile.status()!;
}
private obstacleBindings?: Go2ObstacleAvoidanceBindings;
private deployment?: PolicyDeployment;
private deploymentInitialQpos?: Float64Array;
private rlPolicyLoadGeneration = 0;
private dataRecorder!: DataRecorder;
private controlArbiter?: ControlArbiter;
private agentDriver?: AgentDriver;
private agentScope?: ControllerWriteScope;
private agentWritingScope?: ControllerWriteScope;
private pythonScope?: ControllerWriteScope;
private policyScope?: ControllerWriteScope;
private robotAdapter?: RobotAdapter;
@@ -174,6 +197,9 @@ export class SimulationSession {
}
}, 1000 / 120);
}
get controlOwner(): ControlOwner {
return this.arbiter.current().owner;
}
private get arbiter(): ControlArbiter {
return (this.controlArbiter ??= new ControlArbiter());
}
@@ -207,18 +233,99 @@ export class SimulationSession {
return this.robotRuntime?.latest();
}
private clearControlTargets(): void {
if (this.robotAdapter) this.robotAdapter.safeStop();
// Zero ctrl is a full position jump for mobile arm servos, not a safe stop.
if (this.mobile) this.mobile.env.hold();
else if (this.robotAdapter) this.robotAdapter.safeStop();
else this.data.ctrl.fill(0);
}
private selectControl(owner: ControlOwner): ControlTicket {
private selectControl(owner: ControlOwner, reason = '控制权已切换'): ControlTicket {
this.invalidateAgent(reason);
this.stopExternalClock();
const ticket = this.arbiter.claim(owner);
this.pythonController?.setEnabled(false, Number(this.data.time));
this.rlPolicy?.setEnabled(false, Number(this.data.time));
this.mobile?.setEnabled(false);
this.robotRuntime?.revoke('控制权已切换', false);
this.clearControlTargets();
return ticket;
}
startAgent(factory: AgentFactory): void {
if (this.disposed || (!this.robotAdapter && !this.mobile))
throw new Error('智能任务需要已配置的机器人环境');
this.invalidateAgent('智能任务已替换');
this.agentDriver?.dispose();
this.agentDriver = undefined;
const scope: ControllerWriteScope = { ticket: this.selectControl('agent') };
this.agentScope = scope;
this.clearExternalForce();
try {
this.agentDriver = factory((id, value) => {
if (this.disposed || !this.arbiter.owns(scope.ticket) || this.agentWritingScope !== scope)
throw new Error('智能任务写入票据已失效或不在物理步内');
this.writeActuator(id, value);
});
this.mobile?.env.refreshViews();
this.withAgentWrite(() => this.agentDriver!.hold());
this.accumulator = 0;
this.lastNow = undefined;
} catch (error) {
this.mobile?.env.refreshViews();
this.invalidateAgent('智能任务初始化失败');
this.agentDriver?.dispose();
this.agentDriver = undefined;
this.arbiter.claim('manual');
this.paused = true;
this.clearControlTargets();
throw error;
}
}
stopAgent(reason = '智能任务已停止'): void {
if (this.arbiter.current().owner === 'agent') {
this.selectControl('manual', reason);
this.paused = true;
this.accumulator = 0;
this.lastNow = undefined;
} else this.invalidateAgent(reason);
}
private withAgentWrite<T>(action: () => T): T {
const scope = this.agentScope;
if (!scope || !this.arbiter.owns(scope.ticket)) throw new Error('智能任务控制权失效');
this.agentWritingScope = scope;
try {
return action();
} finally {
this.agentWritingScope = undefined;
}
}
private invalidateAgent(reason: string): void {
if (!this.agentDriver) return;
try {
if (!this.disposed && this.arbiter.owns(this.agentScope?.ticket))
this.withAgentWrite(() => this.agentDriver!.hold());
} catch {
if (!this.disposed) this.clearControlTargets();
console.warn('[MuJoCo] 智能任务保持失败,已使用机器人安全保持');
} finally {
if (this.agentScope) this.agentScope.ticket = undefined;
try {
this.agentDriver.cancel(reason);
} catch {
console.warn('[MuJoCo] 智能任务取消回调失败;写入票据已撤销');
}
}
}
private afterAgentStep(): void {
if (this.arbiter.current().owner !== 'agent' || !this.agentDriver) return;
const opt = this.model.opt,
dt = Number(opt.timestep);
opt.delete();
try {
this.withAgentWrite(() => this.agentDriver!.afterStep(dt));
} catch (error) {
this.stopAgent('智能任务物理回调失败');
throw error;
}
}
setExternalControlEnabled(enabled: boolean): void {
if (!enabled) {
this.stopExternalControl();
@@ -324,9 +431,11 @@ export class SimulationSession {
}
setPaused(paused: boolean): void {
if (paused && this.arbiter.current().owner === 'agent') this.stopAgent('智能任务已暂停');
if (paused && this.arbiter.current().owner === 'external')
this.stopExternalControl('仿真已暂停,请重新授权');
this.paused = paused;
this.mobile?.setPaused(paused);
this.accumulator = 0;
this.lastNow = undefined;
this.robotRuntime?.capture(Number(this.data.time));
@@ -346,17 +455,22 @@ export class SimulationSession {
this.pythonController?.reset(Number(this.data.time));
this.rlPolicy?.reset(Number(this.data.time));
this.robotRuntime?.reset();
this.mobile?.reset();
this.module.mj_forward(this.model, this.data);
this.robotRuntime?.capture(Number(this.data.time));
this.dataRecorder.simulationReset();
}
singleStep(): void {
if (this.arbiter.current().owner === 'agent' && this.paused)
throw new RobotError('PAUSED', '智能任务不能在暂停时单步');
if (this.mobile && this.arbiter.current().owner === 'policy') return;
if (this.arbiter.current().owner === 'external' && this.paused)
throw new RobotError('PAUSED', '外部控制不能在暂停时单步');
if (!this.runController()) return;
this.applyForce();
this.module.mj_step(this.model, this.data);
this.robotRuntime?.afterStep(Number(this.data.time));
this.afterAgentStep();
this.dataRecorder.capture();
}
@@ -377,20 +491,40 @@ export class SimulationSession {
this.lastNow = now;
if (this.paused) return { steps: 0, stepMs: 0, overBudget: false };
this.accumulator += elapsed * this.speed;
const dt = Number(this.model.opt.timestep) || 0.002;
const mobile = this.arbiter.current().owner === 'policy' ? this.mobile : undefined;
const dt = mobile ? MOBILE_TASK.controlDt : Number(this.model.opt.timestep) || 0.002;
if (mobile) this.accumulator = Math.min(this.accumulator, 0.1);
const started = performance.now();
let steps = 0;
while (
!this.paused &&
this.accumulator >= dt &&
steps < this.maxCatchUpSteps &&
performance.now() - started < this.frameBudgetMs
) {
if (!this.runController()) break;
this.applyForce();
this.module.mj_step(this.model, this.data);
this.robotRuntime?.afterStep(Number(this.data.time));
if (mobile) {
try {
if (!mobile.tick()) {
if (mobile.error) this.setPaused(true);
break;
}
if (mobile.ended) {
this.setPaused(true);
break;
}
} catch (error) {
this.setPaused(true);
throw error;
}
} else {
if (!this.runController()) break;
this.applyForce();
this.module.mj_step(this.model, this.data);
this.robotRuntime?.afterStep(Number(this.data.time));
this.afterAgentStep();
}
this.dataRecorder.capture();
this.accumulator -= dt;
this.accumulator = Math.max(0, this.accumulator - dt);
steps++;
}
const overBudget = this.accumulator >= dt;
@@ -527,6 +661,7 @@ export class SimulationSession {
path: string,
deployment?: PolicyDeployment,
): Promise<RLPolicyStatus> {
if (this.mobile) throw new Error('移动操作场景需要匹配的 ONNX + deployment 元数据');
const generation = ++this.rlPolicyLoadGeneration;
if (deployment) deployment = validatePolicyDeployment(deployment);
if (
@@ -565,6 +700,13 @@ export class SimulationSession {
}
setRLPolicyEnabled(enabled: boolean): void {
if (this.mobile) {
if (enabled && this.mobile.status()) {
this.selectControl('policy');
this.mobile.setEnabled(true);
} else if (this.arbiter.current().owner === 'policy') this.selectControl('manual');
return;
}
if (enabled && this.rlPolicy && this.policyScope) {
this.policyScope.ticket = this.selectControl('policy');
this.rlPolicy.setEnabled(true, Number(this.data.time));
@@ -601,6 +743,7 @@ export class SimulationSession {
this.rlPolicyLoadGeneration += 1;
if (this.arbiter.current().owner === 'policy') this.selectControl('manual');
this.policyScope = undefined;
this.mobile?.remove();
this.rlPolicy?.dispose();
this.rlPolicy = undefined;
this.obstacleBindings = undefined;
@@ -621,6 +764,15 @@ export class SimulationSession {
private runController(): boolean {
try {
const owner = this.arbiter.current().owner;
if (
owner === 'agent' &&
(!this.agentDriver || !this.withAgentWrite(() => this.agentDriver!.beforeStep()))
) {
// Model waits freeze physics and discard accumulated wall time.
this.accumulator = 0;
this.lastNow = undefined;
return false;
}
if (owner === 'external' && !this.robotRuntime?.beforeStep()) return false;
if (owner === 'python') this.pythonController?.stepIfDue(Number(this.data.time));
if (owner === 'policy') this.rlPolicy?.step(Number(this.data.time));
@@ -733,8 +885,11 @@ export class SimulationSession {
}
}
private get robotMechanicsLocked(): boolean {
return Boolean(this.robotAdapter || this.mobile);
}
setActuatorParameters(id: number, parameters: ActuatorParameters): boolean {
if (this.arbiter.current().owner !== 'manual' || this.robotAdapter) return false;
if (this.arbiter.current().owner !== 'manual' || this.robotMechanicsLocked) return false;
if (id < 0 || id >= this.model.nactuator) return false;
const finite = (value: number, fallback: number) => (Number.isFinite(value) ? value : fallback);
const ordered = (
@@ -813,7 +968,7 @@ export class SimulationSession {
// Robot profiles use physical drives. Direct qpos writes teleport through
// colliders, even with perfect collision meshes. Keep pose editing for generic
// models; a configured robot is controlled via its associated actuators.
if (this.robotAdapter) return false;
if (this.robotMechanicsLocked) return false;
if (id < 0 || id >= this.model.njnt) return false;
const joint = this.model.jnt(id);
try {
@@ -837,7 +992,7 @@ export class SimulationSession {
resetJoints(): void {
if (this.arbiter.current().owner !== 'manual')
throw new RobotError('CONFLICT', '请先停止当前控制器');
if (this.robotAdapter)
if (this.robotMechanicsLocked)
throw new RobotError('CONFLICT', '机器人 profile 不允许关节瞬移;请使用驱动器或完整仿真重置');
this.setPaused(true);
for (let id = 0; id < this.model.njnt; id += 1) {
@@ -858,7 +1013,7 @@ export class SimulationSession {
setIgnoreJointLimits(ignore: boolean): void {
if (this.arbiter.current().owner !== 'manual')
throw new RobotError('CONFLICT', '请先停止当前控制器');
if (this.robotAdapter)
if (this.robotMechanicsLocked)
throw new RobotError('CONFLICT', '机器人 profile 的关节限位不可修改;请使用物理驱动器控制');
this.ignoreJointLimits = ignore;
for (let id = 0; id < this.model.njnt; id += 1) {
@@ -882,6 +1037,7 @@ export class SimulationSession {
}
setExternalForce(bodyId: number, force: [number, number, number]): void {
if (this.arbiter.current().owner === 'agent') this.stopAgent('外力交互使旧计划失效');
this.forceBody = bodyId > 0 && bodyId < this.model.nbody ? bodyId : -1;
this.force = force;
}
@@ -892,6 +1048,7 @@ export class SimulationSession {
this.perturb.active = 0;
}
initializePerturb(scene: MjvScene, bodyId: number): void {
if (this.arbiter.current().owner === 'agent') this.stopAgent('外力拖动使旧计划失效');
this.perturb.select = bodyId;
this.module.mjv_initPerturb(this.model, this.data, scene, this.perturb);
}
@@ -1176,7 +1333,7 @@ export class SimulationSession {
limited,
limitsIgnored: this.ignoreJointLimits,
editable:
!this.robotAdapter &&
!this.robotMechanicsLocked &&
(type === 2 || type === 3) &&
this.arbiter.current().owner === 'manual',
bodyId: Number(joint.bodyid),
@@ -1213,7 +1370,8 @@ export class SimulationSession {
bodies,
warnings: this.warnings,
controller: this.pythonController?.status(),
rlPolicy: this.rlPolicy?.status(),
rlPolicy: this.mobile?.status() ?? this.rlPolicy?.status(),
mobileRobotId: this.mobile?.env.config.id,
controlOwner: this.arbiter.current().owner,
robot: this.describeRobot(),
externalControl: this.robotRuntime?.status(),
@@ -1242,9 +1400,15 @@ export class SimulationSession {
};
this.stopExternalClock();
this.arbiter.claim('manual');
cleanup('智能任务', () => {
this.invalidateAgent('仿真会话已释放');
this.agentDriver?.dispose();
this.agentDriver = undefined;
});
cleanup('机器人接口', () => this.robotRuntime?.dispose());
cleanup('Python 控制器', () => this.removeController());
cleanup('RL 策略', () => this.removeRLPolicy());
cleanup('移动操作环境', () => this.mobile?.dispose());
cleanup('外力状态', () => this.clearExternalForce());
cleanup('扰动对象', () => this.perturb.delete());
cleanup('仿真数据', () => this.data.delete());
+11
View File
@@ -0,0 +1,11 @@
/** Deliberately repeated triangles: parser tests must preserve every face. */
export function binaryStl(faces: number): Uint8Array {
const data = new Uint8Array(84 + 50 * faces);
const v = new DataView(data.buffer);
v.setUint32(80, faces, true);
for (let i = 0; i < faces; i++) {
v.setFloat32(84 + 50 * i + 24, 1, true);
v.setFloat32(84 + 50 * i + 40, 1, true);
}
return data;
}
@@ -4,6 +4,21 @@ import { LocalTrainingClient } from './LocalTrainingClient';
afterEach(() => vi.unstubAllGlobals());
describe('LocalTrainingClient', () => {
it('移动场景二进制上传与部署元数据均携带认证,作业请求只传服务器快照ID', async () => {
const fetchMock = vi.fn().mockImplementation(() => Promise.resolve(new Response('{}')));
vi.stubGlobal('fetch', fetchMock);
const client = new LocalTrainingClient('http://localhost:8765', 'secret');
const file = new File(['scene'], 'scene.zip');
await client.uploadMobileScene(file);
await client.downloadMobileDeployment('a'.repeat(32));
expect(fetchMock.mock.calls[0][0]).toContain('/api/training/mobile-packages');
const options = fetchMock.mock.calls[0][1] as RequestInit;
expect(options.body).toBe(file);
expect(new Headers(options.headers).get('Content-Type')).toBe('application/zip');
expect(new Headers(options.headers).get('Authorization')).toBe('Bearer secret');
expect(fetchMock.mock.calls[1][0]).toContain('/artifacts/deployment.json');
expect(() => client.uploadMobileScene(new File([], 'empty.zip'))).toThrow('128 MiB');
});
it('规范化服务地址并提交受类型约束的 JSON 请求', async () => {
const fetchMock = vi.fn().mockResolvedValue(
new Response(JSON.stringify({ id: 'a'.repeat(32), state: 'queued' }), {
@@ -1,3 +1,4 @@
import type { MobileDeployment } from '../mobile/training';
import type {
ParameterConstraint,
PretrainedSource,
@@ -79,6 +80,20 @@ export class LocalTrainingClient {
});
}
start(request: TrainingRequest): Promise<TrainingJob> {
return this.startJob(request);
}
uploadMobileScene(file: File): Promise<{ id: string; robotId: string; sceneSha256: string }> {
if (!file.size || file.size > 128 * 1024 ** 2) throw new Error('场景快照上限 128 MiB');
return this.json('/api/training/mobile-packages', {
method: 'POST',
headers: { 'Content-Type': 'application/zip' },
body: file,
});
}
downloadMobileDeployment(id: string): Promise<MobileDeployment> {
return this.json(`/api/training/jobs/${encodeURIComponent(id)}/artifacts/deployment.json`);
}
startJob(request: TrainingRequest): Promise<TrainingJob> {
return this.json('/api/training/jobs', {
method: 'POST',
headers: { 'Content-Type': 'application/json' },

Some files were not shown because too many files have changed in this diff Show More