feat: release v1.0.1 CADWorld 网站与 LeKiwi 智能抓放
web-platform-ci / Standalone decision service (no cloud credentials) (push) Has been cancelled
web-platform-ci / TypeScript, lint, unit, build (push) Has been cancelled
web-platform-ci / Playwright E2E (push) Has been cancelled
lekiwi-compatibility / cpu-compatibility (push) Has been cancelled
web-platform-ci / Standalone decision service (no cloud credentials) (pull_request) Has been cancelled
web-platform-ci / TypeScript, lint, unit, build (pull_request) Has been cancelled
web-platform-ci / Playwright E2E (pull_request) Has been cancelled
lekiwi-compatibility / cpu-compatibility (pull_request) Has been cancelled
web-platform-ci / Standalone decision service (no cloud credentials) (push) Has been cancelled
web-platform-ci / TypeScript, lint, unit, build (push) Has been cancelled
web-platform-ci / Playwright E2E (push) Has been cancelled
lekiwi-compatibility / cpu-compatibility (push) Has been cancelled
web-platform-ci / Standalone decision service (no cloud credentials) (pull_request) Has been cancelled
web-platform-ci / TypeScript, lint, unit, build (pull_request) Has been cancelled
web-platform-ci / Playwright E2E (pull_request) Has been cancelled
lekiwi-compatibility / cpu-compatibility (pull_request) Has been cancelled
集成同源 BYOK 会话隔离、精简模型设置、官方订阅入口和 HTTPS 发布运维;保留本地训练/调参与控制能力。同步 npm 版本及 CHANGELOG,记录公网真实 API 验收仍待用户凭据。
This commit is contained in:
@@ -10,6 +10,7 @@ src/
|
||||
├── components/ui/ 无业务依赖的共享 UI 基础组件
|
||||
├── controller/ Python 控制器运行时、类型和面板
|
||||
├── map/ 地图加载、生成、编辑和面板
|
||||
├── mobile/ 移动操作 RL 底层;agent/ 为独立语义契约、物理技能与任务 UI
|
||||
├── project/ 工程导入、转换、文件树和工作区
|
||||
├── rl/ ONNX 策略运行时、任务绑定、类型和面板
|
||||
├── robot/ 通用机器人契约、profile、实时协议与外部控制面板
|
||||
@@ -42,6 +43,20 @@ SimulationSession snapshot → app → viewer / 各业务面板
|
||||
|
||||
主工作台由 `index.html → src/main.tsx` 启动;自调参工作台由 Vite MPA 入口 `tuning.html → src/tuning/main.tsx` 启动,避免把 MuJoCo/Three.js 主应用依赖打入监控页面。两页仅通过训练 HTTP API和严格同源的短消息交接训练服务凭据/策略导入请求,不在 URL 中传 token。
|
||||
|
||||
## Website 产品边界
|
||||
|
||||
`vite --mode website` 使用 `app/websiteConfig.ts` 关闭云端首版的训练/调参/外控入口,构建到 `website-dist/`,不包含 `tuning.html`;默认 local 构建和底层算法保留。`WebsiteModelSettings` 仅在展开设置后建立同源匿名会话,`DecisionClient.website()` 自动携带 Cookie/CSRF/配置版本,配置与模型请求仍遵守旧的任务失效机制。API_KEY 与设备码不写浏览器持久化。
|
||||
|
||||
后端 `web_server/web_sessions/web_config/model_catalog` 分离公网边界、逐用户内存状态、固定上游配置及模型目录,复用原 `Service` 的有界请求、契约和物理控制解耦。订阅由逐会话 `CodexAccount` 提供官方设备码登录,不共用进程级账号、不放开任意 CLI 执行。部署与验证见 `docs/website-deployment.md`。
|
||||
|
||||
## LeKiwi 智能任务
|
||||
|
||||
主工作台控制台同时承载 RL 与 `mobile/agent/AgentTaskPanel`;独立 `mobile.html` 构建入口已移除,`tuning.html` 和物理夹具保留。`App.loadEntry()` 仍负责同一适配器/查看器的事务与回滚;智能任务只添加场景配置,不另建 RobotManager。任务场景不应用/清空地图草稿,RL 切回时重新编译 RL 场景。
|
||||
|
||||
`AgentTaskController` → gated 本地技能 → 同一个 `SimulationSession.advance()`;模型 Promise 只写邮箱,票据限定同步物理窗口。任务级暂停可继续,全局暂停/外力拖动/地图编辑/模型重载撤销旧请求。完整 CAD 软件渲染时 agent 仅降低绘制频率,空闲帧仍调用同一个 advance,不新建时钟、不减面。
|
||||
|
||||
`DecisionClient` 仅访问本机 `decision_server`;模型 key 只在服务内存,页面临时输入保存后清空,令牌不进入持久存储/任务证据。模型连接、官方 Codex 登录均无隐式付费来源回退。算法、网络和认证 UI 位于各自业务模块,不嵌入 App 的加载算法。验收边界见 [智能抓放](../docs/lekiwi-agent.md)。
|
||||
|
||||
## 机器人接口边界
|
||||
|
||||
`robot/types + validation` 定义跨语言契约,`robot_profiles/` 是唯一参数源;可信 `RobotAdapter` 封装物理模型,`ModelBindings` 为机器人/Python/Go2 共用索引。`ControlArbiter` 在 Session 边界授予身份票据,所有写入都经过所有者检查,不依赖 UI 灰化。
|
||||
|
||||
@@ -1,4 +1,17 @@
|
||||
# 自定义训练与前视射线避障
|
||||
# 统一强化学习训练控制台
|
||||
|
||||
## LeKiwi / LeKiwi Bundle
|
||||
|
||||
1. 主工作台导入机器人 ZIP,在 URDF 选项选对应的「移动操作训练场景」。也可先加载旧 LeKiwi v1 profile,再从控制台选择任务;开始训练会自动加载抓取/搬运场景。
|
||||
2. 启动配置了 `--mobile-python` 的训练服务([独立依赖配置](../training_server/README.md#lekiwi-一键训练与-go2-共用控制台)),在「控制台 → 强化学习任务」连接。面板按当前机器人选择任务,也允许手动切换。
|
||||
3. 默认先选「底盘接近」,配置示教初始化步数(0 可禁用)、位置随机化、独立评估回合,再配置迭代数、环境数(1–64)、每环境采样步数、设备、种子及物体/目标坐标。控制周期固定0.02s,Go2 地形/射线/基础策略/自调参配置隐藏。点击「发起本地训练」自动同步场景并创建后台作业,**不再下载训练包**。
|
||||
4. 共享日志、Loss/平均奖励曲线、进度与停止按钮。完成后点击「导入策略」自动拉取 ONNX 与 deployment,核验形状、任务、变体及 SHA-256,在当前主视口开始推断。暂停/重置/移除策略沿用主控制台。
|
||||
|
||||
5. 根据独立评估决定续训:点击「接续此作业(保留权重)」可同阶段继续。至少10回合、≥80%成功率、无安全终止才允许依次升级到末端接近、抓取放置;场景资产与已评估初态分布必须匹配。未达标策略可以调试导入,但不是成品策略。控制采用 v2 限速增量目标,旧68维 ONNX 会拒绝加载。详见 [训练课程与实测结果](../docs/mobile-training-curriculum.md)。
|
||||
|
||||
两种变体不能互用策略。移动操作任务使用专用物体/目标/地面,不使用 Go2 地图编辑配置;总采样步数为 `迭代 × 环境数 × 每环境采样步数`。物理在 CPU、PPO 网络可选 CPU/单 GPU;短训练只验证运行链路,不保证抓取收敛。独立 `/mobile.html` 产品页已移除;保留主工作台 RL、`tuning.html` 与 `/physics/mobile.html` 夹具。新增的 [LeKiwi 智能抓放](../docs/lekiwi-agent.md) 使用独立任务契约,切回训练会重建 RL 场景,不把双支撑台任务作为原训练快照。普通作业状态在服务重启后不保留,与现有 Go2 一致。
|
||||
|
||||
## Go2 自定义训练与前视射线避障
|
||||
|
||||
1. 启动仓库本地训练服务,在「控制台 → 强化学习任务」输入令牌并连接。
|
||||
2. 选择「前视射线避障导航」,设置地形、种子、障碍物参数、FOV、探测/安全距离及避障奖励权重,发起训练。界面显示进度、最近价值/策略/熵损失和原始日志。
|
||||
|
||||
@@ -0,0 +1,86 @@
|
||||
import { spawn } from 'node:child_process';
|
||||
import { randomBytes } from 'node:crypto';
|
||||
import { mkdtemp, rm } from 'node:fs/promises';
|
||||
import { tmpdir } from 'node:os';
|
||||
import { join, resolve } from 'node:path';
|
||||
import { stopProcess } from './controlBridge';
|
||||
|
||||
/** Explicit live-test helper: keys stay in Python memory, only the ephemeral bridge
|
||||
* token crosses to the browser. Tighten (never bypass) production admission limits. */
|
||||
export async function startLiveDecisionService() {
|
||||
const token = randomBytes(32).toString('hex');
|
||||
const directory = await mkdtemp(join(tmpdir(), 'lekiwi-live-decision-'));
|
||||
const script = `
|
||||
import asyncio, json, os, signal, sys
|
||||
from pathlib import Path
|
||||
from aiohttp import web
|
||||
from decision_server.server import create_app, STATE
|
||||
from decision_server.credentials import deepseek_llm, openrouter_jev
|
||||
from decision_server.protocol import DecisionError
|
||||
async def main():
|
||||
app = create_app(Path(sys.argv[1]), os.environ.pop('DECISION_SMOKE_TOKEN'), origins={'http://127.0.0.1:4176'}, port=0)
|
||||
service = app[STATE]
|
||||
service.connections.values['llm'] = deepseek_llm(Path('.env'))
|
||||
service.connections.values['jev'] = openrouter_jev(Path('.env'))
|
||||
original = service.admit
|
||||
counts = {'llm': 0, 'jev': 0}
|
||||
def admit(role, stamp):
|
||||
if counts[role] >= {'llm': 1, 'jev': 12}[role]:
|
||||
raise DecisionError('live_smoke_budget', 429)
|
||||
original(role, stamp)
|
||||
counts[role] += 1
|
||||
service.admit = admit
|
||||
runner = web.AppRunner(app, access_log=None, handler_cancellation=True)
|
||||
await runner.setup()
|
||||
site = web.TCPSite(runner, '127.0.0.1', 0)
|
||||
await site.start()
|
||||
port = site._server.sockets[0].getsockname()[1]
|
||||
service.hosts = {f'127.0.0.1:{port}'}
|
||||
stop = asyncio.Event()
|
||||
for sig in (signal.SIGTERM, signal.SIGINT):
|
||||
asyncio.get_running_loop().add_signal_handler(sig, stop.set)
|
||||
print('DECISION_READY ' + json.dumps({'endpoint': f'http://127.0.0.1:{port}'}), flush=True)
|
||||
try:
|
||||
await stop.wait()
|
||||
finally:
|
||||
await runner.cleanup()
|
||||
asyncio.run(main())
|
||||
`;
|
||||
const child = spawn(resolve('.venv/bin/python'), ['-u', '-c', script, directory], {
|
||||
env: { ...process.env, DECISION_SMOKE_TOKEN: token, PYTHONNOUSERSITE: '1', PYTHONPATH: '' },
|
||||
stdio: ['ignore', 'pipe', 'ignore'],
|
||||
});
|
||||
const stop = async () => {
|
||||
if (child.pid) await stopProcess(child);
|
||||
await rm(directory, { recursive: true, force: true });
|
||||
};
|
||||
try {
|
||||
const endpoint = await new Promise<string>((resolve, reject) => {
|
||||
let output = '';
|
||||
const timer = setTimeout(
|
||||
() => reject(new Error('真实模型服务启动超时(不输出认证日志)')),
|
||||
15000,
|
||||
);
|
||||
child.once('error', () => {
|
||||
clearTimeout(timer);
|
||||
reject(new Error('真实模型服务启动失败'));
|
||||
});
|
||||
child.once('exit', (code) => {
|
||||
clearTimeout(timer);
|
||||
reject(new Error(`真实模型服务退出:${code}`));
|
||||
});
|
||||
child.stdout.on('data', (chunk: Buffer) => {
|
||||
output = (output + chunk.toString()).slice(-8192);
|
||||
const match = output.match(/DECISION_READY (\{[^\n]+\})/);
|
||||
if (match) {
|
||||
clearTimeout(timer);
|
||||
resolve((JSON.parse(match[1]) as { endpoint: string }).endpoint);
|
||||
}
|
||||
});
|
||||
});
|
||||
return { endpoint, token, stop };
|
||||
} catch (error) {
|
||||
await stop();
|
||||
throw error;
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,43 @@
|
||||
import { createConnection } from 'node:net';
|
||||
import { readFile } from 'node:fs/promises';
|
||||
|
||||
/** Same-user 0600 Unix channel. Never print the connect response or persist its token. */
|
||||
export async function subscriptionOperation(operation: 'connect' | 'status' | 'limits' | 'stop') {
|
||||
const pointer = JSON.parse(
|
||||
await readFile('build/lekiwi-agent/subscription-session.json', 'utf8'),
|
||||
) as { socket?: string; status: string };
|
||||
if (!pointer.socket || pointer.status === 'closed')
|
||||
throw new Error('隔离订阅会话已关闭,请重新发起官方登录');
|
||||
return new Promise<Record<string, unknown>>((resolve, reject) => {
|
||||
const channel = createConnection(pointer.socket!);
|
||||
let buffer = '';
|
||||
channel.setTimeout(20000);
|
||||
channel.once('connect', () => channel.write(JSON.stringify(operation) + '\n'));
|
||||
channel.on('data', (chunk) => {
|
||||
buffer += String(chunk);
|
||||
if (buffer.length > 16384) {
|
||||
channel.destroy();
|
||||
reject(new Error('订阅管理响应过大'));
|
||||
return;
|
||||
}
|
||||
if (buffer.includes('\n')) {
|
||||
try {
|
||||
const result = JSON.parse(buffer.split('\n')[0]) as Record<string, unknown>;
|
||||
if (result.error) reject(new Error('订阅管理请求失败'));
|
||||
else resolve(result);
|
||||
} catch {
|
||||
reject(new Error('订阅管理响应非法'));
|
||||
}
|
||||
channel.end();
|
||||
}
|
||||
});
|
||||
channel.once('timeout', () => {
|
||||
channel.destroy();
|
||||
reject(new Error('订阅管理请求超时'));
|
||||
});
|
||||
channel.once('error', () => reject(new Error('无法连接隔离订阅会话')));
|
||||
channel.once('end', () => {
|
||||
if (!buffer.includes('\n')) reject(new Error('订阅管理连接提前关闭'));
|
||||
});
|
||||
});
|
||||
}
|
||||
@@ -0,0 +1,104 @@
|
||||
import { test, expect } from '@playwright/test';
|
||||
import { resolve } from 'node:path';
|
||||
import { writeFile } from 'node:fs/promises';
|
||||
import type {} from '../physics/agent';
|
||||
|
||||
for (const fault of [
|
||||
'empty-grasp',
|
||||
'empty-once',
|
||||
'alignment',
|
||||
'slip',
|
||||
'jam',
|
||||
'navigation',
|
||||
'unreachable',
|
||||
'overspeed',
|
||||
] as const) {
|
||||
test(`真实 WASM 故障:${fault} 不可冒充成功/无限恢复`, async ({ page }, info) => {
|
||||
await page.goto('/physics/agent.html');
|
||||
await page.waitForFunction(() => Boolean(window.lekiwiAgent));
|
||||
await page.evaluate(
|
||||
(url) => window.lekiwiAgent.boot(url),
|
||||
`/@fs${resolve('build/lekiwi/lekiwi-v1.zip')}`,
|
||||
);
|
||||
try {
|
||||
const r = await page.evaluate(
|
||||
(fault) => window.lekiwiAgent.task(0, undefined, { fault }),
|
||||
fault,
|
||||
);
|
||||
await writeFile(info.outputPath('fault-evidence.json'), JSON.stringify(r, null, 2));
|
||||
console.log('WASM fault', {
|
||||
fault,
|
||||
state: r.state,
|
||||
error: r.error,
|
||||
failures: r.failures,
|
||||
recoveries: r.recoveries,
|
||||
simTime: r.simTime,
|
||||
});
|
||||
expect(r.mode).toBe('mock');
|
||||
expect(r.state).toBe(fault === 'empty-once' ? 'succeeded' : 'failed');
|
||||
expect(r.succeeded).toBe(fault === 'empty-once');
|
||||
expect(r.simTime).toBeLessThanOrEqual(70.01);
|
||||
expect(r.llmCalls).toBe(1);
|
||||
if (fault === 'empty-grasp' || fault === 'empty-once' || fault === 'alignment') {
|
||||
const skill = fault === 'alignment' ? 'pregrasp' : 'close';
|
||||
const code = fault === 'alignment' ? 'alignment_timeout' : 'empty_grasp';
|
||||
const recovered = fault === 'empty-once';
|
||||
expect(r.recoveries[skill]).toBe(recovered ? 1 : 2);
|
||||
expect(r.failures.map((f) => f.code)).toEqual(Array(recovered ? 1 : 3).fill(code));
|
||||
if (!recovered) expect(r.trace.every((s) => !s.secure)).toBe(true);
|
||||
else {
|
||||
expect(r.transported).toBeGreaterThanOrEqual(0.5);
|
||||
expect(r.sample.onGoalSupport).toBe(true);
|
||||
}
|
||||
} else {
|
||||
expect(r.recoveries).toEqual({});
|
||||
expect(r.failures).toHaveLength(1);
|
||||
expect(r.failures[0].code).toBe(
|
||||
{
|
||||
slip: 'lost_grasp',
|
||||
jam: 'joint_stall',
|
||||
navigation: 'navigation_timeout',
|
||||
unreachable: 'unreachable',
|
||||
overspeed: 'joint_velocity',
|
||||
}[fault],
|
||||
);
|
||||
}
|
||||
if (fault === 'slip') {
|
||||
expect(Math.max(...r.trace.map((s) => s.transported))).toBeGreaterThan(0.12);
|
||||
expect(r.transported).toBe(0);
|
||||
}
|
||||
} finally {
|
||||
await page.evaluate(() => window.lekiwiAgent.dispose());
|
||||
}
|
||||
});
|
||||
}
|
||||
|
||||
for (const goal of [
|
||||
[0.237, 0.565, 0.128],
|
||||
[0.277, 0.665, 0.128],
|
||||
]) {
|
||||
test(`真实 WASM 受限目标角点 ${goal.slice(0, 2).join(',')}(非任意目标泛化)`, async ({
|
||||
page,
|
||||
}, info) => {
|
||||
await page.goto('/physics/agent.html');
|
||||
await page.waitForFunction(() => Boolean(window.lekiwiAgent));
|
||||
await page.evaluate(
|
||||
(url) => window.lekiwiAgent.boot(url),
|
||||
`/@fs${resolve('build/lekiwi/lekiwi-v1.zip')}`,
|
||||
);
|
||||
try {
|
||||
const r = await page.evaluate(
|
||||
(goal) => window.lekiwiAgent.task(0, undefined, { goal }),
|
||||
goal,
|
||||
);
|
||||
await writeFile(info.outputPath('goal-evidence.json'), JSON.stringify(r, null, 2));
|
||||
expect(r.state, r.error).toBe('succeeded');
|
||||
expect(r.goal).toEqual(goal);
|
||||
expect(r.succeeded).toBe(true);
|
||||
expect(r.transported).toBeGreaterThanOrEqual(0.5);
|
||||
expect(Math.hypot(...r.sample.object.map((v, i) => v - goal[i]))).toBeLessThan(0.04);
|
||||
} finally {
|
||||
await page.evaluate(() => window.lekiwiAgent.dispose());
|
||||
}
|
||||
});
|
||||
}
|
||||
@@ -0,0 +1,46 @@
|
||||
import { test, expect } from '@playwright/test';
|
||||
import { resolve } from 'node:path';
|
||||
import { writeFile } from 'node:fs/promises';
|
||||
import type {} from '../physics/agent';
|
||||
test('8 次 A 场景与 IK scratch 加载/释放:验证原生句柄,不以 heap 高水位判泄漏', async ({
|
||||
page,
|
||||
}, info) => {
|
||||
await page.goto('/physics/agent.html');
|
||||
await page.waitForFunction(() => Boolean(window.lekiwiAgent));
|
||||
await page.evaluate(
|
||||
(url) => window.lekiwiAgent.boot(url),
|
||||
`/@fs${resolve('build/lekiwi/lekiwi-v1.zip')}`,
|
||||
);
|
||||
const rows = await page.evaluate(() => window.lekiwiAgent.resources());
|
||||
await writeFile(info.outputPath('resource-evidence.json'), JSON.stringify(rows, null, 2));
|
||||
expect(rows).toHaveLength(8);
|
||||
expect(rows.every((row) => row.deleted.length === 6 && row.deleted.every(Boolean))).toBe(true);
|
||||
console.log(
|
||||
'A + scratch heap capacity MiB',
|
||||
rows.map((row) => row.heapBytes / 1024 / 1024),
|
||||
);
|
||||
});
|
||||
for (const action of ['pause-task', 'pause-global', 'stop', 'force', 'reload'] as const) {
|
||||
test(`真实 WASM 等待规划时 ${action}:迟到结果不写控制/不步进`, async ({ page }, info) => {
|
||||
await page.goto('/physics/agent.html');
|
||||
await page.waitForFunction(() => Boolean(window.lekiwiAgent));
|
||||
await page.evaluate(
|
||||
(url) => window.lekiwiAgent.boot(url),
|
||||
`/@fs${resolve('build/lekiwi/lekiwi-v1.zip')}`,
|
||||
);
|
||||
try {
|
||||
const result = await page.evaluate((action) => window.lekiwiAgent.lifecycle(action), action);
|
||||
await writeFile(
|
||||
info.outputPath('late-response-evidence.json'),
|
||||
JSON.stringify(result, null, 2),
|
||||
);
|
||||
expect(result.before).toEqual(result.after);
|
||||
expect(result.before.time).toBe(0);
|
||||
expect(result.jevCalls).toBe(0);
|
||||
expect(result.state).toBe(action === 'pause-task' ? 'paused' : 'cancelled');
|
||||
expect(result.owner).toBe(action === 'pause-task' ? 'agent' : 'manual');
|
||||
} finally {
|
||||
await page.evaluate(() => window.lekiwiAgent.dispose());
|
||||
}
|
||||
});
|
||||
}
|
||||
@@ -0,0 +1,79 @@
|
||||
import { expect, test } from '@playwright/test';
|
||||
import { resolve } from 'node:path';
|
||||
import { readFile, writeFile } from 'node:fs/promises';
|
||||
import { startLiveDecisionService } from './fixtures/decisionService';
|
||||
|
||||
test.use({ trace: 'off', video: 'off' });
|
||||
test('显式 opt-in:主工作台 DeepSeek flash + OpenRouter Jev 真实物理回合', async ({
|
||||
page,
|
||||
}, info) => {
|
||||
test.skip(
|
||||
process.env.LEKIWI_LIVE_API !== '1',
|
||||
'需明确授权 .env 两个角色与一次付费回合;不自动运行',
|
||||
);
|
||||
const service = await startLiveDecisionService();
|
||||
try {
|
||||
await page.goto('/');
|
||||
await page
|
||||
.locator('input[type="file"]')
|
||||
.first()
|
||||
.setInputFiles(resolve('build/lekiwi/lekiwi-v1.zip'));
|
||||
const dialog = page.getByRole('dialog', { name: '配置 URDF 仿真组件' });
|
||||
await expect(dialog).toBeVisible({ timeout: 30000 });
|
||||
await dialog.getByLabel('URDF 机器人 profile').selectOption('lekiwi-v1');
|
||||
await dialog.getByRole('button', { name: '转换并加载' }).click();
|
||||
await expect(page.getByRole('button', { name: '▶ 播放' })).toBeEnabled({ timeout: 60000 });
|
||||
await page.getByRole('tab', { name: '控制台', exact: true }).click();
|
||||
await page.getByRole('button', { name: 'LeKiwi 智能抓放', exact: true }).click();
|
||||
const panel = page.getByLabel('LeKiwi 智能抓放任务', { exact: true });
|
||||
await panel.getByText('模型连接与官方订阅登录', { exact: true }).click();
|
||||
await panel.getByLabel('本机模型服务', { exact: true }).fill(service.endpoint);
|
||||
await panel.getByLabel('本机服务令牌', { exact: true }).fill(service.token);
|
||||
await panel.getByRole('button', { name: '连接本机模型服务', exact: true }).click();
|
||||
await expect(panel).toContainText('本机桥已连接');
|
||||
await expect(panel.getByLabel('LLM 型号', { exact: true })).toHaveValue('deepseek-flash');
|
||||
await expect(panel.getByLabel('JEV 型号', { exact: true })).toHaveValue('typesafe/jev-1.13');
|
||||
await expect(panel.getByLabel('LLM Base URL', { exact: true })).toHaveValue(
|
||||
'https://api.deepseek.com',
|
||||
);
|
||||
await expect(panel.getByLabel('JEV Base URL', { exact: true })).toHaveValue(
|
||||
'https://openrouter.ai/api/alpha/decisions',
|
||||
);
|
||||
await panel.getByText('模型连接与官方订阅登录', { exact: true }).click();
|
||||
await panel.getByLabel('智能任务决策模式').selectOption('api');
|
||||
await panel.getByRole('button', { name: '开始任务(真实 API)', exact: true }).click();
|
||||
const status = panel.getByTestId('agent-task-state');
|
||||
await expect(status).toContainText(/物理验收成功|失败 \/ 安全停止|已取消/, { timeout: 240000 });
|
||||
const pendingDownload = page.waitForEvent('download');
|
||||
await panel.getByRole('button', { name: '导出任务证据', exact: true }).click();
|
||||
const downloaded = await pendingDownload;
|
||||
const data = JSON.parse(await readFile((await downloaded.path())!, 'utf8'));
|
||||
const encoded = JSON.stringify(data, null, 2);
|
||||
if (encoded.includes(service.token)) throw new Error('任务导出包含本机令牌');
|
||||
await writeFile(info.outputPath('live-physical-evidence.json'), encoded);
|
||||
console.log('真实 API 回合(非订阅)', {
|
||||
state: data.state,
|
||||
error: data.error,
|
||||
llm: data.llmCalls,
|
||||
jev: data.jevCalls,
|
||||
simTime: data.sample.time,
|
||||
transported: data.transported,
|
||||
models: [...new Set(data.receipts.map((r: { model: string }) => r.model))],
|
||||
});
|
||||
expect(data.mode).toBe('api');
|
||||
expect(data.state, data.error).toBe('succeeded');
|
||||
expect(data.succeeded).toBe(true);
|
||||
expect(data.transported).toBeGreaterThanOrEqual(0.5);
|
||||
expect(data.llmCalls).toBe(1);
|
||||
expect(data.jevCalls).toBe(11);
|
||||
expect(data.receipts).toHaveLength(12);
|
||||
await page.screenshot({ path: info.outputPath('live-workbench.png') });
|
||||
} catch (error) {
|
||||
// Playwright action errors may quote input values. Close before automatic DOM diagnostics.
|
||||
await page.close().catch(() => undefined);
|
||||
// eslint-disable-next-line preserve-caught-error -- The original cause may contain the private bridge token.
|
||||
throw new Error(String(error).replaceAll(service.token, '[redacted]'));
|
||||
} finally {
|
||||
await service.stop();
|
||||
}
|
||||
});
|
||||
@@ -0,0 +1,103 @@
|
||||
import { test, expect } from '@playwright/test';
|
||||
import { resolve } from 'node:path';
|
||||
import { writeFile } from 'node:fs/promises';
|
||||
import preset from '../../contracts/lekiwi-pick-place-v1.json' with { type: 'json' };
|
||||
import type {} from '../physics/agent';
|
||||
|
||||
for (const seed of preset.frozenSeeds) {
|
||||
test(`规则物理基线:冻结种子 ${seed.id},双指抓持/搬运/释放/撤离`, async ({ page }, info) => {
|
||||
const errors: string[] = [];
|
||||
page.on('pageerror', (error) => errors.push(error.message));
|
||||
await page.goto('/physics/agent.html');
|
||||
await page.waitForFunction(() => Boolean(window.lekiwiAgent));
|
||||
await page.evaluate(
|
||||
(url) => window.lekiwiAgent.boot(url),
|
||||
`/@fs${resolve('build/lekiwi/lekiwi-v1.zip')}`,
|
||||
);
|
||||
try {
|
||||
const result = await page.evaluate((seed) => window.lekiwiAgent.baseline(seed), seed.id);
|
||||
await writeFile(info.outputPath('physical-evidence.json'), JSON.stringify(result, null, 2));
|
||||
console.log('baseline', { ...result, trace: `${result.trace.length} control samples` });
|
||||
expect(result.status, result.error).toBe('succeeded');
|
||||
expect(result.succeeded).toBe(true);
|
||||
expect(result.transported).toBeGreaterThanOrEqual(preset.transportMin);
|
||||
expect(result.maxJointSpeed).toBeLessThanOrEqual(preset.jointSpeedStop);
|
||||
const goal = preset.goal.position.map((v, i) => v + (seed.offset[i] ?? 0));
|
||||
expect(Math.hypot(...result.final.object.map((v, i) => v - goal[i]))).toBeLessThan(
|
||||
preset.placementTolerance,
|
||||
);
|
||||
expect(result.final.objectSpeed).toBeLessThan(preset.placementSpeedMax);
|
||||
expect(result.final.onGoalSupport).toBe(true);
|
||||
expect(result.final.fingerForces.every((force) => force < preset.fingerForceMin)).toBe(true);
|
||||
expect(result.final.opening).toBeGreaterThan(0.65);
|
||||
expect(
|
||||
Math.hypot(...result.final.object.map((v, i) => v - result.final.tcp[i])),
|
||||
).toBeGreaterThanOrEqual(preset.retreatMin);
|
||||
expect(
|
||||
result.trace.filter((s) => s.phase === 'carry' && s.secure && !s.supported).length,
|
||||
).toBeGreaterThan(100);
|
||||
expect(errors).toEqual([]);
|
||||
} finally {
|
||||
await page.evaluate(() => window.lekiwiAgent.dispose());
|
||||
}
|
||||
});
|
||||
}
|
||||
|
||||
for (const seed of preset.frozenSeeds) {
|
||||
test(`mock 决策闭环:冻结种子 ${seed.id},逐技能门禁与独立物理成功`, async ({ page }, info) => {
|
||||
await page.goto('/physics/agent.html');
|
||||
await page.waitForFunction(() => Boolean(window.lekiwiAgent));
|
||||
await page.evaluate(
|
||||
(url) => window.lekiwiAgent.boot(url),
|
||||
`/@fs${resolve('build/lekiwi/lekiwi-v1.zip')}`,
|
||||
);
|
||||
try {
|
||||
const result = await page.evaluate((id) => window.lekiwiAgent.task(id), seed.id);
|
||||
await writeFile(info.outputPath('mock-closed-loop.json'), JSON.stringify(result, null, 2));
|
||||
console.log('mock closed loop', {
|
||||
state: result.state,
|
||||
error: result.error,
|
||||
transported: result.transported,
|
||||
llm: result.llmCalls,
|
||||
jev: result.jevCalls,
|
||||
simTime: result.simTime,
|
||||
});
|
||||
expect(result.ownerAtStart).toBe('agent');
|
||||
expect(result.mode).toBe('mock');
|
||||
expect(result.state, result.error).toBe('succeeded');
|
||||
expect(result.succeeded).toBe(true);
|
||||
expect(result.transported).toBeGreaterThanOrEqual(preset.transportMin);
|
||||
expect(result.maxJointSpeed).toBeLessThanOrEqual(preset.jointSpeedStop);
|
||||
expect(result.llmCalls).toBe(1);
|
||||
expect(result.jevCalls).toBe(11);
|
||||
expect(result.results).toHaveLength(11);
|
||||
expect(result.failures).toEqual([]);
|
||||
expect(result.sample.onGoalSupport).toBe(true);
|
||||
expect(result.sample.fingerForces.every((v) => v < preset.fingerForceMin)).toBe(true);
|
||||
expect(result.sample.opening).toBeGreaterThanOrEqual(0.65);
|
||||
expect(
|
||||
Math.hypot(...result.sample.object.map((v, i) => v - result.sample.tcp[i])),
|
||||
).toBeGreaterThanOrEqual(preset.retreatMin);
|
||||
} finally {
|
||||
await page.evaluate(() => window.lekiwiAgent.dispose());
|
||||
}
|
||||
});
|
||||
}
|
||||
|
||||
test('agent 控制权下同一物理时钟完成基线,不建立第二个环境循环', async ({ page }, info) => {
|
||||
await page.goto('/physics/agent.html');
|
||||
await page.waitForFunction(() => Boolean(window.lekiwiAgent));
|
||||
await page.evaluate(
|
||||
(url) => window.lekiwiAgent.boot(url),
|
||||
`/@fs${resolve('build/lekiwi/lekiwi-v1.zip')}`,
|
||||
);
|
||||
try {
|
||||
const result = await page.evaluate(() => window.lekiwiAgent.baseline(0, true));
|
||||
await writeFile(info.outputPath('governed-baseline.json'), JSON.stringify(result, null, 2));
|
||||
expect(result.ownerAtStart).toBe('agent');
|
||||
expect(result.status, result.error).toBe('succeeded');
|
||||
expect(result.transported).toBeGreaterThanOrEqual(0.5);
|
||||
} finally {
|
||||
await page.evaluate(() => window.lekiwiAgent.dispose());
|
||||
}
|
||||
});
|
||||
@@ -0,0 +1,103 @@
|
||||
import { expect, test } from '@playwright/test';
|
||||
import { resolve } from 'node:path';
|
||||
import { readFile, writeFile } from 'node:fs/promises';
|
||||
import { subscriptionOperation } from './fixtures/subscriptionSession';
|
||||
|
||||
test.use({ trace: 'off', video: 'off' });
|
||||
test('显式 opt-in:已人工登录的 ChatGPT 订阅 + Jev 主视口物理验收与退出', async ({
|
||||
page,
|
||||
}, info) => {
|
||||
test.skip(
|
||||
process.env.LEKIWI_LIVE_SUBSCRIPTION !== '1',
|
||||
'需要用户在隔离官方页面完成登录;不自动发起登录或使用全局账号',
|
||||
);
|
||||
const session = await subscriptionOperation('connect');
|
||||
const token = String(session.token);
|
||||
try {
|
||||
expect(session.llmProtocol).toBe('codex');
|
||||
const before = await subscriptionOperation('status');
|
||||
expect(before.loggedIn).toBe(true);
|
||||
const limitsBefore = await subscriptionOperation('limits');
|
||||
await page.goto('/');
|
||||
await page
|
||||
.locator('input[type="file"]')
|
||||
.first()
|
||||
.setInputFiles(resolve('build/lekiwi/lekiwi-v1.zip'));
|
||||
const dialog = page.getByRole('dialog', { name: '配置 URDF 仿真组件' });
|
||||
await expect(dialog).toBeVisible({ timeout: 30000 });
|
||||
await dialog.getByLabel('URDF 机器人 profile').selectOption('lekiwi-v1');
|
||||
await dialog.getByRole('button', { name: '转换并加载' }).click();
|
||||
await expect(page.getByRole('button', { name: '▶ 播放' })).toBeEnabled({ timeout: 60000 });
|
||||
await page.getByRole('tab', { name: '控制台', exact: true }).click();
|
||||
await page.getByRole('button', { name: 'LeKiwi 智能抓放', exact: true }).click();
|
||||
const panel = page.getByLabel('LeKiwi 智能抓放任务', { exact: true });
|
||||
await panel.getByText('模型连接与官方订阅登录', { exact: true }).click();
|
||||
await panel.getByLabel('本机模型服务', { exact: true }).fill(String(session.address));
|
||||
await panel.getByLabel('本机服务令牌', { exact: true }).fill(token);
|
||||
await panel.getByRole('button', { name: '连接本机模型服务', exact: true }).click();
|
||||
await expect(panel).toContainText('本机桥已连接');
|
||||
await expect(panel.getByLabel('LLM 协议', { exact: true })).toHaveValue('codex');
|
||||
await expect(panel.getByLabel('LLM API Key', { exact: true })).toHaveCount(0);
|
||||
await panel.getByText(/ChatGPT 订阅 · 官方/).click();
|
||||
await panel.getByRole('button', { name: '刷新账号', exact: true }).click();
|
||||
await expect(panel).toContainText('ChatGPT 已登录(仅本次服务会话)');
|
||||
await panel.getByRole('button', { name: '查询官方型号', exact: true }).click();
|
||||
await expect(panel.getByLabel('LLM 型号', { exact: true })).toHaveValue(
|
||||
String(session.llmModel),
|
||||
);
|
||||
await expect(panel.getByLabel('JEV 型号', { exact: true })).toHaveValue('typesafe/jev-1.13');
|
||||
await panel.getByText('模型连接与官方订阅登录', { exact: true }).click();
|
||||
await panel.getByLabel('智能任务决策模式').selectOption('api');
|
||||
await panel.getByRole('button', { name: '开始任务(真实 API)', exact: true }).click();
|
||||
await expect(panel.getByTestId('agent-task-state')).toContainText(
|
||||
/物理验收成功|失败 \/ 安全停止|已取消/,
|
||||
{ timeout: 240000 },
|
||||
);
|
||||
const download = page.waitForEvent('download');
|
||||
await panel.getByRole('button', { name: '导出任务证据', exact: true }).click();
|
||||
const data = JSON.parse(await readFile((await (await download).path())!, 'utf8'));
|
||||
const after = await subscriptionOperation('status');
|
||||
const evidence = {
|
||||
source: 'real-chatgpt-subscription-plus-openrouter-jev',
|
||||
before,
|
||||
after,
|
||||
limitsBefore,
|
||||
...data,
|
||||
};
|
||||
const encoded = JSON.stringify(evidence, null, 2);
|
||||
if (encoded.includes(token) || encoded.includes('auth.openai.com/'))
|
||||
throw new Error('任务证据包含认证信息');
|
||||
await writeFile(info.outputPath('subscription-physical-evidence.json'), encoded);
|
||||
console.log('真实订阅回合', {
|
||||
state: data.state,
|
||||
error: data.error,
|
||||
model: session.llmModel,
|
||||
llm: data.llmCalls,
|
||||
jev: data.jevCalls,
|
||||
simTime: data.sample.time,
|
||||
transported: data.transported,
|
||||
});
|
||||
expect(data.state, data.error).toBe('succeeded');
|
||||
expect(data.succeeded).toBe(true);
|
||||
expect(data.transported).toBeGreaterThanOrEqual(0.5);
|
||||
expect(data.llmCalls).toBe(1);
|
||||
expect(data.jevCalls).toBe(11);
|
||||
expect(data.receipts).toHaveLength(12);
|
||||
expect(data.receipts.find((r: { kind: string }) => r.kind === 'plan').provider).toBe('codex');
|
||||
expect(after.checkedModels).toContain(session.llmModel);
|
||||
await page.screenshot({ path: info.outputPath('subscription-workbench.png') });
|
||||
await panel.getByText('模型连接与官方订阅登录', { exact: true }).click();
|
||||
await panel.getByRole('button', { name: '退出隔离账号', exact: true }).click();
|
||||
await expect(panel).toContainText('已退出本次隔离会话');
|
||||
const loggedOut = await subscriptionOperation('status');
|
||||
expect(loggedOut.loggedIn).toBe(false);
|
||||
await writeFile(
|
||||
info.outputPath('subscription-logout.json'),
|
||||
JSON.stringify(loggedOut, null, 2),
|
||||
);
|
||||
} catch (error) {
|
||||
await page.close().catch(() => undefined);
|
||||
// eslint-disable-next-line preserve-caught-error -- Playwright causes can contain the private bridge token.
|
||||
throw new Error(String(error).replaceAll(token, '[redacted]'));
|
||||
}
|
||||
});
|
||||
@@ -0,0 +1,70 @@
|
||||
import { expect, test } from '@playwright/test';
|
||||
import { resolve } from 'node:path';
|
||||
import { readFile, writeFile } from 'node:fs/promises';
|
||||
|
||||
test('主工作台:A 导入、mock 单技能暂停/继续、物理成功及证据导出', async ({ page }, info) => {
|
||||
const errors: string[] = [];
|
||||
let modelRequests = 0;
|
||||
page.on('pageerror', (e) => errors.push(e.message));
|
||||
page.on('request', (r) => {
|
||||
if (/\/api\/decision\/v1\/(plan|decide)$/.test(r.url())) modelRequests++;
|
||||
});
|
||||
await page.goto('/');
|
||||
await page
|
||||
.locator('input[type="file"]')
|
||||
.first()
|
||||
.setInputFiles(resolve('build/lekiwi/lekiwi-v1.zip'));
|
||||
const dialog = page.getByRole('dialog', { name: '配置 URDF 仿真组件' });
|
||||
await expect(dialog).toBeVisible({ timeout: 30000 });
|
||||
await dialog.getByLabel('URDF 机器人 profile').selectOption('lekiwi-v1');
|
||||
await dialog.getByRole('button', { name: '转换并加载' }).click();
|
||||
await expect(page.getByRole('button', { name: '▶ 播放' })).toBeEnabled({ timeout: 60000 });
|
||||
await page.getByRole('tab', { name: '控制台', exact: true }).click();
|
||||
await page.getByRole('button', { name: 'LeKiwi 智能抓放', exact: true }).click();
|
||||
const panel = page.getByLabel('LeKiwi 智能抓放任务', { exact: true });
|
||||
await expect(panel).toContainText('MOCK(不调用模型 API)');
|
||||
await panel.getByRole('button', { name: '加载 A 任务场景', exact: true }).click();
|
||||
await expect(panel).toContainText('A 抓放场景已加载并暂停', { timeout: 60000 });
|
||||
await panel.getByRole('button', { name: '单技能步进', exact: true }).click();
|
||||
const status = panel.getByTestId('agent-task-state');
|
||||
await expect(status).toContainText('任务已暂停', { timeout: 60000 });
|
||||
await expect(status).toContainText('当前技能:open');
|
||||
await expect(status).toContainText('Jev 1/60');
|
||||
await panel.getByRole('button', { name: '继续任务', exact: true }).click();
|
||||
await expect(status).toContainText('物理验收成功', { timeout: 180000 });
|
||||
await expect(status).toContainText('Jev 11/60');
|
||||
const download = page.waitForEvent('download');
|
||||
await panel.getByRole('button', { name: '导出任务证据', exact: true }).click();
|
||||
const downloaded = await download;
|
||||
const evidence = JSON.parse(await readFile((await downloaded.path())!, 'utf8'));
|
||||
await writeFile(
|
||||
info.outputPath('workbench-physical-evidence.json'),
|
||||
JSON.stringify(evidence, null, 2),
|
||||
);
|
||||
expect(evidence.mode).toBe('mock');
|
||||
expect(evidence.succeeded).toBe(true);
|
||||
expect(evidence.transported).toBeGreaterThanOrEqual(0.5);
|
||||
expect(evidence.sample.onGoalSupport).toBe(true);
|
||||
expect(modelRequests).toBe(0);
|
||||
expect(errors).toEqual([]);
|
||||
await page.screenshot({ path: info.outputPath('agent-workbench.png') });
|
||||
await panel.getByRole('button', { name: '重置任务场景', exact: true }).click();
|
||||
await expect(panel.getByTestId('agent-task-state')).toHaveCount(0, { timeout: 60000 });
|
||||
await panel.getByLabel('目标 Y', { exact: true }).fill('0.617');
|
||||
await panel.getByRole('button', { name: '单技能步进', exact: true }).click();
|
||||
await expect(status).toContainText('任务已暂停', { timeout: 60000 });
|
||||
const changedDownload = page.waitForEvent('download');
|
||||
await panel.getByRole('button', { name: '导出任务证据', exact: true }).click();
|
||||
const changed = JSON.parse(await readFile((await (await changedDownload).path())!, 'utf8'));
|
||||
expect(changed.goal).toEqual([0.257, 0.617, 0.128]);
|
||||
expect(changed.runId).not.toBe(evidence.runId);
|
||||
expect(changed.sample.time).toBeLessThan(2.01);
|
||||
expect(changed.sample.object[1]).toBeLessThan(0.02); // New source episode, not the old placed cube.
|
||||
await writeFile(info.outputPath('edited-goal-episode.json'), JSON.stringify(changed, null, 2));
|
||||
await panel.getByLabel('目标 Y', { exact: true }).fill('0.615');
|
||||
await expect(status).toContainText('已取消');
|
||||
await expect(status).toContainText('目标编辑使旧计划失效');
|
||||
expect(modelRequests).toBe(0);
|
||||
expect(errors).toEqual([]);
|
||||
await expect(page.getByRole('button', { name: '强化学习任务', exact: true })).toBeVisible();
|
||||
});
|
||||
@@ -0,0 +1,76 @@
|
||||
import { test, expect } from '@playwright/test';
|
||||
import { resolve } from 'node:path';
|
||||
import { writeFile } from 'node:fs/promises';
|
||||
|
||||
test('完整 CAD 主视口:60 秒渲染/物理性能记录(不承诺固定 FPS)', async ({ page }, info) => {
|
||||
const errors: string[] = [];
|
||||
page.on('pageerror', (error) => errors.push(error.message));
|
||||
await page.goto('/');
|
||||
const started = performance.now();
|
||||
await page
|
||||
.getByLabel('打开文件', { exact: true })
|
||||
.setInputFiles(resolve('build/lekiwi/lekiwi-v1.zip'));
|
||||
const dialog = page.getByRole('dialog', { name: '配置 URDF 仿真组件' });
|
||||
await expect(dialog).toBeVisible({ timeout: 30000 });
|
||||
await dialog.getByLabel('URDF 机器人 profile').selectOption('lekiwi-v1');
|
||||
await dialog.getByRole('button', { name: '转换并加载' }).click();
|
||||
await expect(page.getByRole('button', { name: '▶ 播放' })).toBeEnabled({ timeout: 60000 });
|
||||
const loadMs = performance.now() - started;
|
||||
await page.getByRole('button', { name: '▶ 播放' }).click();
|
||||
const result = await page.evaluate(async () => {
|
||||
const path = '/src/stores/useAppStore.ts';
|
||||
const { useAppStore } = (await import(
|
||||
/* @vite-ignore */ path
|
||||
)) as typeof import('../src/stores/useAppStore');
|
||||
const canvas = document.querySelector('main canvas') as HTMLCanvasElement;
|
||||
const gl = canvas.getContext('webgl2')!,
|
||||
ext = gl.getExtension('WEBGL_debug_renderer_info');
|
||||
const graphics = {
|
||||
size: [canvas.width, canvas.height],
|
||||
renderer: ext ? String(gl.getParameter(ext.UNMASKED_RENDERER_WEBGL)) : 'unknown',
|
||||
};
|
||||
const start = performance.now(),
|
||||
frames: number[] = [],
|
||||
steps: number[] = [];
|
||||
const longTasks: number[] = [];
|
||||
const observer = new PerformanceObserver((list) => {
|
||||
longTasks.push(...list.getEntries().map((e) => e.duration));
|
||||
});
|
||||
observer.observe({ type: 'longtask' });
|
||||
let previous = start;
|
||||
await new Promise<void>((resolve) => {
|
||||
const tick = (now: number) => {
|
||||
frames.push(now - previous);
|
||||
previous = now;
|
||||
steps.push(useAppStore.getState().stepMs);
|
||||
if (now - start >= 60000) resolve();
|
||||
else requestAnimationFrame(tick);
|
||||
};
|
||||
requestAnimationFrame(tick);
|
||||
});
|
||||
observer.disconnect();
|
||||
const p95 = (values: number[]) =>
|
||||
[...values].sort((a, b) => a - b)[Math.floor(values.length * 0.95)] ?? 0;
|
||||
const state = useAppStore.getState();
|
||||
return {
|
||||
graphics,
|
||||
frames: frames.length,
|
||||
fps: (1000 * frames.length) / (previous - start),
|
||||
frameP95Ms: p95(frames),
|
||||
stepP95Ms: p95(steps),
|
||||
longTasks,
|
||||
heap: state.snapshot?.wasmHeapBytes,
|
||||
simTime: state.snapshot?.time,
|
||||
diagnostic: state.diagnostic?.summary,
|
||||
};
|
||||
});
|
||||
await writeFile(
|
||||
info.outputPath('full-cad-performance.json'),
|
||||
JSON.stringify({ loadMs, ...result }, null, 2),
|
||||
);
|
||||
console.log('full CAD performance', { loadMs, ...result });
|
||||
await page.screenshot({ path: info.outputPath('full-cad.png') });
|
||||
expect(errors).toEqual([]);
|
||||
expect(result.diagnostic).toBeUndefined();
|
||||
expect(result.simTime).toBeGreaterThan(0);
|
||||
});
|
||||
@@ -0,0 +1,58 @@
|
||||
import { expect, test } from '@playwright/test';
|
||||
import { resolve } from 'node:path';
|
||||
import { writeFile } from 'node:fs/promises';
|
||||
import type {} from '../physics/runner';
|
||||
|
||||
test('完整 314244 面三轮:真实 WASM、随轴运动、ZIP 重导与有界资源', async ({ page }, info) => {
|
||||
const errors: string[] = [];
|
||||
page.on('pageerror', (error) => errors.push(error.message));
|
||||
await page.goto('/physics/runner.html');
|
||||
await page.waitForFunction(() => Boolean(window.lekiwiPhysics));
|
||||
await page.evaluate((path) => window.lekiwiPhysics.boot(path), `/@fs${resolve('build/lekiwi')}`);
|
||||
try {
|
||||
const before = await page.evaluate(() => window.lekiwiPhysics.visualStats());
|
||||
// The three original filenames contain identical bytes; one content-addressed
|
||||
// OBJ is shared by three independent visual instances.
|
||||
expect(before.generatedMeshes).toHaveLength(1);
|
||||
expect(before.wheels).toHaveLength(3);
|
||||
for (const wheel of before.wheels) {
|
||||
expect(wheel.faces).toBe(314244);
|
||||
expect(wheel.vertices).toBeGreaterThan(150000);
|
||||
expect(wheel.centerError).toBeLessThan(1e-6);
|
||||
expect(wheel.scale).toEqual([0.001, 0.001, 0.001]);
|
||||
expect(wheel.collision).toBe(false);
|
||||
expect(wheel.mass).toBe(0);
|
||||
}
|
||||
await page.evaluate(() => window.lekiwiPhysics.drive(0, 0.1, 0, 0.25));
|
||||
const after = await page.evaluate(() => window.lekiwiPhysics.visualStats());
|
||||
for (let i = 0; i < 3; i++) {
|
||||
expect(after.wheels[i].centerError).toBeLessThan(1e-6);
|
||||
expect(
|
||||
after.wheels[i].rotation.some((v, j) => Math.abs(v - before.wheels[i].rotation[j]) > 0.01),
|
||||
).toBe(true);
|
||||
}
|
||||
const traces: Awaited<ReturnType<typeof window.lekiwiPhysics.packageRoundtrip>>[] = [];
|
||||
for (let i = 0; i < 8; i++) {
|
||||
const result = await page.evaluate(() => window.lekiwiPhysics.packageRoundtrip());
|
||||
expect(result.oldModelDeleted).toBe(true);
|
||||
expect(result.oldDataDeleted).toBe(true);
|
||||
expect(result.workspaces).toHaveLength(1);
|
||||
expect(result.generatedMeshes).toHaveLength(1);
|
||||
expect(result.wheels.every((wheel) => wheel.faces === 314244 && !wheel.collision)).toBe(true);
|
||||
traces.push(result);
|
||||
}
|
||||
console.log(
|
||||
'mesh resources',
|
||||
traces.map((t) => ({ heap: t.heap, bytes: t.projectBytes, xml: t.xmlFiles })),
|
||||
);
|
||||
await writeFile(
|
||||
info.outputPath('full-mesh-roundtrip.json'),
|
||||
JSON.stringify({ before, after, traces }, null, 2),
|
||||
);
|
||||
expect(traces.every((t) => t.projectBytes === traces[0].projectBytes)).toBe(true);
|
||||
expect(traces[7].heap).toBeLessThanOrEqual(traces[5].heap + 32 * 1024 * 1024);
|
||||
expect(errors).toEqual([]);
|
||||
} finally {
|
||||
await page.evaluate(() => window.lekiwiPhysics.dispose());
|
||||
}
|
||||
});
|
||||
@@ -0,0 +1,187 @@
|
||||
import { expect, test } from '@playwright/test';
|
||||
import { spawn } from 'node:child_process';
|
||||
import { createServer } from 'node:net';
|
||||
import { existsSync } from 'node:fs';
|
||||
import { resolve } from 'node:path';
|
||||
import { mkdir, readFile, readdir, writeFile } from 'node:fs/promises';
|
||||
|
||||
const mobilePython = process.env.MOBILE_TRAINING_PYTHON ?? resolve('build/venvs/mobile/bin/python');
|
||||
const bundle =
|
||||
process.env.MOBILE_BUNDLE_ZIP ??
|
||||
resolve('../Reference_Projects/LeKiwi/New_urdf/robot_urdf_bundle (1).zip');
|
||||
for (const [taskId, file] of [
|
||||
['MobileManipulator-LeKiwi-v1', resolve('build/lekiwi/lekiwi-v1.zip')],
|
||||
['MobileManipulator-LeKiwi-Bundle', bundle],
|
||||
]) {
|
||||
test(`主控制台真实一键训练及策略导入:${taskId}`, async ({ page }, info) => {
|
||||
test.skip(
|
||||
!existsSync(mobilePython) || !existsSync(file),
|
||||
'需要 MuJoCo 3.11/SB3 独立环境及机器人资产',
|
||||
);
|
||||
const port = await new Promise<number>((resolvePort) => {
|
||||
const listener = createServer();
|
||||
listener.listen(0, '127.0.0.1', () => {
|
||||
const port = (listener.address() as { port: number }).port;
|
||||
listener.close(() => resolvePort(port));
|
||||
});
|
||||
});
|
||||
const root = info.outputPath('training');
|
||||
await mkdir(root, { recursive: true });
|
||||
const token = 'oneclick-test-token';
|
||||
const endpoint = `http://127.0.0.1:${port}`;
|
||||
const process = spawn(
|
||||
resolve('.venv/bin/python'),
|
||||
[
|
||||
'-u',
|
||||
'training_server/server.py',
|
||||
'--port',
|
||||
String(port),
|
||||
'--trainer-root',
|
||||
root,
|
||||
'--mobile-python',
|
||||
mobilePython,
|
||||
'--token',
|
||||
token,
|
||||
],
|
||||
{ stdio: ['ignore', 'pipe', 'pipe'] },
|
||||
);
|
||||
let output = '';
|
||||
process.stdout.on('data', (data) => {
|
||||
output += String(data);
|
||||
});
|
||||
process.stderr.on('data', (data) => {
|
||||
output += String(data);
|
||||
});
|
||||
const errors: string[] = [];
|
||||
page.on('pageerror', (error) => errors.push(error.message));
|
||||
try {
|
||||
await expect
|
||||
.poll(
|
||||
async () => {
|
||||
try {
|
||||
return (
|
||||
await fetch(`${endpoint}/api/training/health`, {
|
||||
headers: { Authorization: `Bearer ${token}` },
|
||||
})
|
||||
).ok;
|
||||
} catch {
|
||||
return false;
|
||||
}
|
||||
},
|
||||
{ timeout: 30000 },
|
||||
)
|
||||
.toBe(true);
|
||||
await page.goto('/');
|
||||
await page.locator('input[type="file"]').first().setInputFiles(file);
|
||||
const dialog = page.getByRole('dialog', { name: '配置 URDF 仿真组件' });
|
||||
await expect(dialog).toBeVisible({ timeout: 30000 });
|
||||
// v1 also covers migration from the existing external-control profile at start.
|
||||
await dialog
|
||||
.getByLabel('URDF 机器人 profile')
|
||||
.selectOption(taskId === 'MobileManipulator-LeKiwi-v1' ? 'lekiwi-v1' : taskId);
|
||||
await dialog.getByRole('button', { name: '转换并加载' }).click();
|
||||
await expect(page.getByRole('button', { name: '▶ 播放' })).toBeEnabled({ timeout: 60000 });
|
||||
await page.getByRole('tab', { name: '控制台', exact: true }).click();
|
||||
if (taskId === 'MobileManipulator-LeKiwi-v1') {
|
||||
await page.getByRole('button', { name: 'LeKiwi 智能抓放', exact: true }).click();
|
||||
const agent = page.getByLabel('LeKiwi 智能抓放任务', { exact: true });
|
||||
await agent.getByRole('button', { name: '单技能步进', exact: true }).click();
|
||||
await expect(agent.getByTestId('agent-task-state')).toContainText('任务已暂停', {
|
||||
timeout: 60000,
|
||||
});
|
||||
}
|
||||
await page.getByRole('button', { name: '强化学习任务', exact: true }).click();
|
||||
await page.getByLabel('本地训练服务地址').fill(endpoint);
|
||||
await page.getByLabel('训练服务访问令牌').fill(token);
|
||||
await page.getByRole('button', { name: '连接', exact: true }).click();
|
||||
await expect(page.getByLabel('训练任务')).toHaveValue(taskId);
|
||||
await page.getByLabel('训练迭代', { exact: true }).fill('2');
|
||||
await page.getByLabel('每环境采样步数').fill('8');
|
||||
await expect(page.getByLabel('移动操作训练阶段')).toHaveValue('navigate');
|
||||
await page.getByLabel('独立评估回合').fill('2');
|
||||
await page.getByLabel('导航启动示教步数(仅初训)').fill('0');
|
||||
const jobResponse = page.waitForResponse(
|
||||
(response) =>
|
||||
response.url() === `${endpoint}/api/training/jobs` &&
|
||||
response.request().method() === 'POST',
|
||||
);
|
||||
await page.getByRole('button', { name: '发起本地训练' }).click();
|
||||
const created = (await (await jobResponse).json()) as { id: string };
|
||||
await expect(page.getByRole('button', { name: '导入策略', exact: true })).toBeEnabled({
|
||||
timeout: 60000,
|
||||
});
|
||||
await page.getByRole('button', { name: '导入策略', exact: true }).click();
|
||||
await expect
|
||||
.poll(
|
||||
async () =>
|
||||
page.evaluate(async () => {
|
||||
const path = '/src/stores/useAppStore.ts';
|
||||
const { useAppStore } = (await import(
|
||||
/* @vite-ignore */ path
|
||||
)) as typeof import('../src/stores/useAppStore');
|
||||
return useAppStore.getState().snapshot?.rlPolicy?.inferenceCount ?? 0;
|
||||
}),
|
||||
{ timeout: 30000 },
|
||||
)
|
||||
.toBeGreaterThan(0);
|
||||
const job = await (
|
||||
await fetch(`${endpoint}/api/training/jobs/${created.id}`, {
|
||||
headers: { Authorization: `Bearer ${token}` },
|
||||
})
|
||||
).json();
|
||||
expect(job.state).toBe('succeeded');
|
||||
expect(job.logs.join('\n')).toContain('Mean reward:');
|
||||
expect(job.deployment.input.shape).toEqual([1, 92]);
|
||||
expect(job.deployment.trainingStage).toBe('navigate');
|
||||
expect(job.deployment.evaluation.episodes).toBe(2);
|
||||
expect(job.deployment.evaluation.safetyStops).toBe(0);
|
||||
expect(job.deployment.evaluation.maxJointVelocity).toBeLessThan(2);
|
||||
expect(job.deployment.output.shape).toEqual([1, 12]);
|
||||
if (taskId === 'MobileManipulator-LeKiwi-v1') {
|
||||
const scenes = (await readdir(root, { recursive: true })).filter((name) =>
|
||||
name.endsWith('.training.xml'),
|
||||
);
|
||||
expect(scenes.length).toBeGreaterThan(0);
|
||||
for (const scene of scenes)
|
||||
expect(await readFile(resolve(root, scene), 'utf8')).not.toContain('__agent_');
|
||||
await expect(page.getByTestId('agent-task-state')).toContainText('已取消');
|
||||
}
|
||||
await writeFile(info.outputPath('job.json'), JSON.stringify(job, null, 2));
|
||||
await page.screenshot({ path: info.outputPath('unified-training.png') });
|
||||
expect(errors).toEqual([]);
|
||||
if (taskId === 'MobileManipulator-LeKiwi-v1') {
|
||||
await page.getByRole('button', { name: '接续此作业(保留权重)' }).click();
|
||||
await expect(page.getByLabel('接续作业 ID')).toHaveValue(created.id);
|
||||
const resumedResponse = page.waitForResponse(
|
||||
(response) =>
|
||||
response.url() === `${endpoint}/api/training/jobs` &&
|
||||
response.request().method() === 'POST',
|
||||
);
|
||||
await page.getByRole('button', { name: '发起本地训练' }).click();
|
||||
const response = await resumedResponse;
|
||||
expect(response.status()).toBe(202);
|
||||
const resumed = await response.json();
|
||||
await expect(page.getByRole('button', { name: '导入策略', exact: true })).toBeEnabled({
|
||||
timeout: 60000,
|
||||
});
|
||||
const second = await (
|
||||
await fetch(`${endpoint}/api/training/jobs/${resumed.id}`, {
|
||||
headers: { Authorization: `Bearer ${token}` },
|
||||
})
|
||||
).json();
|
||||
expect(second.state).toBe('succeeded');
|
||||
expect(second.deployment.initialization).toBe('resumed-checkpoint');
|
||||
expect(second.deployment.trainedTimesteps).toBeGreaterThan(job.deployment.trainedTimesteps);
|
||||
await writeFile(info.outputPath('resumed-job.json'), JSON.stringify(second, null, 2));
|
||||
}
|
||||
} finally {
|
||||
await page.close();
|
||||
process.kill('SIGTERM');
|
||||
await new Promise<void>((resolveExit) => {
|
||||
if (process.exitCode !== null) resolveExit();
|
||||
else process.once('exit', () => resolveExit());
|
||||
});
|
||||
await writeFile(info.outputPath('server.log'), output.replaceAll(token, '<redacted>'));
|
||||
}
|
||||
});
|
||||
}
|
||||
@@ -0,0 +1,137 @@
|
||||
import { expect, test } from '@playwright/test';
|
||||
import { createHash } from 'node:crypto';
|
||||
import { readFile, writeFile } from 'node:fs/promises';
|
||||
import { readdirSync } from 'node:fs';
|
||||
import { resolve } from 'node:path';
|
||||
import type {} from '../physics/runner';
|
||||
import type { ProjectManifest } from '../src/project/types';
|
||||
|
||||
const assetBase = `/@fs${resolve(process.cwd(), 'build/lekiwi')}`;
|
||||
const hash = (text: string) => createHash('sha256').update(text).digest('hex');
|
||||
|
||||
test('等价 URDF 变体:真实 WASM 控制、输入溯源及 MJCF 导出重载', async ({ page }, info) => {
|
||||
const original = await readFile('robot_profiles/lekiwi-reference.urdf', 'utf8');
|
||||
const variant = original
|
||||
.replace('name="LeKiwi"', 'name="LeKiwi 换色变体"')
|
||||
.replace('</visual>', '<material name="red"><color rgba="1 0 0 1"/></material></visual>');
|
||||
await page.route('**/URDF/LeKiwi.urdf', (route) =>
|
||||
route.fulfill({ body: variant, contentType: 'application/xml' }),
|
||||
);
|
||||
await page.goto('/physics/runner.html');
|
||||
await page.waitForFunction(() => Boolean(window.lekiwiPhysics));
|
||||
const boot = await page.evaluate((path) => window.lekiwiPhysics.boot(path), assetBase);
|
||||
expect(boot.nu).toBe(9);
|
||||
expect(boot.njnt).toBe(46);
|
||||
const stable = await page.evaluate(() => window.lekiwiPhysics.step(3));
|
||||
expect(stable.finite).toBe(true);
|
||||
const drive = await page.evaluate(() => window.lekiwiPhysics.drive(0.1, 0, 0, 2));
|
||||
expect(drive.position[0] - stable.position[0]).toBeGreaterThan(0.15);
|
||||
const xml = await page.evaluate(() => window.lekiwiPhysics.xml());
|
||||
expect(xml).toContain(`name="platform_robot_source_sha256" data="${hash(original)}"`);
|
||||
expect(xml).toContain(`name="platform_robot_input_sha256" data="${hash(variant)}"`);
|
||||
const reloaded = await page.evaluate(() => window.lekiwiPhysics.reimport());
|
||||
expect(reloaded.robot?.profileId).toBe('lekiwi-v1');
|
||||
expect(reloaded.nu).toBe(9);
|
||||
expect(await page.evaluate(() => window.lekiwiPhysics.xml())).toContain(hash(variant));
|
||||
await writeFile(info.outputPath('variant.xml'), xml);
|
||||
await writeFile(
|
||||
info.outputPath('variant-control.json'),
|
||||
JSON.stringify({ boot, stable, drive, reloaded }, null, 2),
|
||||
);
|
||||
await page.evaluate(() => window.lekiwiPhysics.dispose());
|
||||
});
|
||||
|
||||
test('同名底盘 STL 被改动也拒绝旧 profile,不只校验机械臂网格', async ({ page }) => {
|
||||
await page.route('**/URDF/meshes/base_plate_layer1-v5.stl', (route) =>
|
||||
route.fulfill({ body: 'changed mesh bytes' }),
|
||||
);
|
||||
await page.goto('/physics/runner.html');
|
||||
await page.waitForFunction(() => Boolean(window.lekiwiPhysics));
|
||||
await expect(page.evaluate((path) => window.lekiwiPhysics.boot(path), assetBase)).rejects.toThrow(
|
||||
/SHA-256 不匹配.*base_plate_layer1/,
|
||||
);
|
||||
await page.evaluate(() => window.lekiwiPhysics.dispose());
|
||||
});
|
||||
|
||||
// Optional local evidence: assets stay outside the repository. Setting the env
|
||||
// explicitly makes missing/invalid samples fail, rather than silently skipping.
|
||||
const localDirectory = process.env.LEKIWI_VARIANT_DIR;
|
||||
const localSamples = localDirectory
|
||||
? readdirSync(localDirectory)
|
||||
.filter((file) => file.endsWith('.zip'))
|
||||
.sort()
|
||||
: [];
|
||||
if (localDirectory && !localSamples.length) throw new Error('LEKIWI_VARIANT_DIR 中没有 ZIP 样本');
|
||||
for (const name of localSamples) {
|
||||
test(`用户变体包诊断(不适配旧控制):${name}`, async ({ page }, info) => {
|
||||
await page.route('**/__local_variant.zip', (route) =>
|
||||
route.fulfill({ path: resolve(localDirectory!, name), contentType: 'application/zip' }),
|
||||
);
|
||||
await page.goto('/physics/runner.html');
|
||||
const result = await page.evaluate(async () => {
|
||||
const importerPath = '/src/project/importer.ts';
|
||||
const profilesPath = '/src/project/robotProfiles/index.ts';
|
||||
const physicsPath = '/src/simulation/PhysicsAdapter.ts';
|
||||
const registryPath = '/src/robot/registry.ts';
|
||||
const { importBrowserFiles, prepareProjectForMujoco } = await import(
|
||||
/* @vite-ignore */ importerPath
|
||||
);
|
||||
const { prepareRobotProject } = await import(/* @vite-ignore */ profilesPath);
|
||||
const { MainThreadPhysicsAdapter } = await import(/* @vite-ignore */ physicsPath);
|
||||
const { sha256 } = await import(/* @vite-ignore */ registryPath);
|
||||
const bytes = await (await fetch('/__local_variant.zip')).arrayBuffer();
|
||||
const manifest: ProjectManifest = await importBrowserFiles([
|
||||
new File([bytes], 'variant.zip'),
|
||||
]);
|
||||
const entry = manifest.selectedEntry!;
|
||||
const data = manifest.files.find((file) => file.path === entry)!.data;
|
||||
const sourceHash = await sha256(data);
|
||||
const doc = new DOMParser().parseFromString(
|
||||
new TextDecoder().decode(data),
|
||||
'application/xml',
|
||||
);
|
||||
const movable = Array.from(doc.querySelectorAll('robot > joint')).filter(
|
||||
(j) => j.getAttribute('type') !== 'fixed',
|
||||
);
|
||||
let profileError = '';
|
||||
try {
|
||||
await prepareRobotProject(manifest, entry, 'lekiwi-v1');
|
||||
} catch (error) {
|
||||
profileError = String(error);
|
||||
}
|
||||
const prepared = await prepareProjectForMujoco(manifest, entry);
|
||||
const fixedText = new TextDecoder().decode(
|
||||
prepared.manifest.files.find((file: { path: string }) => file.path === entry).data,
|
||||
);
|
||||
const adapter = new MainThreadPhysicsAdapter();
|
||||
let genericCompileError = '';
|
||||
try {
|
||||
await adapter.load(manifest, entry, { urdfMode: 'mjcf', baseMode: 'floating' });
|
||||
} catch (error) {
|
||||
genericCompileError = String(error);
|
||||
} finally {
|
||||
adapter.dispose();
|
||||
}
|
||||
return {
|
||||
entry,
|
||||
sourceHash,
|
||||
profileError,
|
||||
genericCompileError,
|
||||
links: doc.querySelectorAll('robot > link').length,
|
||||
movable: movable.map((j) => j.getAttribute('name')),
|
||||
warnings: prepared.warnings,
|
||||
unresolvedPackageUri: fixedText.includes('package://'),
|
||||
sourceUnmodified: (await sha256(data)) === sourceHash,
|
||||
};
|
||||
});
|
||||
expect(result.links).toBe(35);
|
||||
expect(result.movable).toHaveLength(10);
|
||||
expect(result.profileError).toContain('需要 9 个活动关节,实际 10 个');
|
||||
expect(result.profileError).toContain('Link7');
|
||||
expect(result.unresolvedPackageUri).toBe(false);
|
||||
expect(result.sourceUnmodified).toBe(true);
|
||||
// This is an import repair, not an unreviewed wheel-mesh simplification.
|
||||
expect(result.genericCompileError).toMatch(/STL|faces/i);
|
||||
await writeFile(info.outputPath('local-variant-audit.json'), JSON.stringify(result, null, 2));
|
||||
});
|
||||
}
|
||||
@@ -0,0 +1,181 @@
|
||||
import { expect, test, type Page } from '@playwright/test';
|
||||
import { existsSync } from 'node:fs';
|
||||
import { mkdir, readFile, writeFile } from 'node:fs/promises';
|
||||
import { resolve } from 'node:path';
|
||||
import type {} from '../physics/mobile';
|
||||
|
||||
const original = resolve('build/lekiwi/lekiwi-v1.zip');
|
||||
const bundle =
|
||||
process.env.MOBILE_BUNDLE_ZIP ??
|
||||
resolve('../Reference_Projects/LeKiwi/New_urdf/robot_urdf_bundle (1).zip');
|
||||
|
||||
async function exportRollout(page: Page, zipName: string, traceName: string) {
|
||||
await page.evaluate(() => window.mobilePhysics.reset());
|
||||
await mkdir('build/mobile-validation', { recursive: true });
|
||||
const downloadPromise = page.waitForEvent('download');
|
||||
await page.evaluate(() => window.mobilePhysics.download());
|
||||
await (await downloadPromise).saveAs(`build/mobile-validation/${zipName}`);
|
||||
const actions = Array.from({ length: 12 }, (_, step) => {
|
||||
const a = new Array<number>(12).fill(0);
|
||||
a[0] = 0.1;
|
||||
a[3] = 0.03 * Math.sin(step);
|
||||
a[11] = 1;
|
||||
return a;
|
||||
});
|
||||
const rollout = [];
|
||||
for (const action of actions)
|
||||
rollout.push(await page.evaluate((a) => window.mobilePhysics.step(1, a), action));
|
||||
await writeFile(
|
||||
`build/mobile-validation/${traceName}`,
|
||||
JSON.stringify({ actions, rollout }, null, 2),
|
||||
);
|
||||
}
|
||||
|
||||
test('真实 WASM:两种机器人、控制、拖动、失败回滚、重复切换与训练包导出', async ({ page }) => {
|
||||
test.skip(!existsSync(original) || !existsSync(bundle), '需要本地两个机器人 ZIP');
|
||||
const errors: string[] = [];
|
||||
page.on('pageerror', (e) => errors.push(e.message));
|
||||
await page.route('**/original.zip', (route) => route.fulfill({ path: original }));
|
||||
await page.route('**/bundle.zip', (route) => route.fulfill({ path: bundle }));
|
||||
await page.goto('/physics/mobile.html');
|
||||
await page.waitForFunction(() => Boolean(window.mobilePhysics));
|
||||
const traces = [];
|
||||
for (const id of [
|
||||
'lekiwi-v1',
|
||||
'lekiwi-bundle',
|
||||
'lekiwi-v1',
|
||||
'lekiwi-bundle',
|
||||
'lekiwi-v1',
|
||||
'lekiwi-bundle',
|
||||
'lekiwi-v1',
|
||||
'lekiwi-bundle',
|
||||
// Complete CAD meshes increase allocator/cache warm-up; exercise extra
|
||||
// cycles without raising the peak-memory bound or weakening final stability.
|
||||
'lekiwi-v1',
|
||||
'lekiwi-bundle',
|
||||
'lekiwi-v1',
|
||||
'lekiwi-bundle',
|
||||
]) {
|
||||
const initial = await page.evaluate(
|
||||
(id) => window.mobilePhysics.boot(id === 'lekiwi-v1' ? '/original.zip' : '/bundle.zip', id),
|
||||
id,
|
||||
);
|
||||
console.log('mobile heap', id, initial.heap / 1048576);
|
||||
expect(initial.heap).toBeLessThan(1600 * 1048576);
|
||||
expect(initial.observation).toHaveLength(92);
|
||||
expect(initial.observation.every(Number.isFinite)).toBe(true);
|
||||
expect(initial.ctrl).toHaveLength(9);
|
||||
expect(initial.retiredModelDeleted).toBe(true);
|
||||
expect(initial.retiredDataDeleted).toBe(true);
|
||||
const stable = await page.evaluate(() => window.mobilePhysics.step(10));
|
||||
expect(stable.time).toBeCloseTo(0.2, 8);
|
||||
expect(stable.qpos.every(Number.isFinite)).toBe(true);
|
||||
const action = new Array<number>(12).fill(0);
|
||||
action[0] = 0.3;
|
||||
action[3] = 0.2;
|
||||
action[11] = 1;
|
||||
const moved = await page.evaluate((a) => window.mobilePhysics.step(10, a, true), action);
|
||||
expect(moved.ctrl.some((v, i) => Math.abs(v - initial.ctrl[i]) > 0.01)).toBe(true);
|
||||
expect(moved.qpos.some((v, i) => Math.abs(v - stable.qpos[i]) > 0.001)).toBe(true);
|
||||
action[0] = 0;
|
||||
action[11] = -1;
|
||||
const closed = await page.evaluate((a) => window.mobilePhysics.step(80, a, true), action);
|
||||
expect(closed.state[29]).toBeLessThan(initial.state[29] - 0.2);
|
||||
expect(closed.qpos.every(Number.isFinite)).toBe(true);
|
||||
expect(await page.evaluate(() => window.mobilePhysics.workspaces())).toHaveLength(1);
|
||||
const edited = await page.evaluate(() => window.mobilePhysics.move('goal', [0.4, -0.2, 0.019]));
|
||||
expect(edited.state.slice(47, 50)).toEqual([0.4, -0.2, 0.019]);
|
||||
expect(edited.stage).toBe('reach');
|
||||
traces.push({ initial, stable, moved, closed, edited });
|
||||
if (traces.length === 1)
|
||||
await exportRollout(page, 'lekiwi-training.zip', 'lekiwi-rollout.json');
|
||||
}
|
||||
const previous = await page.evaluate(() => window.mobilePhysics.state());
|
||||
const failure = await page.evaluate(async () => {
|
||||
try {
|
||||
await window.mobilePhysics.boot('/original.zip', 'lekiwi-bundle');
|
||||
return '';
|
||||
} catch (e) {
|
||||
return String(e);
|
||||
}
|
||||
});
|
||||
expect(failure).toContain('Link1');
|
||||
expect((await page.evaluate(() => window.mobilePhysics.state())).robotId).toBe(previous.robotId);
|
||||
expect(await page.evaluate(() => window.mobilePhysics.workspaces())).toHaveLength(1);
|
||||
const compileFailure = await page.evaluate(async () => {
|
||||
try {
|
||||
await window.mobilePhysics.invalidCandidate('/bundle.zip');
|
||||
return '';
|
||||
} catch (e) {
|
||||
return String(e);
|
||||
}
|
||||
});
|
||||
expect(compileFailure).toContain('missing_actuator');
|
||||
expect(await page.evaluate(() => window.mobilePhysics.workspaces())).toHaveLength(1);
|
||||
expect(
|
||||
(await page.evaluate(() => window.mobilePhysics.step(1))).observation.every(Number.isFinite),
|
||||
).toBe(true);
|
||||
await writeFile(
|
||||
'build/mobile-validation/heap-warmup.json',
|
||||
JSON.stringify(
|
||||
traces.map((t) => ({ robot: t.initial.robotId, heap: t.initial.heap })),
|
||||
null,
|
||||
2,
|
||||
),
|
||||
);
|
||||
expect(traces[11].initial.heap).toBe(traces[9].initial.heap);
|
||||
await exportRollout(page, 'bundle-training.zip', 'rollout.json');
|
||||
await writeFile('build/mobile-validation/traces.json', JSON.stringify(traces, null, 2));
|
||||
await page.evaluate(() => {
|
||||
window.mobilePhysics.reset();
|
||||
window.mobilePhysics.show();
|
||||
});
|
||||
await page.waitForFunction(() => document.querySelector('canvas')!.width > 500);
|
||||
// Use actual camera projection, not hard-coded screenshot pixel coordinates.
|
||||
for (const entity of ['object', 'goal'] as const) {
|
||||
const before = await page.evaluate(() => window.mobilePhysics.state());
|
||||
const p = await page.evaluate((entity) => window.mobilePhysics.screen(entity), entity);
|
||||
await page.keyboard.down('Shift');
|
||||
await page.mouse.move(p.x, p.y);
|
||||
await page.mouse.down();
|
||||
await page.mouse.move(p.x + 45, p.y + 12, { steps: 5 });
|
||||
await page.mouse.up();
|
||||
await page.keyboard.up('Shift');
|
||||
const after = await page.evaluate(() => window.mobilePhysics.state());
|
||||
const offset = entity === 'object' ? 37 : 47;
|
||||
expect(
|
||||
Math.hypot(
|
||||
after.state[offset] - before.state[offset],
|
||||
after.state[offset + 1] - before.state[offset + 1],
|
||||
),
|
||||
).toBeGreaterThan(0.01);
|
||||
}
|
||||
expect(errors).toEqual([]);
|
||||
await page.evaluate(() => window.mobilePhysics.dispose());
|
||||
});
|
||||
|
||||
// 独立产品页已移除;导入/控制/暂停/重置/状态 UI 在主工作台用例验收:
|
||||
// lekiwi.agent.workspace.spec.ts。这里保留两模型物理、训练快照及 ORT 夹具。
|
||||
|
||||
test('真实 ORT WASM:导出的 smoke 权重锁步执行(不代表训练成功)', async ({ page }) => {
|
||||
const model = resolve('build/mobile-validation/smoke.onnx');
|
||||
test.skip(!existsSync(model) || !existsSync(bundle), '先按文档导出 smoke.onnx');
|
||||
const metadata = JSON.parse(await readFile('build/mobile-validation/smoke.json', 'utf8'));
|
||||
await page.route('**/bundle.zip', (route) => route.fulfill({ path: bundle }));
|
||||
await page.route('**/smoke.onnx', (route) => route.fulfill({ path: model }));
|
||||
await page.goto('/physics/mobile.html');
|
||||
await page.waitForFunction(() => Boolean(window.mobilePhysics));
|
||||
await page.evaluate(() => window.mobilePhysics.boot('/bundle.zip', 'lekiwi-bundle'));
|
||||
await page.evaluate((metadata) => window.mobilePhysics.policy('/smoke.onnx', metadata), metadata);
|
||||
await expect
|
||||
.poll(
|
||||
async () => (await page.evaluate(() => window.mobilePhysics.tick(performance.now()))).time,
|
||||
{ timeout: 30_000, intervals: [20] },
|
||||
)
|
||||
.toBeGreaterThanOrEqual(0.1);
|
||||
const status = await page.evaluate(() => window.mobilePhysics.tick(performance.now()));
|
||||
expect(status.metrics?.error).toBe('');
|
||||
expect(status.metrics!.inferenceCount).toBeGreaterThanOrEqual(5);
|
||||
await writeFile('build/mobile-validation/onnx-metrics.json', JSON.stringify(status, null, 2));
|
||||
await page.evaluate(() => window.mobilePhysics.dispose());
|
||||
});
|
||||
@@ -0,0 +1,141 @@
|
||||
import { expect, test } from '@playwright/test';
|
||||
import { resolve } from 'node:path';
|
||||
import { existsSync } from 'node:fs';
|
||||
import { readFile, writeFile } from 'node:fs/promises';
|
||||
|
||||
test.skip(
|
||||
process.env.CADWORLD_PRODUCTION_TEST !== '1',
|
||||
'真实站点检查须显式 opt-in;不使用任何现存密钥',
|
||||
);
|
||||
|
||||
test('生产 HTTPS:模型导入、Pyodide、精简设置、安全会话与订阅不可用边界', async ({
|
||||
page,
|
||||
}, info) => {
|
||||
const errors: string[] = [];
|
||||
const external: string[] = [];
|
||||
page.on('pageerror', (e) => errors.push(e.message));
|
||||
page.on('request', (r) => {
|
||||
if (
|
||||
r.url().startsWith('http') &&
|
||||
new URL(r.url()).origin !== 'https://cadworld-sim.robotquan.com'
|
||||
)
|
||||
external.push(r.url());
|
||||
});
|
||||
const response = await page.goto('/');
|
||||
expect(response?.status()).toBe(200);
|
||||
await expect(page).toHaveTitle(/CADWorld Sim/);
|
||||
await page
|
||||
.locator('input[type=file]')
|
||||
.first()
|
||||
.setInputFiles(resolve('web_platform/fixtures/python_controller/model.xml'));
|
||||
await expect(page.getByRole('button', { name: '▶ 播放' })).toBeEnabled({ timeout: 90000 });
|
||||
await page.getByRole('tab', { name: '控制台', exact: true }).click();
|
||||
await expect(page.getByRole('button', { name: '强化学习任务', exact: true })).toHaveCount(0);
|
||||
await page.getByRole('button', { name: /Python 脚本控制/ }).click();
|
||||
await page
|
||||
.locator('input[accept=".py,text/x-python"]')
|
||||
.setInputFiles(resolve('web_platform/fixtures/python_controller/balance.py'));
|
||||
await expect(page.getByText('倒立摆 PD 平衡控制', { exact: true })).toBeVisible({
|
||||
timeout: 90000,
|
||||
});
|
||||
await page.getByRole('button', { name: '启用', exact: true }).click();
|
||||
await expect(page.getByRole('button', { name: /Python 脚本控制/ })).toContainText('运行');
|
||||
await page.getByRole('button', { name: 'LeKiwi 智能抓放', exact: true }).click();
|
||||
await page.getByText('模型设置 · API_KEY / ChatGPT 订阅', { exact: true }).click();
|
||||
await expect(page.getByRole('button', { name: '保存设置' })).toBeEnabled({ timeout: 20000 });
|
||||
await expect(page.getByLabel('本机服务令牌')).toHaveCount(0);
|
||||
const cookie = (await page.context().cookies()).find((c) => c.name === '__Host-cadworld-session');
|
||||
expect(cookie?.secure).toBe(true);
|
||||
expect(cookie?.httpOnly).toBe(true);
|
||||
expect(cookie?.sameSite).toBe('Strict');
|
||||
await page.screenshot({ path: info.outputPath('website-api-settings.png') });
|
||||
await page.getByRole('button', { name: 'ChatGPT 订阅', exact: true }).click();
|
||||
// No login or model invocation. Availability must be explicit; user's login stays interactive.
|
||||
await expect(page.getByText(/当前服务器可能无法连接官方服务/)).toBeVisible();
|
||||
await expect(page.getByRole('button', { name: '登录 ChatGPT 订阅', exact: true })).toBeVisible();
|
||||
await page.screenshot({ path: info.outputPath('website-subscription-entry.png') });
|
||||
await page.getByText('连接测试与凭据管理', { exact: true }).click();
|
||||
await page.getByRole('button', { name: '清除本次凭据', exact: true }).click();
|
||||
await expect(page.getByText(/本次凭据已清除/)).toBeVisible();
|
||||
expect(errors).toEqual([]);
|
||||
expect(external).toEqual([]);
|
||||
});
|
||||
|
||||
test('官方订阅入口网络探测:不登录账号、不推理,完成后销毁会话', async ({ page }, info) => {
|
||||
await page.goto('/');
|
||||
const evidence = await page.evaluate(async () => {
|
||||
const prefix = '/api/decision/v1';
|
||||
const headers = { 'Content-Type': 'application/json' };
|
||||
const bootstrap = await fetch(prefix + '/session', { method: 'POST', headers, body: '{}' });
|
||||
const session = await bootstrap.json();
|
||||
const protectedHeaders = {
|
||||
...headers,
|
||||
'X-CSRF-Token': session.csrfToken,
|
||||
'X-Config-Version': String(session.configVersion),
|
||||
};
|
||||
const response = await fetch(prefix + '/codex/login', {
|
||||
method: 'POST',
|
||||
headers: protectedHeaders,
|
||||
body: '{}',
|
||||
});
|
||||
const result = await response.json();
|
||||
const cleared = await fetch(prefix + '/session', {
|
||||
method: 'DELETE',
|
||||
headers: protectedHeaders,
|
||||
body: '{}',
|
||||
});
|
||||
return {
|
||||
status: response.status,
|
||||
error: typeof result.error === 'string' ? result.error : undefined,
|
||||
deviceFlowOffered: Boolean(result.verificationUrl && result.userCode),
|
||||
cleared: cleared.status,
|
||||
accountLoginPerformed: false,
|
||||
inferenceCalls: 0,
|
||||
};
|
||||
});
|
||||
expect(evidence.cleared).toBe(200);
|
||||
expect([200, 409, 429, 502, 504]).toContain(evidence.status);
|
||||
if (evidence.status !== 200) expect(evidence.error).toMatch(/^[a-z0-9_]+$/);
|
||||
await writeFile(
|
||||
info.outputPath('subscription-network-status.json'),
|
||||
JSON.stringify(evidence, null, 2),
|
||||
);
|
||||
});
|
||||
|
||||
test('公网真实 WASM A 抓放(明确 mock,不调用付费 API)', async ({ page }, info) => {
|
||||
test.skip(!existsSync('build/lekiwi/lekiwi-v1.zip'), '无本地 A 资产');
|
||||
let modelCalls = 0;
|
||||
page.on('request', (r) => {
|
||||
if (/\/api\/decision\/v1\/(plan|decide|test)$/.test(r.url())) modelCalls++;
|
||||
});
|
||||
await page.goto('/');
|
||||
await page
|
||||
.locator('input[type=file]')
|
||||
.first()
|
||||
.setInputFiles(resolve('build/lekiwi/lekiwi-v1.zip'));
|
||||
const dialog = page.getByRole('dialog', { name: '配置 URDF 仿真组件' });
|
||||
await expect(dialog).toBeVisible({ timeout: 30000 });
|
||||
await dialog.getByLabel('URDF 机器人 profile').selectOption('lekiwi-v1');
|
||||
await dialog.getByRole('button', { name: '转换并加载' }).click();
|
||||
await expect(page.getByRole('button', { name: '▶ 播放' })).toBeEnabled({ timeout: 90000 });
|
||||
await page.getByRole('tab', { name: '控制台', exact: true }).click();
|
||||
await page.getByRole('button', { name: 'LeKiwi 智能抓放', exact: true }).click();
|
||||
const panel = page.getByLabel('LeKiwi 智能抓放任务', { exact: true });
|
||||
await panel.getByRole('button', { name: '开始任务(mock)', exact: true }).click();
|
||||
await expect(panel.getByTestId('agent-task-state')).toContainText('物理验收成功', {
|
||||
timeout: 180000,
|
||||
});
|
||||
await panel.getByText('高级操作与证据导出', { exact: true }).click();
|
||||
const download = page.waitForEvent('download');
|
||||
await panel.getByRole('button', { name: '导出任务证据', exact: true }).click();
|
||||
const data = JSON.parse(await readFile((await (await download).path())!, 'utf8'));
|
||||
expect(data.succeeded).toBe(true);
|
||||
expect(data.transported).toBeGreaterThanOrEqual(0.5);
|
||||
expect(data.mode).toBe('mock');
|
||||
expect(modelCalls).toBe(0);
|
||||
await writeFile(
|
||||
info.outputPath('production-mock-physical-evidence.json'),
|
||||
JSON.stringify(data, null, 2),
|
||||
);
|
||||
await page.screenshot({ path: info.outputPath('production-physical-success.png') });
|
||||
});
|
||||
@@ -0,0 +1,119 @@
|
||||
import { expect, test } from '@playwright/test';
|
||||
import { resolve } from 'node:path';
|
||||
import { existsSync } from 'node:fs';
|
||||
import { readFile, writeFile } from 'node:fs/promises';
|
||||
|
||||
const prefix = '/api/decision/v1';
|
||||
test('网站生产构建:精简设置、同源 Cookie/CSRF、双浏览器隔离与假 HTTP 上游', async ({
|
||||
page,
|
||||
browser,
|
||||
}, info) => {
|
||||
const errors: string[] = [];
|
||||
page.on('pageerror', (e) => errors.push(e.message));
|
||||
await page.goto('/');
|
||||
await expect(page).toHaveTitle(/CADWorld Sim/);
|
||||
await page
|
||||
.locator('input[type=file]')
|
||||
.first()
|
||||
.setInputFiles(resolve('web_platform/fixtures/python_controller/model.xml'));
|
||||
await expect(page.getByRole('button', { name: '▶ 播放' })).toBeEnabled({ timeout: 60000 });
|
||||
await page.getByRole('tab', { name: '控制台', exact: true }).click();
|
||||
await expect(page.getByRole('button', { name: '强化学习任务', exact: true })).toHaveCount(0);
|
||||
await expect(page.getByRole('button', { name: '开源项目 / 外部控制', exact: true })).toHaveCount(
|
||||
0,
|
||||
);
|
||||
await page.getByRole('button', { name: 'LeKiwi 智能抓放', exact: true }).click();
|
||||
await page.getByText('模型设置 · API_KEY / ChatGPT 订阅', { exact: true }).click();
|
||||
await expect(page.getByRole('button', { name: '保存设置' })).toBeEnabled();
|
||||
await expect(page.getByLabel('本机服务令牌')).toHaveCount(0);
|
||||
await page.getByLabel('LLM API_KEY', { exact: true }).fill('e2e-llm-fixture-not-real');
|
||||
await page.getByLabel('Jev API_KEY(OpenRouter)').fill('e2e-jev-fixture-not-real');
|
||||
await page.getByRole('button', { name: '保存设置', exact: true }).click();
|
||||
await expect(page.getByRole('status').filter({ hasText: '设置已保存' })).toBeVisible();
|
||||
await expect(page.getByLabel('LLM API_KEY', { exact: true })).toHaveValue('');
|
||||
await page.getByText('连接测试与凭据管理', { exact: true }).click();
|
||||
await page.getByRole('button', { name: '测试 LLM(可能计费)', exact: true }).click();
|
||||
await expect(page.getByText(/LLM 连接测试通过/)).toBeVisible();
|
||||
const other = await browser.newContext();
|
||||
try {
|
||||
const second = await other.newPage();
|
||||
await second.goto('/');
|
||||
const fresh = await second.evaluate(async (prefix) => {
|
||||
const response = await fetch(prefix + '/session', {
|
||||
method: 'POST',
|
||||
headers: { 'Content-Type': 'application/json' },
|
||||
body: '{}',
|
||||
});
|
||||
return response.json();
|
||||
}, prefix);
|
||||
expect(fresh.ready).toBe(false);
|
||||
const state = await page.evaluate(
|
||||
async (prefix) => (await fetch(prefix + '/status')).json(),
|
||||
prefix,
|
||||
);
|
||||
expect(state.ready).toBe(true);
|
||||
const forbidden = await page.evaluate(
|
||||
async (prefix) =>
|
||||
(
|
||||
await fetch(prefix + '/configuration', {
|
||||
method: 'PUT',
|
||||
headers: { 'Content-Type': 'application/json' },
|
||||
body: '{}',
|
||||
})
|
||||
).status,
|
||||
prefix,
|
||||
);
|
||||
expect(forbidden).toBe(403);
|
||||
expect(await page.evaluate(() => document.cookie)).not.toContain('cadworld');
|
||||
expect(
|
||||
await page.evaluate(() => JSON.stringify({ ...localStorage, ...sessionStorage })),
|
||||
).not.toContain('e2e-llm');
|
||||
} finally {
|
||||
await other.close();
|
||||
}
|
||||
await page.getByRole('button', { name: 'ChatGPT 订阅', exact: true }).click();
|
||||
await expect(page.getByRole('button', { name: '登录 ChatGPT 订阅', exact: true })).toBeVisible();
|
||||
await page.screenshot({ path: info.outputPath('website-settings.png') });
|
||||
expect(errors).toEqual([]);
|
||||
});
|
||||
|
||||
test('网站 A 模型:真实 WASM、显式 mock 抓放和证据导出(无远端模型调用)', async ({
|
||||
page,
|
||||
}, info) => {
|
||||
test.skip(!existsSync('build/lekiwi/lekiwi-v1.zip'), '需要本地 A 模型,不自动公开或下载资产');
|
||||
let calls = 0;
|
||||
page.on('request', (r) => {
|
||||
if (/\/api\/decision\/v1\/(plan|decide)$/.test(r.url())) calls++;
|
||||
});
|
||||
await page.goto('/');
|
||||
await page
|
||||
.locator('input[type=file]')
|
||||
.first()
|
||||
.setInputFiles(resolve('build/lekiwi/lekiwi-v1.zip'));
|
||||
const dialog = page.getByRole('dialog', { name: '配置 URDF 仿真组件' });
|
||||
await expect(dialog).toBeVisible({ timeout: 30000 });
|
||||
await dialog.getByLabel('URDF 机器人 profile').selectOption('lekiwi-v1');
|
||||
await dialog.getByRole('button', { name: '转换并加载' }).click();
|
||||
await expect(page.getByRole('button', { name: '▶ 播放' })).toBeEnabled({ timeout: 60000 });
|
||||
await page.getByRole('tab', { name: '控制台', exact: true }).click();
|
||||
await page.getByRole('button', { name: 'LeKiwi 智能抓放', exact: true }).click();
|
||||
const panel = page.getByLabel('LeKiwi 智能抓放任务', { exact: true });
|
||||
await panel.getByRole('button', { name: '开始任务(mock)', exact: true }).click();
|
||||
await expect(panel.getByTestId('agent-task-state')).toContainText('物理验收成功', {
|
||||
timeout: 180000,
|
||||
});
|
||||
await panel.getByText('高级操作与证据导出', { exact: true }).click();
|
||||
const downloaded = page.waitForEvent('download');
|
||||
await panel.getByRole('button', { name: '导出任务证据', exact: true }).click();
|
||||
const evidence = JSON.parse(await readFile((await (await downloaded).path())!, 'utf8'));
|
||||
expect(evidence.mode).toBe('mock');
|
||||
expect(evidence.succeeded).toBe(true);
|
||||
expect(evidence.transported).toBeGreaterThanOrEqual(0.5);
|
||||
expect(evidence.sample.onGoalSupport).toBe(true);
|
||||
expect(calls).toBe(0);
|
||||
await writeFile(
|
||||
info.outputPath('website-physical-evidence.json'),
|
||||
JSON.stringify(evidence, null, 2),
|
||||
);
|
||||
await page.screenshot({ path: info.outputPath('website-grasp.png') });
|
||||
});
|
||||
@@ -0,0 +1,10 @@
|
||||
<!doctype html>
|
||||
<html>
|
||||
<head>
|
||||
<meta charset="UTF-8" />
|
||||
<title>LeKiwi 真实接触验收夹具</title>
|
||||
</head>
|
||||
<body>
|
||||
<script type="module" src="./agent.ts"></script>
|
||||
</body>
|
||||
</html>
|
||||
@@ -0,0 +1,356 @@
|
||||
import { MainThreadPhysicsAdapter } from '../src/simulation/PhysicsAdapter';
|
||||
import { importBrowserFiles } from '../src/project/importer';
|
||||
import type { ProjectManifest } from '../src/project/types';
|
||||
import { composePickPlaceScene, PICK_PLACE_ROBOT } from '../src/mobile/agent/PickPlaceScene';
|
||||
import { DeterministicBaseline } from '../src/mobile/agent/DeterministicBaseline';
|
||||
import { AgentTaskController } from '../src/mobile/agent/AgentTaskController';
|
||||
import { MockDecisionProvider } from '../src/mobile/agent/MockDecisionProvider';
|
||||
import { DecisionClient } from '../src/mobile/agent/DecisionClient';
|
||||
import { ModelBindings } from '../src/simulation/ModelBindings';
|
||||
import { PICK_PLACE as P } from '../src/mobile/agent/PickPlaceScene';
|
||||
type Fault =
|
||||
| 'empty-grasp'
|
||||
| 'empty-once'
|
||||
| 'alignment'
|
||||
| 'slip'
|
||||
| 'jam'
|
||||
| 'navigation'
|
||||
| 'unreachable'
|
||||
| 'overspeed';
|
||||
|
||||
const adapter = new MainThreadPhysicsAdapter();
|
||||
let source: ProjectManifest | undefined;
|
||||
let active: DeterministicBaseline | undefined;
|
||||
let task: AgentTaskController | undefined;
|
||||
let generation = 0;
|
||||
const api = {
|
||||
async boot(url: string) {
|
||||
const response = await fetch(url);
|
||||
if (!response.ok) throw new Error(`A 资产 HTTP ${response.status}`);
|
||||
source = await importBrowserFiles([new File([await response.blob()], 'lekiwi-v1.zip')]);
|
||||
},
|
||||
async baseline(seed: number, governed = false) {
|
||||
const epoch = ++generation;
|
||||
task?.dispose();
|
||||
task = undefined;
|
||||
active?.dispose();
|
||||
active = undefined;
|
||||
if (!source) throw new Error('尚未导入 A 资产');
|
||||
await adapter.load(source, 'URDF/LeKiwi.urdf', {
|
||||
robotProfileId: 'lekiwi-v1',
|
||||
configureRobotRuntime: false,
|
||||
mobileRobot: governed ? PICK_PLACE_ROBOT : undefined,
|
||||
urdfMode: 'mjcf',
|
||||
baseMode: 'floating',
|
||||
map: { kind: 'none' },
|
||||
sceneComposer: (xml) => composePickPlaceScene(xml, seed, governed),
|
||||
});
|
||||
adapter.releaseRetired();
|
||||
const session = adapter.session!;
|
||||
let baseline!: DeterministicBaseline;
|
||||
let ownerAtStart = 'manual';
|
||||
if (governed) {
|
||||
session.startAgent((write) => {
|
||||
baseline = new DeterministicBaseline(session, seed, write);
|
||||
let cancelled = false;
|
||||
return {
|
||||
beforeStep: () => {
|
||||
if (cancelled) return false;
|
||||
baseline.beforeStep();
|
||||
return baseline.status === 'running';
|
||||
},
|
||||
afterStep: (dt) => {
|
||||
baseline.afterStep(dt);
|
||||
if (baseline.status !== 'running') session.stopAgent(baseline.status);
|
||||
},
|
||||
hold: () => baseline.hold(),
|
||||
cancel: () => {
|
||||
cancelled = true;
|
||||
},
|
||||
dispose: () => baseline.dispose(),
|
||||
};
|
||||
});
|
||||
session.setPaused(false);
|
||||
ownerAtStart = session.snapshot().controlOwner!;
|
||||
} else
|
||||
baseline = new DeterministicBaseline(session, seed, (id, value) =>
|
||||
session.setActuator(id, value),
|
||||
);
|
||||
active = baseline;
|
||||
const opt = session.model.opt,
|
||||
dt = Number(opt.timestep);
|
||||
opt.delete();
|
||||
let steps = 0;
|
||||
const began = performance.now();
|
||||
while (baseline.status === 'running') {
|
||||
if (epoch !== generation) throw new Error('基线已取消');
|
||||
if (!governed) baseline.beforeStep();
|
||||
if (baseline.status !== 'running') break;
|
||||
session.singleStep();
|
||||
if (++steps % 20 === 0) session.module.mj_forward(session.model, session.data);
|
||||
if (!governed) baseline.afterStep(dt);
|
||||
if (steps % 200 === 0) await new Promise<void>((resolve) => setTimeout(resolve, 0));
|
||||
}
|
||||
const result = {
|
||||
seed,
|
||||
ownerAtStart,
|
||||
status: baseline.status,
|
||||
error: baseline.error,
|
||||
steps,
|
||||
simTime: Number(session.data.time),
|
||||
wallMs: performance.now() - began,
|
||||
transported: baseline.evaluator.transported,
|
||||
succeeded: baseline.evaluator.succeeded,
|
||||
maxJointSpeed: baseline.maxJointSpeed,
|
||||
final: baseline.sample,
|
||||
trace: baseline.trace,
|
||||
};
|
||||
baseline.dispose();
|
||||
active = undefined;
|
||||
return result;
|
||||
},
|
||||
async task(
|
||||
seed: number,
|
||||
connection?: { address: string; token: string },
|
||||
options: { fault?: Fault; goal?: number[] } = {},
|
||||
) {
|
||||
if (connection && options.fault) throw new Error('故障夹具禁止真实 API');
|
||||
const epoch = ++generation;
|
||||
task?.dispose();
|
||||
active?.dispose();
|
||||
if (!source) throw new Error('尚未导入 A 资产');
|
||||
await adapter.load(source, 'URDF/LeKiwi.urdf', {
|
||||
robotProfileId: 'lekiwi-v1',
|
||||
configureRobotRuntime: false,
|
||||
mobileRobot: PICK_PLACE_ROBOT,
|
||||
urdfMode: 'mjcf',
|
||||
baseMode: 'floating',
|
||||
map: { kind: 'none' },
|
||||
sceneComposer: (xml) => composePickPlaceScene(xml, seed, true, options.goal),
|
||||
});
|
||||
adapter.releaseRetired();
|
||||
const session = adapter.session!;
|
||||
const provider = connection
|
||||
? new DecisionClient(connection.address, connection.token)
|
||||
: new MockDecisionProvider();
|
||||
let controller!: AgentTaskController;
|
||||
let skills!: DeterministicBaseline;
|
||||
session.startAgent((write) => {
|
||||
const b = new ModelBindings(session.model),
|
||||
r = PICK_PLACE_ROBOT;
|
||||
const grip = b.scalarActuator(
|
||||
r.gripperActuators[0].name,
|
||||
r.gripperJoint,
|
||||
'position',
|
||||
).actuatorId;
|
||||
const jam = b.scalarActuator(r.armActuators[1], r.armJoints[1].name, 'position').actuatorId;
|
||||
const wheels = new Set(
|
||||
r.baseActuators.map(
|
||||
(name, i) => b.scalarActuator(name, r.baseJoints[i], 'velocity').actuatorId,
|
||||
),
|
||||
);
|
||||
let jamTarget = 0,
|
||||
slipAt: number | undefined,
|
||||
slipOpening = 0;
|
||||
// Test-only actuator faults: no live qpos writes, attachments or object teleportation.
|
||||
const disturbedWrite = (id: number, value: number) => {
|
||||
if (options.fault === 'jam' && id === jam) value = jamTarget;
|
||||
if (options.fault === 'overspeed' && id === jam) value = r.armJoints[1].max;
|
||||
if (options.fault === 'navigation' && wheels.has(id)) value = 0;
|
||||
if (
|
||||
id === grip &&
|
||||
(options.fault === 'empty-grasp' ||
|
||||
(options.fault === 'empty-once' && controller.failures.length === 0))
|
||||
)
|
||||
value = r.gripperOpen;
|
||||
if (id === grip && options.fault === 'slip') {
|
||||
if (
|
||||
slipAt === undefined &&
|
||||
skills.phase === 'carry' &&
|
||||
skills.evaluator.transported > 0.12
|
||||
) {
|
||||
slipAt = skills.sample.time;
|
||||
slipOpening = skills.sample.opening;
|
||||
}
|
||||
if (slipAt !== undefined) {
|
||||
const opening = Math.min(
|
||||
1,
|
||||
slipOpening + P.gripperOpeningRate * (skills.sample.time - slipAt),
|
||||
);
|
||||
value = r.gripperClosed + opening * (r.gripperOpen - r.gripperClosed);
|
||||
}
|
||||
}
|
||||
write(id, value);
|
||||
};
|
||||
skills = new DeterministicBaseline(session, seed, disturbedWrite, true, options.goal);
|
||||
jamTarget = skills.sample.joints[1];
|
||||
if (options.fault === 'unreachable' || options.fault === 'alignment') {
|
||||
const solve = skills.ik.solve.bind(skills.ik);
|
||||
// Actual scratch-data IK, not fake success/failure results or live qpos writes.
|
||||
skills.ik.solve = (target) =>
|
||||
solve(
|
||||
options.fault === 'unreachable' ? [2, 2, 2] : [target[0] + 0.02, target[1], target[2]],
|
||||
);
|
||||
}
|
||||
controller = new AgentTaskController(
|
||||
skills,
|
||||
provider,
|
||||
'将方块抓起,使用底盘搬运到指定支撑台并释放、撤离。',
|
||||
);
|
||||
return controller;
|
||||
});
|
||||
task = controller;
|
||||
active = skills;
|
||||
session.setPaused(false);
|
||||
const ownerAtStart = session.snapshot().controlOwner;
|
||||
const began = performance.now();
|
||||
let frames = 0;
|
||||
while (!controller.terminal) {
|
||||
if (epoch !== generation) throw new Error('任务已取消');
|
||||
session.singleStep();
|
||||
if (++frames % 20 === 0 && controller.state === 'executing')
|
||||
session.module.mj_forward(session.model, session.data);
|
||||
if (frames % 200 === 0) await new Promise<void>((resolve) => setTimeout(resolve, 0));
|
||||
}
|
||||
const result = {
|
||||
...controller.snapshot(),
|
||||
fault: options.fault ?? null,
|
||||
ownerAtStart,
|
||||
seed,
|
||||
simTime: Number(session.data.time),
|
||||
wallMs: performance.now() - began,
|
||||
maxJointSpeed: skills.maxJointSpeed,
|
||||
results: controller.results,
|
||||
failures: controller.failures,
|
||||
trace: skills.trace,
|
||||
receipts: provider instanceof DecisionClient ? provider.receipts : [],
|
||||
};
|
||||
session.stopAgent('任务夹具已结束');
|
||||
task = undefined;
|
||||
active = undefined;
|
||||
return result;
|
||||
},
|
||||
async lifecycle(action: 'pause-task' | 'pause-global' | 'stop' | 'force' | 'reload') {
|
||||
task?.dispose();
|
||||
active?.dispose();
|
||||
if (!source) throw new Error('尚未导入 A 资产');
|
||||
const load = () =>
|
||||
adapter.load(source!, 'URDF/LeKiwi.urdf', {
|
||||
robotProfileId: 'lekiwi-v1',
|
||||
configureRobotRuntime: false,
|
||||
mobileRobot: PICK_PLACE_ROBOT,
|
||||
urdfMode: 'mjcf',
|
||||
baseMode: 'floating',
|
||||
map: { kind: 'none' },
|
||||
sceneComposer: (xml) => composePickPlaceScene(xml, 0, true),
|
||||
});
|
||||
await load();
|
||||
adapter.releaseRetired();
|
||||
const session = adapter.session!,
|
||||
provider = new MockDecisionProvider();
|
||||
let release!: () => void;
|
||||
const delayed = new Promise<void>((resolve) => {
|
||||
release = resolve;
|
||||
});
|
||||
const plan = provider.plan.bind(provider);
|
||||
// Deliberately ignore abort AFTER creating the reply: prove the local epoch gate,
|
||||
// not just HTTP cancellation, prevents a broken provider from moving real WASM.
|
||||
provider.plan = async (...args) => {
|
||||
const reply = await plan(...args);
|
||||
await delayed;
|
||||
return reply;
|
||||
};
|
||||
let controller!: AgentTaskController, skills!: DeterministicBaseline;
|
||||
session.startAgent((write) => {
|
||||
skills = new DeterministicBaseline(session, 0, write, true);
|
||||
controller = new AgentTaskController(skills, provider, '抓取并搬运方块');
|
||||
return controller;
|
||||
});
|
||||
task = controller;
|
||||
active = skills;
|
||||
session.setPaused(false);
|
||||
session.singleStep();
|
||||
await new Promise<void>((resolve) => setTimeout(resolve, 0));
|
||||
if (controller.state !== 'planning') throw new Error('生命周期夹具未进入等待规划');
|
||||
if (action === 'pause-task') controller.pause();
|
||||
if (action === 'pause-global') session.setPaused(true);
|
||||
if (action === 'stop') session.stopAgent('夹具停止');
|
||||
if (action === 'force') session.setExternalForce(1, [1, 0, 0]);
|
||||
if (action === 'reload') {
|
||||
await load();
|
||||
adapter.releaseRetired();
|
||||
}
|
||||
const current = adapter.session!;
|
||||
const before = {
|
||||
time: Number(current.data.time),
|
||||
ctrl: Array.from(current.data.ctrl),
|
||||
qpos: Array.from(current.data.qpos),
|
||||
};
|
||||
release();
|
||||
await new Promise<void>((resolve) => setTimeout(resolve, 20));
|
||||
for (let i = 0; i < 10; i++) current.advance(performance.now() + i * 20);
|
||||
const after = {
|
||||
time: Number(current.data.time),
|
||||
ctrl: Array.from(current.data.ctrl),
|
||||
qpos: Array.from(current.data.qpos),
|
||||
};
|
||||
const result = {
|
||||
action,
|
||||
source: 'delayed-mock-plus-real-WASM',
|
||||
state: controller.state,
|
||||
owner: current.controlOwner,
|
||||
jevCalls: controller.jevCalls,
|
||||
before,
|
||||
after,
|
||||
};
|
||||
controller.cancel('生命周期验收结束');
|
||||
return result;
|
||||
},
|
||||
async resources() {
|
||||
const rows: { heapBytes: number; resources: { isDeleted(): boolean }[] }[] = [];
|
||||
for (let i = 0; i < 8; i++) {
|
||||
await api.lifecycle('stop');
|
||||
const current = adapter.session!;
|
||||
// Test-only reflection retains handles, not their typed-array views, to verify
|
||||
// native deletions rather than inferring a leak from linear-memory capacity.
|
||||
const ik = active!.ik as unknown as {
|
||||
scratch: { isDeleted(): boolean };
|
||||
positionJacobian: { isDeleted(): boolean };
|
||||
rotationJacobian: { isDeleted(): boolean };
|
||||
};
|
||||
const sampler = active! as unknown as { sampler: { force: { isDeleted(): boolean } } };
|
||||
rows.push({
|
||||
heapBytes: (current.data.qpos as Float64Array).buffer.byteLength,
|
||||
resources: [
|
||||
current.model,
|
||||
current.data,
|
||||
ik.scratch,
|
||||
ik.positionJacobian,
|
||||
ik.rotationJacobian,
|
||||
sampler.sampler.force,
|
||||
],
|
||||
});
|
||||
task?.dispose();
|
||||
active?.dispose();
|
||||
}
|
||||
api.dispose();
|
||||
return rows.map((row) => ({
|
||||
heapBytes: row.heapBytes,
|
||||
deleted: row.resources.map((resource) => resource.isDeleted()),
|
||||
}));
|
||||
},
|
||||
dispose() {
|
||||
generation++;
|
||||
task?.dispose();
|
||||
task = undefined;
|
||||
active?.dispose();
|
||||
active = undefined;
|
||||
adapter.dispose();
|
||||
},
|
||||
};
|
||||
declare global {
|
||||
interface Window {
|
||||
lekiwiAgent: typeof api;
|
||||
}
|
||||
}
|
||||
window.lekiwiAgent = api;
|
||||
@@ -0,0 +1,10 @@
|
||||
<!doctype html>
|
||||
<html lang="zh-CN">
|
||||
<head>
|
||||
<meta charset="UTF-8" />
|
||||
<title>Mobile WASM contract runner</title>
|
||||
</head>
|
||||
<body>
|
||||
<script type="module" src="./mobile.ts"></script>
|
||||
</body>
|
||||
</html>
|
||||
@@ -0,0 +1,136 @@
|
||||
import { RobotManager } from '../src/mobile/RobotManager';
|
||||
import { MOBILE_ROBOTS } from '../src/mobile/RobotDescriptor';
|
||||
import type { MobilePolicyMetadata } from '../src/mobile/ONNXPolicyRunner';
|
||||
import { MuJoCoViewer } from '../src/viewer/MuJoCoViewer';
|
||||
import { TaskDragController } from '../src/mobile/TaskDragController';
|
||||
import { Vector3 } from 'three';
|
||||
|
||||
let viewer: MuJoCoViewer | undefined;
|
||||
let drag: TaskDragController | undefined;
|
||||
const manager = new RobotManager({
|
||||
attach(session) {
|
||||
viewer?.attach(session);
|
||||
},
|
||||
});
|
||||
function snapshot() {
|
||||
const env = manager.env!;
|
||||
const observation = env.observe();
|
||||
return {
|
||||
robotId: env.config.id,
|
||||
state: Array.from(env.state),
|
||||
observation: Array.from(observation),
|
||||
ctrl: Array.from(env.session.data.ctrl, Number),
|
||||
qpos: Array.from(env.session.data.qpos, Number),
|
||||
time: Number(env.session.data.time),
|
||||
reward: env.kernel.result.reward,
|
||||
stage: env.kernel.result.info.stage,
|
||||
revision: env.revision,
|
||||
heap: env.wasmHeapBytes,
|
||||
version: env.session.module.mj_versionString(),
|
||||
workspace: manager.adapter.workspace!.root,
|
||||
};
|
||||
}
|
||||
const api = {
|
||||
async boot(url: string, id: string) {
|
||||
const config = MOBILE_ROBOTS.find((c) => c.id === id)!;
|
||||
const response = await fetch(url);
|
||||
if (!response.ok) throw new Error(`asset HTTP ${response.status}`);
|
||||
const previous = manager.adapter.session;
|
||||
await manager.loadZip(new File([await response.blob()], 'robot.zip'), config);
|
||||
return {
|
||||
...snapshot(),
|
||||
retiredModelDeleted: previous?.model.isDeleted() ?? true,
|
||||
retiredDataDeleted: previous?.data.isDeleted() ?? true,
|
||||
};
|
||||
},
|
||||
async invalidCandidate(url: string) {
|
||||
const config = structuredClone(MOBILE_ROBOTS[1]);
|
||||
const wrong = {
|
||||
...config,
|
||||
armActuators: ['missing_actuator', ...config.armActuators.slice(1)],
|
||||
};
|
||||
await manager.loadZip(new File([await (await fetch(url)).blob()], 'robot.zip'), wrong);
|
||||
},
|
||||
step(n: number, action?: number[], manual = false) {
|
||||
const env = manager.env!;
|
||||
if (action) env.action.set(action);
|
||||
const started = performance.now();
|
||||
for (let i = 0; i < n; i++) env.step(env.action, manual);
|
||||
return { ...snapshot(), wallMs: performance.now() - started };
|
||||
},
|
||||
reset() {
|
||||
manager.reset();
|
||||
return snapshot();
|
||||
},
|
||||
move(entity: 'object' | 'goal', position: number[]) {
|
||||
manager.move(entity, position);
|
||||
return snapshot();
|
||||
},
|
||||
state: snapshot,
|
||||
workspaces() {
|
||||
const module = manager.adapter.session!.module as unknown as {
|
||||
FS: { readdir(p: string): string[] };
|
||||
};
|
||||
return module.FS.readdir('/workspace').filter((s) => s !== '.' && s !== '..');
|
||||
},
|
||||
async download() {
|
||||
const bytes = await manager.exportTrainingBundle();
|
||||
const href = URL.createObjectURL(new Blob([bytes.slice().buffer]));
|
||||
const link = document.createElement('a');
|
||||
link.href = href;
|
||||
link.download = 'training.zip';
|
||||
link.click();
|
||||
setTimeout(() => URL.revokeObjectURL(href), 1000);
|
||||
},
|
||||
async policy(modelUrl: string, metadata: MobilePolicyMetadata) {
|
||||
await manager.loadPolicy(new Uint8Array(await (await fetch(modelUrl)).arrayBuffer()), metadata);
|
||||
manager.setMode('policy');
|
||||
manager.setRunning(true);
|
||||
},
|
||||
tick(now: number) {
|
||||
return {
|
||||
...manager.advance(now),
|
||||
metrics: manager.policy?.metrics,
|
||||
time: Number(manager.env!.session.data.time),
|
||||
};
|
||||
},
|
||||
show() {
|
||||
const host = document.createElement('div');
|
||||
host.style.cssText = 'width:1000px;height:650px;position:relative';
|
||||
document.body.append(host);
|
||||
const noop = () => {};
|
||||
viewer = new MuJoCoViewer(host, {
|
||||
advance: (now) => manager.advance(now),
|
||||
onSelection: noop,
|
||||
onFrame: noop,
|
||||
onError: (e) => {
|
||||
throw e;
|
||||
},
|
||||
onMapEditorSelect: noop,
|
||||
onMapEditorPreviewSelect: noop,
|
||||
onParametricMapSelect: noop,
|
||||
onParametricMapTransform: noop,
|
||||
onMapEditorTransform: noop,
|
||||
});
|
||||
viewer.attach(manager.adapter.session);
|
||||
drag = new TaskDragController(viewer, manager);
|
||||
},
|
||||
screen(entity: 'object' | 'goal') {
|
||||
const s = manager.env!.state,
|
||||
offset = entity === 'object' ? 37 : 47;
|
||||
const p = new Vector3(s[offset], s[offset + 1], s[offset + 2]).project(viewer!.camera);
|
||||
const r = viewer!.renderer.domElement.getBoundingClientRect();
|
||||
return { x: r.left + (p.x + 1) * 0.5 * r.width, y: r.top + (1 - p.y) * 0.5 * r.height };
|
||||
},
|
||||
async dispose() {
|
||||
drag?.dispose();
|
||||
await manager.dispose();
|
||||
viewer?.dispose();
|
||||
},
|
||||
};
|
||||
declare global {
|
||||
interface Window {
|
||||
mobilePhysics: typeof api;
|
||||
}
|
||||
}
|
||||
window.mobilePhysics = api;
|
||||
@@ -1,6 +1,9 @@
|
||||
import { MainThreadPhysicsAdapter } from '../src/simulation/PhysicsAdapter';
|
||||
import { SimulationSession } from '../src/simulation/SimulationSession';
|
||||
import { ModelBindings } from '../src/simulation/ModelBindings';
|
||||
import { mergeCachedFiles } from '../src/project/cachedFiles';
|
||||
import { compiledProjectPackage } from '../src/project/exportPackage';
|
||||
import { importBrowserFiles } from '../src/project/importer';
|
||||
import { LEKIWI_PROFILE as p, bodyToWheels } from '../src/robot/profiles/lekiwi';
|
||||
import type { ProjectManifest } from '../src/project/types';
|
||||
import { ExternalControlClient } from '../src/robot/ExternalControlClient';
|
||||
@@ -13,6 +16,7 @@ import { surfaceDistance, type Surface } from './meshDistance';
|
||||
const adapter = new MainThreadPhysicsAdapter();
|
||||
let bindings: ModelBindings;
|
||||
let currentManifest: ProjectManifest;
|
||||
let roundtripPackage: ReturnType<typeof compiledProjectPackage> | undefined;
|
||||
let frame: number | undefined;
|
||||
let bridge: ExternalControlClient | undefined;
|
||||
function session() {
|
||||
@@ -169,6 +173,7 @@ const harness = {
|
||||
bridge?.sync();
|
||||
},
|
||||
async boot(assetBase: string) {
|
||||
roundtripPackage = undefined;
|
||||
const response = await fetch(`${assetBase}/source-manifest.json`);
|
||||
if (!response.ok) throw new Error('Run examples/lekiwi/prepare_assets.py first');
|
||||
const source = (await response.json()) as { files: Record<string, string> };
|
||||
@@ -198,6 +203,8 @@ const harness = {
|
||||
map: { kind: 'none' },
|
||||
enhancements: { addActuators: false, addSensors: false, sensorType: 'camera' },
|
||||
});
|
||||
currentManifest = mergeCachedFiles(currentManifest, adapter.cachedSupportFiles());
|
||||
adapter.releaseRetired();
|
||||
bindings = new ModelBindings(session().model);
|
||||
return { version: session().module.mj_versionString(), ...read() };
|
||||
},
|
||||
@@ -778,6 +785,86 @@ const harness = {
|
||||
cancel: await cancel,
|
||||
};
|
||||
},
|
||||
visualStats() {
|
||||
const s = session(),
|
||||
m = s.model,
|
||||
d = s.data;
|
||||
s.module.mj_forward(m, d);
|
||||
const wheels = p.wheels.map((name) => {
|
||||
const wheelBody = bindings.resolve(bindings.bodies, `__lekiwi_${name}`);
|
||||
for (let id = 0; id < m.ngeom; id++) {
|
||||
const geom = m.geom(id);
|
||||
try {
|
||||
if (
|
||||
!geom.name.startsWith('4-Omni-Directional-Wheel_Single_Body') ||
|
||||
Number(m.geom_group[id]) !== 1
|
||||
)
|
||||
continue;
|
||||
let body = Number(m.geom_bodyid[id]);
|
||||
while (body && body !== wheelBody) body = Number(m.body_parentid[body]);
|
||||
if (body !== wheelBody) continue;
|
||||
const meshId = Number(m.geom_dataid[id]);
|
||||
const centerError = Math.hypot(
|
||||
...[0, 1, 2].map((i) => d.geom_xpos[id * 3 + i] - d.xpos[wheelBody * 3 + i]),
|
||||
);
|
||||
return {
|
||||
name,
|
||||
faces: Number(m.mesh_facenum[meshId]),
|
||||
vertices: Number(m.mesh_vertnum[meshId]),
|
||||
scale: Array.from(m.mesh_scale.slice(meshId * 3, meshId * 3 + 3)),
|
||||
centerError,
|
||||
rotation: Array.from(d.geom_xmat.slice(id * 9, id * 9 + 9), Number),
|
||||
collision: Boolean(m.geom_contype[id] || m.geom_conaffinity[id]),
|
||||
mass: Number(m.body_mass[Number(m.geom_bodyid[id])]),
|
||||
};
|
||||
} finally {
|
||||
geom.delete();
|
||||
}
|
||||
}
|
||||
throw new Error(`missing CAD wheel ${name}`);
|
||||
});
|
||||
const fs = s.module as unknown as { FS: { readdir(path: string): string[] } };
|
||||
return {
|
||||
wheels,
|
||||
heap: (d.qpos as Float64Array).buffer.byteLength,
|
||||
projectBytes: currentManifest.files.reduce((sum, file) => sum + file.data.byteLength, 0),
|
||||
xmlFiles: currentManifest.files
|
||||
.filter((f) => f.path.endsWith('.xml'))
|
||||
.map((f) => ({ path: f.path, bytes: f.size })),
|
||||
workspaces: fs.FS.readdir('/workspace').filter((name) => name !== '.' && name !== '..'),
|
||||
generatedMeshes: currentManifest.files
|
||||
.filter((f) => /stl-obj-v1-.*\.obj$/.test(f.path))
|
||||
.map((f) => ({ path: f.path, bytes: f.size })),
|
||||
};
|
||||
},
|
||||
async packageRoundtrip() {
|
||||
// Reload the SAME exported package. Re-exporting into the imported sources
|
||||
// would intentionally append a new XML snapshot each round, not test leaks.
|
||||
const packaged = (roundtripPackage ??= compiledProjectPackage(
|
||||
currentManifest,
|
||||
'URDF/LeKiwi.urdf',
|
||||
adapter.exportMjcf(),
|
||||
adapter.cachedSupportFiles(),
|
||||
));
|
||||
const imported = await importBrowserFiles([
|
||||
new File([packaged.data.slice().buffer], 'complete.zip'),
|
||||
]);
|
||||
const old = session();
|
||||
await adapter.load(imported, packaged.entryPath, {
|
||||
robotProfileId: p.id,
|
||||
urdfMode: 'mjcf',
|
||||
baseMode: 'floating',
|
||||
map: { kind: 'none' },
|
||||
});
|
||||
adapter.releaseRetired();
|
||||
currentManifest = imported;
|
||||
bindings = new ModelBindings(session().model);
|
||||
return {
|
||||
...harness.visualStats(),
|
||||
oldModelDeleted: old.model.isDeleted(),
|
||||
oldDataDeleted: old.data.isDeleted(),
|
||||
};
|
||||
},
|
||||
async reimport() {
|
||||
const data = adapter.exportMjcf();
|
||||
const file = {
|
||||
|
||||
@@ -0,0 +1,14 @@
|
||||
import { defineConfig } from '@playwright/test';
|
||||
export default defineConfig({
|
||||
testDir: './e2e',
|
||||
testMatch: /lekiwi\.agent(?:\.(?:workspace|live|faults|lifecycle|subscription))?\.spec\.ts/,
|
||||
outputDir: '../build/e2e/lekiwi-agent',
|
||||
timeout: 300_000,
|
||||
workers: 1,
|
||||
use: { baseURL: 'http://127.0.0.1:4176' },
|
||||
webServer: {
|
||||
command: 'npm run dev --prefix .. -- --host 127.0.0.1 --port 4176 --strictPort',
|
||||
url: 'http://127.0.0.1:4176',
|
||||
reuseExistingServer: false,
|
||||
},
|
||||
});
|
||||
@@ -1,7 +1,7 @@
|
||||
import { defineConfig } from '@playwright/test';
|
||||
export default defineConfig({
|
||||
testDir: './e2e',
|
||||
testIgnore: /lekiwi.*\.spec\.ts/,
|
||||
testIgnore: /(?:lekiwi.*|mobile|website.*)\.spec\.ts/,
|
||||
timeout: 120_000,
|
||||
use: { baseURL: 'http://127.0.0.1:4173' },
|
||||
webServer: [
|
||||
|
||||
@@ -3,6 +3,7 @@ export default defineConfig({
|
||||
testDir: './e2e',
|
||||
outputDir: '../build/e2e/lekiwi',
|
||||
testMatch: /lekiwi.*\.spec\.ts/,
|
||||
testIgnore: /lekiwi\.agent.*\.spec\.ts/, // Dedicated clock/Origin fixture: playwright.agent.config.ts
|
||||
timeout: 180_000,
|
||||
workers: 1,
|
||||
use: { baseURL: 'http://127.0.0.1:4174' },
|
||||
|
||||
@@ -0,0 +1,14 @@
|
||||
import { defineConfig } from '@playwright/test';
|
||||
export default defineConfig({
|
||||
testDir: './e2e',
|
||||
testMatch: /mobile\.spec\.ts/,
|
||||
outputDir: '../build/e2e/mobile',
|
||||
timeout: 240_000,
|
||||
workers: 1,
|
||||
use: { baseURL: 'http://127.0.0.1:4175' },
|
||||
webServer: {
|
||||
command: 'npm run dev --prefix .. -- --host 127.0.0.1 --port 4175 --strictPort',
|
||||
url: 'http://127.0.0.1:4175',
|
||||
reuseExistingServer: false,
|
||||
},
|
||||
});
|
||||
@@ -0,0 +1,21 @@
|
||||
import { defineConfig } from '@playwright/test';
|
||||
export default defineConfig({
|
||||
testDir: './e2e',
|
||||
testMatch: 'website.production.spec.ts',
|
||||
workers: 1,
|
||||
timeout: 240_000,
|
||||
use: {
|
||||
baseURL: 'https://cadworld-sim.robotquan.com',
|
||||
viewport: { width: 1440, height: 1000 },
|
||||
ignoreHTTPSErrors: false,
|
||||
trace: 'off',
|
||||
screenshot: 'only-on-failure',
|
||||
launchOptions: {
|
||||
args: [
|
||||
'--use-angle=swiftshader',
|
||||
'--enable-unsafe-swiftshader',
|
||||
'--host-resolver-rules=MAP cadworld-sim.robotquan.com 47.93.31.109',
|
||||
],
|
||||
},
|
||||
},
|
||||
});
|
||||
@@ -0,0 +1,27 @@
|
||||
import { defineConfig } from '@playwright/test';
|
||||
export default defineConfig({
|
||||
testDir: './e2e',
|
||||
testMatch: 'website.spec.ts',
|
||||
workers: 1,
|
||||
timeout: 240_000,
|
||||
use: {
|
||||
baseURL: 'http://127.0.0.1:4180',
|
||||
viewport: { width: 1440, height: 1000 },
|
||||
launchOptions: { args: ['--use-angle=swiftshader', '--enable-unsafe-swiftshader'] },
|
||||
screenshot: 'only-on-failure',
|
||||
trace: 'retain-on-failure',
|
||||
},
|
||||
webServer: [
|
||||
{
|
||||
command: 'cd .. && CADWORLD_E2E=1 .venv/bin/python -m decision_server.tests.website_fixture',
|
||||
url: 'http://127.0.0.1:8769/healthz',
|
||||
reuseExistingServer: false,
|
||||
},
|
||||
{
|
||||
command:
|
||||
'cd .. && CADWORLD_DEV_API=http://127.0.0.1:8769 npm run preview -- --mode website --host 127.0.0.1 --port 4180 --strictPort',
|
||||
url: 'http://127.0.0.1:4180',
|
||||
reuseExistingServer: false,
|
||||
},
|
||||
],
|
||||
});
|
||||
+189
-11
@@ -1,3 +1,11 @@
|
||||
import {
|
||||
mobileTrainingRobot,
|
||||
mobileSceneSnapshot,
|
||||
isMobileTrainingTask,
|
||||
type MobileDeployment,
|
||||
} from '../mobile/training';
|
||||
import type { RobotConfig } from '../mobile/RobotDescriptor';
|
||||
import { sha256 } from '../robot/registry';
|
||||
import { useThemePreference } from './hooks/useThemePreference';
|
||||
import { useExternalControl } from './hooks/useExternalControl';
|
||||
import { ExternalControlPanel } from '../robot/ExternalControlPanel';
|
||||
@@ -48,7 +56,13 @@ import {
|
||||
type UrdfEnhancementOptions,
|
||||
type UrdfLoadMode,
|
||||
} from '../simulation/PhysicsAdapter';
|
||||
import type { ActuatorParameters } from '../simulation/SimulationSession';
|
||||
import type { ActuatorParameters, SimulationSession } from '../simulation/SimulationSession';
|
||||
import { AgentTaskPanel } from '../mobile/agent/AgentTaskPanel';
|
||||
import {
|
||||
composePickPlaceScene,
|
||||
PICK_PLACE_ROBOT,
|
||||
type PickPlaceTaskConfig,
|
||||
} from '../mobile/agent/PickPlaceScene';
|
||||
import type { DataRecorderConfig } from '../telemetry/DataRecorder';
|
||||
import type { ControllerCommand, ControllerStatus } from '../controller/types';
|
||||
import type { RLCommand, RLPolicyStatus } from '../rl/types';
|
||||
@@ -99,6 +113,7 @@ import { DiagnosticsDrawer } from './components/DiagnosticsDrawer';
|
||||
import { ToolbarOverflowMenu } from './components/ToolbarOverflowMenu';
|
||||
|
||||
import { UrdfImportOptionsDialog } from './components/UrdfImportOptionsDialog';
|
||||
import { compiledProjectPackage } from '../project/exportPackage';
|
||||
import {
|
||||
downloadBytes,
|
||||
exportedFileName,
|
||||
@@ -291,6 +306,9 @@ export function App() {
|
||||
setRightOpen,
|
||||
} = useSidebarLayout();
|
||||
const robotProfilesRef = useRef(new Map<string, string>());
|
||||
const agentSceneSession = useRef<SimulationSession | undefined>(undefined);
|
||||
const lastAgentConfig = useRef<PickPlaceTaskConfig>({ seed: 0 });
|
||||
const [agentScene, setAgentScene] = useState(false);
|
||||
const sensorCameraFrame = useRef<HTMLDivElement>(null);
|
||||
const orientationHost = useRef<HTMLDivElement>(null);
|
||||
const [mapCommitState, setMapCommitState] = useState<'idle' | 'submitting' | 'failed'>('idle');
|
||||
@@ -572,10 +590,18 @@ export function App() {
|
||||
}, [showVisualMap, showMapCollision]);
|
||||
useEffect(() => {
|
||||
viewer.current?.setParametricMapAssets(
|
||||
trainingDeployment ? [] : placedMapAssets,
|
||||
trainingDeployment || agentScene ? [] : placedMapAssets,
|
||||
mapSceneDraft.changedIds,
|
||||
);
|
||||
}, [placedMapAssets, mapSceneDraft.changedIds, trainingDeployment]);
|
||||
}, [placedMapAssets, mapSceneDraft.changedIds, trainingDeployment, agentScene]);
|
||||
useEffect(() => {
|
||||
const session = adapter.current.session;
|
||||
if (session?.snapshot().controlOwner === 'agent') {
|
||||
session.stopAgent('地图草稿编辑使旧计划失效');
|
||||
session.setPaused(true);
|
||||
publishExternalSnapshot();
|
||||
}
|
||||
}, [placedMapAssets, publishExternalSnapshot]);
|
||||
useEffect(() => {
|
||||
viewer.current?.setShowPerceptionRays(showPerceptionRays);
|
||||
}, [showPerceptionRays]);
|
||||
@@ -597,6 +623,8 @@ export function App() {
|
||||
requestedSceneAssets?: readonly PlacedMapAsset[],
|
||||
requestedDeployment?: PolicyDeployment,
|
||||
requestedPolicy?: { data: Uint8Array; path: string },
|
||||
requestedMobile?: RobotConfig,
|
||||
requestedTask?: PickPlaceTaskConfig,
|
||||
) => {
|
||||
if (!manifest.current || loadInFlight.current) return false;
|
||||
// 普通模型重载始终使用上次成功应用的地图基线。只有场景提交入口可以显式
|
||||
@@ -634,12 +662,30 @@ export function App() {
|
||||
try {
|
||||
const activeViewer = viewer.current ?? (await viewerReady.current);
|
||||
if (!activeViewer) throw new Error('三维视口尚未就绪,请重试');
|
||||
const selectedProfile = robotProfilesRef.current.get(path);
|
||||
const mobileRobot =
|
||||
(requestedTask ? PICK_PLACE_ROBOT : requestedMobile) ??
|
||||
(selectedProfile && isMobileTrainingTask(selectedProfile)
|
||||
? mobileTrainingRobot(selectedProfile)
|
||||
: undefined);
|
||||
const snapshot = await adapter.current.load(manifest.current, path, {
|
||||
urdfMode: requestedMode ?? urdfModeRef.current,
|
||||
baseMode: baseModeRef.current,
|
||||
enhancements: urdfEnhancementsRef.current,
|
||||
robotProfileId: robotProfilesRef.current.get(path),
|
||||
mapAssets: sceneAssets,
|
||||
baseMode: requestedTask ? 'floating' : baseModeRef.current,
|
||||
enhancements: requestedTask
|
||||
? { addActuators: true, addSensors: false, sensorType: 'camera' }
|
||||
: urdfEnhancementsRef.current,
|
||||
robotProfileId: mobileRobot
|
||||
? mobileRobot.recipe === 'lekiwi-v1'
|
||||
? 'lekiwi-v1'
|
||||
: undefined
|
||||
: selectedProfile,
|
||||
mobileRobot,
|
||||
configureRobotRuntime: requestedTask ? false : undefined,
|
||||
sceneComposer: requestedTask
|
||||
? (xml) => composePickPlaceScene(xml, requestedTask.seed, true, requestedTask.goal)
|
||||
: undefined,
|
||||
mapAssets: mobileRobot ? [] : sceneAssets,
|
||||
map: requestedTask ? { kind: 'none' } : undefined,
|
||||
trainingDeployment: requestedDeployment,
|
||||
trainingPolicy: requestedPolicy,
|
||||
onProgress: ({ value, label }) =>
|
||||
@@ -691,7 +737,7 @@ export function App() {
|
||||
let visualMapWarning: string | undefined;
|
||||
try {
|
||||
const assets =
|
||||
manifest.current && !requestedDeployment
|
||||
manifest.current && !requestedDeployment && !requestedTask
|
||||
? visualMapAssets(manifest.current, placedMapAssetsRef.current)
|
||||
: [];
|
||||
await activeViewer.setVisualMaps(assets);
|
||||
@@ -727,12 +773,17 @@ export function App() {
|
||||
setAppliedMapAssets(committedMapAssets);
|
||||
}
|
||||
activeViewer.setParametricMapAssets(
|
||||
requestedDeployment ? [] : placedMapAssetsRef.current,
|
||||
requestedDeployment || requestedTask ? [] : placedMapAssetsRef.current,
|
||||
summarizeMapSceneDraft(placedMapAssetsRef.current, appliedMapAssetsRef.current)
|
||||
.changedIds,
|
||||
);
|
||||
adapter.current.releaseRetired();
|
||||
setTrainingDeployment(requestedDeployment);
|
||||
agentSceneSession.current = requestedTask
|
||||
? (adapter.current.session ?? undefined)
|
||||
: undefined;
|
||||
if (requestedTask) lastAgentConfig.current = requestedTask;
|
||||
setAgentScene(Boolean(requestedTask));
|
||||
sessionSwapped = false;
|
||||
return true;
|
||||
} catch (error) {
|
||||
@@ -1205,6 +1256,12 @@ export function App() {
|
||||
const current = manifest.current;
|
||||
const activeViewer = viewer.current;
|
||||
if (!current || !activeViewer) return;
|
||||
if (agentSceneSession.current && agentSceneSession.current === adapter.current.session) {
|
||||
agentSceneSession.current.stopAgent('地图编辑使旧计划失效');
|
||||
agentSceneSession.current.setPaused(true);
|
||||
publishExternalSnapshot();
|
||||
return;
|
||||
}
|
||||
try {
|
||||
const visuals = visualMapAssets(current, assets);
|
||||
if (reload)
|
||||
@@ -1935,6 +1992,10 @@ export function App() {
|
||||
adapter.current.setPaused(value);
|
||||
};
|
||||
const reset = () => {
|
||||
if (agentSceneSession.current && agentSceneSession.current === adapter.current.session) {
|
||||
void prepareAgentScene(lastAgentConfig.current).catch(() => undefined);
|
||||
return;
|
||||
}
|
||||
adapter.current.setPaused(true);
|
||||
adapter.current.reset();
|
||||
state.setSnapshot(adapter.current.snapshot() ?? undefined);
|
||||
@@ -1950,6 +2011,7 @@ export function App() {
|
||||
};
|
||||
const jointPoseEditingDisabled = Boolean(
|
||||
state.snapshot?.robot ||
|
||||
state.snapshot?.mobileRobotId ||
|
||||
(state.snapshot?.controlOwner && state.snapshot.controlOwner !== 'manual'),
|
||||
);
|
||||
const mode = (value: InteractionMode) => {
|
||||
@@ -2145,6 +2207,75 @@ export function App() {
|
||||
setSelectedPolicyPath(path);
|
||||
void loadPolicyBytes(file.data, path).catch(() => {});
|
||||
};
|
||||
const prepareAgentScene = async (config: PickPlaceTaskConfig): Promise<SimulationSession> => {
|
||||
const entry = manifest.current?.entries.find(
|
||||
(e) =>
|
||||
e.format === 'urdf' &&
|
||||
(robotProfilesRef.current.get(e.path) === 'lekiwi-v1' ||
|
||||
/(^|\/)URDF\/LeKiwi\.urdf$/.test(e.path)),
|
||||
);
|
||||
if (!entry) throw new Error('请先导入 A / lekiwi-v1 ZIP;不会使用其他机器人替代');
|
||||
if (
|
||||
!(await loadEntry(
|
||||
entry.path,
|
||||
'mjcf',
|
||||
undefined,
|
||||
undefined,
|
||||
undefined,
|
||||
PICK_PLACE_ROBOT,
|
||||
config,
|
||||
))
|
||||
)
|
||||
throw new Error('A 任务场景加载失败,请查看编译诊断');
|
||||
return adapter.current.session!;
|
||||
};
|
||||
const prepareMobileScene = async (taskId: string) => {
|
||||
if (loadInFlight.current || policyLoadInFlight.current) throw new Error('模型/策略正在加载');
|
||||
const config = mobileTrainingRobot(taskId);
|
||||
const entry = useAppStore.getState().selectedEntry;
|
||||
if (!entry || !manifest.current) throw new Error('请先导入对应机器人 URDF/ZIP');
|
||||
if (
|
||||
agentSceneSession.current === adapter.current.session ||
|
||||
adapter.current.session?.mobile?.env.config.id !== config.id
|
||||
) {
|
||||
if (!(await loadEntry(entry, 'mjcf', undefined, undefined, undefined, config)))
|
||||
throw new Error(useAppStore.getState().diagnostic?.detail ?? '移动操作场景加载失败');
|
||||
}
|
||||
return { config, entry };
|
||||
};
|
||||
const importMobilePolicy = async (file: File, deployment: MobileDeployment) => {
|
||||
if (deployment.robotId !== mobileTrainingRobot(deployment.trainingTaskId).id)
|
||||
throw new Error('部署变体不匹配');
|
||||
await prepareMobileScene(deployment.trainingTaskId);
|
||||
const session = adapter.current.session!;
|
||||
policyLoadInFlight.current = true;
|
||||
state.setLoading(true);
|
||||
const previousPaused = session.paused;
|
||||
session.setPaused(true);
|
||||
try {
|
||||
if (deployment.sceneSha256 !== (await sha256(adapter.current.exportMjcf())))
|
||||
throw new Error('当前场景与训练场景 SHA-256 不匹配,请恢复训练时的模型');
|
||||
if (file.size > 64 * 1024 ** 2) throw new Error('策略超过 64 MiB');
|
||||
const data = new Uint8Array(await file.arrayBuffer());
|
||||
if (session !== adapter.current.session) throw new Error('场景已切换');
|
||||
await session.loadMobilePolicy(data, file.name, deployment);
|
||||
if (session !== adapter.current.session) throw new Error('场景已切换');
|
||||
session.setPaused(false);
|
||||
state.setPaused(false);
|
||||
setPolicyStatus(session.snapshot().rlPolicy);
|
||||
state.setSnapshot(session.snapshot());
|
||||
notify('移动操作策略已导入并运行', deployment.trainingTaskId);
|
||||
} catch (error) {
|
||||
if (session === adapter.current.session) {
|
||||
session.setPaused(previousPaused);
|
||||
state.setPaused(previousPaused);
|
||||
}
|
||||
throw error;
|
||||
} finally {
|
||||
policyLoadInFlight.current = false;
|
||||
state.setLoading(false);
|
||||
}
|
||||
};
|
||||
const importPolicy = async (file: File, expected?: PolicyDeployment) => {
|
||||
try {
|
||||
if (!/\.onnx$/i.test(file.name)) throw new Error('请选择 .onnx 文件');
|
||||
@@ -2280,11 +2411,26 @@ export function App() {
|
||||
try {
|
||||
const data = adapter.current.exportMjcf();
|
||||
downloadBytes(data, exportedFileName(manifest.current?.name ?? 'model', 'xml'));
|
||||
notify('MJCF 已导出', '导出内容来自当前已编译模型');
|
||||
notify('MJCF 已导出', '仅 XML;如需携带原始及派生网格,请使用“导出完整仿真工程 ZIP”');
|
||||
} catch (error) {
|
||||
state.setDiagnostic(diagnostic('模型编译', error, state.selectedEntry));
|
||||
}
|
||||
};
|
||||
const exportProjectPackage = () => {
|
||||
if (!manifest.current || !state.selectedEntry) return;
|
||||
try {
|
||||
const result = compiledProjectPackage(
|
||||
manifest.current,
|
||||
state.selectedEntry,
|
||||
adapter.current.exportMjcf(),
|
||||
adapter.current.cachedSupportFiles(),
|
||||
);
|
||||
downloadBytes(result.data, exportedFileName(manifest.current.name, 'zip'), 'application/zip');
|
||||
notify('完整仿真工程已导出', `包含原始资源、派生网格及 ${result.entryPath}`);
|
||||
} catch (error) {
|
||||
state.setDiagnostic(diagnostic('文件系统', error, state.selectedEntry));
|
||||
}
|
||||
};
|
||||
const toggleFullscreen = () => {
|
||||
if (document.fullscreenElement) void document.exitFullscreen().catch(() => {});
|
||||
else if (root.current) void root.current.requestFullscreen().catch(() => {});
|
||||
@@ -2430,6 +2576,14 @@ export function App() {
|
||||
disabled: !state.snapshot,
|
||||
run: exportMjcf,
|
||||
},
|
||||
{
|
||||
id: 'export-project-package',
|
||||
label: '导出完整仿真工程 ZIP',
|
||||
group: '工程',
|
||||
icon: <Download className="h-4 w-4" />,
|
||||
disabled: !state.snapshot || state.loading,
|
||||
run: exportProjectPackage,
|
||||
},
|
||||
{
|
||||
id: 'workspace-controls',
|
||||
label: '在右侧打开控制台',
|
||||
@@ -2500,6 +2654,21 @@ export function App() {
|
||||
active={workspaceTool ?? 'controls'}
|
||||
snapshot={state.snapshot}
|
||||
loading={state.loading}
|
||||
agentTask={
|
||||
<AgentTaskPanel
|
||||
loading={state.loading}
|
||||
bridge={{
|
||||
loadScene: prepareAgentScene,
|
||||
getSession: () => adapter.current.session ?? undefined,
|
||||
isTaskScene: () =>
|
||||
Boolean(
|
||||
agentSceneSession.current &&
|
||||
agentSceneSession.current === adapter.current.session,
|
||||
),
|
||||
publish: publishExternalSnapshot,
|
||||
}}
|
||||
/>
|
||||
}
|
||||
externalControl={
|
||||
state.snapshot && (
|
||||
<ExternalControlPanel
|
||||
@@ -2542,10 +2711,19 @@ export function App() {
|
||||
onSelectPolicyPath={setSelectedPolicyPath}
|
||||
onLoadPolicyPath={loadPolicyPath}
|
||||
onImportPolicy={importPolicy}
|
||||
mobileTraining={{
|
||||
robotId: state.snapshot?.mobileRobotId ?? state.snapshot?.robot?.profileId,
|
||||
prepare: async (taskId) => {
|
||||
const { config, entry } = await prepareMobileScene(taskId);
|
||||
return mobileSceneSnapshot(adapter.current, manifest.current!, entry, config);
|
||||
},
|
||||
importPolicy: importMobilePolicy,
|
||||
}}
|
||||
compileTrainingScene={(coordinates) => {
|
||||
if (
|
||||
mapSceneDirty ||
|
||||
trainingDeployment ||
|
||||
agentScene ||
|
||||
useAppStore.getState().loading ||
|
||||
loadInFlight.current
|
||||
)
|
||||
@@ -2553,7 +2731,7 @@ export function App() {
|
||||
return adapter.current.exportTrainingTerrain(appliedMapAssets, coordinates);
|
||||
}}
|
||||
trainingSceneMaps={appliedMapAssets}
|
||||
trainingSceneDirty={mapSceneDirty || Boolean(trainingDeployment)}
|
||||
trainingSceneDirty={mapSceneDirty || Boolean(trainingDeployment) || agentScene}
|
||||
onTogglePolicy={togglePolicy}
|
||||
onPolicyCommand={setPolicyCommand}
|
||||
navigationTargetMode={navigationTargetMode}
|
||||
|
||||
@@ -113,6 +113,10 @@ export function UrdfImportOptionsDialog({
|
||||
onChange={(e) => setProfileId(e.target.value)}
|
||||
>
|
||||
<option value="">不启用(通用转换)</option>
|
||||
<option value="MobileManipulator-LeKiwi-v1">LeKiwi v1 · 移动操作训练场景</option>
|
||||
<option value="MobileManipulator-LeKiwi-Bundle">
|
||||
LeKiwi Bundle · 移动操作训练场景
|
||||
</option>
|
||||
{ROBOT_PROFILES.map((p) => (
|
||||
<option key={p.id} value={p.id}>
|
||||
{p.label}
|
||||
@@ -120,9 +124,15 @@ export function UrdfImportOptionsDialog({
|
||||
))}
|
||||
</Select>
|
||||
</label>
|
||||
{profileId && (
|
||||
{profileId.startsWith('MobileManipulator-') && (
|
||||
<p className="text-xs text-warning">
|
||||
将校验固定模型版本,采用浮动基座、简化被动滚子/碰撞和九路伺服;不使用下方通用组件,不启用相机。
|
||||
加载抓取物体与搬运目标;进入控制台 →
|
||||
强化学习任务即可一键训练。移动操作模式不启用外部控制桥。
|
||||
</p>
|
||||
)}
|
||||
{profileId && !profileId.startsWith('MobileManipulator-') && (
|
||||
<p className="text-xs text-warning">
|
||||
将校验模型语义与网格:允许换色、格式等非机械变体,拒绝关节拓扑、轴向、尺寸等变化。采用浮动基座、简化被动滚子/碰撞和九路伺服;不使用下方通用组件,不启用相机。
|
||||
</p>
|
||||
)}
|
||||
<fieldset
|
||||
|
||||
@@ -1,6 +1,7 @@
|
||||
import type { ChangeEvent, ReactNode } from 'react';
|
||||
import { Boxes, Code2, FolderOpen, PanelLeft, PanelRight, Search, Upload } from 'lucide-react';
|
||||
import { IconButton } from '../../components/ui';
|
||||
import { WEBSITE_MODE } from '../websiteConfig';
|
||||
const fileActionClass =
|
||||
'cyber-action inline-flex h-8 cursor-pointer items-center gap-1.5 rounded-md border border-border bg-surface/80 px-2 text-xs font-medium text-text-primary shadow-sm transition-[background-color,border-color,transform] hover:-translate-y-px hover:border-border-strong hover:bg-element-hover focus-within:ring-2 focus-within:ring-accent/30';
|
||||
const fileActionLabelClass = 'whitespace-nowrap';
|
||||
@@ -39,7 +40,7 @@ export function WorkbenchHeader({
|
||||
<Boxes aria-hidden="true" className="h-3.5 w-3.5" />
|
||||
</span>
|
||||
<h1 className="truncate text-sm font-semibold tracking-tight text-text-primary">
|
||||
MuJoCo
|
||||
{WEBSITE_MODE ? 'CADWorld Sim' : 'MuJoCo'}
|
||||
</h1>
|
||||
</div>
|
||||
<label
|
||||
@@ -103,6 +104,7 @@ export function WorkbenchHeader({
|
||||
>
|
||||
<PanelRight className="h-4 w-4" />
|
||||
</IconButton>
|
||||
|
||||
{endActions}
|
||||
{compactMenu}
|
||||
<IconButton tooltip="命令面板(Ctrl+K)" aria-label="打开命令面板" onClick={onCommands}>
|
||||
|
||||
@@ -56,6 +56,17 @@ function props(): ComponentProps<typeof WorkspaceToolsPanel> {
|
||||
}
|
||||
|
||||
describe('WorkspaceToolsPanel', () => {
|
||||
it('旧描述符未声明 training 也可展开统一训练入口', () => {
|
||||
const values = props();
|
||||
values.snapshot = {
|
||||
...snapshot,
|
||||
robot: { capabilities: { training: false } },
|
||||
} as SimulationSnapshot;
|
||||
render(<WorkspaceToolsPanel {...values} />);
|
||||
fireEvent.click(screen.getByRole('button', { name: '强化学习任务' }));
|
||||
expect(screen.getByLabelText('训练服务访问令牌')).toBeVisible();
|
||||
expect(screen.queryByText(/不支持训练/)).not.toBeInTheDocument();
|
||||
});
|
||||
it('首次展开后折叠训练组不丢连接凭据与状态', () => {
|
||||
render(<WorkspaceToolsPanel {...props()} />);
|
||||
expect(screen.queryByLabelText('训练服务访问令牌')).not.toBeInTheDocument();
|
||||
|
||||
@@ -1,3 +1,5 @@
|
||||
import type { MobileTrainingBridge } from '../../mobile/training';
|
||||
import { websiteCapabilities } from '../websiteConfig';
|
||||
import { useState, type ReactNode } from 'react';
|
||||
import type { TrainingSceneCompiler } from '../../map/trainingMap';
|
||||
import type { PolicyDeployment } from '../../rl/deployment';
|
||||
@@ -38,6 +40,7 @@ export function WorkspaceToolsPanel({
|
||||
snapshot,
|
||||
loading,
|
||||
externalControl,
|
||||
agentTask,
|
||||
ignoreJointLimits,
|
||||
jointAdvanced,
|
||||
angleUnit,
|
||||
@@ -65,6 +68,7 @@ export function WorkspaceToolsPanel({
|
||||
onSelectPolicyPath,
|
||||
onLoadPolicyPath,
|
||||
onImportPolicy,
|
||||
mobileTraining,
|
||||
compileTrainingScene,
|
||||
trainingSceneMaps,
|
||||
trainingSceneDirty,
|
||||
@@ -84,6 +88,7 @@ export function WorkspaceToolsPanel({
|
||||
snapshot?: SimulationSnapshot;
|
||||
loading: boolean;
|
||||
externalControl?: ReactNode;
|
||||
agentTask?: ReactNode;
|
||||
ignoreJointLimits: boolean;
|
||||
jointAdvanced: boolean;
|
||||
angleUnit: 'rad' | 'deg';
|
||||
@@ -114,6 +119,7 @@ export function WorkspaceToolsPanel({
|
||||
onSelectPolicyPath: (path: string) => void;
|
||||
onLoadPolicyPath: (path: string) => void;
|
||||
onImportPolicy: (file: File, deployment?: PolicyDeployment) => void | Promise<void>;
|
||||
mobileTraining?: MobileTrainingBridge;
|
||||
compileTrainingScene?: TrainingSceneCompiler;
|
||||
trainingSceneMaps?: readonly PlacedMapAsset[];
|
||||
trainingSceneDirty?: boolean;
|
||||
@@ -133,7 +139,12 @@ export function WorkspaceToolsPanel({
|
||||
const resolvedPolicyStatus = policyStatus ?? snapshot?.rlPolicy;
|
||||
const controls = snapshot ? (
|
||||
<>
|
||||
{externalControl && (
|
||||
{agentTask && (
|
||||
<CollapsibleSection title="LeKiwi 智能抓放" defaultOpen={false} keepMounted>
|
||||
{agentTask}
|
||||
</CollapsibleSection>
|
||||
)}
|
||||
{websiteCapabilities.externalControl && externalControl && (
|
||||
<CollapsibleSection
|
||||
title="开源项目 / 外部控制"
|
||||
defaultOpen={Boolean(snapshot.robot)}
|
||||
@@ -157,7 +168,7 @@ export function WorkspaceToolsPanel({
|
||||
key={actuator.id}
|
||||
showParameterNotice={false}
|
||||
disabled={manualDisabled}
|
||||
parametersDisabled={Boolean(snapshot.robot)}
|
||||
parametersDisabled={Boolean(snapshot.robot || snapshot.mobileRobotId)}
|
||||
actuator={actuator}
|
||||
onControl={(value) => onActuator(actuator.id, value)}
|
||||
onParameters={(parameters) => onActuatorParameters(actuator.id, parameters)}
|
||||
@@ -174,13 +185,16 @@ export function WorkspaceToolsPanel({
|
||||
badge={<ConsoleSectionBadges count={snapshot.joints.length} />}
|
||||
>
|
||||
<div className="mb-4 grid grid-cols-2 gap-2">
|
||||
<Button onClick={onResetJoints} disabled={manualDisabled}>
|
||||
<Button
|
||||
onClick={onResetJoints}
|
||||
disabled={manualDisabled || Boolean(snapshot.robot || snapshot.mobileRobotId)}
|
||||
>
|
||||
重置关节
|
||||
</Button>
|
||||
<Button
|
||||
variant={ignoreJointLimits ? 'primary' : 'secondary'}
|
||||
aria-pressed={ignoreJointLimits}
|
||||
disabled={manualDisabled}
|
||||
disabled={manualDisabled || Boolean(snapshot.robot || snapshot.mobileRobotId)}
|
||||
onClick={onToggleJointLimits}
|
||||
>
|
||||
忽略关节限位
|
||||
@@ -309,31 +323,28 @@ export function WorkspaceToolsPanel({
|
||||
onRemove={onRemovePolicy}
|
||||
/>
|
||||
</CollapsibleSection>
|
||||
<CollapsibleSection
|
||||
title="强化学习任务"
|
||||
defaultOpen={false}
|
||||
keepMounted
|
||||
badge={
|
||||
<ConsoleSectionBadges
|
||||
state={trainingSummary || undefined}
|
||||
running={trainingSummary === '训练中'}
|
||||
/>
|
||||
}
|
||||
>
|
||||
{snapshot.robot && !snapshot.robot.capabilities.training ? (
|
||||
<p className="text-xs text-text-secondary">
|
||||
此机器人接口 V1 不支持训练。既有 Go2 训练任务不受影响,请使用对应模型。
|
||||
</p>
|
||||
) : (
|
||||
{websiteCapabilities.training && (
|
||||
<CollapsibleSection
|
||||
title="强化学习任务"
|
||||
defaultOpen={false}
|
||||
keepMounted
|
||||
badge={
|
||||
<ConsoleSectionBadges
|
||||
state={trainingSummary || undefined}
|
||||
running={trainingSummary === '训练中'}
|
||||
/>
|
||||
}
|
||||
>
|
||||
<LocalTrainingPanel
|
||||
mobileTraining={mobileTraining}
|
||||
onStatusChange={setTrainingSummary}
|
||||
onPolicyReady={onImportPolicy}
|
||||
compileScene={compileTrainingScene}
|
||||
sceneMaps={trainingSceneMaps}
|
||||
sceneDirty={trainingSceneDirty}
|
||||
/>
|
||||
)}
|
||||
</CollapsibleSection>
|
||||
</CollapsibleSection>
|
||||
)}
|
||||
</>
|
||||
) : (
|
||||
<p className="p-4 text-sm text-text-tertiary">导入模型后显示控制工具</p>
|
||||
|
||||
@@ -0,0 +1,7 @@
|
||||
/** Build-time product boundary; never contains credentials or upstream URLs. */
|
||||
export const WEBSITE_MODE = import.meta.env.MODE === 'website';
|
||||
export const websiteCapabilities = {
|
||||
training: !WEBSITE_MODE,
|
||||
externalControl: !WEBSITE_MODE,
|
||||
subscription: true,
|
||||
} as const;
|
||||
@@ -0,0 +1,335 @@
|
||||
import type { SimulationSession } from '../simulation/SimulationSession';
|
||||
import { ModelBindings, type ScalarActuatorBinding } from '../simulation/ModelBindings';
|
||||
import { MOBILE_TASK as T, validateRobotConfig, type RobotConfig } from './RobotDescriptor';
|
||||
import {
|
||||
clamp,
|
||||
TaskKernel,
|
||||
TRAINING_STAGES,
|
||||
type TrainingStage,
|
||||
type StepResult,
|
||||
} from './TaskKernel';
|
||||
import { SafeActionController } from './SafeActionController';
|
||||
|
||||
export class MobileManipulatorWasmEnv {
|
||||
readonly kernel: TaskKernel;
|
||||
readonly motion: SafeActionController;
|
||||
readonly state = new Float64Array(T.stateSize);
|
||||
readonly action = new Float32Array(T.actionSize);
|
||||
readonly control: Float64Array;
|
||||
readonly controlAddresses: Int32Array;
|
||||
readonly objectBodyId: number;
|
||||
readonly goalBodyId: number;
|
||||
readonly frameSkip: number;
|
||||
revision = 0;
|
||||
private disposed = false;
|
||||
private readonly arm: ScalarActuatorBinding[];
|
||||
private readonly gripperQ: number;
|
||||
private readonly grippers: ScalarActuatorBinding[];
|
||||
private readonly baseQ: number;
|
||||
private readonly baseV: number;
|
||||
private readonly objectQ: number;
|
||||
private readonly objectV: number;
|
||||
private readonly eefBody: number;
|
||||
private readonly eefSite: number;
|
||||
private readonly goalMocap: number;
|
||||
private qpos!: Float64Array;
|
||||
private qvel!: Float64Array;
|
||||
private ctrl!: Float64Array;
|
||||
private xpos!: Float64Array;
|
||||
private xquat!: Float64Array;
|
||||
private sitePos!: Float64Array;
|
||||
private siteMat!: Float64Array;
|
||||
private mocapPos!: Float64Array;
|
||||
private mocapQuat!: Float64Array;
|
||||
|
||||
constructor(
|
||||
readonly session: SimulationSession,
|
||||
readonly config: RobotConfig,
|
||||
) {
|
||||
validateRobotConfig(config);
|
||||
const { model, module } = session,
|
||||
b = new ModelBindings(model);
|
||||
const free = (name: string): [number, number] => {
|
||||
const id = b.resolve(b.joints, name);
|
||||
if (Number(model.jnt_type[id]) !== 0) throw new Error(`${name} 必须是 freejoint`);
|
||||
return [Number(model.jnt_qposadr[id]), Number(model.jnt_dofadr[id])];
|
||||
};
|
||||
[this.baseQ, this.baseV] = free(config.baseJointName);
|
||||
if (
|
||||
Number(model.jnt_bodyid[b.resolve(b.joints, config.baseJointName)]) !==
|
||||
b.resolve(b.bodies, config.baseBodyName)
|
||||
)
|
||||
throw new Error('底座自由关节不属于配置的 baseBodyName');
|
||||
[this.objectQ, this.objectV] = free('__mm_object_joint');
|
||||
this.objectBodyId = b.resolve(b.bodies, '__mm_object');
|
||||
this.goalBodyId = b.resolve(b.bodies, '__mm_goal');
|
||||
this.goalMocap = Number(model.body_mocapid[this.goalBodyId]);
|
||||
if (this.goalMocap < 0) throw new Error('目标必须是 mocap body');
|
||||
this.eefBody = b.resolve(b.bodies, config.eefBodyName);
|
||||
this.eefSite = config.eefSiteName ? module.mj_name2id(model, 6, config.eefSiteName) : -1;
|
||||
if (config.eefSiteName && this.eefSite < 0) throw new Error('找不到 EEF site');
|
||||
this.arm = config.armJoints.map((j, i) =>
|
||||
b.scalarActuator(config.armActuators[i], j.name, j.mode),
|
||||
);
|
||||
this.gripperQ = b.scalarJoint(config.gripperJoint).qposAddress;
|
||||
this.grippers = config.gripperActuators.map((g) =>
|
||||
b.scalarActuator(g.name, g.joint ?? config.gripperJoint, 'position'),
|
||||
);
|
||||
const bindings = [
|
||||
...config.baseActuators.map((n, i) => b.scalarActuator(n, config.baseJoints[i], 'velocity')),
|
||||
...this.arm,
|
||||
...this.grippers,
|
||||
];
|
||||
config.armJoints.forEach((spec, i) => {
|
||||
const joint = model.jnt(this.arm[i].id);
|
||||
try {
|
||||
if (
|
||||
!joint.limited ||
|
||||
Math.abs(Number(joint.range[0]) - spec.min) > 1e-5 ||
|
||||
Math.abs(Number(joint.range[1]) - spec.max) > 1e-5
|
||||
)
|
||||
throw new Error(`关节限位与 RobotConfig 不匹配:${spec.name}`);
|
||||
} finally {
|
||||
joint.delete();
|
||||
}
|
||||
});
|
||||
this.controlAddresses = Int32Array.from(bindings.map((a) => a.controlAddress));
|
||||
this.motion = new SafeActionController(config);
|
||||
this.control = this.motion.control;
|
||||
// Fail closed instead of silently clipping against a different compiled controller.
|
||||
const ranges = [
|
||||
...config.baseActuators.map(() => [-config.wheelLimit, config.wheelLimit]),
|
||||
...config.armJoints.map((j) =>
|
||||
j.mode === 'position' ? [j.min, j.max] : [-j.velocityLimit, j.velocityLimit],
|
||||
),
|
||||
...config.gripperActuators.map((g) => [
|
||||
Math.min(g.closed, g.open),
|
||||
Math.max(g.closed, g.open),
|
||||
]),
|
||||
];
|
||||
bindings.forEach((binding, i) => {
|
||||
// 3.11's bool-array memory_view binding is unavailable; use the scalar view.
|
||||
const actuator = model.actuator(binding.actuatorId);
|
||||
try {
|
||||
if (
|
||||
!actuator.ctrllimited ||
|
||||
Math.abs(Number(actuator.ctrlrange[0]) - ranges[i][0]) > 1e-5 ||
|
||||
Math.abs(Number(actuator.ctrlrange[1]) - ranges[i][1]) > 1e-5
|
||||
)
|
||||
throw new Error('RobotConfig 与编译模型 ctrlrange 不匹配');
|
||||
} finally {
|
||||
actuator.delete();
|
||||
}
|
||||
});
|
||||
const opt = model.opt;
|
||||
const dt = Number(opt.timestep);
|
||||
opt.delete();
|
||||
this.frameSkip = Math.round(T.controlDt / dt);
|
||||
if (this.frameSkip < 1 || Math.abs(this.frameSkip * dt - T.controlDt) > 1e-9)
|
||||
throw new Error('controlDt 必须是物理 dt 的整数倍');
|
||||
this.kernel = new TaskKernel(config);
|
||||
this.refreshViews();
|
||||
this.reset();
|
||||
}
|
||||
|
||||
/** Only at load/resume barriers: another model compilation may grow shared WASM memory. */
|
||||
refreshViews(): void {
|
||||
const d = this.session.data;
|
||||
this.qpos = d.qpos;
|
||||
this.qvel = d.qvel;
|
||||
this.ctrl = d.ctrl;
|
||||
this.xpos = d.xpos;
|
||||
this.xquat = d.xquat;
|
||||
this.sitePos = d.site_xpos;
|
||||
this.siteMat = d.site_xmat;
|
||||
this.mocapPos = d.mocap_pos;
|
||||
this.mocapQuat = d.mocap_quat;
|
||||
}
|
||||
get wasmHeapBytes(): number {
|
||||
return this.qpos.buffer.byteLength;
|
||||
}
|
||||
private check(): void {
|
||||
if (this.disposed) throw new Error('环境已释放');
|
||||
}
|
||||
private forward(): void {
|
||||
this.session.module.mj_forward(this.session.model, this.session.data);
|
||||
}
|
||||
|
||||
setTrainingStage(stage: TrainingStage): void {
|
||||
if (!TRAINING_STAGES.includes(stage)) throw new Error('未知训练阶段');
|
||||
if (this.kernel.trainingStage !== stage) {
|
||||
this.kernel.trainingStage = stage;
|
||||
this.kernel.reset();
|
||||
this.hold();
|
||||
this.revision++;
|
||||
}
|
||||
}
|
||||
reset(options?: { object?: readonly number[]; goal?: readonly number[] }): Float32Array {
|
||||
this.check();
|
||||
const { module, model, data } = this.session;
|
||||
module.mj_resetData(model, data);
|
||||
this.action.fill(0);
|
||||
for (let i = 0; i < this.arm.length; i++) {
|
||||
const spec = this.config.armJoints[i];
|
||||
this.qpos[this.arm[i].qposAddress] = spec.neutral;
|
||||
}
|
||||
this.qpos[this.gripperQ] = this.config.gripperOpen;
|
||||
for (let i = 0; i < this.grippers.length; i++)
|
||||
this.qpos[this.grippers[i].qposAddress] = this.config.gripperActuators[i].open;
|
||||
this.kernel.reset();
|
||||
this.revision++;
|
||||
this.forward();
|
||||
this.hold(false);
|
||||
if (options?.object) this.moveTaskEntity('object', options.object);
|
||||
if (options?.goal) this.moveTaskEntity('goal', options.goal);
|
||||
return this.observe();
|
||||
}
|
||||
|
||||
/** Preserve safe servo targets (not gravity-sagged qpos); zero velocity commands.
|
||||
* Re-anchoring to measured q on every pause would make a loaded arm progressively sag. */
|
||||
hold(preserveTargets = true): void {
|
||||
this.check();
|
||||
this.action.fill(0);
|
||||
this.capture();
|
||||
this.motion.reset(
|
||||
this.state,
|
||||
preserveTargets ? Float64Array.from(this.controlAddresses, (i) => this.ctrl[i]) : undefined,
|
||||
);
|
||||
this.kernel.recordAction(this.motion.applied, this.motion.targets);
|
||||
for (let i = 0; i < this.control.length; i++)
|
||||
this.ctrl[this.controlAddresses[i]] = this.control[i];
|
||||
}
|
||||
applyAction(action: Float32Array, manual = false): void {
|
||||
this.check();
|
||||
this.motion.apply(
|
||||
action,
|
||||
this.state,
|
||||
manual ? 'pick-place' : this.kernel.trainingStage,
|
||||
manual || this.kernel.hasLifted,
|
||||
);
|
||||
this.kernel.recordAction(this.motion.applied, this.motion.targets);
|
||||
for (let i = 0; i < this.control.length; i++)
|
||||
this.ctrl[this.controlAddresses[i]] = this.control[i];
|
||||
}
|
||||
observe(): Float32Array {
|
||||
this.check();
|
||||
this.capture();
|
||||
return this.kernel.observe(this.state);
|
||||
}
|
||||
step(action: Float32Array, manual = false): StepResult {
|
||||
this.check();
|
||||
if (this.kernel.result.terminated || this.kernel.result.truncated)
|
||||
throw new Error('回合结束,请 reset');
|
||||
this.applyAction(action, manual);
|
||||
let peak = 0,
|
||||
safety = '';
|
||||
for (let i = 0; i < this.frameSkip; i++) {
|
||||
this.session.module.mj_step(this.session.model, this.session.data);
|
||||
for (const joint of this.arm) peak = Math.max(peak, Math.abs(this.qvel[joint.qvelAddress]));
|
||||
for (const joint of this.grippers)
|
||||
peak = Math.max(peak, Math.abs(this.qvel[joint.qvelAddress]));
|
||||
if (peak > T.jointSpeedStop) {
|
||||
safety = 'joint_velocity';
|
||||
break;
|
||||
}
|
||||
}
|
||||
// mj_step leaves position-derived fields at the integration boundary; both runtimes forward.
|
||||
this.forward();
|
||||
this.capture();
|
||||
return this.kernel.evaluate(this.state, safety, peak);
|
||||
}
|
||||
private capture(): void {
|
||||
const s = this.state;
|
||||
for (let i = 0; i < 7; i++) {
|
||||
s[i] = this.qpos[this.baseQ + i];
|
||||
s[37 + i] = this.qpos[this.objectQ + i];
|
||||
}
|
||||
for (let i = 0; i < 3; i++) {
|
||||
s[7 + i] = this.qvel[this.baseV + i];
|
||||
s[44 + i] = this.qvel[this.objectV + i];
|
||||
}
|
||||
// MuJoCo freejoint angular velocity is local; expose world angular velocity in SI.
|
||||
const w = s[3],
|
||||
x = s[4],
|
||||
y = s[5],
|
||||
z = s[6];
|
||||
const vx = this.qvel[this.baseV + 3],
|
||||
vy = this.qvel[this.baseV + 4],
|
||||
vz = this.qvel[this.baseV + 5];
|
||||
const tx = 2 * (y * vz - z * vy),
|
||||
ty = 2 * (z * vx - x * vz),
|
||||
tz = 2 * (x * vy - y * vx);
|
||||
s[10] = vx + w * tx + y * tz - z * ty;
|
||||
s[11] = vy + w * ty + z * tx - x * tz;
|
||||
s[12] = vz + w * tz + x * ty - y * tx;
|
||||
for (let i = 0; i < this.arm.length; i++) {
|
||||
s[13 + i] = this.qpos[this.arm[i].qposAddress];
|
||||
s[21 + i] = this.qvel[this.arm[i].qvelAddress];
|
||||
}
|
||||
s[29] = clamp(
|
||||
(this.qpos[this.gripperQ] - this.config.gripperClosed) /
|
||||
(this.config.gripperOpen - this.config.gripperClosed),
|
||||
0,
|
||||
1,
|
||||
);
|
||||
for (let i = 0; i < 3; i++) {
|
||||
s[30 + i] =
|
||||
this.eefSite >= 0 ? this.sitePos[3 * this.eefSite + i] : this.xpos[3 * this.eefBody + i];
|
||||
s[47 + i] = this.mocapPos[3 * this.goalMocap + i];
|
||||
}
|
||||
if (this.eefSite >= 0) matrixQuaternion(this.siteMat, 9 * this.eefSite, s, 33);
|
||||
else for (let i = 0; i < 4; i++) s[33 + i] = this.xquat[4 * this.eefBody + i];
|
||||
for (let i = 0; i < 4; i++) s[50 + i] = this.mocapQuat[4 * this.goalMocap + i];
|
||||
for (let i = 0; i < s.length; i++)
|
||||
if (!Number.isFinite(s[i])) throw new Error('仿真产生非有限状态,已停止');
|
||||
}
|
||||
/** World XY dragging on a fixed plane; editing begins a new task episode, not a robot reset. */
|
||||
moveTaskEntity(entity: 'object' | 'goal', position: readonly number[]): void {
|
||||
this.check();
|
||||
if (
|
||||
position.length !== 3 ||
|
||||
position.some((v) => !Number.isFinite(v) || Math.abs(v) > T.positionScale)
|
||||
)
|
||||
throw new Error('拖动位置超出工作空间');
|
||||
const target = entity === 'object' ? this.qpos : this.mocapPos;
|
||||
const offset = entity === 'object' ? this.objectQ : this.goalMocap * 3;
|
||||
for (let i = 0; i < 3; i++)
|
||||
target[offset + i] = i === 2 ? Math.max(T.objectStart[2], position[i]) : position[i];
|
||||
if (entity === 'object') {
|
||||
for (let i = 0; i < 6; i++) this.qvel[this.objectV + i] = 0;
|
||||
}
|
||||
this.session.data.qacc_warmstart.fill(0);
|
||||
this.kernel.reset();
|
||||
this.revision++;
|
||||
this.forward();
|
||||
this.hold();
|
||||
this.observe();
|
||||
}
|
||||
dispose(): void {
|
||||
this.disposed = true;
|
||||
this.revision++;
|
||||
}
|
||||
}
|
||||
|
||||
/** Matrix to unit quaternion, wxyz; no temporary array or native scratch allocation. */
|
||||
function matrixQuaternion(m: Float64Array, a: number, out: Float64Array, b: number): void {
|
||||
const trace = m[a] + m[a + 4] + m[a + 8];
|
||||
if (trace > 0) {
|
||||
const r = Math.sqrt(1 + trace) * 2;
|
||||
out[b] = r / 4;
|
||||
out[b + 1] = (m[a + 7] - m[a + 5]) / r;
|
||||
out[b + 2] = (m[a + 2] - m[a + 6]) / r;
|
||||
out[b + 3] = (m[a + 3] - m[a + 1]) / r;
|
||||
} else {
|
||||
const i = m[a] > m[a + 4] && m[a] > m[a + 8] ? 0 : m[a + 4] > m[a + 8] ? 1 : 2;
|
||||
const j = (i + 1) % 3,
|
||||
k = (i + 2) % 3;
|
||||
const r = Math.sqrt(1 + m[a + i * 3 + i] - m[a + j * 3 + j] - m[a + k * 3 + k]) * 2;
|
||||
out[b] = (m[a + k * 3 + j] - m[a + j * 3 + k]) / r;
|
||||
out[b + 1 + i] = r / 4;
|
||||
out[b + 1 + j] = (m[a + j * 3 + i] + m[a + i * 3 + j]) / r;
|
||||
out[b + 1 + k] = (m[a + k * 3 + i] + m[a + i * 3 + k]) / r;
|
||||
}
|
||||
}
|
||||
|
||||
export { MobileManipulatorWasmEnv as InteractiveManipulatorEnv };
|
||||
@@ -0,0 +1,120 @@
|
||||
import { expect, it, vi } from 'vitest';
|
||||
import { MOBILE_ROBOTS, MOBILE_TASK as T } from './RobotDescriptor';
|
||||
import type { MobileDeployment } from './training';
|
||||
import { SimulationSession } from '../simulation/SimulationSession';
|
||||
import { ControlArbiter } from '../simulation/ControlArbiter';
|
||||
const mocks = vi.hoisted(() => ({ load: vi.fn() }));
|
||||
vi.mock('./MobileManipulatorWasmEnv', () => ({
|
||||
MobileManipulatorWasmEnv: class {
|
||||
config = MOBILE_ROBOTS[0];
|
||||
reset = vi.fn();
|
||||
setTrainingStage = vi.fn();
|
||||
moveTaskEntity = vi.fn();
|
||||
dispose = vi.fn();
|
||||
kernel = { result: { terminated: false, truncated: false, info: { safety_stop: '' } } };
|
||||
},
|
||||
}));
|
||||
vi.mock('./ONNXPolicyRunner', () => ({ ONNXPolicyRunner: { load: mocks.load } }));
|
||||
import { MobilePolicyController } from './MobilePolicyController';
|
||||
const deployment = {
|
||||
trainingTaskId: 'MobileManipulator-LeKiwi-v1',
|
||||
trainingStage: 'navigate',
|
||||
resetOptions: { object: [0.25, 0, 0.019], goal: [0.55, 0.15, 0.019] },
|
||||
input: { name: 'observation', dtype: 'float32', shape: [1, T.observationSize] },
|
||||
output: { name: 'action', dtype: 'float32', shape: [1, 12] },
|
||||
} as MobileDeployment;
|
||||
const runner = () => ({
|
||||
setEnabled: vi.fn(),
|
||||
dispose: vi.fn().mockResolvedValue(undefined),
|
||||
tick: vi.fn().mockReturnValue(true),
|
||||
metrics: { inferenceCount: 1, lastInferenceMs: 1, error: '' },
|
||||
});
|
||||
it('主会话策略加载、暂停恢复、reset目标和销毁', async () => {
|
||||
const r = runner();
|
||||
mocks.load.mockResolvedValue(r);
|
||||
const controller = new MobilePolicyController({} as SimulationSession, MOBILE_ROBOTS[0]);
|
||||
await controller.load(new Uint8Array(), 'policy.onnx', deployment);
|
||||
expect(controller.env.moveTaskEntity).toHaveBeenCalledWith('goal', deployment.resetOptions.goal);
|
||||
controller.setEnabled(true);
|
||||
expect(r.setEnabled).toHaveBeenLastCalledWith(false);
|
||||
controller.setPaused(false);
|
||||
expect(r.setEnabled).toHaveBeenLastCalledWith(true);
|
||||
expect(controller.status()).toMatchObject({
|
||||
enabled: true,
|
||||
observationSize: T.observationSize,
|
||||
actionSize: 12,
|
||||
});
|
||||
controller.setPaused(true);
|
||||
expect(r.setEnabled).toHaveBeenLastCalledWith(false);
|
||||
controller.dispose();
|
||||
expect(r.dispose).toHaveBeenCalledOnce();
|
||||
expect(controller.env.dispose).toHaveBeenCalledOnce();
|
||||
});
|
||||
it('移除/销毁期间迟到ORT加载不得重置旧场景', async () => {
|
||||
let finish!: (value: ReturnType<typeof runner>) => void;
|
||||
mocks.load.mockImplementation(
|
||||
() =>
|
||||
new Promise((resolve) => {
|
||||
finish = resolve;
|
||||
}),
|
||||
);
|
||||
const c = new MobilePolicyController({} as SimulationSession, MOBILE_ROBOTS[0]);
|
||||
const loading = c.load(new Uint8Array(), 'policy.onnx', deployment);
|
||||
await vi.waitFor(() => expect(finish).toBeDefined());
|
||||
c.dispose();
|
||||
const r = runner();
|
||||
finish(r);
|
||||
await expect(loading).rejects.toThrow('取消');
|
||||
expect(r.dispose).toHaveBeenCalledOnce();
|
||||
expect(c.env.reset).not.toHaveBeenCalled();
|
||||
});
|
||||
it('移动策略交出控制权时保持关节位置,不能把位置舵机 ctrl 清零', () => {
|
||||
const ctrl = new Float64Array(9).fill(1);
|
||||
const hold = vi.fn(() => {
|
||||
ctrl.fill(0);
|
||||
ctrl[3] = 0.6;
|
||||
});
|
||||
const session = Object.assign(Object.create(SimulationSession.prototype), {
|
||||
mobile: { env: { hold } },
|
||||
data: { ctrl },
|
||||
}) as { clearControlTargets(): void };
|
||||
session.clearControlTargets();
|
||||
expect(hold).toHaveBeenCalledOnce();
|
||||
expect(ctrl[3]).toBe(0.6);
|
||||
});
|
||||
it('主会话移动策略不再调用普通 mj_step,等待单飞输出;终止即暂停', () => {
|
||||
const arbiter = new ControlArbiter();
|
||||
arbiter.claim('policy');
|
||||
const mobile = {
|
||||
tick: vi.fn().mockReturnValue(false),
|
||||
ended: false,
|
||||
error: undefined,
|
||||
setPaused: vi.fn(),
|
||||
};
|
||||
const mj_step = vi.fn();
|
||||
const capture = vi.fn();
|
||||
const s = Object.assign(Object.create(SimulationSession.prototype) as object, {
|
||||
mobile,
|
||||
controlArbiter: arbiter,
|
||||
model: { opt: { timestep: 0.002 } },
|
||||
data: { time: 0 },
|
||||
module: { mj_step },
|
||||
dataRecorder: { capture },
|
||||
paused: false,
|
||||
lastNow: 0,
|
||||
accumulator: 0,
|
||||
speed: 1,
|
||||
maxCatchUpSteps: 10,
|
||||
frameBudgetMs: 100,
|
||||
}) as unknown as SimulationSession;
|
||||
expect(s.advance(20).steps).toBe(0);
|
||||
expect(mj_step).not.toHaveBeenCalled();
|
||||
mobile.tick.mockReturnValue(true);
|
||||
expect(s.advance(21).steps).toBe(1);
|
||||
expect(capture).toHaveBeenCalledOnce();
|
||||
expect(mj_step).not.toHaveBeenCalled();
|
||||
mobile.ended = true;
|
||||
s.advance(41);
|
||||
expect(s.paused).toBe(true);
|
||||
expect(mobile.setPaused).toHaveBeenCalledWith(true);
|
||||
});
|
||||
@@ -0,0 +1,118 @@
|
||||
import type { SimulationSession } from '../simulation/SimulationSession';
|
||||
import type { RLPolicyStatus } from '../rl/types';
|
||||
import { MobileManipulatorWasmEnv } from './MobileManipulatorWasmEnv';
|
||||
import { MOBILE_TASK as T, type RobotConfig } from './RobotDescriptor';
|
||||
import type { ONNXPolicyRunner } from './ONNXPolicyRunner';
|
||||
import type { MobileDeployment } from './training';
|
||||
|
||||
/** Owned by the main SimulationSession; the ONNX runner is the sole physics writer
|
||||
* while policy control is active. In-flight results never advance retired sessions. */
|
||||
export class MobilePolicyController {
|
||||
readonly env: MobileManipulatorWasmEnv;
|
||||
private runner?: ONNXPolicyRunner;
|
||||
private generation = 0;
|
||||
private disposed = false;
|
||||
private enabled = false;
|
||||
private paused = true;
|
||||
private path = '';
|
||||
private deployment?: MobileDeployment;
|
||||
constructor(session: SimulationSession, config: RobotConfig) {
|
||||
this.env = new MobileManipulatorWasmEnv(session, structuredClone(config));
|
||||
}
|
||||
async load(bytes: Uint8Array, path: string, deployment: MobileDeployment): Promise<void> {
|
||||
const generation = ++this.generation;
|
||||
for (const position of [deployment.resetOptions?.object, deployment.resetOptions?.goal]) {
|
||||
if (
|
||||
!Array.isArray(position) ||
|
||||
position.length !== 3 ||
|
||||
position.some((v) => !Number.isFinite(v) || Math.abs(v) > T.positionScale) ||
|
||||
position[2] < T.objectStart[2]
|
||||
)
|
||||
throw new Error('策略初始目标坐标无效');
|
||||
}
|
||||
if (
|
||||
deployment.input?.name !== 'observation' ||
|
||||
deployment.output?.name !== 'action' ||
|
||||
deployment.input.dtype !== 'float32' ||
|
||||
deployment.output.dtype !== 'float32' ||
|
||||
JSON.stringify(deployment.input.shape) !== JSON.stringify([1, T.observationSize]) ||
|
||||
JSON.stringify(deployment.output.shape) !== '[1,12]'
|
||||
)
|
||||
throw new Error('移动操作部署张量契约不匹配');
|
||||
const { ONNXPolicyRunner } = await import('./ONNXPolicyRunner');
|
||||
const runner = await ONNXPolicyRunner.load(bytes, deployment, this.env);
|
||||
if (generation !== this.generation || this.disposed) {
|
||||
await runner.dispose();
|
||||
throw new Error('移动操作策略加载已取消');
|
||||
}
|
||||
this.removeRunner();
|
||||
this.runner = runner;
|
||||
this.path = path;
|
||||
this.deployment = structuredClone(deployment);
|
||||
this.reset();
|
||||
}
|
||||
private removeRunner(): void {
|
||||
this.runner?.setEnabled(false);
|
||||
void this.runner?.dispose();
|
||||
this.runner = undefined;
|
||||
this.enabled = false;
|
||||
}
|
||||
remove(): void {
|
||||
this.generation++;
|
||||
this.removeRunner();
|
||||
}
|
||||
setEnabled(value: boolean): void {
|
||||
this.enabled = value && !!this.runner;
|
||||
this.runner?.setEnabled(this.enabled && !this.paused);
|
||||
}
|
||||
setPaused(value: boolean): void {
|
||||
this.paused = value;
|
||||
this.runner?.setEnabled(this.enabled && !value);
|
||||
}
|
||||
reset(): void {
|
||||
if (this.deployment) this.env.setTrainingStage(this.deployment.trainingStage);
|
||||
this.env.reset();
|
||||
if (this.deployment) {
|
||||
this.env.moveTaskEntity('object', this.deployment.resetOptions.object);
|
||||
this.env.moveTaskEntity('goal', this.deployment.resetOptions.goal);
|
||||
}
|
||||
}
|
||||
tick(): boolean {
|
||||
return this.runner?.tick() ?? false;
|
||||
}
|
||||
get ended(): boolean {
|
||||
return this.env.kernel.result.terminated || this.env.kernel.result.truncated;
|
||||
}
|
||||
get error(): string | undefined {
|
||||
return (
|
||||
this.runner?.metrics.error ||
|
||||
(this.env.kernel.result.info.safety_stop
|
||||
? `安全停止:${this.env.kernel.result.info.safety_stop},请重置并检查模型/策略`
|
||||
: undefined)
|
||||
);
|
||||
}
|
||||
status(): RLPolicyStatus | undefined {
|
||||
if (!this.runner) return undefined;
|
||||
return {
|
||||
taskId: this.deployment!.trainingTaskId,
|
||||
taskName: `${this.env.config.label} · ${{ navigate: '底盘接近', reach: '末端接近', 'pick-place': '抓取放置' }[this.deployment!.trainingStage]}`,
|
||||
path: this.path,
|
||||
loaded: true,
|
||||
enabled: this.enabled && !this.error,
|
||||
controlHz: 1 / T.controlDt,
|
||||
observationSize: T.observationSize,
|
||||
actionSize: T.actionSize,
|
||||
inputName: 'observation',
|
||||
outputName: 'action',
|
||||
command: { linearX: 0, linearY: 0, angularZ: 0 },
|
||||
inferenceCount: this.runner.metrics.inferenceCount,
|
||||
lastInferenceMs: this.runner.metrics.lastInferenceMs,
|
||||
error: this.error,
|
||||
};
|
||||
}
|
||||
dispose(): void {
|
||||
this.disposed = true;
|
||||
this.remove();
|
||||
this.env.dispose();
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,135 @@
|
||||
import { webcrypto } from 'node:crypto';
|
||||
import { ONNXPolicyRunner, type MobilePolicyMetadata } from './ONNXPolicyRunner';
|
||||
import { MOBILE_ROBOTS, MOBILE_TASK as T } from './RobotDescriptor';
|
||||
import type { MobileManipulatorWasmEnv } from './MobileManipulatorWasmEnv';
|
||||
import { sha256 } from '../robot/registry';
|
||||
|
||||
const ort = vi.hoisted(() => ({
|
||||
create: vi.fn(),
|
||||
tensors: [] as { dispose: ReturnType<typeof vi.fn> }[],
|
||||
}));
|
||||
vi.mock('onnxruntime-web/wasm', () => ({
|
||||
env: { wasm: {} },
|
||||
InferenceSession: { create: ort.create },
|
||||
Tensor: class {
|
||||
dispose = vi.fn();
|
||||
constructor() {
|
||||
ort.tensors.push(this);
|
||||
}
|
||||
},
|
||||
}));
|
||||
const flush = async () => {
|
||||
for (let i = 0; i < 10; i++) await Promise.resolve();
|
||||
};
|
||||
async function setup() {
|
||||
Object.defineProperty(globalThis, 'crypto', { configurable: true, value: webcrypto });
|
||||
const session = {
|
||||
inputNames: ['observation'],
|
||||
outputNames: ['action'],
|
||||
inputMetadata: [{ isTensor: true, type: 'float32', shape: [1, T.observationSize] }],
|
||||
outputMetadata: [{ isTensor: true, type: 'float32', shape: [1, 12] }],
|
||||
run: vi.fn(),
|
||||
release: vi.fn(async () => {}),
|
||||
};
|
||||
ort.create.mockResolvedValue(session);
|
||||
const env = {
|
||||
config: MOBILE_ROBOTS[0],
|
||||
revision: 0,
|
||||
observe: vi.fn(() => new Float32Array(T.observationSize)),
|
||||
setTrainingStage: vi.fn(),
|
||||
step: vi.fn(),
|
||||
hold: vi.fn(),
|
||||
};
|
||||
const bytes = new Uint8Array([1, 2, 3]);
|
||||
const metadata: MobilePolicyMetadata = {
|
||||
taskId: T.id,
|
||||
robotId: env.config.id,
|
||||
observationSize: T.observationSize,
|
||||
actionSemantics: T.actionSemantics,
|
||||
trainingStage: 'navigate',
|
||||
actionSize: 12,
|
||||
controlDt: 0.02,
|
||||
normalized: true,
|
||||
modelSha256: await sha256(bytes),
|
||||
robotConfigSha256: await sha256(new TextEncoder().encode(JSON.stringify(env.config))),
|
||||
};
|
||||
const output = { type: 'float32', dims: [1, 12], data: new Float32Array(12), dispose: vi.fn() };
|
||||
return {
|
||||
session,
|
||||
env,
|
||||
bytes,
|
||||
metadata,
|
||||
output,
|
||||
load: () => ONNXPolicyRunner.load(bytes, metadata, env as unknown as MobileManipulatorWasmEnv),
|
||||
};
|
||||
}
|
||||
it('旧动作语义即使伪装新张量尺寸也拒绝加载', async () => {
|
||||
const x = await setup();
|
||||
ort.create.mockClear();
|
||||
x.metadata.taskId = 'mobile-manipulator-v1';
|
||||
await expect(x.load()).rejects.toThrow('重新训练');
|
||||
x.metadata.taskId = T.id;
|
||||
x.metadata.actionSemantics = 'absolute-position';
|
||||
await expect(x.load()).rejects.toThrow('重新训练');
|
||||
expect(ort.create).not.toHaveBeenCalled();
|
||||
});
|
||||
it('单飞推理;过期/reset/disable 结果不写入;释放等待 run 完成', async () => {
|
||||
const x = await setup(),
|
||||
runner = await x.load();
|
||||
let finish!: (v: { action: typeof x.output }) => void;
|
||||
x.session.run.mockImplementation(
|
||||
() =>
|
||||
new Promise((r) => {
|
||||
finish = r;
|
||||
}),
|
||||
);
|
||||
runner.setEnabled(true);
|
||||
runner.tick();
|
||||
runner.tick();
|
||||
expect(x.session.run).toHaveBeenCalledTimes(1);
|
||||
x.env.revision++;
|
||||
finish({ action: x.output });
|
||||
await flush();
|
||||
expect(x.env.step).not.toHaveBeenCalled();
|
||||
expect(x.output.dispose).toHaveBeenCalledOnce();
|
||||
runner.tick();
|
||||
runner.setEnabled(false);
|
||||
finish({ action: x.output });
|
||||
await flush();
|
||||
runner.setEnabled(true);
|
||||
runner.tick();
|
||||
const releasing = runner.dispose();
|
||||
expect(x.session.release).not.toHaveBeenCalled();
|
||||
finish({ action: x.output });
|
||||
await releasing;
|
||||
expect(x.session.release).toHaveBeenCalledOnce();
|
||||
expect(x.env.step).not.toHaveBeenCalled();
|
||||
});
|
||||
it('成功输出仅在下一 rAF 消费;无效输出安全停机', async () => {
|
||||
const x = await setup(),
|
||||
runner = await x.load();
|
||||
x.session.run.mockResolvedValue({ action: x.output });
|
||||
runner.setEnabled(true);
|
||||
runner.tick();
|
||||
await flush();
|
||||
expect(x.env.step).not.toHaveBeenCalled();
|
||||
expect(runner.tick()).toBe(true);
|
||||
expect(x.env.step).toHaveBeenCalledOnce();
|
||||
x.output.data[3] = NaN;
|
||||
runner.tick();
|
||||
await flush();
|
||||
expect(runner.metrics.error).toContain('NaN');
|
||||
expect(runner.tick()).toBe(false);
|
||||
await runner.dispose();
|
||||
});
|
||||
it('拒绝错误维度、机器人、权重哈希并释放已创建的 ORT session', async () => {
|
||||
const x = await setup();
|
||||
x.session.inputMetadata[0].shape = [1, 67];
|
||||
await expect(x.load()).rejects.toThrow('float32');
|
||||
expect(x.session.release).toHaveBeenCalledOnce();
|
||||
x.metadata.robotId = 'other';
|
||||
await expect(x.load()).rejects.toThrow('元数据');
|
||||
x.metadata.robotId = x.env.config.id;
|
||||
x.metadata.modelSha256 = 'wrong';
|
||||
await expect(x.load()).rejects.toThrow('SHA-256');
|
||||
});
|
||||
@@ -0,0 +1,190 @@
|
||||
import * as ort from 'onnxruntime-web/wasm';
|
||||
import { sha256 } from '../robot/registry';
|
||||
import { MOBILE_TASK as T } from './RobotDescriptor';
|
||||
import { TRAINING_STAGES, type TrainingStage } from './TaskKernel';
|
||||
import type { MobileManipulatorWasmEnv } from './MobileManipulatorWasmEnv';
|
||||
|
||||
ort.env.wasm.numThreads = 1;
|
||||
ort.env.wasm.proxy = false;
|
||||
|
||||
export interface MobilePolicyMetadata {
|
||||
taskId: string;
|
||||
actionSemantics: string;
|
||||
trainingStage: TrainingStage;
|
||||
robotId: string;
|
||||
observationSize: number;
|
||||
actionSize: number;
|
||||
controlDt: number;
|
||||
normalized: boolean;
|
||||
modelSha256: string;
|
||||
robotConfigSha256: string;
|
||||
}
|
||||
/** Single-flight, lockstep inference. Rendering continues while inference runs; physics
|
||||
* waits, so training and deployment never differ by an implicit action delay. */
|
||||
export class ONNXPolicyRunner {
|
||||
readonly action = new Float32Array(T.actionSize);
|
||||
readonly input = new Float32Array(T.observationSize);
|
||||
readonly metrics = { inferenceCount: 0, lastInferenceMs: 0, maxInferenceMs: 0, error: '' };
|
||||
private readonly tensor: ort.Tensor;
|
||||
private readonly feeds: Record<string, ort.Tensor>;
|
||||
private inFlight?: Promise<void>;
|
||||
private epoch = 0;
|
||||
private ready = false;
|
||||
private enabled = false;
|
||||
private disposed = false;
|
||||
private revision: number;
|
||||
private releasePromise?: Promise<void>;
|
||||
|
||||
private constructor(
|
||||
private readonly session: ort.InferenceSession,
|
||||
readonly env: MobileManipulatorWasmEnv,
|
||||
private readonly stage: TrainingStage,
|
||||
) {
|
||||
this.tensor = new ort.Tensor('float32', this.input, [1, T.observationSize]);
|
||||
this.feeds = { observation: this.tensor };
|
||||
this.revision = env.revision;
|
||||
}
|
||||
static async load(
|
||||
model: Uint8Array,
|
||||
metadata: MobilePolicyMetadata,
|
||||
env: MobileManipulatorWasmEnv,
|
||||
): Promise<ONNXPolicyRunner> {
|
||||
if (
|
||||
metadata.taskId !== T.id ||
|
||||
metadata.actionSemantics !== T.actionSemantics ||
|
||||
!TRAINING_STAGES.includes(metadata.trainingStage) ||
|
||||
metadata.robotId !== env.config.id ||
|
||||
metadata.observationSize !== T.observationSize ||
|
||||
metadata.actionSize !== T.actionSize ||
|
||||
metadata.controlDt !== T.controlDt ||
|
||||
metadata.normalized !== true
|
||||
)
|
||||
throw new Error('策略元数据与 v2 安全控制契约/机器人不匹配;旧全行程位置策略必须重新训练');
|
||||
if (
|
||||
metadata.modelSha256 !== (await sha256(model)) ||
|
||||
metadata.robotConfigSha256 !==
|
||||
(await sha256(new TextEncoder().encode(JSON.stringify(env.config))))
|
||||
)
|
||||
throw new Error('策略权重/机器人配置 SHA-256 不匹配');
|
||||
const session = await ort.InferenceSession.create(model.slice(), {
|
||||
executionProviders: ['wasm'],
|
||||
graphOptimizationLevel: 'all',
|
||||
});
|
||||
try {
|
||||
if (
|
||||
session.inputNames.length !== 1 ||
|
||||
session.outputNames.length !== 1 ||
|
||||
session.inputNames[0] !== 'observation' ||
|
||||
session.outputNames[0] !== 'action'
|
||||
)
|
||||
throw new Error('ONNX 必须为 observation → action');
|
||||
for (const [m, size] of [
|
||||
[session.inputMetadata[0], T.observationSize],
|
||||
[session.outputMetadata[0], T.actionSize],
|
||||
] as const)
|
||||
if (
|
||||
!m?.isTensor ||
|
||||
m.type !== 'float32' ||
|
||||
m.shape.length !== 2 ||
|
||||
m.shape[0] !== 1 ||
|
||||
m.shape[1] !== size
|
||||
)
|
||||
throw new Error('ONNX 必须声明固定 float32 [1,92] → [1,12]');
|
||||
return new ONNXPolicyRunner(session, env, metadata.trainingStage);
|
||||
} catch (error) {
|
||||
await session.release();
|
||||
throw error;
|
||||
}
|
||||
}
|
||||
setEnabled(enabled: boolean): void {
|
||||
if (this.disposed) return;
|
||||
const wasEnabled = this.enabled;
|
||||
this.enabled = enabled;
|
||||
this.epoch++;
|
||||
this.ready = false;
|
||||
if (enabled) {
|
||||
this.env.setTrainingStage(this.stage);
|
||||
this.metrics.error = '';
|
||||
}
|
||||
this.revision = this.env.revision;
|
||||
if (enabled || wasEnabled) this.env.hold();
|
||||
}
|
||||
/** Called from the existing viewer's rAF. Returns true only after exactly one env step. */
|
||||
tick(): boolean {
|
||||
if (!this.enabled || this.disposed) return false;
|
||||
if (this.revision !== this.env.revision) {
|
||||
this.epoch++;
|
||||
this.ready = false;
|
||||
this.revision = this.env.revision;
|
||||
}
|
||||
if (this.ready) {
|
||||
this.ready = false;
|
||||
this.env.step(this.action);
|
||||
return true;
|
||||
}
|
||||
if (this.inFlight) return false;
|
||||
this.input.set(this.env.observe());
|
||||
const epoch = this.epoch,
|
||||
revision = this.env.revision,
|
||||
started = performance.now();
|
||||
this.inFlight = this.session
|
||||
.run(this.feeds)
|
||||
.then((outputs) => {
|
||||
try {
|
||||
const output = outputs.action;
|
||||
if (
|
||||
!output ||
|
||||
output.type !== 'float32' ||
|
||||
output.dims.length !== 2 ||
|
||||
output.dims[0] !== 1 ||
|
||||
output.dims[1] !== T.actionSize
|
||||
)
|
||||
throw new Error('策略输出形状错误');
|
||||
const values = output.data as Float32Array;
|
||||
for (let i = 0; i < values.length; i++)
|
||||
if (!Number.isFinite(values[i])) throw new Error('策略输出包含 NaN/Infinity');
|
||||
if (
|
||||
this.disposed ||
|
||||
!this.enabled ||
|
||||
epoch !== this.epoch ||
|
||||
revision !== this.env.revision
|
||||
)
|
||||
return;
|
||||
this.action.set(values);
|
||||
this.ready = true;
|
||||
this.metrics.inferenceCount++;
|
||||
this.metrics.lastInferenceMs = performance.now() - started;
|
||||
this.metrics.maxInferenceMs = Math.max(
|
||||
this.metrics.maxInferenceMs,
|
||||
this.metrics.lastInferenceMs,
|
||||
);
|
||||
} finally {
|
||||
for (const name in outputs) outputs[name].dispose();
|
||||
}
|
||||
})
|
||||
.catch((error: unknown) => {
|
||||
if (this.disposed || epoch !== this.epoch || revision !== this.env.revision) return;
|
||||
this.metrics.error = error instanceof Error ? error.message : String(error);
|
||||
this.enabled = false;
|
||||
this.ready = false;
|
||||
this.env.hold();
|
||||
})
|
||||
.finally(() => {
|
||||
this.inFlight = undefined;
|
||||
});
|
||||
return false;
|
||||
}
|
||||
async dispose(): Promise<void> {
|
||||
if (this.releasePromise) return this.releasePromise;
|
||||
this.disposed = true;
|
||||
this.enabled = false;
|
||||
this.epoch++;
|
||||
this.ready = false;
|
||||
// Never release ORT while run() still owns its input; never touch a retired env on completion.
|
||||
this.releasePromise = (this.inFlight ?? Promise.resolve()).finally(async () => {
|
||||
this.tensor.dispose();
|
||||
await this.session.release();
|
||||
});
|
||||
return this.releasePromise;
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,99 @@
|
||||
import robots from '../../../contracts/mobile-robots-v1.json';
|
||||
import task from '../../../contracts/mobile-manipulator-v2.json';
|
||||
|
||||
export interface ArmJointConfig {
|
||||
name: string;
|
||||
min: number;
|
||||
max: number;
|
||||
neutral: number;
|
||||
mode: 'position' | 'velocity';
|
||||
velocityLimit: number;
|
||||
}
|
||||
/** Names, not file-order indices. Resolved once against the compiled model. */
|
||||
export interface RobotConfig {
|
||||
id: string;
|
||||
label: string;
|
||||
recipe: 'lekiwi-v1' | 'lekiwi-bundle' | 'mjcf';
|
||||
baseBodyName: string;
|
||||
baseJointName: string;
|
||||
baseJoints: readonly string[];
|
||||
baseActuators: readonly string[];
|
||||
/** wheel rad/s = matrix * body-frame [vx m/s, vy m/s, wz rad/s]. */
|
||||
baseMix: readonly (readonly number[])[];
|
||||
baseLimits: readonly number[];
|
||||
wheelLimit: number;
|
||||
armJoints: readonly ArmJointConfig[];
|
||||
armActuators: readonly string[];
|
||||
gripperJoint: string;
|
||||
gripperActuators: readonly { name: string; joint?: string; closed: number; open: number }[];
|
||||
gripperClosed: number;
|
||||
gripperOpen: number;
|
||||
eefBodyName: string;
|
||||
eefSiteName?: string;
|
||||
eefOffset: readonly number[];
|
||||
}
|
||||
export const MOBILE_TASK = Object.freeze(task);
|
||||
export const MOBILE_ROBOTS: readonly RobotConfig[] = robots as RobotConfig[];
|
||||
|
||||
export function validateRobotConfig(c: RobotConfig): void {
|
||||
if (
|
||||
!c.id ||
|
||||
!c.baseBodyName ||
|
||||
!c.baseJointName ||
|
||||
!c.eefBodyName ||
|
||||
!['lekiwi-v1', 'lekiwi-bundle', 'mjcf'].includes(c.recipe) ||
|
||||
!c.armJoints.length ||
|
||||
c.armJoints.length > task.maxArmJoints ||
|
||||
c.armJoints.length !== c.armActuators.length ||
|
||||
!c.baseJoints.length ||
|
||||
c.baseJoints.length !== c.baseActuators.length ||
|
||||
c.baseMix.length !== c.baseJoints.length ||
|
||||
c.baseMix.some((row) => row.length !== 3 || row.some((v) => !Number.isFinite(v))) ||
|
||||
c.baseLimits.length !== 3 ||
|
||||
c.baseLimits.some((v) => !Number.isFinite(v) || v <= 0) ||
|
||||
!Number.isFinite(c.wheelLimit) ||
|
||||
c.wheelLimit <= 0 ||
|
||||
c.eefOffset.length !== 3 ||
|
||||
c.eefOffset.some((v) => !Number.isFinite(v)) ||
|
||||
!Number.isFinite(c.gripperClosed) ||
|
||||
!Number.isFinite(c.gripperOpen) ||
|
||||
c.gripperClosed === c.gripperOpen ||
|
||||
!c.gripperActuators.length ||
|
||||
c.gripperActuators.some(
|
||||
(v) =>
|
||||
!v.name || !Number.isFinite(v.closed) || !Number.isFinite(v.open) || v.closed === v.open,
|
||||
) ||
|
||||
c.armJoints.some(
|
||||
(j) =>
|
||||
!j.name ||
|
||||
!Number.isFinite(j.min) ||
|
||||
!Number.isFinite(j.max) ||
|
||||
j.min >= j.max ||
|
||||
!Number.isFinite(j.neutral) ||
|
||||
j.neutral < j.min ||
|
||||
j.neutral > j.max ||
|
||||
!['position', 'velocity'].includes(j.mode) ||
|
||||
!Number.isFinite(j.velocityLimit) ||
|
||||
j.velocityLimit <= 0,
|
||||
)
|
||||
)
|
||||
throw new Error('RobotConfig 无效或超过 mobile-manipulator-v2 的固定槽位');
|
||||
const primary = c.gripperActuators.filter((g) => (g.joint ?? c.gripperJoint) === c.gripperJoint);
|
||||
if (
|
||||
!primary.length ||
|
||||
primary.some((g) => g.closed !== c.gripperClosed || g.open !== c.gripperOpen)
|
||||
)
|
||||
throw new Error('夹爪观测关节与执行器行程不一致');
|
||||
const joints = [
|
||||
...c.baseJoints,
|
||||
...c.armJoints.map((j) => j.name),
|
||||
...new Set([c.gripperJoint, ...c.gripperActuators.map((g) => g.joint ?? c.gripperJoint)]),
|
||||
];
|
||||
const actuators = [
|
||||
...c.baseActuators,
|
||||
...c.armActuators,
|
||||
...c.gripperActuators.map((a) => a.name),
|
||||
];
|
||||
if (new Set(joints).size !== joints.length || new Set(actuators).size !== actuators.length)
|
||||
throw new Error('RobotConfig 存在重复关节/执行器');
|
||||
}
|
||||
@@ -0,0 +1,145 @@
|
||||
import { RobotManager } from './RobotManager';
|
||||
import { MOBILE_ROBOTS } from './RobotDescriptor';
|
||||
|
||||
const fakes = vi.hoisted(() => ({
|
||||
imports: vi.fn(),
|
||||
gate: undefined as Promise<void> | undefined,
|
||||
loads: [] as string[],
|
||||
deleted: [] as string[],
|
||||
}));
|
||||
vi.mock('../project/importer', () => ({ importBrowserFiles: fakes.imports }));
|
||||
vi.mock('./ONNXPolicyRunner', () => ({ ONNXPolicyRunner: { load: vi.fn() } }));
|
||||
vi.mock('../simulation/PhysicsAdapter', () => ({
|
||||
MainThreadPhysicsAdapter: class {
|
||||
session: { name: string } | null = null;
|
||||
workspace: { root: string } | null = null;
|
||||
retired: { name: string } | null = null;
|
||||
oldWorkspace: { root: string } | null = null;
|
||||
async load(manifest: { name: string }, _entry: string, options: { workspaceSlot: string }) {
|
||||
fakes.loads.push(manifest.name);
|
||||
await fakes.gate;
|
||||
this.retired = this.session;
|
||||
this.oldWorkspace = this.workspace;
|
||||
this.session = { name: manifest.name };
|
||||
this.workspace = { root: `/workspace/${options.workspaceSlot}` };
|
||||
}
|
||||
releaseRetired() {
|
||||
if (this.retired) fakes.deleted.push(this.retired.name);
|
||||
this.retired = null;
|
||||
this.oldWorkspace = null;
|
||||
}
|
||||
rollbackRetired() {
|
||||
if (this.session) fakes.deleted.push(this.session.name);
|
||||
this.session = this.retired;
|
||||
this.workspace = this.oldWorkspace;
|
||||
this.retired = null;
|
||||
}
|
||||
dispose() {
|
||||
this.releaseRetired();
|
||||
if (this.session) fakes.deleted.push(this.session.name);
|
||||
this.session = this.workspace = null;
|
||||
}
|
||||
},
|
||||
}));
|
||||
vi.mock('./MobileManipulatorWasmEnv', () => ({
|
||||
MobileManipulatorWasmEnv: class {
|
||||
refreshViews = vi.fn();
|
||||
hold = vi.fn();
|
||||
reset = vi.fn();
|
||||
dispose = vi.fn();
|
||||
constructor(
|
||||
readonly session: unknown,
|
||||
readonly config: { id: string },
|
||||
) {
|
||||
if (config.id === 'bad') throw new Error('binding mismatch');
|
||||
}
|
||||
},
|
||||
}));
|
||||
function manifest(name: string) {
|
||||
return { id: name, name, entries: [{ path: 'scene.xml', format: 'mjcf' }], files: [] };
|
||||
}
|
||||
const file = (name: string) => new File([], name);
|
||||
const config = { ...MOBILE_ROBOTS[0], recipe: 'mjcf' as const };
|
||||
const flush = async () => {
|
||||
for (let i = 0; i < 5; i++) await Promise.resolve();
|
||||
};
|
||||
beforeEach(() => {
|
||||
fakes.loads.length = fakes.deleted.length = 0;
|
||||
fakes.gate = undefined;
|
||||
fakes.imports.mockReset().mockImplementation(async ([f]: File[]) => manifest(f.name));
|
||||
});
|
||||
it('使用空闲双槽,先 viewer attach 后销毁旧模型;绑定失败恢复旧环境', async () => {
|
||||
const attach = vi.fn(),
|
||||
manager = new RobotManager({ attach });
|
||||
await manager.loadZip(file('first'), config);
|
||||
const old = manager.env,
|
||||
slot = manager.adapter.workspace!.root;
|
||||
await expect(manager.loadZip(file('bad'), { ...config, id: 'bad' })).rejects.toThrow(
|
||||
'binding mismatch',
|
||||
);
|
||||
expect(manager.env).toBe(old);
|
||||
expect(manager.adapter.workspace!.root).toBe(slot);
|
||||
expect(fakes.deleted).toEqual(['bad']);
|
||||
expect(old!.refreshViews).toHaveBeenCalled();
|
||||
await manager.loadZip(file('next'), config);
|
||||
expect(manager.adapter.workspace!.root).not.toBe(slot);
|
||||
expect(old!.dispose).toHaveBeenCalledOnce();
|
||||
expect(fakes.deleted).toEqual(['bad', 'first']);
|
||||
expect(attach).toHaveBeenLastCalledWith(manager.adapter.session);
|
||||
await manager.dispose();
|
||||
expect(fakes.deleted).toEqual(['bad', 'first', 'next']);
|
||||
});
|
||||
it('编译屏障期间的失焦/暂停不会访问可能失效的旧 WASM views', async () => {
|
||||
const manager = new RobotManager({ attach: vi.fn() });
|
||||
await manager.loadZip(file('first'), config);
|
||||
const old = manager.env!;
|
||||
let release!: () => void;
|
||||
fakes.gate = new Promise<void>((r) => {
|
||||
release = r;
|
||||
});
|
||||
const loading = manager.loadZip(file('next'), config);
|
||||
const calls = vi.mocked(old.hold).mock.calls.length;
|
||||
manager.setRunning(false);
|
||||
expect(vi.mocked(old.hold).mock.calls.length).toBe(calls);
|
||||
expect(() => manager.reset()).toThrow('正在切换');
|
||||
expect(() => manager.setMode('teleop')).toThrow('正在切换');
|
||||
await expect(manager.exportTrainingBundle()).rejects.toThrow();
|
||||
release();
|
||||
await loading;
|
||||
await manager.dispose();
|
||||
});
|
||||
it('最新切换优先,取消的解包不进入 native compile', async () => {
|
||||
let resolve!: (value: ReturnType<typeof manifest>) => void;
|
||||
fakes.imports.mockImplementationOnce(
|
||||
() =>
|
||||
new Promise((r) => {
|
||||
resolve = r;
|
||||
}),
|
||||
);
|
||||
const manager = new RobotManager({ attach: vi.fn() });
|
||||
const first = manager.loadZip(file('first'), config).catch((e: Error) => e.message);
|
||||
await flush();
|
||||
const next = manager.loadZip(file('next'), config);
|
||||
resolve(manifest('first'));
|
||||
await next;
|
||||
expect(await first).toContain('取消');
|
||||
expect(fakes.loads).toEqual(['next']);
|
||||
expect(manager.loading).toBe(false);
|
||||
await manager.dispose();
|
||||
});
|
||||
it('加载途中 dispose 不提交迟到模型,所有候选被释放', async () => {
|
||||
let release!: () => void;
|
||||
fakes.gate = new Promise<void>((r) => {
|
||||
release = r;
|
||||
});
|
||||
const manager = new RobotManager({ attach: vi.fn() });
|
||||
const loading = manager.loadZip(file('late'), config).catch((e: Error) => e.message);
|
||||
await flush();
|
||||
expect(() => manager.setRunning(true)).toThrow();
|
||||
const closing = manager.dispose();
|
||||
release();
|
||||
await closing;
|
||||
expect(await loading).toContain('取消');
|
||||
expect(fakes.deleted).toEqual(['late']);
|
||||
expect(manager.env).toBeUndefined();
|
||||
});
|
||||
@@ -0,0 +1,224 @@
|
||||
import { importBrowserFiles } from '../project/importer';
|
||||
import type { ProjectManifest } from '../project/types';
|
||||
import { MainThreadPhysicsAdapter } from '../simulation/PhysicsAdapter';
|
||||
import type { FrameResult, SimulationSession } from '../simulation/SimulationSession';
|
||||
import { MobileManipulatorWasmEnv } from './MobileManipulatorWasmEnv';
|
||||
import { ONNXPolicyRunner, type MobilePolicyMetadata } from './ONNXPolicyRunner';
|
||||
import { composeMobileScene, prepareBundle } from './SceneComposer';
|
||||
import { MOBILE_TASK as T, validateRobotConfig, type RobotConfig } from './RobotDescriptor';
|
||||
|
||||
/** One physics owner. Viewer GPU objects detach before native model/data are deleted. */
|
||||
export class RobotManager {
|
||||
readonly adapter = new MainThreadPhysicsAdapter();
|
||||
env?: MobileManipulatorWasmEnv;
|
||||
policy?: ONNXPolicyRunner;
|
||||
mode: 'teleop' | 'policy' = 'teleop';
|
||||
running = false;
|
||||
loading = false;
|
||||
private readonly workspacePrefix = `mobile_${crypto.randomUUID().replaceAll('-', '_')}`;
|
||||
private generation = 0;
|
||||
private policyGeneration = 0;
|
||||
private disposed = false;
|
||||
private queue: Promise<unknown> = Promise.resolve();
|
||||
private project?: ProjectManifest;
|
||||
private entryPath = '';
|
||||
private lastNow?: number;
|
||||
private accumulator = 0;
|
||||
private readonly frame: FrameResult = { steps: 0, stepMs: 0, overBudget: false };
|
||||
constructor(private readonly viewer: { attach(session: SimulationSession | null): void }) {}
|
||||
|
||||
async loadZip(file: File, config: RobotConfig, entryPath?: string): Promise<void> {
|
||||
if (this.disposed) throw new Error('RobotManager 已释放');
|
||||
validateRobotConfig(config);
|
||||
// Clone at the boundary: changing a caller's config cannot mutate a running contract.
|
||||
const descriptor = structuredClone(config);
|
||||
const generation = ++this.generation;
|
||||
++this.policyGeneration;
|
||||
this.setRunning(false);
|
||||
this.loading = true;
|
||||
const load = this.queue
|
||||
.catch(() => {})
|
||||
.then(async () => {
|
||||
if (generation !== this.generation || this.disposed) throw new Error('机器人切换已取消');
|
||||
let manifest = await importBrowserFiles([file]);
|
||||
const entry =
|
||||
entryPath ??
|
||||
(manifest.entries.length === 1 ? manifest.entries[0].path : manifest.selectedEntry);
|
||||
if (!entry || !manifest.entries.some((e) => e.path === entry))
|
||||
throw new Error('请显式指定 ZIP 中的模型入口');
|
||||
if (descriptor.recipe === 'lekiwi-bundle') {
|
||||
if (!entry.toLowerCase().endsWith('.urdf')) throw new Error('bundle 适配器要求原始 URDF');
|
||||
manifest = prepareBundle(manifest, entry, descriptor);
|
||||
}
|
||||
if (generation !== this.generation || this.disposed) throw new Error('机器人切换已取消');
|
||||
const oldEnv = this.env;
|
||||
let candidate: MobileManipulatorWasmEnv | undefined;
|
||||
let staged = false;
|
||||
try {
|
||||
await this.adapter.load(manifest, entry, {
|
||||
robotProfileId: descriptor.recipe === 'lekiwi-v1' ? 'lekiwi-v1' : undefined,
|
||||
configureRobotRuntime: false,
|
||||
// Unique per manager, but bounded to two paths. MuJoCo caches meshes by
|
||||
// resource path; monotonically new mount roots grow the native asset cache.
|
||||
workspaceSlot: `${this.workspacePrefix}_${this.adapter.workspace?.root.endsWith('_0') ? 1 : 0}`,
|
||||
map: { kind: 'none' },
|
||||
sceneComposer: (xml) => composeMobileScene(xml, descriptor),
|
||||
});
|
||||
staged = true;
|
||||
oldEnv?.refreshViews();
|
||||
if (generation !== this.generation || this.disposed) throw new Error('机器人切换已取消');
|
||||
candidate = new MobileManipulatorWasmEnv(this.adapter.session!, descriptor);
|
||||
this.viewer.attach(this.adapter.session);
|
||||
const oldPolicy = this.policy;
|
||||
this.policy = undefined;
|
||||
await oldPolicy?.dispose();
|
||||
if (generation !== this.generation || this.disposed) throw new Error('机器人切换已取消');
|
||||
this.env = candidate;
|
||||
this.mode = 'teleop';
|
||||
this.project = manifest;
|
||||
this.entryPath = entry;
|
||||
oldEnv?.dispose();
|
||||
this.adapter.releaseRetired();
|
||||
} catch (error) {
|
||||
candidate?.dispose();
|
||||
if (staged) {
|
||||
// A failed viewer attachment may own candidate mjvScene objects.
|
||||
this.viewer.attach(null);
|
||||
this.adapter.rollbackRetired();
|
||||
}
|
||||
oldEnv?.refreshViews();
|
||||
if (!this.disposed) this.viewer.attach(this.adapter.session);
|
||||
throw error;
|
||||
}
|
||||
});
|
||||
this.queue = load;
|
||||
return load.finally(() => {
|
||||
if (generation === this.generation) this.loading = false;
|
||||
});
|
||||
}
|
||||
|
||||
async loadPolicy(bytes: Uint8Array, metadata: MobilePolicyMetadata): Promise<void> {
|
||||
const env = this.env,
|
||||
generation = ++this.policyGeneration;
|
||||
if (!env || this.disposed || this.loading) throw new Error('请先完成机器人加载');
|
||||
this.setRunning(false);
|
||||
const runner = await ONNXPolicyRunner.load(bytes, metadata, env);
|
||||
if (generation !== this.policyGeneration || env !== this.env || this.disposed) {
|
||||
await runner.dispose();
|
||||
throw new Error('策略加载已取消');
|
||||
}
|
||||
const previous = this.policy;
|
||||
this.policy = runner;
|
||||
await previous?.dispose();
|
||||
}
|
||||
setMode(mode: 'teleop' | 'policy'): void {
|
||||
if (this.loading) throw new Error('正在切换机器人');
|
||||
if (mode === 'policy' && !this.policy) throw new Error('请先加载匹配策略及元数据');
|
||||
this.setRunning(false);
|
||||
this.mode = mode;
|
||||
if (mode === 'teleop') this.env?.setTrainingStage('pick-place');
|
||||
this.env?.hold();
|
||||
}
|
||||
setRunning(value: boolean): void {
|
||||
if (value && (!this.env || this.disposed || this.loading))
|
||||
throw new Error('没有可运行的环境,或正在切换模型');
|
||||
this.running = value;
|
||||
this.accumulator = 0;
|
||||
this.lastNow = undefined;
|
||||
this.policy?.setEnabled(value && this.mode === 'policy');
|
||||
// During compile an old WASM view can be detached. It was already held before
|
||||
// loading began; blur/visibility callbacks must not touch it until the barrier.
|
||||
if (!value && !this.loading) this.env?.hold();
|
||||
}
|
||||
reset(): void {
|
||||
if (this.loading) throw new Error('正在切换机器人');
|
||||
this.setRunning(false);
|
||||
this.env?.reset();
|
||||
}
|
||||
move(entity: 'object' | 'goal', position: readonly number[]): void {
|
||||
if (this.loading) return;
|
||||
this.setRunning(false);
|
||||
this.env?.moveTaskEntity(entity, position);
|
||||
}
|
||||
/** Passed into MuJoCoViewer; never also call SimulationSession.advance. */
|
||||
advance(now: number): FrameResult {
|
||||
const f = this.frame;
|
||||
f.steps = 0;
|
||||
f.stepMs = 0;
|
||||
f.overBudget = false;
|
||||
const elapsed =
|
||||
this.lastNow === undefined ? 0 : Math.min(0.1, Math.max(0, (now - this.lastNow) / 1000));
|
||||
this.lastNow = now;
|
||||
if (!this.running || !this.env) return f;
|
||||
this.accumulator = Math.min(0.1, this.accumulator + elapsed);
|
||||
const started = performance.now();
|
||||
try {
|
||||
while (
|
||||
this.accumulator + 1e-9 >= T.controlDt &&
|
||||
f.steps < 4 &&
|
||||
performance.now() - started < 8
|
||||
) {
|
||||
if (this.mode === 'policy') {
|
||||
if (!this.policy?.tick()) {
|
||||
if (this.policy?.metrics.error) this.setRunning(false);
|
||||
break;
|
||||
}
|
||||
} else this.env.step(this.env.action, true); // Teleop bypasses phase masks, never speed limits.
|
||||
this.accumulator -= T.controlDt;
|
||||
f.steps++;
|
||||
if (this.env.kernel.result.terminated || this.env.kernel.result.truncated) {
|
||||
this.setRunning(false);
|
||||
break;
|
||||
}
|
||||
}
|
||||
} catch (error) {
|
||||
this.setRunning(false);
|
||||
throw error;
|
||||
}
|
||||
f.stepMs = performance.now() - started;
|
||||
f.overBudget = this.accumulator >= T.controlDt;
|
||||
return f;
|
||||
}
|
||||
|
||||
/** Same composed MJCF and assets for native training; no second URDF converter. */
|
||||
async exportTrainingBundle(): Promise<Uint8Array> {
|
||||
if (!this.env || !this.project || !this.adapter.workspace || this.loading)
|
||||
throw new Error('尚未完成机器人加载');
|
||||
const generation = this.generation;
|
||||
const { zipSync } = await import('fflate');
|
||||
if (this.disposed || this.loading || generation !== this.generation)
|
||||
throw new Error('导出已取消');
|
||||
const encoder = new TextEncoder();
|
||||
const scene = `${this.entryPath}.training.xml`;
|
||||
const files: Record<string, Uint8Array> = {};
|
||||
for (const f of [...this.project.files, ...this.adapter.cachedSupportFiles()])
|
||||
files[f.path] = f.data;
|
||||
files[scene] = this.adapter.exportMjcf();
|
||||
files['robot.json'] = encoder.encode(JSON.stringify(this.env.config));
|
||||
files['task.json'] = encoder.encode(JSON.stringify(T));
|
||||
files['environment.json'] = encoder.encode(
|
||||
JSON.stringify({
|
||||
scene,
|
||||
mujoco: this.adapter.session!.module.mj_versionString(),
|
||||
taskId: T.id,
|
||||
robotId: this.env.config.id,
|
||||
}),
|
||||
);
|
||||
return zipSync(files, { level: 1 });
|
||||
}
|
||||
async dispose(): Promise<void> {
|
||||
if (this.disposed) return;
|
||||
this.setRunning(false);
|
||||
this.disposed = true;
|
||||
this.generation++;
|
||||
this.policyGeneration++;
|
||||
await this.queue.catch(() => {});
|
||||
await this.policy?.dispose();
|
||||
this.policy = undefined;
|
||||
this.viewer.attach(null);
|
||||
this.env?.dispose();
|
||||
this.env = undefined;
|
||||
this.adapter.dispose();
|
||||
this.project = undefined;
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,100 @@
|
||||
import golden from '../../../contracts/fixtures/mobile-motion-v2-golden.json';
|
||||
import { MOBILE_ROBOTS, MOBILE_TASK as T } from './RobotDescriptor';
|
||||
import { SafeActionController } from './SafeActionController';
|
||||
import { TaskKernel, type TrainingStage } from './TaskKernel';
|
||||
|
||||
it('两个变体、三个阶段多步与 Python 控制/滤波状态一致;目标速度有界', () => {
|
||||
for (const sample of golden) {
|
||||
const config = MOBILE_ROBOTS.find((r) => r.id === sample.robotId)!;
|
||||
const controller = new SafeActionController(config);
|
||||
const state = Float64Array.from(sample.state);
|
||||
controller.reset(state);
|
||||
for (const frame of sample.frames) {
|
||||
const previous = controller.control.slice();
|
||||
controller.apply(
|
||||
Float32Array.from(frame.action),
|
||||
state,
|
||||
sample.stage as TrainingStage,
|
||||
false,
|
||||
);
|
||||
frame.control.forEach((v, i) => expect(controller.control[i]).toBeCloseTo(v, 6));
|
||||
frame.applied.forEach((v, i) => expect(controller.applied[i]).toBeCloseTo(v, 6));
|
||||
frame.targets.forEach((v, i) => expect(controller.targets[i]).toBeCloseTo(v, 6));
|
||||
config.armJoints.forEach((joint, i) => {
|
||||
expect(Math.abs(controller.control[3 + i] - previous[3 + i])).toBeLessThanOrEqual(
|
||||
Math.min(joint.velocityLimit, T.armSpeedLimit) * T.controlDt + 1e-12,
|
||||
);
|
||||
});
|
||||
if (sample.stage === 'navigate')
|
||||
expect(Array.from(controller.applied.slice(3))).toEqual(Array(9).fill(0));
|
||||
}
|
||||
const previous = controller.control.slice();
|
||||
expect(() => controller.apply(new Float32Array(12).fill(NaN), state, 'reach', false)).toThrow();
|
||||
expect(controller.control).toEqual(previous);
|
||||
const kernel = new TaskKernel(config);
|
||||
kernel.recordAction(controller.applied, controller.targets);
|
||||
expect(Array.from(kernel.observe(state).slice(68, 80))).toEqual(Array.from(controller.applied));
|
||||
expect(Array.from(kernel.observe(state).slice(80))).toEqual(Array.from(controller.targets));
|
||||
}
|
||||
});
|
||||
|
||||
it('底盘加速度、速度模式和夹爪开度速率受限;复位清除滤波历史', () => {
|
||||
const c = {
|
||||
...MOBILE_ROBOTS[0],
|
||||
armJoints: MOBILE_ROBOTS[0].armJoints.map((j, i) =>
|
||||
i === 0 ? { ...j, mode: 'velocity' as const } : j,
|
||||
),
|
||||
};
|
||||
const controller = new SafeActionController(c);
|
||||
const state = new Float64Array(T.stateSize);
|
||||
state[3] = 1;
|
||||
state[37] = 1;
|
||||
state[29] = 1;
|
||||
controller.reset(state);
|
||||
for (let i = 0; i < 50; i++) {
|
||||
const previous = controller.applied.slice();
|
||||
const opening = controller.targets[11];
|
||||
controller.apply(new Float32Array(12).fill(i < 20 ? -1 : 1), state, 'pick-place', true);
|
||||
for (let j = 0; j < 3; j++)
|
||||
expect(
|
||||
Math.abs(controller.applied[j] - previous[j]) * T.baseSpeedLimits[j],
|
||||
).toBeLessThanOrEqual(T.baseAccelerationLimits[j] * T.controlDt + 1e-7);
|
||||
expect(Math.abs(controller.control[3])).toBeLessThanOrEqual(T.armSpeedLimit);
|
||||
expect(Math.abs(controller.targets[11] - opening)).toBeLessThanOrEqual(
|
||||
2 * T.gripperOpeningRate * T.controlDt + 1e-7,
|
||||
);
|
||||
}
|
||||
controller.reset(state);
|
||||
expect(controller.applied).toEqual(new Float32Array(12));
|
||||
});
|
||||
|
||||
it('暂停保持既有安全目标,不反复追随重力下垂后的 qpos', () => {
|
||||
const controller = new SafeActionController(MOBILE_ROBOTS[0]);
|
||||
const state = new Float64Array(T.stateSize);
|
||||
state[29] = 1;
|
||||
controller.reset(state);
|
||||
state[13] = -0.1;
|
||||
for (let i = 0; i < 10; i++) {
|
||||
controller.reset(state, controller.control);
|
||||
expect(controller.control[3]).toBe(0);
|
||||
expect(controller.applied).toEqual(new Float32Array(12));
|
||||
}
|
||||
});
|
||||
it('导航需要接近位、朝向、停车和稳定驻留;超速不能算成功', () => {
|
||||
const kernel = new TaskKernel(MOBILE_ROBOTS[0], 'navigate');
|
||||
const s = new Float64Array(T.stateSize);
|
||||
s[3] = 1;
|
||||
s[37] = T.navigationOffset[0];
|
||||
s[7] = 0.2;
|
||||
for (let i = 0; i < T.settleSteps; i++) expect(kernel.evaluate(s).info.is_success).toBe(false);
|
||||
s[7] = 0;
|
||||
for (let i = 0; i < T.settleSteps - 1; i++) expect(kernel.evaluate(s).terminated).toBe(false);
|
||||
expect(kernel.evaluate(s).info.is_success).toBe(true);
|
||||
kernel.reset();
|
||||
s[21] = T.jointSpeedStop + 0.1;
|
||||
const result = kernel.evaluate(s);
|
||||
expect(result.terminated).toBe(true);
|
||||
expect(result.info.safety_stop).toBe('joint_velocity');
|
||||
expect(result.info.is_success).toBe(false);
|
||||
expect(result.info.reward_components.joint_velocity).toBeLessThan(0);
|
||||
});
|
||||
@@ -0,0 +1,115 @@
|
||||
import { MOBILE_TASK as T, type RobotConfig } from './RobotDescriptor';
|
||||
import { clamp, navigationError, type TrainingStage } from './TaskKernel';
|
||||
|
||||
/** v2: integrated position setpoints, bounded speeds/accelerations, no full-range jumps.
|
||||
* Mirrored by training_server/mobile_manipulator/motion.py. All filter state is observed. */
|
||||
export class SafeActionController {
|
||||
readonly applied = new Float32Array(T.actionSize);
|
||||
readonly targets = new Float32Array(T.actionSize);
|
||||
readonly control: Float64Array;
|
||||
constructor(readonly config: RobotConfig) {
|
||||
this.control = new Float64Array(
|
||||
config.baseJoints.length + config.armJoints.length + config.gripperActuators.length,
|
||||
);
|
||||
}
|
||||
reset(state: Float64Array, previousControl?: Float64Array): void {
|
||||
previousControl = previousControl?.slice();
|
||||
this.applied.fill(0);
|
||||
this.targets.fill(0);
|
||||
this.control.fill(0);
|
||||
const n = this.config.baseJoints.length;
|
||||
this.config.armJoints.forEach((j, i) => {
|
||||
if (j.mode === 'position') {
|
||||
let q = state[13 + i];
|
||||
if (previousControl && Number.isFinite(previousControl[n + i]))
|
||||
q = clamp(previousControl[n + i], q - T.armTrackingError, q + T.armTrackingError);
|
||||
q = clamp(q, j.min, j.max);
|
||||
this.control[n + i] = q;
|
||||
this.targets[3 + i] = (2 * (q - j.min)) / (j.max - j.min) - 1;
|
||||
}
|
||||
});
|
||||
let opening = state[29];
|
||||
for (let i = 0; i < this.config.gripperActuators.length; i++) {
|
||||
const g = this.config.gripperActuators[i],
|
||||
index = n + this.config.armJoints.length + i;
|
||||
if (
|
||||
previousControl &&
|
||||
(g.joint ?? this.config.gripperJoint) === this.config.gripperJoint &&
|
||||
Number.isFinite(previousControl[index])
|
||||
) {
|
||||
opening = clamp((previousControl[index] - g.closed) / (g.open - g.closed), 0, 1);
|
||||
break;
|
||||
}
|
||||
}
|
||||
this.targets[11] = 2 * opening - 1;
|
||||
this.gripper(opening);
|
||||
}
|
||||
private gripper(opening: number): void {
|
||||
const n = this.config.baseJoints.length + this.config.armJoints.length;
|
||||
this.config.gripperActuators.forEach((g, i) => {
|
||||
this.control[n + i] = g.closed + opening * (g.open - g.closed);
|
||||
});
|
||||
}
|
||||
apply(action: Float32Array, state: Float64Array, stage: TrainingStage, lifted: boolean): void {
|
||||
if (action.length !== T.actionSize || action.some((v) => !Number.isFinite(v)))
|
||||
throw new Error('动作必须是有限 [12]');
|
||||
const { distance, yaw } = navigationError(state);
|
||||
const near = distance < T.navigationTolerance && Math.abs(yaw) < T.navigationYawTolerance;
|
||||
const manipulate = stage !== 'navigate' && (near || lifted);
|
||||
const c = this.config,
|
||||
dt = T.controlDt;
|
||||
for (let i = 0; i < 3; i++) {
|
||||
const limit = Math.min(c.baseLimits[i], T.baseSpeedLimits[i]);
|
||||
const desired = near && !lifted ? 0 : clamp(action[i]);
|
||||
const delta = (T.baseAccelerationLimits[i] * dt) / limit;
|
||||
this.applied[i] = clamp(desired, this.applied[i] - delta, this.applied[i] + delta);
|
||||
}
|
||||
let largest = c.wheelLimit;
|
||||
c.baseMix.forEach((row, i) => {
|
||||
this.control[i] = 0;
|
||||
for (let j = 0; j < 3; j++)
|
||||
this.control[i] +=
|
||||
row[j] * this.applied[j] * Math.min(c.baseLimits[j], T.baseSpeedLimits[j]);
|
||||
largest = Math.max(largest, Math.abs(this.control[i]));
|
||||
});
|
||||
const n = c.baseJoints.length;
|
||||
for (let i = 0; i < n; i++) this.control[i] *= c.wheelLimit / largest;
|
||||
c.armJoints.forEach((j, i) => {
|
||||
const k = 3 + i;
|
||||
const speed = Math.min(j.velocityLimit, T.armSpeedLimit);
|
||||
const delta = (T.armAccelerationLimit * dt) / speed;
|
||||
const a = clamp(
|
||||
manipulate ? clamp(action[k]) : 0,
|
||||
this.applied[k] - delta,
|
||||
this.applied[k] + delta,
|
||||
);
|
||||
if (j.mode === 'position') {
|
||||
const old = this.control[n + i];
|
||||
const desired = clamp(
|
||||
old + a * speed * dt,
|
||||
state[13 + i] - T.armTrackingError,
|
||||
state[13 + i] + T.armTrackingError,
|
||||
);
|
||||
const target = manipulate
|
||||
? clamp(clamp(desired, old - speed * dt, old + speed * dt), j.min, j.max)
|
||||
: old;
|
||||
this.control[n + i] = target;
|
||||
this.applied[k] = (target - old) / (speed * dt);
|
||||
this.targets[k] = (2 * (target - j.min)) / (j.max - j.min) - 1;
|
||||
} else {
|
||||
this.applied[k] = manipulate ? a : 0;
|
||||
this.control[n + i] = this.applied[k] * speed;
|
||||
}
|
||||
});
|
||||
const opening = (this.targets[11] + 1) / 2;
|
||||
const target = clamp(
|
||||
opening +
|
||||
(manipulate && stage === 'pick-place' ? clamp(action[11]) : 0) * T.gripperOpeningRate * dt,
|
||||
0,
|
||||
1,
|
||||
);
|
||||
this.applied[11] = (target - opening) / (T.gripperOpeningRate * dt);
|
||||
this.targets[11] = 2 * target - 1;
|
||||
this.gripper(target);
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,58 @@
|
||||
import { composeMobileScene, prepareBundle } from './SceneComposer';
|
||||
import { MOBILE_ROBOTS } from './RobotDescriptor';
|
||||
import type { ProjectManifest } from '../project/types';
|
||||
|
||||
const encode = (xml: string) => new TextEncoder().encode(xml);
|
||||
const xml =
|
||||
'<mujoco><compiler angle="radian"/><worldbody><geom type="plane" size="1 1 .1"/><body name="Wrist_Roll_08c-v1"/></worldbody><keyframe><key/></keyframe></mujoco>';
|
||||
const config = { ...MOBILE_ROBOTS[0], recipe: 'mjcf' as const };
|
||||
it('组合现有模型,任务视觉不被查看器 collision-group 过滤;重复组合拒绝', () => {
|
||||
const output = composeMobileScene(encode(xml), config);
|
||||
const doc = new DOMParser().parseFromString(new TextDecoder().decode(output), 'application/xml');
|
||||
expect(doc.querySelectorAll('geom[type="plane"]')).toHaveLength(1);
|
||||
expect(doc.querySelector('geom[name="__mm_floor"]')?.getAttribute('group')).toBe('2');
|
||||
expect(doc.querySelector('geom[name="__mm_object_geom"]')?.getAttribute('group')).toBe('1');
|
||||
expect(doc.querySelector('body[name="__mm_goal"]')?.getAttribute('mocap')).toBe('true');
|
||||
expect(doc.querySelector('site[name="__mm_eef"]')).not.toBeNull();
|
||||
expect(doc.querySelector('keyframe')).toBeNull();
|
||||
expect(() => composeMobileScene(output, config)).toThrow('命名空间');
|
||||
});
|
||||
it('bundle 不把原始 LeKiwi 五轴名称按顺序误认成 Link1..4', () => {
|
||||
const data = encode(
|
||||
'<robot><joint name="base_left_wheel"/><joint name="base_back_wheel"/><joint name="base_right_wheel"/><joint name="arm_shoulder_pan"/></robot>',
|
||||
);
|
||||
const manifest = {
|
||||
files: [{ path: 'robot.urdf', data, size: data.length }],
|
||||
entries: [{ path: 'robot.urdf', format: 'urdf' }],
|
||||
} as ProjectManifest;
|
||||
expect(() => prepareBundle(manifest, 'robot.urdf', MOBILE_ROBOTS[1])).toThrow('Link1');
|
||||
});
|
||||
it('Bundle 关节级限力不能覆盖仿真舵机限力,否则零位保持也会坍塌', () => {
|
||||
const c = MOBILE_ROBOTS[1];
|
||||
const joints = [...c.armJoints.map((j) => j.name), c.gripperJoint]
|
||||
.map(
|
||||
(name) =>
|
||||
`<joint name="${name}" axis="0 0 1" actuatorfrclimited="true" actuatorfrcrange="-1 1"/>`,
|
||||
)
|
||||
.join('');
|
||||
const wheels = c.baseJoints.map((name) => `<body><joint name="${name}"/></body>`).join('');
|
||||
const input = `<mujoco><worldbody><body name="${c.baseBodyName}">${wheels}<body name="base_link">${joints}<body name="${c.eefBodyName}"/></body></body></worldbody></mujoco>`;
|
||||
const output = new DOMParser().parseFromString(
|
||||
new TextDecoder().decode(composeMobileScene(encode(input), c)),
|
||||
'application/xml',
|
||||
);
|
||||
for (const j of c.armJoints)
|
||||
expect(output.querySelector(`joint[name="${j.name}"]`)?.getAttribute('actuatorfrcrange')).toBe(
|
||||
'-8 8',
|
||||
);
|
||||
expect(
|
||||
output.querySelector(`joint[name="${c.gripperJoint}"]`)?.getAttribute('actuatorfrcrange'),
|
||||
).toBe('-2 2');
|
||||
});
|
||||
it('拒绝无效 XML/DOCTYPE/未知 eefBodyName,不输出无法绑定的任务', () => {
|
||||
expect(() => composeMobileScene(encode('<broken>'), config)).toThrow();
|
||||
expect(() => composeMobileScene(encode('<!DOCTYPE mujoco><mujoco/>'), config)).toThrow();
|
||||
expect(() => composeMobileScene(encode(xml), { ...config, eefBodyName: 'missing' })).toThrow(
|
||||
'missing',
|
||||
);
|
||||
});
|
||||
@@ -0,0 +1,289 @@
|
||||
import type { ProjectManifest } from '../project/types';
|
||||
import { MOBILE_TASK as T, type RobotConfig } from './RobotDescriptor';
|
||||
|
||||
const encoder = new TextEncoder();
|
||||
function parse(data: Uint8Array): Document {
|
||||
const doc = new DOMParser().parseFromString(new TextDecoder().decode(data), 'application/xml');
|
||||
if (doc.querySelector('parsererror') || doc.doctype) throw new Error('模型 XML 无效');
|
||||
return doc;
|
||||
}
|
||||
function node(
|
||||
doc: Document,
|
||||
tag: string,
|
||||
attributes: Record<string, string | number> = {},
|
||||
): Element {
|
||||
const n = doc.createElement(tag);
|
||||
for (const [key, value] of Object.entries(attributes)) n.setAttribute(key, String(value));
|
||||
return n;
|
||||
}
|
||||
function named(doc: Document, tag: string, name: string): Element {
|
||||
const result = Array.from(doc.getElementsByTagName(tag)).filter(
|
||||
(e) => e.getAttribute('name') === name,
|
||||
);
|
||||
if (result.length !== 1) throw new Error(`模型缺少/重复 ${tag}: ${name}`);
|
||||
return result[0];
|
||||
}
|
||||
|
||||
/** The new bundle has a different arm. Never route it through the old source/FK gate. */
|
||||
export function prepareBundle(
|
||||
manifest: ProjectManifest,
|
||||
entryPath: string,
|
||||
c: RobotConfig,
|
||||
): ProjectManifest {
|
||||
const source = manifest.files.find((f) => f.path === entryPath);
|
||||
if (!source) throw new Error('缺少 bundle URDF');
|
||||
const doc = parse(source.data);
|
||||
for (const name of [...c.baseJoints, ...c.armJoints.map((j) => j.name), c.gripperJoint])
|
||||
named(doc, 'joint', name);
|
||||
for (const mesh of doc.querySelectorAll('mesh[filename]')) {
|
||||
if (mesh.getAttribute('filename')!.includes('4-Omni-Directional-Wheel_Single_Body'))
|
||||
mesh.replaceWith(node(doc, 'sphere', { radius: 0.05 }));
|
||||
}
|
||||
// Exporter duplicated collisions. Keep one representation per visual, not overlapping copies.
|
||||
for (const link of doc.querySelectorAll('link')) {
|
||||
const seen = new Set<string>();
|
||||
for (const collision of link.querySelectorAll(':scope > collision')) {
|
||||
const mesh = collision.querySelector('mesh');
|
||||
if (!mesh) continue;
|
||||
const origin = collision.querySelector('origin');
|
||||
const transform = [
|
||||
mesh.getAttribute('scale') ?? '1 1 1',
|
||||
origin?.getAttribute('xyz') ?? '0 0 0',
|
||||
origin?.getAttribute('rpy') ?? '0 0 0',
|
||||
]
|
||||
.map((text) =>
|
||||
text
|
||||
.trim()
|
||||
.split(/\s+/)
|
||||
.map((v) => Math.round(Number(v) * 1e9) / 1e9)
|
||||
.join(' '),
|
||||
)
|
||||
.join('|');
|
||||
const key = `${mesh.getAttribute('filename')}|${transform}`;
|
||||
if (seen.has(key)) collision.remove();
|
||||
else seen.add(key);
|
||||
}
|
||||
}
|
||||
const data = encoder.encode(new XMLSerializer().serializeToString(doc));
|
||||
return {
|
||||
...manifest,
|
||||
files: manifest.files.map((f) => (f === source ? { ...f, data, size: data.length } : f)),
|
||||
};
|
||||
}
|
||||
|
||||
/** Simulation-only ideal omni base; preserves Link1..4 arm frames/inertias/limits.
|
||||
* Native arm mesh contacts are convex approximations, not the old CAD hull recipe.
|
||||
*/
|
||||
function adaptBundle(doc: Document, c: RobotConfig): void {
|
||||
const root = doc.documentElement,
|
||||
cad = named(doc, 'body', c.baseBodyName);
|
||||
for (const name of c.baseJoints) named(doc, 'joint', name).parentElement!.remove();
|
||||
for (const j of cad.querySelectorAll(':scope > joint, :scope > freejoint')) j.remove();
|
||||
cad.setAttribute('name', '__mm_bundle_cad');
|
||||
cad.setAttribute('pos', '0 0 -0.01786');
|
||||
cad.setAttribute('quat', '0.7071067811865476 0 0 -0.7071067811865476');
|
||||
const base = node(doc, 'body', { name: c.baseBodyName, pos: '0 0 0.0515' });
|
||||
base.append(node(doc, 'freejoint', { name: c.baseJointName }));
|
||||
base.append(node(doc, 'inertial', { pos: '0 0 .04', mass: 2.2, diaginertia: '.014 .014 .022' }));
|
||||
cad.replaceWith(base);
|
||||
base.append(cad);
|
||||
// Keep collision on the arm, simplify chassis to avoid convex CAD covering wheel wells.
|
||||
const arm = named(doc, 'body', 'base_link');
|
||||
for (const geom of cad.querySelectorAll('geom')) {
|
||||
if (!arm.contains(geom)) {
|
||||
geom.setAttribute('contype', '0');
|
||||
geom.setAttribute('conaffinity', '0');
|
||||
} else {
|
||||
geom.setAttribute('friction', '.8 .005 .0001');
|
||||
}
|
||||
}
|
||||
base.append(
|
||||
node(doc, 'geom', { type: 'cylinder', size: '.112 .025', pos: '0 0 .04', group: 3, mass: 0 }),
|
||||
);
|
||||
root.querySelector('actuator')?.remove();
|
||||
const actuators = node(doc, 'actuator');
|
||||
root.append(actuators);
|
||||
const specs = [
|
||||
...c.armJoints,
|
||||
{ name: c.gripperJoint, min: c.gripperClosed, max: c.gripperOpen, mode: 'position' },
|
||||
];
|
||||
for (const spec of specs) {
|
||||
const joint = named(doc, 'joint', spec.name);
|
||||
joint.setAttribute('limited', 'true');
|
||||
joint.setAttribute('range', `${spec.min} ${spec.max}`);
|
||||
joint.setAttribute('damping', '.02');
|
||||
joint.setAttribute('armature', '.002');
|
||||
// URDF conversion leaves a joint-level ±1 N·m clamp. It otherwise overrides
|
||||
// the simulation servos below and lets the loaded arm collapse under gravity.
|
||||
joint.setAttribute('actuatorfrclimited', 'true');
|
||||
joint.setAttribute('actuatorfrcrange', spec.name === c.gripperJoint ? '-2 2' : '-8 8');
|
||||
if (spec.name === c.gripperJoint)
|
||||
joint.setAttribute(
|
||||
'axis',
|
||||
joint
|
||||
.getAttribute('axis')!
|
||||
.split(/\s+/)
|
||||
.map((v) => -Number(v))
|
||||
.join(' '),
|
||||
);
|
||||
actuators.append(
|
||||
node(doc, 'position', {
|
||||
name: `${spec.name}_servo`,
|
||||
joint: spec.name,
|
||||
kp: spec.name === c.gripperJoint ? 15 : 40,
|
||||
kv: 2,
|
||||
ctrllimited: 'true',
|
||||
ctrlrange: `${spec.min} ${spec.max}`,
|
||||
forcelimited: 'true',
|
||||
forcerange: spec.name === c.gripperJoint ? '-2 2' : '-8 8',
|
||||
}),
|
||||
);
|
||||
}
|
||||
for (let i = 0; i < c.baseJoints.length; i++) {
|
||||
const name = c.baseJoints[i],
|
||||
tx = c.baseMix[i][0] * 0.05,
|
||||
ty = c.baseMix[i][1] * 0.05;
|
||||
const axle = `${-ty} ${tx} 0`;
|
||||
const wheel = node(doc, 'body', {
|
||||
name: `__mm_${name}`,
|
||||
pos: `${0.125 * ty} ${-0.125 * tx} 0`,
|
||||
});
|
||||
wheel.append(
|
||||
node(doc, 'joint', {
|
||||
name,
|
||||
axis: axle,
|
||||
limited: 'false',
|
||||
damping: '.0002',
|
||||
armature: '.00005',
|
||||
}),
|
||||
);
|
||||
wheel.append(
|
||||
node(doc, 'inertial', { pos: '0 0 0', mass: '.06', diaginertia: '.00004 .00004 .00004' }),
|
||||
);
|
||||
wheel.append(
|
||||
node(doc, 'geom', {
|
||||
type: 'cylinder',
|
||||
size: '.033 .012',
|
||||
zaxis: axle,
|
||||
contype: 0,
|
||||
conaffinity: 0,
|
||||
mass: 0,
|
||||
group: 1,
|
||||
rgba: '.15 .18 .2 1',
|
||||
}),
|
||||
);
|
||||
for (let j = 0; j < 12; j++) {
|
||||
const angle = (2 * Math.PI * j) / 12,
|
||||
s = Math.sin(angle),
|
||||
k = Math.cos(angle);
|
||||
const axis = `${k * tx} ${k * ty} ${-s}`;
|
||||
const roller = node(doc, 'body', { pos: `${0.041 * s * tx} ${0.041 * s * ty} ${0.041 * k}` });
|
||||
roller.append(
|
||||
node(doc, 'joint', { axis, limited: 'false', damping: '.000005', armature: '.0000001' }),
|
||||
);
|
||||
roller.append(
|
||||
node(doc, 'geom', {
|
||||
type: 'capsule',
|
||||
size: '.009 .006',
|
||||
zaxis: axis,
|
||||
mass: '.003',
|
||||
contype: 4,
|
||||
conaffinity: 1,
|
||||
group: 1,
|
||||
friction: '1 .001 .0001',
|
||||
condim: 3,
|
||||
solref: '.008 1',
|
||||
rgba: '.3 .3 .3 1',
|
||||
}),
|
||||
);
|
||||
wheel.append(roller);
|
||||
}
|
||||
base.append(wheel);
|
||||
actuators.append(
|
||||
node(doc, 'velocity', {
|
||||
name: c.baseActuators[i],
|
||||
joint: name,
|
||||
kv: '.5',
|
||||
ctrllimited: 'true',
|
||||
ctrlrange: `${-c.wheelLimit} ${c.wheelLimit}`,
|
||||
forcelimited: 'true',
|
||||
forcerange: '-2 2',
|
||||
}),
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
export function composeMobileScene(data: Uint8Array, config: RobotConfig): Uint8Array {
|
||||
const doc = parse(data),
|
||||
root = doc.documentElement;
|
||||
if (root.tagName !== 'mujoco') throw new Error('场景组合需要已转换的 MJCF');
|
||||
if (
|
||||
Array.from(doc.querySelectorAll('[name]')).some((e) =>
|
||||
e.getAttribute('name')!.startsWith('__mm_'),
|
||||
)
|
||||
)
|
||||
throw new Error('任务命名空间 __mm_ 冲突(请勿重复组合)');
|
||||
if (config.recipe === 'lekiwi-bundle') adaptBundle(doc, config);
|
||||
const world = root.querySelector('worldbody');
|
||||
if (!world) throw new Error('缺少 worldbody');
|
||||
// A single task floor, no inherited robot-scene floors.
|
||||
for (const geom of world.querySelectorAll(':scope > geom[type="plane"]')) geom.remove();
|
||||
world.append(
|
||||
node(doc, 'geom', {
|
||||
name: '__mm_floor',
|
||||
type: 'plane',
|
||||
group: 2,
|
||||
size: '3 3 .1',
|
||||
friction: '.8 .005 .0001',
|
||||
contype: 1,
|
||||
conaffinity: 7,
|
||||
rgba: '.25 .3 .35 1',
|
||||
}),
|
||||
);
|
||||
const object = node(doc, 'body', { name: '__mm_object', pos: T.objectStart.join(' ') });
|
||||
object.append(node(doc, 'freejoint', { name: '__mm_object_joint' }));
|
||||
object.append(
|
||||
node(doc, 'geom', {
|
||||
name: '__mm_object_geom',
|
||||
type: 'box',
|
||||
group: 1,
|
||||
size: `${T.objectHalfSize} ${T.objectHalfSize} ${T.objectHalfSize}`,
|
||||
mass: '.04',
|
||||
friction: '1 .005 .0001',
|
||||
condim: 4,
|
||||
contype: 1,
|
||||
conaffinity: 7,
|
||||
rgba: '1 .35 .08 1',
|
||||
}),
|
||||
);
|
||||
world.append(object);
|
||||
const goal = node(doc, 'body', { name: '__mm_goal', mocap: 'true', pos: T.goalStart.join(' ') });
|
||||
goal.append(
|
||||
node(doc, 'geom', {
|
||||
name: '__mm_goal_geom',
|
||||
type: 'cylinder',
|
||||
group: 1,
|
||||
size: '.05 .003',
|
||||
contype: 0,
|
||||
conaffinity: 0,
|
||||
rgba: '.1 1 .35 .45',
|
||||
}),
|
||||
);
|
||||
world.append(goal);
|
||||
if (config.eefSiteName)
|
||||
named(doc, 'body', config.eefBodyName).append(
|
||||
node(doc, 'site', {
|
||||
name: config.eefSiteName,
|
||||
pos: config.eefOffset.join(' '),
|
||||
size: '.007',
|
||||
rgba: '.2 .8 1 1',
|
||||
}),
|
||||
);
|
||||
const option = root.querySelector('option') ?? node(doc, 'option');
|
||||
if (!option.parentElement) root.prepend(option);
|
||||
// Preserve reviewed source physics (the full LeKiwi collision recipe uses 1 ms).
|
||||
if (!option.hasAttribute('timestep')) option.setAttribute('timestep', '.002');
|
||||
if (!option.hasAttribute('integrator')) option.setAttribute('integrator', 'implicitfast');
|
||||
root.querySelector('keyframe')?.remove(); // nq grew: old keys are no longer valid.
|
||||
return encoder.encode(new XMLSerializer().serializeToString(doc));
|
||||
}
|
||||
@@ -0,0 +1,107 @@
|
||||
import * as THREE from 'three';
|
||||
import type { MuJoCoViewer } from '../viewer/MuJoCoViewer';
|
||||
import type { RobotManager } from './RobotManager';
|
||||
|
||||
/** Shift + primary drag uses the viewer's actual rendered meshes and world XY plane. */
|
||||
export class TaskDragController {
|
||||
private readonly ray = new THREE.Raycaster();
|
||||
private readonly pointer = new THREE.Vector2();
|
||||
private readonly plane = new THREE.Plane(new THREE.Vector3(0, 0, 1));
|
||||
private readonly point = new THREE.Vector3();
|
||||
private readonly offset = new THREE.Vector3();
|
||||
private readonly position = [0, 0, 0];
|
||||
private entity?: 'object' | 'goal';
|
||||
private pointerId?: number;
|
||||
private generation?: number;
|
||||
private dragEnv?: RobotManager['env'];
|
||||
private readonly canvas: HTMLCanvasElement;
|
||||
constructor(
|
||||
private readonly viewer: MuJoCoViewer,
|
||||
private readonly manager: RobotManager,
|
||||
) {
|
||||
this.canvas = viewer.renderer.domElement;
|
||||
this.canvas.addEventListener('pointerdown', this.down, true);
|
||||
this.canvas.addEventListener('pointermove', this.move, true);
|
||||
this.canvas.addEventListener('pointerup', this.up, true);
|
||||
this.canvas.addEventListener('pointercancel', this.up, true);
|
||||
this.canvas.addEventListener('lostpointercapture', this.up, true);
|
||||
window.addEventListener('blur', this.up);
|
||||
}
|
||||
private setRay(e: PointerEvent): void {
|
||||
const r = this.canvas.getBoundingClientRect();
|
||||
this.pointer.set(
|
||||
(2 * (e.clientX - r.left)) / r.width - 1,
|
||||
1 - (2 * (e.clientY - r.top)) / r.height,
|
||||
);
|
||||
this.ray.setFromCamera(this.pointer, this.viewer.camera);
|
||||
}
|
||||
private down = (e: PointerEvent): void => {
|
||||
const env = this.manager.env;
|
||||
if (!env || this.manager.loading || e.button !== 0 || !e.shiftKey) return;
|
||||
this.setRay(e);
|
||||
const hit = this.ray
|
||||
.intersectObjects(this.viewer.scene.children, true)
|
||||
.find(
|
||||
(h) =>
|
||||
h.object.visible &&
|
||||
(h.object.userData.bodyId === env.objectBodyId ||
|
||||
h.object.userData.bodyId === env.goalBodyId),
|
||||
);
|
||||
if (!hit) return;
|
||||
this.entity = hit.object.userData.bodyId === env.objectBodyId ? 'object' : 'goal';
|
||||
this.manager.setRunning(false);
|
||||
env.observe();
|
||||
const index = this.entity === 'object' ? 37 : 47;
|
||||
const z = env.state[index + 2];
|
||||
this.plane.constant = -z;
|
||||
if (!this.ray.ray.intersectPlane(this.plane, this.point)) {
|
||||
this.entity = undefined;
|
||||
return;
|
||||
}
|
||||
this.offset.set(env.state[index], env.state[index + 1], z).sub(this.point);
|
||||
this.generation = env.revision;
|
||||
this.dragEnv = env;
|
||||
this.pointerId = e.pointerId;
|
||||
this.viewer.controls.enabled = false;
|
||||
this.canvas.setPointerCapture(e.pointerId);
|
||||
e.preventDefault();
|
||||
e.stopImmediatePropagation();
|
||||
};
|
||||
private move = (e: PointerEvent): void => {
|
||||
if (!this.entity || e.pointerId !== this.pointerId) return;
|
||||
const env = this.manager.env;
|
||||
if (!env || this.manager.loading || env !== this.dragEnv || env.revision !== this.generation) {
|
||||
this.up();
|
||||
return;
|
||||
}
|
||||
this.setRay(e);
|
||||
if (this.ray.ray.intersectPlane(this.plane, this.point)) {
|
||||
this.point.add(this.offset);
|
||||
this.position[0] = Math.max(-1.9, Math.min(1.9, this.point.x));
|
||||
this.position[1] = Math.max(-1.9, Math.min(1.9, this.point.y));
|
||||
this.position[2] = Math.max(0.019, Math.min(1.9, this.point.z));
|
||||
this.manager.move(this.entity, this.position);
|
||||
this.generation = env.revision;
|
||||
}
|
||||
e.preventDefault();
|
||||
e.stopImmediatePropagation();
|
||||
};
|
||||
private up = (): void => {
|
||||
const id = this.pointerId;
|
||||
this.entity = undefined;
|
||||
this.pointerId = undefined;
|
||||
this.dragEnv = undefined;
|
||||
this.viewer.controls.enabled = true;
|
||||
if (id !== undefined && this.canvas.hasPointerCapture(id))
|
||||
this.canvas.releasePointerCapture(id);
|
||||
};
|
||||
dispose(): void {
|
||||
this.up();
|
||||
this.canvas.removeEventListener('pointerdown', this.down, true);
|
||||
this.canvas.removeEventListener('pointermove', this.move, true);
|
||||
this.canvas.removeEventListener('pointerup', this.up, true);
|
||||
this.canvas.removeEventListener('pointercancel', this.up, true);
|
||||
this.canvas.removeEventListener('lostpointercapture', this.up, true);
|
||||
window.removeEventListener('blur', this.up);
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,95 @@
|
||||
import golden from '../../../contracts/fixtures/mobile-golden.json';
|
||||
import { MOBILE_ROBOTS, MOBILE_TASK as T, validateRobotConfig } from './RobotDescriptor';
|
||||
import { decodeAction, TaskKernel } from './TaskKernel';
|
||||
|
||||
function state() {
|
||||
const s = new Float64Array(T.stateSize);
|
||||
for (const i of [3, 33, 40, 50]) s[i] = 1;
|
||||
s[39] = s[49] = T.objectStart[2];
|
||||
return s;
|
||||
}
|
||||
describe('mobile-manipulator-v2 数学契约', () => {
|
||||
it('两个描述符有效,名称/维度错误拒绝', () => {
|
||||
for (const c of MOBILE_ROBOTS) validateRobotConfig(c);
|
||||
expect(() => validateRobotConfig({ ...MOBILE_ROBOTS[0], baseMix: [[1]] })).toThrow();
|
||||
expect(() =>
|
||||
validateRobotConfig({
|
||||
...MOBILE_ROBOTS[0],
|
||||
armJoints: Array(9).fill(MOBILE_ROBOTS[0].armJoints[0]),
|
||||
}),
|
||||
).toThrow();
|
||||
});
|
||||
it('与 Python 40 组独立 golden 状态、旋转、控制及奖励一致', () => {
|
||||
for (const sample of golden) {
|
||||
const config = MOBILE_ROBOTS.find((r) => r.id === sample.robotId)!;
|
||||
const kernel = new TaskKernel(config);
|
||||
kernel.hasLifted = sample.lifted;
|
||||
const result = kernel.evaluate(Float64Array.from(sample.state));
|
||||
sample.observation.forEach((v, i) => expect(result.observation[i]).toBeCloseTo(v, 6));
|
||||
expect(result.reward).toBeCloseTo(sample.reward, 12);
|
||||
expect(result.info.stage).toBe(sample.info.stage);
|
||||
expect(result.terminated).toBe(sample.terminated);
|
||||
expect(result.truncated).toBe(sample.truncated);
|
||||
const control = new Float64Array(9);
|
||||
decodeAction(config, Float32Array.from(sample.action), control);
|
||||
sample.control.forEach((v, i) => expect(control[i]).toBeCloseTo(v, 10));
|
||||
}
|
||||
});
|
||||
it('退化四元数采用与 Python 相同的单位姿态回退', () => {
|
||||
const k = new TaskKernel(MOBILE_ROBOTS[0]);
|
||||
const o = k.observe(new Float64Array(T.stateSize));
|
||||
for (const start of [3, 41, 48, 55, 62])
|
||||
expect(Array.from(o.slice(start, start + 4))).toEqual([1, 0, 0, 0]);
|
||||
});
|
||||
it('成功必须经过抓取抬升、张开夹爪、末端撤离、连续稳定放置', () => {
|
||||
expect(
|
||||
T.controlDt * T.maxSteps * (T.reachWeight + T.liftWeight + T.transportWeight),
|
||||
).toBeLessThan(T.successBonus);
|
||||
const k = new TaskKernel(MOBILE_ROBOTS[0]),
|
||||
s = state();
|
||||
s[29] = 1;
|
||||
s[30] = 1;
|
||||
for (let i = 0; i < 20; i++) expect(k.evaluate(s).terminated).toBe(false);
|
||||
s[29] = 0;
|
||||
s[30] = 0;
|
||||
s[39] = s[32] = 0.12;
|
||||
expect(k.evaluate(s).info.stage).toBe('transport');
|
||||
s[29] = 1;
|
||||
s[30] = 1;
|
||||
s[39] = T.objectStart[2];
|
||||
for (let i = 0; i < T.settleSteps - 1; i++) expect(k.evaluate(s).terminated).toBe(false);
|
||||
expect(k.evaluate(s).info.reward_components.success).toBe(T.successBonus);
|
||||
expect(() => k.evaluate(s)).toThrow('回合');
|
||||
k.reset();
|
||||
expect(k.hasLifted).toBe(false);
|
||||
expect(k.result.terminated).toBe(false);
|
||||
k.steps = T.maxSteps - 1;
|
||||
expect(k.evaluate(s).truncated).toBe(true);
|
||||
});
|
||||
it('复用所有输出缓冲区,位置/速度控制,拒绝 NaN 不部分写入', () => {
|
||||
const c = structuredClone(MOBILE_ROBOTS[1]);
|
||||
const config = {
|
||||
...c,
|
||||
armJoints: c.armJoints.map((j, i) => (i === 0 ? { ...j, mode: 'velocity' as const } : j)),
|
||||
};
|
||||
const action = new Float32Array(12),
|
||||
ctrl = new Float64Array(9);
|
||||
action[3] = 0.5;
|
||||
action[11] = 1;
|
||||
decodeAction(config, action, ctrl);
|
||||
expect(ctrl[3]).toBe(1);
|
||||
expect(ctrl[8]).toBeCloseTo(0.9, 12);
|
||||
const old = ctrl.slice();
|
||||
action[11] = NaN;
|
||||
expect(() => decodeAction(config, action, ctrl)).toThrow();
|
||||
expect(ctrl).toEqual(old);
|
||||
const k = new TaskKernel(config),
|
||||
s = state(),
|
||||
first = k.evaluate(s);
|
||||
for (let i = 0; i < 100; i++) {
|
||||
expect(k.evaluate(s)).toBe(first);
|
||||
expect(k.observe(s)).toBe(first.observation);
|
||||
expect(k.observation.slice(34, 37)).toEqual(new Float32Array(3));
|
||||
}
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,301 @@
|
||||
import { MOBILE_TASK as T, type RobotConfig } from './RobotDescriptor';
|
||||
|
||||
export const clamp = (v: number, lo = -1, hi = 1): number => Math.max(lo, Math.min(hi, v));
|
||||
export const TRAINING_STAGES = ['navigate', 'reach', 'pick-place'] as const;
|
||||
export type TrainingStage = (typeof TRAINING_STAGES)[number];
|
||||
export type TaskStage = 'navigate' | 'reach' | 'lift' | 'transport' | 'success' | 'safety-stop';
|
||||
export function navigationError(s: Float64Array): { distance: number; yaw: number } {
|
||||
return {
|
||||
distance: Math.hypot(
|
||||
s[37] - T.navigationOffset[0] - s[0],
|
||||
s[38] - T.navigationOffset[1] - s[1],
|
||||
),
|
||||
yaw: Math.atan2(2 * (s[3] * s[6] + s[4] * s[5]), 1 - 2 * (s[5] ** 2 + s[6] ** 2)),
|
||||
};
|
||||
}
|
||||
export interface TaskInfo {
|
||||
reward_components: {
|
||||
reach: number;
|
||||
lift: number;
|
||||
transport: number;
|
||||
success: number;
|
||||
navigation: number;
|
||||
action_rate: number;
|
||||
joint_velocity: number;
|
||||
safety: number;
|
||||
};
|
||||
safety_stop: string;
|
||||
navigation_distance: number;
|
||||
max_joint_velocity: number;
|
||||
is_success: boolean;
|
||||
stage: TaskStage;
|
||||
}
|
||||
export interface StepResult {
|
||||
observation: Float32Array;
|
||||
reward: number;
|
||||
terminated: boolean;
|
||||
truncated: boolean;
|
||||
info: TaskInfo;
|
||||
}
|
||||
|
||||
/** v1 raw state layout (SI, world frame, quaternions wxyz):
|
||||
* base pose 0:7, world linear/angular velocity 7:13, arm q 13:21, qd 21:29,
|
||||
* opening [0,1] 29, EEF pose 30:37, object pose 37:44, object world velocity 44:47,
|
||||
* goal pose 47:54. Padding slots are zero. See docs/mobile-manipulator.md.
|
||||
*/
|
||||
export class TaskKernel {
|
||||
readonly observation = new Float32Array(T.observationSize);
|
||||
readonly lastAction = new Float32Array(T.actionSize);
|
||||
readonly targets = new Float32Array(T.actionSize);
|
||||
private actionRate = 0;
|
||||
private previousDistance?: number;
|
||||
readonly result: StepResult = {
|
||||
observation: this.observation,
|
||||
reward: 0,
|
||||
terminated: false,
|
||||
truncated: false,
|
||||
info: {
|
||||
reward_components: {
|
||||
reach: 0,
|
||||
lift: 0,
|
||||
transport: 0,
|
||||
success: 0,
|
||||
navigation: 0,
|
||||
action_rate: 0,
|
||||
joint_velocity: 0,
|
||||
safety: 0,
|
||||
},
|
||||
safety_stop: '',
|
||||
navigation_distance: 0,
|
||||
max_joint_velocity: 0,
|
||||
is_success: false,
|
||||
stage: 'reach',
|
||||
},
|
||||
};
|
||||
steps = 0;
|
||||
hasLifted = false;
|
||||
settle = 0;
|
||||
constructor(
|
||||
readonly config: RobotConfig,
|
||||
public trainingStage: TrainingStage = 'pick-place',
|
||||
) {}
|
||||
|
||||
reset(): void {
|
||||
this.steps = this.settle = 0;
|
||||
this.hasLifted = false;
|
||||
this.result.reward = 0;
|
||||
this.result.terminated = this.result.truncated = this.result.info.is_success = false;
|
||||
this.result.info.stage = this.trainingStage === 'navigate' ? 'navigate' : 'reach';
|
||||
this.result.info.safety_stop = '';
|
||||
this.result.info.max_joint_velocity = this.result.info.navigation_distance = 0;
|
||||
this.lastAction.fill(0);
|
||||
this.targets.fill(0);
|
||||
this.actionRate = 0;
|
||||
this.previousDistance = undefined;
|
||||
const r = this.result.info.reward_components;
|
||||
r.reach =
|
||||
r.lift =
|
||||
r.transport =
|
||||
r.success =
|
||||
r.navigation =
|
||||
r.action_rate =
|
||||
r.joint_velocity =
|
||||
r.safety =
|
||||
0;
|
||||
}
|
||||
recordAction(applied: Float32Array, targets: Float32Array): void {
|
||||
this.actionRate = 0;
|
||||
for (let i = 0; i < T.actionSize; i++)
|
||||
this.actionRate += (applied[i] - this.lastAction[i]) ** 2;
|
||||
this.lastAction.set(applied);
|
||||
this.targets.set(targets);
|
||||
}
|
||||
|
||||
observe(s: Float64Array): Float32Array {
|
||||
const o = this.observation;
|
||||
o.fill(0);
|
||||
for (let i = 0; i < 3; i++) o[i] = clamp(s[i] / T.positionScale);
|
||||
this.quat(s, 3, 3);
|
||||
for (let i = 0; i < 6; i++)
|
||||
o[7 + i] = clamp(s[7 + i] / (i < 3 ? T.linearVelocityScale : T.angularVelocityScale));
|
||||
for (let i = 0; i < this.config.armJoints.length; i++) {
|
||||
const j = this.config.armJoints[i];
|
||||
o[13 + i] = clamp((2 * (s[13 + i] - j.min)) / (j.max - j.min) - 1);
|
||||
o[21 + i] = clamp(s[21 + i] / T.jointVelocityScale);
|
||||
o[29 + i] = 1;
|
||||
}
|
||||
o[37] = clamp(2 * s[29] - 1);
|
||||
for (let i = 0; i < 3; i++) {
|
||||
o[38 + i] = clamp(s[30 + i] / T.positionScale);
|
||||
o[59 + i] = clamp(s[47 + i] / T.positionScale);
|
||||
}
|
||||
this.quat(s, 33, 41);
|
||||
this.relative(s, 30, 37, 45);
|
||||
this.relative(s, 47, 37, 52);
|
||||
this.quat(s, 50, 62);
|
||||
o[66] = Number(this.hasLifted);
|
||||
o[67] = this.settle / T.settleSteps;
|
||||
o.set(this.lastAction, 68);
|
||||
o.set(this.targets, 80);
|
||||
return o;
|
||||
}
|
||||
|
||||
private quat(s: Float64Array, start: number, target: number): void {
|
||||
const n = Math.hypot(s[start], s[start + 1], s[start + 2], s[start + 3]);
|
||||
const sign = s[start] < 0 ? -1 : 1;
|
||||
for (let i = 0; i < 4; i++)
|
||||
this.observation[target + i] = n > 1e-12 ? (sign * s[start + i]) / n : Number(i === 0);
|
||||
}
|
||||
|
||||
/** inverse(parent) * child, including position rotated into parent frame. */
|
||||
private relative(s: Float64Array, parent: number, child: number, out: number): void {
|
||||
const w = s[parent + 3],
|
||||
x = -s[parent + 4],
|
||||
y = -s[parent + 5],
|
||||
z = -s[parent + 6];
|
||||
const vx = s[child] - s[parent],
|
||||
vy = s[child + 1] - s[parent + 1],
|
||||
vz = s[child + 2] - s[parent + 2];
|
||||
const tx = 2 * (y * vz - z * vy),
|
||||
ty = 2 * (z * vx - x * vz),
|
||||
tz = 2 * (x * vy - y * vx);
|
||||
const o = this.observation;
|
||||
o[out] = clamp((vx + w * tx + y * tz - z * ty) / T.positionScale);
|
||||
o[out + 1] = clamp((vy + w * ty + z * tx - x * tz) / T.positionScale);
|
||||
o[out + 2] = clamp((vz + w * tz + x * ty - y * tx) / T.positionScale);
|
||||
const a = s[child + 3],
|
||||
b = s[child + 4],
|
||||
c = s[child + 5],
|
||||
d = s[child + 6];
|
||||
const rw = w * a - x * b - y * c - z * d;
|
||||
const rx = w * b + x * a + y * d - z * c;
|
||||
const ry = w * c - x * d + y * a + z * b;
|
||||
const rz = w * d + x * c - y * b + z * a;
|
||||
const n = Math.hypot(rw, rx, ry, rz),
|
||||
k = n > 1e-12 ? (rw < 0 ? -1 : 1) / n : 0;
|
||||
o[out + 3] = n > 1e-12 ? rw * k : 1;
|
||||
o[out + 4] = rx * k;
|
||||
o[out + 5] = ry * k;
|
||||
o[out + 6] = rz * k;
|
||||
}
|
||||
|
||||
evaluate(s: Float64Array, safetyStop = '', peakVelocity = 0): StepResult {
|
||||
if (this.result.terminated || this.result.truncated) throw new Error('回合已结束,请 reset');
|
||||
const reach = Math.hypot(s[37] - s[30], s[38] - s[31], s[39] - s[32]);
|
||||
const goal = Math.hypot(s[37] - s[47], s[38] - s[48], s[39] - s[49]);
|
||||
const lift = clamp((s[39] - T.objectStart[2]) / T.liftHeight, 0, 1);
|
||||
if (this.trainingStage === 'pick-place' && lift >= 1 && reach < T.graspDistance && s[29] < 0.4)
|
||||
this.hasLifted = true;
|
||||
const { distance, yaw } = navigationError(s);
|
||||
const near = distance < T.navigationTolerance && Math.abs(yaw) < T.navigationYawTolerance;
|
||||
const stopped =
|
||||
Math.hypot(s[7], s[8], s[9]) < T.navigationSpeedTolerance &&
|
||||
Math.hypot(s[10], s[11], s[12]) < 0.1;
|
||||
let settled =
|
||||
this.hasLifted &&
|
||||
goal < T.goalTolerance &&
|
||||
s[29] > T.releaseOpening &&
|
||||
reach > T.graspDistance &&
|
||||
Math.hypot(s[44], s[45], s[46]) < T.settleSpeed;
|
||||
if (this.trainingStage === 'navigate') settled = near && stopped;
|
||||
else if (this.trainingStage === 'reach')
|
||||
settled =
|
||||
near && stopped && reach < T.graspDistance && Math.hypot(...s.subarray(21, 29)) < 0.15;
|
||||
this.settle = settled ? Math.min(T.settleSteps, this.settle + 1) : 0;
|
||||
let success = this.settle === T.settleSteps;
|
||||
const r = this.result.info.reward_components;
|
||||
// Dense rates are integrated over time; action-rate cost acts on control changes.
|
||||
r.reach = T.controlDt * T.reachWeight * Math.exp(-T.reachGain * reach);
|
||||
r.lift = T.controlDt * T.liftWeight * lift;
|
||||
r.transport = this.hasLifted
|
||||
? T.controlDt * T.transportWeight * Math.exp(-T.transportGain * goal)
|
||||
: 0;
|
||||
if (this.trainingStage === 'navigate') r.reach = r.lift = r.transport = 0;
|
||||
else if (this.trainingStage === 'reach') r.lift = r.transport = 0;
|
||||
const progress = this.previousDistance === undefined ? 0 : this.previousDistance - distance;
|
||||
this.previousDistance = distance;
|
||||
r.navigation = this.hasLifted
|
||||
? 0
|
||||
: T.navigationProgressWeight * progress - T.controlDt * (distance + 0.1 * Math.abs(yaw));
|
||||
r.action_rate = -T.actionRateWeight * this.actionRate;
|
||||
r.joint_velocity = 0;
|
||||
for (let i = 21; i < 29; i++) {
|
||||
r.joint_velocity -= T.controlDt * T.jointVelocityWeight * s[i] ** 2;
|
||||
peakVelocity = Math.max(peakVelocity, Math.abs(s[i]));
|
||||
}
|
||||
if (!safetyStop && peakVelocity > T.jointSpeedStop) safetyStop = 'joint_velocity';
|
||||
if (
|
||||
!safetyStop &&
|
||||
(1 - 2 * (s[4] ** 2 + s[5] ** 2) < 0.5 ||
|
||||
Math.max(Math.abs(s[0]), Math.abs(s[1])) > T.positionScale)
|
||||
)
|
||||
safetyStop = 'base_pose';
|
||||
if (safetyStop) success = false;
|
||||
r.success = success ? T.successBonus : 0;
|
||||
r.safety = safetyStop ? -T.safetyPenalty : 0;
|
||||
this.result.reward =
|
||||
r.reach +
|
||||
r.lift +
|
||||
r.transport +
|
||||
r.success +
|
||||
r.navigation +
|
||||
r.action_rate +
|
||||
r.joint_velocity +
|
||||
r.safety;
|
||||
this.result.terminated = success || !!safetyStop;
|
||||
this.result.info.safety_stop = safetyStop;
|
||||
this.result.info.navigation_distance = distance;
|
||||
this.result.info.max_joint_velocity = Math.max(
|
||||
this.result.info.max_joint_velocity,
|
||||
peakVelocity,
|
||||
);
|
||||
this.result.truncated = ++this.steps >= T.maxSteps && !success;
|
||||
this.result.info.is_success = success;
|
||||
this.result.info.stage = safetyStop
|
||||
? 'safety-stop'
|
||||
: success
|
||||
? 'success'
|
||||
: this.trainingStage === 'navigate' || (!near && !this.hasLifted)
|
||||
? 'navigate'
|
||||
: this.hasLifted
|
||||
? 'transport'
|
||||
: reach < T.graspDistance
|
||||
? 'lift'
|
||||
: 'reach';
|
||||
this.observe(s);
|
||||
return this.result;
|
||||
}
|
||||
}
|
||||
|
||||
/** @deprecated v1 math oracle retained ONLY for historical regression fixtures.
|
||||
* Production v2 environments use SafeActionController, never this full-range mapper.
|
||||
* Output order: base actuators, arm actuators, gripper actuators. */
|
||||
export function decodeAction(
|
||||
config: RobotConfig,
|
||||
action: Float32Array,
|
||||
output: Float64Array,
|
||||
): void {
|
||||
if (action.length !== T.actionSize) throw new Error('动作必须是 [12]');
|
||||
for (let i = 0; i < action.length; i++)
|
||||
if (!Number.isFinite(action[i])) throw new Error('动作包含非有限数');
|
||||
let largest = config.wheelLimit;
|
||||
for (let i = 0; i < config.baseMix.length; i++) {
|
||||
const row = config.baseMix[i];
|
||||
output[i] =
|
||||
row[0] * clamp(action[0]) * config.baseLimits[0] +
|
||||
row[1] * clamp(action[1]) * config.baseLimits[1] +
|
||||
row[2] * clamp(action[2]) * config.baseLimits[2];
|
||||
largest = Math.max(largest, Math.abs(output[i]));
|
||||
}
|
||||
for (let i = 0; i < config.baseMix.length; i++) output[i] *= config.wheelLimit / largest;
|
||||
let offset = config.baseMix.length;
|
||||
for (let i = 0; i < config.armJoints.length; i++) {
|
||||
const j = config.armJoints[i],
|
||||
a = clamp(action[3 + i]);
|
||||
output[offset++] =
|
||||
j.mode === 'position' ? j.min + (a + 1) * 0.5 * (j.max - j.min) : a * j.velocityLimit;
|
||||
}
|
||||
const opening = (clamp(action[11]) + 1) * 0.5;
|
||||
for (const g of config.gripperActuators)
|
||||
output[offset++] = g.closed + opening * (g.open - g.closed);
|
||||
}
|
||||
@@ -0,0 +1,336 @@
|
||||
import { describe, expect, it, vi } from 'vitest';
|
||||
import { AgentTaskController, type SkillExecutor } from './AgentTaskController';
|
||||
import { MockDecisionProvider } from './MockDecisionProvider';
|
||||
import { PickPlaceEvaluator, type PhysicalSample } from './PhysicalEvidence';
|
||||
import { SKILLS, type Skill } from './protocol';
|
||||
|
||||
class FakeSkills implements SkillExecutor {
|
||||
phase: Skill = 'open';
|
||||
waiting = true;
|
||||
status: 'running' | 'succeeded' | 'failed' = 'running';
|
||||
error = '';
|
||||
completed = false;
|
||||
fault = '';
|
||||
goal = [0.257, 0.615, 0.128];
|
||||
evaluator = new PickPlaceEvaluator([0.257, 0.015, 0.128], this.goal);
|
||||
sample: PhysicalSample = {
|
||||
time: 0,
|
||||
base: [0, 0, 0.05],
|
||||
baseSpeed: 0,
|
||||
yaw: 0,
|
||||
tilt: 0,
|
||||
joints: [0, 0, 0, 0, 0],
|
||||
peakJointSpeed: 0,
|
||||
opening: 1,
|
||||
tcp: [0.257, 0.015, 0.2],
|
||||
object: [0.257, 0.015, 0.128],
|
||||
objectSpeed: 0,
|
||||
fingerForces: [0, 0],
|
||||
supported: true,
|
||||
onGoalSupport: false,
|
||||
};
|
||||
authorized: Skill[] = [];
|
||||
hold = vi.fn();
|
||||
dispose = vi.fn();
|
||||
beforeStep = vi.fn();
|
||||
get candidate(): Skill {
|
||||
return this.completed ? (SKILLS[SKILLS.indexOf(this.phase) + 1] ?? 'settle') : this.phase;
|
||||
}
|
||||
authorize(skill: Skill) {
|
||||
if (!this.waiting || skill !== this.candidate) throw new Error('bad switch');
|
||||
if (skill === 'carry' && !this.evaluator.secure) throw new Error('unsafe');
|
||||
this.authorized.push(skill);
|
||||
this.phase = skill;
|
||||
this.completed = false;
|
||||
this.waiting = false;
|
||||
}
|
||||
afterStep(dt: number) {
|
||||
this.sample.time += dt;
|
||||
if (this.fault && this.phase === 'close') {
|
||||
this.status = 'failed';
|
||||
this.error = this.fault;
|
||||
return;
|
||||
}
|
||||
if (this.phase === 'verify') this.evaluator.secure = true;
|
||||
if (this.phase === 'carry') this.evaluator.transported = 0.6;
|
||||
if (this.phase === 'release') this.evaluator.secure = false;
|
||||
this.completed = true;
|
||||
this.waiting = true;
|
||||
if (this.phase === 'settle') {
|
||||
this.evaluator.succeeded = true;
|
||||
this.status = 'succeeded';
|
||||
}
|
||||
}
|
||||
canRecover() {
|
||||
return this.status === 'failed' && ['empty_grasp', 'alignment_timeout'].includes(this.error);
|
||||
}
|
||||
recover() {
|
||||
if (!this.canRecover()) throw new Error('unsafe recovery');
|
||||
this.phase = 'open';
|
||||
this.status = 'running';
|
||||
this.error = '';
|
||||
this.waiting = true;
|
||||
this.completed = false;
|
||||
}
|
||||
}
|
||||
async function flush() {
|
||||
for (let i = 0; i < 12; i++) await Promise.resolve();
|
||||
}
|
||||
async function run(driver: AgentTaskController) {
|
||||
for (let i = 0; i < 300 && !driver.terminal; i++) {
|
||||
if (driver.beforeStep()) driver.afterStep(0.001);
|
||||
await flush();
|
||||
}
|
||||
expect(driver.terminal).toBe(true);
|
||||
}
|
||||
describe('模型闭环调度(技能为 mock,物理另有真实 WASM 验收)', () => {
|
||||
it('LLM 一次,Jev 每个边界一次;异步回调不授权,独立证据才成功', async () => {
|
||||
const skills = new FakeSkills();
|
||||
const driver = new AgentTaskController(skills, new MockDecisionProvider(), '搬运方块');
|
||||
expect(driver.beforeStep()).toBe(false);
|
||||
await flush();
|
||||
expect(skills.authorized).toEqual([]);
|
||||
expect(skills.sample.time).toBe(0);
|
||||
await run(driver);
|
||||
expect(driver.state).toBe('succeeded');
|
||||
expect(driver.llmCalls).toBe(1);
|
||||
expect(driver.jevCalls).toBe(11);
|
||||
expect(skills.authorized).toEqual(SKILLS);
|
||||
expect(driver.results).toHaveLength(11);
|
||||
driver.dispose();
|
||||
});
|
||||
it('支持可恢复空抓,但跨重规划也不能重置每技能 2 次上限', async () => {
|
||||
const skills = new FakeSkills();
|
||||
skills.fault = 'empty_grasp';
|
||||
const driver = new AgentTaskController(skills, new MockDecisionProvider(), '搬运方块');
|
||||
await run(driver);
|
||||
expect(driver.state).toBe('failed');
|
||||
expect(driver.recoveries.close).toBe(2);
|
||||
expect(driver.failures).toHaveLength(3);
|
||||
expect(driver.jevCalls).toBeLessThanOrEqual(60);
|
||||
expect(skills.authorized).not.toContain('carry');
|
||||
driver.dispose();
|
||||
});
|
||||
it('关节卡住/失抓/不可达等硬失败立即停,不询问模型放行', async () => {
|
||||
for (const fault of ['joint_velocity', 'lost_grasp', 'unreachable', 'navigation_timeout']) {
|
||||
const skills = new FakeSkills();
|
||||
skills.fault = fault;
|
||||
const driver = new AgentTaskController(skills, new MockDecisionProvider(), '搬运方块');
|
||||
await run(driver);
|
||||
expect(driver.error).toBe(fault);
|
||||
expect(driver.recoveries).toEqual({});
|
||||
expect(driver.jevCalls).toBe(4);
|
||||
driver.dispose();
|
||||
}
|
||||
});
|
||||
it('物理成功不能由模型的 secure 声明替代', async () => {
|
||||
const skills = new FakeSkills(),
|
||||
provider = new MockDecisionProvider();
|
||||
const normal = provider.decide.bind(provider);
|
||||
provider.decide = async (...args) => {
|
||||
const value = await normal(...args);
|
||||
value.value.grasp = 'secure';
|
||||
return value;
|
||||
};
|
||||
const driver = new AgentTaskController(skills, provider, '搬运方块');
|
||||
await run(driver);
|
||||
expect(driver.state).toBe('failed');
|
||||
expect(driver.error).toContain('矛盾');
|
||||
expect(skills.authorized).toEqual([]);
|
||||
driver.dispose();
|
||||
});
|
||||
it('未知候选/循环计划拒绝;真实 API 模式错误不会降级 mock', async () => {
|
||||
const skills = new FakeSkills(),
|
||||
provider = new MockDecisionProvider();
|
||||
const normal = provider.plan.bind(provider);
|
||||
provider.plan = async (...args) => {
|
||||
const value = await normal(...args);
|
||||
value.value.steps.reverse();
|
||||
return value;
|
||||
};
|
||||
const driver = new AgentTaskController(skills, provider, '搬运方块');
|
||||
await run(driver);
|
||||
expect(driver.state).toBe('failed');
|
||||
expect(driver.jevCalls).toBe(0);
|
||||
driver.dispose();
|
||||
const api = {
|
||||
mode: 'api' as const,
|
||||
plan: vi.fn().mockRejectedValue(new Error('upstream_http_401')),
|
||||
decide: vi.fn(),
|
||||
};
|
||||
const failed = new AgentTaskController(new FakeSkills(), api, '搬运方块');
|
||||
await run(failed);
|
||||
expect(failed.error).toBe('upstream_http_401');
|
||||
expect(api.decide).not.toHaveBeenCalled();
|
||||
failed.dispose();
|
||||
});
|
||||
it('正常边界可请求重规划但最多 2 次,不暗中继续动作', async () => {
|
||||
const skills = new FakeSkills(),
|
||||
provider = new MockDecisionProvider();
|
||||
const normal = provider.decide.bind(provider);
|
||||
provider.decide = async (...args) => {
|
||||
const value = await normal(...args);
|
||||
value.value.recovery = 'replan';
|
||||
return value;
|
||||
};
|
||||
const driver = new AgentTaskController(skills, provider, '搬运方块');
|
||||
await run(driver);
|
||||
expect(driver.replans).toBe(2);
|
||||
expect(driver.llmCalls).toBe(3);
|
||||
expect(driver.error).toContain('重规划预算');
|
||||
expect(skills.authorized).toEqual([]);
|
||||
driver.dispose();
|
||||
});
|
||||
it('LLM 要求重规划时,Jev 的重试建议不能绕过该分支', async () => {
|
||||
const skills = new FakeSkills();
|
||||
skills.fault = 'empty_grasp';
|
||||
const provider = new MockDecisionProvider(),
|
||||
normal = provider.plan.bind(provider);
|
||||
provider.plan = async (...args) => {
|
||||
const result = await normal(...args);
|
||||
result.value.steps.forEach((step) => {
|
||||
step.onFailure = 'replan';
|
||||
});
|
||||
return result;
|
||||
};
|
||||
const driver = new AgentTaskController(skills, provider, '搬运');
|
||||
await run(driver);
|
||||
expect(driver.replans).toBe(2);
|
||||
expect(driver.llmCalls).toBe(3);
|
||||
expect(driver.recoveries.close).toBe(2);
|
||||
driver.dispose();
|
||||
});
|
||||
it('恢复中的重规划也受技能次数限制', async () => {
|
||||
const skills = new FakeSkills();
|
||||
skills.fault = 'empty_grasp';
|
||||
const provider = new MockDecisionProvider(),
|
||||
normal = provider.decide.bind(provider);
|
||||
provider.decide = async (...args) => {
|
||||
const value = await normal(...args);
|
||||
if (args[2] !== 'none') value.value.recovery = 'replan';
|
||||
return value;
|
||||
};
|
||||
const driver = new AgentTaskController(skills, provider, '搬运方块');
|
||||
await run(driver);
|
||||
expect(driver.replans).toBe(2);
|
||||
expect(driver.recoveries.close).toBe(2);
|
||||
expect(driver.state).toBe('failed');
|
||||
driver.dispose();
|
||||
});
|
||||
it('请求量和墙钟期限同时有限;模型等待不推进技能/物理时间', async () => {
|
||||
const skills = new FakeSkills();
|
||||
let now = 0;
|
||||
const driver = new AgentTaskController(skills, new MockDecisionProvider(), '搬运方块', {
|
||||
limits: { maxJev: 1 },
|
||||
now: () => now,
|
||||
});
|
||||
await run(driver);
|
||||
expect(driver.error).toContain('Jev 请求预算');
|
||||
expect(driver.jevCalls).toBe(1);
|
||||
driver.dispose();
|
||||
const limited = new AgentTaskController(
|
||||
new FakeSkills(),
|
||||
new MockDecisionProvider(),
|
||||
'搬运方块',
|
||||
{ limits: { wallMs: 10 }, now: () => now },
|
||||
);
|
||||
limited.beforeStep();
|
||||
now = 11;
|
||||
limited.beforeStep();
|
||||
expect(limited.error).toContain('墙钟期限');
|
||||
limited.dispose();
|
||||
});
|
||||
it('任务暂停取消旧响应;继续只能处理新请求,支持单技能步进', async () => {
|
||||
const skills = new FakeSkills(),
|
||||
provider = new MockDecisionProvider();
|
||||
const normal = provider.plan.bind(provider);
|
||||
let resolve!: () => void;
|
||||
provider.plan = async (...args) => {
|
||||
await new Promise<void>((r) => {
|
||||
resolve = r;
|
||||
});
|
||||
return normal(...args);
|
||||
};
|
||||
const driver = new AgentTaskController(skills, provider, '搬运方块');
|
||||
driver.beforeStep();
|
||||
driver.pause();
|
||||
resolve();
|
||||
await flush();
|
||||
expect(driver.state).toBe('paused');
|
||||
expect(skills.sample.time).toBe(0);
|
||||
provider.plan = normal;
|
||||
driver.resume(true);
|
||||
for (let i = 0; i < 30 && driver.state !== 'paused'; i++) {
|
||||
if (driver.beforeStep()) driver.afterStep(0.001);
|
||||
await flush();
|
||||
}
|
||||
expect(driver.state).toBe('paused');
|
||||
expect(skills.authorized).toEqual(['open']);
|
||||
const time = skills.sample.time;
|
||||
driver.beforeStep();
|
||||
expect(skills.sample.time).toBe(time);
|
||||
driver.resume();
|
||||
await run(driver);
|
||||
expect(driver.state).toBe('succeeded');
|
||||
driver.dispose();
|
||||
});
|
||||
it('模型超时/错误候选均冻结并失败,禁止生成额外物理步', async () => {
|
||||
const skills = new FakeSkills(),
|
||||
provider = new MockDecisionProvider();
|
||||
provider.plan = () => new Promise(() => undefined);
|
||||
const driver = new AgentTaskController(skills, provider, '搬运', { limits: { requestMs: 1 } });
|
||||
driver.beforeStep();
|
||||
await new Promise((r) => setTimeout(r, 5));
|
||||
await flush();
|
||||
driver.beforeStep();
|
||||
expect(driver.error).toContain('超时');
|
||||
expect(skills.sample.time).toBe(0);
|
||||
driver.dispose();
|
||||
const bad = new MockDecisionProvider(),
|
||||
normal = bad.decide.bind(bad);
|
||||
bad.decide = async (...args) => {
|
||||
const value = await normal(...args);
|
||||
value.value.choice = 'carry';
|
||||
return value;
|
||||
};
|
||||
const invalid = new AgentTaskController(new FakeSkills(), bad, '搬运');
|
||||
await run(invalid);
|
||||
expect(invalid.error).toContain('候选');
|
||||
invalid.dispose();
|
||||
});
|
||||
it('技能自报成功但缺少独立评估证据仍然失败', async () => {
|
||||
const skills = new FakeSkills(),
|
||||
normal = skills.afterStep.bind(skills);
|
||||
skills.afterStep = (dt) => {
|
||||
normal(dt);
|
||||
if (skills.status === 'succeeded') skills.evaluator.succeeded = false;
|
||||
};
|
||||
const driver = new AgentTaskController(skills, new MockDecisionProvider(), '搬运');
|
||||
await run(driver);
|
||||
expect(driver.state).toBe('failed');
|
||||
expect(driver.results.at(-1)?.code).toBe('placement_unverified');
|
||||
driver.dispose();
|
||||
});
|
||||
it('取消/销毁后迟到结果不能授权;模型忽略 abort 也不能越过门禁', async () => {
|
||||
const skills = new FakeSkills(),
|
||||
provider = new MockDecisionProvider();
|
||||
const normal = provider.plan.bind(provider);
|
||||
let resolve!: () => void;
|
||||
provider.plan = async (obs, instruction, remaining) => {
|
||||
await new Promise<void>((r) => {
|
||||
resolve = r;
|
||||
});
|
||||
return normal(obs, instruction, remaining, new AbortController().signal);
|
||||
};
|
||||
const driver = new AgentTaskController(skills, provider, '搬运方块');
|
||||
driver.beforeStep();
|
||||
driver.cancel('scene changed');
|
||||
driver.dispose();
|
||||
resolve();
|
||||
await flush();
|
||||
expect(driver.beforeStep()).toBe(false);
|
||||
expect(skills.authorized).toEqual([]);
|
||||
expect(skills.dispose).toHaveBeenCalledOnce();
|
||||
expect(driver.state).toBe('cancelled');
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,412 @@
|
||||
import type { AgentDriver } from '../../simulation/AgentDriver';
|
||||
import type { DeterministicBaseline } from './DeterministicBaseline';
|
||||
import type { DecisionProvider } from './DecisionClient';
|
||||
import { RequestGate, type RequestStamp } from './RequestGate';
|
||||
import {
|
||||
AGENT_VERSION,
|
||||
SKILLS,
|
||||
observation,
|
||||
validateJevDecision,
|
||||
validatePlan,
|
||||
skillErrorCode,
|
||||
type JevDecision,
|
||||
type Plan,
|
||||
type Skill,
|
||||
type SkillResult,
|
||||
} from './protocol';
|
||||
|
||||
export type SkillExecutor = Pick<
|
||||
DeterministicBaseline,
|
||||
| 'sample'
|
||||
| 'evaluator'
|
||||
| 'phase'
|
||||
| 'candidate'
|
||||
| 'goal'
|
||||
| 'status'
|
||||
| 'error'
|
||||
| 'waiting'
|
||||
| 'beforeStep'
|
||||
| 'afterStep'
|
||||
| 'authorize'
|
||||
| 'canRecover'
|
||||
| 'recover'
|
||||
| 'hold'
|
||||
| 'dispose'
|
||||
>;
|
||||
export type AgentState =
|
||||
'idle' | 'planning' | 'deciding' | 'executing' | 'paused' | 'succeeded' | 'failed' | 'cancelled';
|
||||
export const AGENT_LIMITS = {
|
||||
maxRecoveries: 2,
|
||||
maxReplans: 2,
|
||||
maxJev: 60,
|
||||
maxLlm: 3,
|
||||
wallMs: 1_200_000,
|
||||
requestMs: 65_000,
|
||||
} as const;
|
||||
type Limits = { [K in keyof typeof AGENT_LIMITS]: number };
|
||||
type Outcome =
|
||||
| { kind: 'plan'; value: unknown; remaining: Skill[] }
|
||||
| { kind: 'jev'; value: unknown; candidates: (Skill | 'stop')[] }
|
||||
| { kind: 'error'; error: string };
|
||||
|
||||
/** The sole driver of already-validated local skills. Promises only populate a mailbox;
|
||||
* authorize/recover/servo/hold operations happen in the session's synchronous write window. */
|
||||
export class AgentTaskController implements AgentDriver {
|
||||
state: AgentState = 'idle';
|
||||
error = '';
|
||||
plan?: Plan;
|
||||
lastDecision?: JevDecision;
|
||||
readonly results: SkillResult[] = [];
|
||||
readonly failures: { skill: Skill; code: string; sequence: number }[] = [];
|
||||
readonly recoveries: Partial<Record<Skill, number>> = {};
|
||||
llmCalls = 0;
|
||||
jevCalls = 0;
|
||||
replans = 0;
|
||||
private readonly gate = new RequestGate();
|
||||
private readonly limits: Limits;
|
||||
private readonly now: () => number;
|
||||
private readonly began: number;
|
||||
private finishedAt?: number;
|
||||
private readonly runId: string;
|
||||
private readonly sceneRevision: number;
|
||||
private sequence = 0;
|
||||
private requestId = 0;
|
||||
private planRevision = 0;
|
||||
private epoch = 0;
|
||||
private inflight = false;
|
||||
private outcome?: Outcome;
|
||||
private needPlan = true;
|
||||
private recoveryPlan = false;
|
||||
private paused = false;
|
||||
private single = false;
|
||||
private disposed = false;
|
||||
private failureRecorded = false;
|
||||
constructor(
|
||||
readonly skills: SkillExecutor,
|
||||
readonly provider: DecisionProvider,
|
||||
private readonly instruction: string,
|
||||
options: {
|
||||
runId?: string;
|
||||
sceneRevision?: number;
|
||||
now?: () => number;
|
||||
limits?: Partial<Limits>;
|
||||
} = {},
|
||||
) {
|
||||
this.runId = options.runId ?? crypto.randomUUID();
|
||||
this.sceneRevision = options.sceneRevision ?? 0;
|
||||
this.now = options.now ?? (() => performance.now());
|
||||
this.began = this.now();
|
||||
this.limits = { ...AGENT_LIMITS, ...options.limits };
|
||||
for (const [key, value] of Object.entries(this.limits))
|
||||
if (
|
||||
!Number.isInteger(value) ||
|
||||
value < (['maxRecoveries', 'maxReplans'].includes(key) ? 0 : 1) ||
|
||||
value > AGENT_LIMITS[key as keyof Limits]
|
||||
)
|
||||
throw new Error('智能任务预算非法或超过硬上限');
|
||||
if (!instruction.trim() || Array.from(instruction).length > 2000)
|
||||
throw new Error('任务指令为空或过长');
|
||||
}
|
||||
get terminal(): boolean {
|
||||
return ['succeeded', 'failed', 'cancelled'].includes(this.state);
|
||||
}
|
||||
snapshot() {
|
||||
return {
|
||||
version: AGENT_VERSION,
|
||||
goal: [...this.skills.goal],
|
||||
mode: this.provider.mode,
|
||||
wallMs: (this.finishedAt ?? this.now()) - this.began,
|
||||
state: this.state,
|
||||
error: this.error,
|
||||
phase: this.skills.phase,
|
||||
candidate: this.skills.candidate,
|
||||
plan: this.plan,
|
||||
decision: this.lastDecision,
|
||||
llmCalls: this.llmCalls,
|
||||
jevCalls: this.jevCalls,
|
||||
replans: this.replans,
|
||||
recoveries: { ...this.recoveries },
|
||||
sample: this.skills.sample,
|
||||
secure: this.skills.evaluator.secure,
|
||||
transported: this.skills.evaluator.transported,
|
||||
succeeded: this.skills.evaluator.succeeded,
|
||||
runId: this.runId,
|
||||
sceneRevision: this.sceneRevision,
|
||||
sequence: this.sequence,
|
||||
};
|
||||
}
|
||||
/** Task pause keeps the owner and freezes physics. Global session pause cancels the task. */
|
||||
pause(): void {
|
||||
if (this.terminal || this.disposed) return;
|
||||
this.paused = true;
|
||||
this.invalidate();
|
||||
this.state = 'paused';
|
||||
}
|
||||
resume(singleSkill = false): void {
|
||||
if (this.terminal || this.disposed) return;
|
||||
this.paused = false;
|
||||
this.single = singleSkill;
|
||||
this.state = 'idle';
|
||||
}
|
||||
private invalidate(): void {
|
||||
this.epoch++;
|
||||
this.gate.invalidate('任务版本已失效');
|
||||
this.inflight = false;
|
||||
this.outcome = undefined;
|
||||
}
|
||||
private stamp(): RequestStamp {
|
||||
return {
|
||||
runId: this.runId,
|
||||
sceneRevision: this.sceneRevision,
|
||||
sequence: this.sequence,
|
||||
planRevision: this.planRevision,
|
||||
requestId: `${this.runId}:${++this.requestId}`,
|
||||
};
|
||||
}
|
||||
private finish(state: 'failed' | 'succeeded', code = ''): void {
|
||||
this.invalidate();
|
||||
this.state = state;
|
||||
this.finishedAt = this.now();
|
||||
this.error = code;
|
||||
this.skills.hold();
|
||||
}
|
||||
private recordFailure(): void {
|
||||
if (this.failureRecorded) return;
|
||||
this.failureRecorded = true;
|
||||
this.failures.push({
|
||||
skill: this.skills.phase,
|
||||
code: this.skills.error,
|
||||
sequence: this.sequence,
|
||||
});
|
||||
this.results.push({
|
||||
version: AGENT_VERSION,
|
||||
skill: this.skills.phase,
|
||||
status: 'failed',
|
||||
code: skillErrorCode(this.skills.error),
|
||||
evidenceSequence: this.sequence,
|
||||
});
|
||||
}
|
||||
private recoveryAllowed(): boolean {
|
||||
return (
|
||||
this.skills.canRecover() &&
|
||||
(this.recoveries[this.skills.phase] ?? 0) < this.limits.maxRecoveries
|
||||
);
|
||||
}
|
||||
private recover(): void {
|
||||
if (!this.recoveryAllowed()) throw new Error('恢复不安全或次数已耗尽');
|
||||
const phase = this.skills.phase;
|
||||
this.skills.recover();
|
||||
this.recoveries[phase] = (this.recoveries[phase] ?? 0) + 1;
|
||||
this.failureRecorded = false;
|
||||
}
|
||||
private replan(recover: boolean): void {
|
||||
if (this.replans >= this.limits.maxReplans) throw new Error('重规划预算耗尽');
|
||||
if (recover && !this.recoveryAllowed()) throw new Error('恢复不安全或次数已耗尽');
|
||||
this.replans++;
|
||||
this.recoveryPlan = recover;
|
||||
this.needPlan = true;
|
||||
}
|
||||
private requestPlan(): void {
|
||||
if (this.llmCalls >= this.limits.maxLlm) throw new Error('LLM 请求预算耗尽');
|
||||
this.llmCalls++;
|
||||
this.planRevision++;
|
||||
const remaining = this.recoveryPlan
|
||||
? [...SKILLS]
|
||||
: SKILLS.slice(SKILLS.indexOf(this.skills.candidate));
|
||||
const stamp = this.stamp();
|
||||
const obs = observation(
|
||||
this.skills.sample,
|
||||
this.skills.evaluator,
|
||||
this.skills.goal,
|
||||
stamp,
|
||||
this.skills.phase,
|
||||
);
|
||||
const epoch = this.epoch;
|
||||
this.state = 'planning';
|
||||
this.inflight = true;
|
||||
void this.gate
|
||||
.request(stamp, this.limits.requestMs, (signal) =>
|
||||
this.provider.plan(obs, this.instruction, remaining, signal),
|
||||
)
|
||||
.then((value) => this.deliver(epoch, { kind: 'plan', value, remaining }))
|
||||
.catch((error: unknown) =>
|
||||
this.deliver(epoch, {
|
||||
kind: 'error',
|
||||
error: error instanceof Error ? error.message : '模型规划失败',
|
||||
}),
|
||||
);
|
||||
}
|
||||
private requestJev(): void {
|
||||
if (this.jevCalls >= this.limits.maxJev) throw new Error('Jev 请求预算耗尽');
|
||||
this.jevCalls++;
|
||||
const failed = this.skills.status === 'failed';
|
||||
const candidates: (Skill | 'stop')[] = [failed ? 'open' : this.skills.candidate, 'stop'];
|
||||
const stamp = this.stamp();
|
||||
const obs = observation(
|
||||
this.skills.sample,
|
||||
this.skills.evaluator,
|
||||
this.skills.goal,
|
||||
stamp,
|
||||
this.skills.phase,
|
||||
);
|
||||
const epoch = this.epoch;
|
||||
this.state = 'deciding';
|
||||
this.inflight = true;
|
||||
void this.gate
|
||||
.request(stamp, this.limits.requestMs, (signal) =>
|
||||
this.provider.decide(obs, candidates, failed ? this.skills.error : 'none', signal),
|
||||
)
|
||||
.then((value) => this.deliver(epoch, { kind: 'jev', value, candidates }))
|
||||
.catch((error: unknown) =>
|
||||
this.deliver(epoch, {
|
||||
kind: 'error',
|
||||
error: error instanceof Error ? error.message : 'Jev 判定失败',
|
||||
}),
|
||||
);
|
||||
}
|
||||
private deliver(epoch: number, outcome: Outcome): void {
|
||||
if (this.epoch !== epoch || this.terminal || this.disposed || this.paused) return;
|
||||
this.inflight = false;
|
||||
this.outcome = outcome; // No writes, physical steps or re-entrant ownership changes here.
|
||||
}
|
||||
private apply(outcome: Outcome): void {
|
||||
if (outcome.kind === 'error') throw new Error(outcome.error);
|
||||
if (outcome.kind === 'plan') {
|
||||
this.plan = validatePlan(outcome.value, outcome.remaining);
|
||||
this.needPlan = false;
|
||||
if (this.recoveryPlan) {
|
||||
this.recover();
|
||||
this.recoveryPlan = false;
|
||||
}
|
||||
return;
|
||||
}
|
||||
const value = validateJevDecision(outcome.value, outcome.candidates);
|
||||
this.lastDecision = value;
|
||||
if (value.choice === 'stop' || value.recovery === 'stop') throw new Error('Jev 请求安全停止');
|
||||
if (value.grasp === 'secure' && !this.skills.evaluator.secure)
|
||||
throw new Error('Jev 抓持判断与本地物理证据矛盾');
|
||||
const failed = this.skills.status === 'failed';
|
||||
if (value.recovery === 'replan') {
|
||||
this.replan(failed);
|
||||
return;
|
||||
}
|
||||
if (failed) {
|
||||
if (value.recovery !== 'retry' || value.choice !== 'open')
|
||||
throw new Error('故障没有合法恢复选择');
|
||||
const policy = this.plan?.steps.find((s) => s.skill === this.skills.phase)?.onFailure;
|
||||
if (policy === 'replan') {
|
||||
this.replan(true);
|
||||
return;
|
||||
}
|
||||
if (policy !== 'retry') throw new Error('当前计划不允许直接重试');
|
||||
this.recover();
|
||||
} else if (value.recovery !== 'continue') throw new Error('无故障时不能重复技能');
|
||||
if (
|
||||
['carry', 'stop-base'].includes(value.choice) &&
|
||||
(value.grasp !== 'secure' || !this.skills.evaluator.secure)
|
||||
)
|
||||
throw new Error('搬运前未取得一致抓持证据');
|
||||
this.skills.authorize(value.choice);
|
||||
this.state = 'executing';
|
||||
}
|
||||
beforeStep(): boolean {
|
||||
if (this.disposed) return false;
|
||||
try {
|
||||
if (this.terminal) {
|
||||
this.skills.hold();
|
||||
return false;
|
||||
}
|
||||
if (this.now() - this.began > this.limits.wallMs) throw new Error('任务墙钟期限耗尽');
|
||||
if (this.paused) {
|
||||
this.skills.hold();
|
||||
return false;
|
||||
}
|
||||
if (this.outcome) {
|
||||
const outcome = this.outcome;
|
||||
this.outcome = undefined;
|
||||
this.apply(outcome);
|
||||
}
|
||||
if (this.needPlan && !this.inflight) this.requestPlan();
|
||||
if (this.inflight) {
|
||||
this.skills.hold();
|
||||
return false;
|
||||
}
|
||||
if (this.skills.status === 'failed') {
|
||||
this.recordFailure();
|
||||
if (
|
||||
!this.recoveryAllowed() ||
|
||||
this.plan?.steps.find((s) => s.skill === this.skills.phase)?.onFailure === 'stop'
|
||||
)
|
||||
throw new Error(this.skills.error || '技能失败');
|
||||
this.requestJev();
|
||||
this.skills.hold();
|
||||
return false;
|
||||
}
|
||||
if (this.skills.status === 'succeeded') {
|
||||
if (!this.skills.evaluator.succeeded) throw new Error('缺少独立物理成功证据');
|
||||
this.finish('succeeded');
|
||||
return false;
|
||||
}
|
||||
if (this.skills.waiting) {
|
||||
this.requestJev();
|
||||
this.skills.hold();
|
||||
return false;
|
||||
}
|
||||
this.state = 'executing';
|
||||
this.skills.beforeStep();
|
||||
return this.skills.status === 'running';
|
||||
} catch (error) {
|
||||
this.finish('failed', error instanceof Error ? error.message : '智能任务执行失败');
|
||||
return false;
|
||||
}
|
||||
}
|
||||
afterStep(dt: number): void {
|
||||
if (this.disposed || this.terminal || this.paused) return;
|
||||
this.sequence++;
|
||||
this.skills.afterStep(dt);
|
||||
if (this.skills.status === 'failed') this.recordFailure();
|
||||
else if (this.skills.waiting || this.skills.status === 'succeeded') {
|
||||
if (this.skills.status === 'succeeded' && !this.skills.evaluator.succeeded) {
|
||||
this.results.push({
|
||||
version: AGENT_VERSION,
|
||||
skill: this.skills.phase,
|
||||
status: 'failed',
|
||||
code: 'placement_unverified',
|
||||
evidenceSequence: this.sequence,
|
||||
});
|
||||
this.finish('failed', '缺少独立物理成功证据');
|
||||
return;
|
||||
}
|
||||
this.results.push({
|
||||
version: AGENT_VERSION,
|
||||
skill: this.skills.phase,
|
||||
status: 'completed',
|
||||
code: 'none',
|
||||
evidenceSequence: this.sequence,
|
||||
});
|
||||
if (this.skills.status === 'succeeded') {
|
||||
this.finish(
|
||||
this.skills.evaluator.succeeded ? 'succeeded' : 'failed',
|
||||
this.skills.evaluator.succeeded ? '' : '缺少独立物理成功证据',
|
||||
);
|
||||
} else if (this.single) this.pause();
|
||||
}
|
||||
}
|
||||
hold(): void {
|
||||
if (!this.disposed) this.skills.hold();
|
||||
}
|
||||
cancel(reason: string): void {
|
||||
this.invalidate();
|
||||
if (!this.terminal) {
|
||||
this.state = 'cancelled';
|
||||
this.finishedAt = this.now();
|
||||
this.error = reason;
|
||||
}
|
||||
}
|
||||
dispose(): void {
|
||||
if (this.disposed) return;
|
||||
this.cancel('任务已释放');
|
||||
this.disposed = true;
|
||||
this.skills.dispose();
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,418 @@
|
||||
import { useEffect, useRef, useState } from 'react';
|
||||
import { Button } from '../../components/ui';
|
||||
import { WEBSITE_MODE } from '../../app/websiteConfig';
|
||||
import { downloadBytes } from '../../project/cachedFiles';
|
||||
import type { SimulationSession } from '../../simulation/SimulationSession';
|
||||
import { AgentTaskController } from './AgentTaskController';
|
||||
import { DecisionClient } from './DecisionClient';
|
||||
import { DeterministicBaseline } from './DeterministicBaseline';
|
||||
import { MockDecisionProvider } from './MockDecisionProvider';
|
||||
import { ModelConnectionPanel } from './ModelConnectionPanel';
|
||||
import {
|
||||
PICK_PLACE,
|
||||
TASK_GOAL_LIMITS,
|
||||
seedPositions,
|
||||
type PickPlaceTaskConfig,
|
||||
} from './PickPlaceScene';
|
||||
|
||||
export interface AgentWorkbenchBridge {
|
||||
loadScene(config: PickPlaceTaskConfig): Promise<SimulationSession>;
|
||||
getSession(): SimulationSession | undefined;
|
||||
isTaskScene(): boolean;
|
||||
publish(): void;
|
||||
}
|
||||
const states = {
|
||||
idle: '准备',
|
||||
planning: '等待 LLM 规划',
|
||||
deciding: '等待 Jev 判定',
|
||||
executing: '执行技能',
|
||||
paused: '任务已暂停',
|
||||
succeeded: '物理验收成功',
|
||||
failed: '失败 / 安全停止',
|
||||
cancelled: '已取消',
|
||||
};
|
||||
export function AgentTaskPanel({
|
||||
bridge,
|
||||
loading,
|
||||
}: {
|
||||
bridge: AgentWorkbenchBridge;
|
||||
loading: boolean;
|
||||
}) {
|
||||
const liveBridge = useRef(bridge);
|
||||
useEffect(() => {
|
||||
liveBridge.current = bridge;
|
||||
}, [bridge]);
|
||||
const task = useRef<AgentTaskController | undefined>(undefined);
|
||||
const ownSession = useRef<SimulationSession | undefined>(undefined);
|
||||
const [snapshot, setSnapshot] = useState<ReturnType<AgentTaskController['snapshot']>>();
|
||||
const [client, setClient] = useState<DecisionClient>();
|
||||
const [settingsOpen, setSettingsOpen] = useState(false);
|
||||
const [mode, setMode] = useState<'mock' | 'api'>('mock');
|
||||
const [instruction, setInstruction] = useState(
|
||||
'把方块抓起,使用底盘搬运到指定支撑台,放下、松开并撤离。',
|
||||
);
|
||||
const [goal, setGoal] = useState<number[]>([...PICK_PLACE.goal.position]);
|
||||
const [busy, setBusy] = useState(false);
|
||||
const [message, setMessage] = useState('需要 A / lekiwi-v1 资产;不会自动下载或替换机器人。');
|
||||
const epoch = useRef(0);
|
||||
const sceneDirty = useRef(true);
|
||||
const alive = useRef(true);
|
||||
function publish() {
|
||||
const current = task.current;
|
||||
if (current) setSnapshot(current.snapshot());
|
||||
liveBridge.current.publish();
|
||||
}
|
||||
function stop(reason = '用户停止任务') {
|
||||
epoch.current++;
|
||||
const current = task.current,
|
||||
session = ownSession.current;
|
||||
if (current) current.cancel(reason);
|
||||
if (
|
||||
session &&
|
||||
session === liveBridge.current.getSession() &&
|
||||
session.snapshot().controlOwner === 'agent'
|
||||
) {
|
||||
session.stopAgent(reason);
|
||||
session.setPaused(true);
|
||||
}
|
||||
if (alive.current) publish();
|
||||
}
|
||||
useEffect(() => {
|
||||
alive.current = true;
|
||||
const timer = setInterval(() => {
|
||||
const current = task.current,
|
||||
session = ownSession.current;
|
||||
if (!current) return;
|
||||
setSnapshot(current.snapshot());
|
||||
if (
|
||||
current.terminal &&
|
||||
session === liveBridge.current.getSession() &&
|
||||
session?.snapshot().controlOwner === 'agent'
|
||||
) {
|
||||
session.stopAgent('任务已结束');
|
||||
session.setPaused(true);
|
||||
liveBridge.current.publish();
|
||||
}
|
||||
}, 250);
|
||||
const invalidatePending = () => {
|
||||
epoch.current++;
|
||||
};
|
||||
return () => {
|
||||
alive.current = false;
|
||||
invalidatePending();
|
||||
clearInterval(timer);
|
||||
const session = ownSession.current;
|
||||
if (
|
||||
session === liveBridge.current.getSession() &&
|
||||
session?.snapshot().controlOwner === 'agent'
|
||||
) {
|
||||
session.stopAgent('任务面板已卸载');
|
||||
session.setPaused(true);
|
||||
}
|
||||
task.current?.dispose();
|
||||
};
|
||||
}, []);
|
||||
async function load() {
|
||||
stop('任务场景重新加载');
|
||||
const generation = epoch.current;
|
||||
setBusy(true);
|
||||
try {
|
||||
seedPositions(0, goal);
|
||||
const session = await liveBridge.current.loadScene({ seed: 0, goal: [...goal] });
|
||||
if (generation !== epoch.current || !alive.current) return;
|
||||
ownSession.current = session;
|
||||
task.current = undefined;
|
||||
sceneDirty.current = false;
|
||||
setSnapshot(undefined);
|
||||
setMessage('A 抓放场景已加载并暂停;用户地图草稿保留,但不用于本次空旷平地任务。');
|
||||
} catch (error) {
|
||||
if (generation === epoch.current && alive.current)
|
||||
setMessage(error instanceof Error ? error.message : '加载失败');
|
||||
} finally {
|
||||
if (alive.current) setBusy(false);
|
||||
}
|
||||
}
|
||||
async function start(single = false) {
|
||||
if (busy || loading) return;
|
||||
if (mode === 'api' && !client) {
|
||||
setMessage(
|
||||
WEBSITE_MODE
|
||||
? '请在模型设置中保存 LLM 与 Jev 连接;不会自动转为演示。'
|
||||
: '真实模式必须先连接本机模型服务;不会转为 mock。',
|
||||
);
|
||||
return;
|
||||
}
|
||||
if (task.current) {
|
||||
stop('开始新回合');
|
||||
task.current.dispose();
|
||||
task.current = undefined;
|
||||
sceneDirty.current = true;
|
||||
}
|
||||
setSnapshot(undefined);
|
||||
const generation = ++epoch.current;
|
||||
setBusy(true);
|
||||
try {
|
||||
seedPositions(0, goal);
|
||||
let session = liveBridge.current.getSession();
|
||||
if (sceneDirty.current || task.current || !liveBridge.current.isTaskScene())
|
||||
session = await liveBridge.current.loadScene({ seed: 0, goal: [...goal] });
|
||||
if (generation !== epoch.current || !alive.current) return;
|
||||
if (!session) throw new Error('尚未加载任务场景');
|
||||
const provider = mode === 'api' ? client! : new MockDecisionProvider();
|
||||
session.startAgent((write) => {
|
||||
const skills = new DeterministicBaseline(session!, 0, write, true, goal);
|
||||
try {
|
||||
const controller = new AgentTaskController(skills, provider, instruction);
|
||||
if (single) controller.resume(true);
|
||||
task.current = controller;
|
||||
return controller;
|
||||
} catch (error) {
|
||||
skills.dispose();
|
||||
throw error;
|
||||
}
|
||||
});
|
||||
ownSession.current = session;
|
||||
sceneDirty.current = false;
|
||||
session.setPaused(false);
|
||||
setMessage(
|
||||
mode === 'mock'
|
||||
? '显式 mock 决策;执行与验收为真实 MuJoCo 物理。'
|
||||
: '真实模型模式,可能计费;失败不降级规则。',
|
||||
);
|
||||
publish();
|
||||
} catch (error) {
|
||||
if (generation === epoch.current && alive.current)
|
||||
setMessage(error instanceof Error ? error.message : '启动失败');
|
||||
} finally {
|
||||
if (alive.current) setBusy(false);
|
||||
}
|
||||
}
|
||||
function resume(single = false) {
|
||||
const current = task.current;
|
||||
if (current && !current.terminal && ownSession.current === liveBridge.current.getSession()) {
|
||||
current.resume(single);
|
||||
publish();
|
||||
} else void start(single);
|
||||
}
|
||||
function changeGoal(axis: number, value: number) {
|
||||
stop('目标编辑使旧计划失效');
|
||||
sceneDirty.current = true;
|
||||
setGoal((previous) => previous.map((v, i) => (i === axis ? value : v)));
|
||||
// Any edit requires a new scene before starting, even if values are subsequently restored.
|
||||
task.current?.dispose();
|
||||
setMessage('目标已编辑;开始时会重新加载场景与方块,旧计划不可复用。');
|
||||
}
|
||||
const running = Boolean(
|
||||
snapshot && !['succeeded', 'failed', 'cancelled', 'paused'].includes(snapshot.state),
|
||||
);
|
||||
const locked = busy || loading;
|
||||
const receipts = client?.receipts.filter((r) => r.stamp.runId === snapshot?.runId) ?? [];
|
||||
const costs = receipts.filter((r) => r.usage.cost !== undefined);
|
||||
const tokens = receipts.filter((r) =>
|
||||
['total_tokens', 'input_tokens', 'prompt_tokens'].some((key) => r.usage[key] !== undefined),
|
||||
);
|
||||
return (
|
||||
<div className="space-y-3 text-xs" aria-label="LeKiwi 智能抓放任务">
|
||||
<p>
|
||||
仅仿真 · MuJoCo 结构化真值(非视觉) · A /
|
||||
lekiwi-v1。空旷平地单方块,不承诺障碍导航或任意目标泛化。
|
||||
</p>
|
||||
<p>任务模式:{mode === 'mock' ? 'MOCK(不调用模型 API)' : '真实 LLM + Jev(可能计费)'}</p>
|
||||
<label className="block">
|
||||
决策模式
|
||||
<select
|
||||
aria-label="智能任务决策模式"
|
||||
disabled={locked}
|
||||
value={mode}
|
||||
onChange={(e) => {
|
||||
stop('决策模式改变');
|
||||
setMode(e.target.value as 'mock' | 'api');
|
||||
}}
|
||||
className="w-full bg-surface p-1"
|
||||
>
|
||||
<option value="mock">显式 mock LLM + Jev</option>
|
||||
<option value="api">真实模型服务(无自动回退)</option>
|
||||
</select>
|
||||
</label>
|
||||
<label className="block">
|
||||
中文任务指令
|
||||
<textarea
|
||||
className="w-full rounded border border-border bg-surface p-2"
|
||||
rows={3}
|
||||
maxLength={2000}
|
||||
value={instruction}
|
||||
disabled={locked || running}
|
||||
onChange={(e) => {
|
||||
stop('任务指令修改');
|
||||
setInstruction(e.target.value);
|
||||
}}
|
||||
/>
|
||||
</label>
|
||||
<details open={!WEBSITE_MODE}>
|
||||
<summary>高级:目标位置与实验范围</summary>
|
||||
<p>语言计划必须遵循安全技能顺序;放置位置以以下场景坐标为准。</p>
|
||||
<fieldset disabled={locked || running} className="space-y-1">
|
||||
<legend>目标位置(米,暂停后编辑;重新载入回合)</legend>
|
||||
{[0, 1].map((axis) => (
|
||||
<label key={axis} className="block">
|
||||
目标 {axis ? 'Y' : 'X'}
|
||||
<input
|
||||
className="ml-2 w-24 bg-surface p-1"
|
||||
type="number"
|
||||
min={TASK_GOAL_LIMITS.min[axis]}
|
||||
max={TASK_GOAL_LIMITS.max[axis]}
|
||||
step="0.001"
|
||||
value={Number.isNaN(goal[axis]) ? '' : goal[axis]}
|
||||
onChange={(e) => changeGoal(axis, e.target.valueAsNumber)}
|
||||
/>
|
||||
</label>
|
||||
))}
|
||||
<p>Z 固定 {goal[2]};坐标编辑为受限实验,默认冻结场景已验收。</p>
|
||||
</fieldset>
|
||||
</details>
|
||||
<div className="flex flex-wrap gap-2">
|
||||
<Button disabled={locked || running} onClick={() => void start()}>
|
||||
开始任务({mode === 'api' ? '真实 API' : 'mock'})
|
||||
</Button>
|
||||
<Button
|
||||
disabled={locked || !running}
|
||||
onClick={() => {
|
||||
task.current?.pause();
|
||||
publish();
|
||||
}}
|
||||
>
|
||||
暂停任务
|
||||
</Button>
|
||||
<Button disabled={locked || snapshot?.state !== 'paused'} onClick={() => resume()}>
|
||||
继续任务
|
||||
</Button>
|
||||
<Button onClick={() => stop()}>停止任务</Button>
|
||||
</div>
|
||||
<details open={!WEBSITE_MODE}>
|
||||
<summary>高级操作与证据导出</summary>
|
||||
<div className="mt-2 flex flex-wrap gap-2">
|
||||
<Button disabled={locked || running} onClick={() => void load()}>
|
||||
加载 A 任务场景
|
||||
</Button>
|
||||
<Button disabled={locked || running} onClick={() => resume(true)}>
|
||||
单技能步进
|
||||
</Button>
|
||||
<Button disabled={locked} onClick={() => void load()}>
|
||||
重置任务场景
|
||||
</Button>
|
||||
<Button
|
||||
disabled={!snapshot}
|
||||
onClick={() => {
|
||||
const current = task.current;
|
||||
if (!current) return;
|
||||
downloadBytes(
|
||||
new TextEncoder().encode(
|
||||
JSON.stringify(
|
||||
{
|
||||
...current.snapshot(),
|
||||
results: current.results,
|
||||
failures: current.failures,
|
||||
trace: (current.skills as DeterministicBaseline).trace,
|
||||
receipts,
|
||||
},
|
||||
null,
|
||||
2,
|
||||
),
|
||||
),
|
||||
`lekiwi-agent-${current.snapshot().runId}.json`,
|
||||
);
|
||||
}}
|
||||
>
|
||||
导出任务证据
|
||||
</Button>
|
||||
</div>
|
||||
</details>
|
||||
<p role="status">{message}</p>
|
||||
{snapshot && (
|
||||
<div data-testid="agent-task-state" className="space-y-1">
|
||||
<p>
|
||||
状态:{states[snapshot.state]} · 当前技能:{snapshot.phase}
|
||||
</p>
|
||||
{snapshot.error && (
|
||||
<p role="alert" className="text-warning">
|
||||
{snapshot.error}
|
||||
</p>
|
||||
)}
|
||||
<details open={!WEBSITE_MODE}>
|
||||
<summary>运行详情与物理证据</summary>
|
||||
<p>
|
||||
规划 {snapshot.llmCalls}/3 · Jev {snapshot.jevCalls}/60 · 重规划 {snapshot.replans}/2
|
||||
</p>
|
||||
<p>
|
||||
墙钟 {(snapshot.wallMs / 1000).toFixed(1)} s · 仿真 {snapshot.sample.time.toFixed(2)}{' '}
|
||||
s · 安全抓持 {snapshot.secure ? '是' : '否'} · 有效搬运{' '}
|
||||
{snapshot.transported.toFixed(3)} m
|
||||
</p>
|
||||
<p>
|
||||
双指力 {snapshot.sample.fingerForces.map((v) => v.toFixed(2)).join(' / ')} N ·
|
||||
目标支撑 {snapshot.sample.onGoalSupport ? '是' : '否'}
|
||||
</p>
|
||||
<p>
|
||||
Jev:
|
||||
{snapshot.decision
|
||||
? `${snapshot.decision.choice} / ${snapshot.decision.grasp} / ${snapshot.decision.diagnosis} / ${snapshot.decision.recovery}`
|
||||
: '尚未判定'}
|
||||
</p>
|
||||
<p>
|
||||
已返回费用:
|
||||
{costs.length
|
||||
? `$${costs.reduce((sum, r) => sum + r.usage.cost, 0).toFixed(6)}(部分服务报告;非完整账单)`
|
||||
: '未报告,不估算为零'}
|
||||
</p>
|
||||
<p>
|
||||
已报告 Token:
|
||||
{tokens.length
|
||||
? tokens.reduce(
|
||||
(sum, r) =>
|
||||
sum +
|
||||
(r.usage.total_tokens ??
|
||||
(r.usage.input_tokens ?? r.usage.prompt_tokens ?? 0) +
|
||||
(r.usage.output_tokens ?? r.usage.completion_tokens ?? 0)),
|
||||
0,
|
||||
)
|
||||
: '未报告'}{' '}
|
||||
· 最近调用耗时{' '}
|
||||
{receipts.at(-1)?.elapsedMs === undefined
|
||||
? '未报告'
|
||||
: `${receipts.at(-1)!.elapsedMs!.toFixed(0)} ms`}
|
||||
</p>
|
||||
<p>最近模型:{receipts.at(-1)?.model ?? (mode === 'mock' ? 'mock' : '尚无有效响应')}</p>
|
||||
</details>
|
||||
{snapshot.plan && (
|
||||
<details open>
|
||||
<summary>LLM 受约束计划</summary>
|
||||
<p>{snapshot.plan.summary}</p>
|
||||
<ol className="list-inside list-decimal">
|
||||
{snapshot.plan.steps.map((s) => (
|
||||
<li key={s.skill}>
|
||||
{s.skill} · 失败时 {s.onFailure}
|
||||
</li>
|
||||
))}
|
||||
</ol>
|
||||
</details>
|
||||
)}
|
||||
</div>
|
||||
)}
|
||||
<p>
|
||||
全局暂停/隐藏/重载会取消任务;任务级暂停可继续。重置前请导出当前证据。
|
||||
{!WEBSITE_MODE && 'RL/外控与任务互斥,后台训练作业不会被停止。'}
|
||||
</p>
|
||||
<details onToggle={(event) => setSettingsOpen(event.currentTarget.open)}>
|
||||
<summary>
|
||||
{WEBSITE_MODE ? '模型设置 · API_KEY / ChatGPT 订阅' : '模型连接与官方订阅登录'}
|
||||
</summary>
|
||||
{(!WEBSITE_MODE || settingsOpen) && (
|
||||
<ModelConnectionPanel
|
||||
onClient={setClient}
|
||||
onInvalidate={() => stop('模型配置或账号操作使任务失效')}
|
||||
/>
|
||||
)}
|
||||
</details>
|
||||
</div>
|
||||
);
|
||||
}
|
||||
@@ -0,0 +1,116 @@
|
||||
import { describe, expect, it, vi } from 'vitest';
|
||||
import { DecisionClient } from './DecisionClient';
|
||||
import { MockDecisionProvider } from './MockDecisionProvider';
|
||||
import { PickPlaceEvaluator, type PhysicalSample } from './PhysicalEvidence';
|
||||
import { observation, SKILLS } from './protocol';
|
||||
|
||||
const sample: PhysicalSample = {
|
||||
time: 0,
|
||||
base: [0, 0, 0.05],
|
||||
baseSpeed: 0,
|
||||
yaw: 0,
|
||||
tilt: 0,
|
||||
joints: [0, 0, 0, 0, 0],
|
||||
peakJointSpeed: 0,
|
||||
opening: 1,
|
||||
tcp: [0.257, 0.015, 0.2],
|
||||
object: [0.257, 0.015, 0.128],
|
||||
objectSpeed: 0,
|
||||
fingerForces: [0, 0],
|
||||
supported: true,
|
||||
onGoalSupport: false,
|
||||
};
|
||||
const stamp = {
|
||||
runId: 'run',
|
||||
sceneRevision: 1,
|
||||
sequence: 0,
|
||||
planRevision: 1,
|
||||
requestId: 'request',
|
||||
};
|
||||
const obs = observation(
|
||||
sample,
|
||||
new PickPlaceEvaluator(sample.object, [0.257, 0.615, 0.128]),
|
||||
[0.257, 0.615, 0.128],
|
||||
stamp,
|
||||
'open',
|
||||
);
|
||||
describe('DecisionClient', () => {
|
||||
it('只连接本机桥;令牌仅 Header,usage 仅数值白名单,不持久化', async () => {
|
||||
const result = await new MockDecisionProvider().plan(
|
||||
obs,
|
||||
'搬运',
|
||||
SKILLS,
|
||||
new AbortController().signal,
|
||||
);
|
||||
const fetcher = vi.fn<typeof fetch>(async () =>
|
||||
Response.json({
|
||||
...result,
|
||||
model: 'fixture',
|
||||
provider: 'responses',
|
||||
usage: { input_tokens: 10, cost: 0.01, raw: 'discard' },
|
||||
}),
|
||||
);
|
||||
const persist = vi.spyOn(Storage.prototype, 'setItem');
|
||||
const client = new DecisionClient('http://127.0.0.1:8768', 'service-token-fixture', fetcher);
|
||||
expect((await client.plan(obs, '搬运', SKILLS, new AbortController().signal)).stamp).toEqual(
|
||||
stamp,
|
||||
);
|
||||
const [url, init] = fetcher.mock.calls[0];
|
||||
expect(url).toContain('/api/decision/v1/plan');
|
||||
expect(init?.headers).toMatchObject({ Authorization: 'Bearer service-token-fixture' });
|
||||
expect(init?.body).not.toContain('service-token-fixture');
|
||||
expect(init?.redirect).toBe('error');
|
||||
expect(client.receipts[0].usage).toEqual({ input_tokens: 10, cost: 0.01 });
|
||||
expect(persist).not.toHaveBeenCalled();
|
||||
persist.mockRestore();
|
||||
});
|
||||
it('浏览器原生 fetch 必须使用全局 receiver,不能绑定到客户端实例', async () => {
|
||||
const fetcher: typeof fetch = async function (this: unknown) {
|
||||
if (this !== globalThis) throw new TypeError('Illegal invocation');
|
||||
return Response.json({ ok: true });
|
||||
};
|
||||
const client = new DecisionClient('http://localhost:8768', 'fixture', fetcher);
|
||||
expect(await client.call('/status')).toEqual({ ok: true });
|
||||
});
|
||||
it('拒绝非本机地址、用户名、路径、query 等令牌外泄入口', () => {
|
||||
for (const url of [
|
||||
'https://example.com',
|
||||
'http://localhost.evil.test',
|
||||
'http://x:y@localhost',
|
||||
'http://localhost/path',
|
||||
'http://localhost?secret=value',
|
||||
])
|
||||
expect(() => new DecisionClient(url, 'fixture')).toThrow();
|
||||
});
|
||||
it('abort 同时取消 HTTP 和服务端指定请求', async () => {
|
||||
const fetcher = vi.fn<typeof fetch>(async (url, init) => {
|
||||
if (String(url).endsWith('/cancel')) return Response.json({ cancelled: true });
|
||||
return await new Promise<Response>((_, reject) =>
|
||||
init?.signal?.addEventListener('abort', () => reject(new Error('aborted'))),
|
||||
);
|
||||
});
|
||||
const client = new DecisionClient('http://localhost:8768', 'fixture', fetcher);
|
||||
const controller = new AbortController();
|
||||
const pending = client.plan(obs, '搬运', SKILLS, controller.signal);
|
||||
controller.abort();
|
||||
await expect(pending).rejects.toThrow('aborted');
|
||||
expect(fetcher.mock.calls).toHaveLength(2);
|
||||
expect(JSON.parse(String(fetcher.mock.calls[1][1]?.body))).toEqual({
|
||||
runId: 'run',
|
||||
requestId: 'request',
|
||||
});
|
||||
});
|
||||
it('错误原文/非法 JSON/超长响应均不能成为决策或泄露到错误消息', async () => {
|
||||
for (const response of [
|
||||
new Response('not-json-private', { status: 502 }),
|
||||
Response.json({ error: 'sensitive raw error sk-example' }, { status: 401 }),
|
||||
new Response('x'.repeat(131073)),
|
||||
]) {
|
||||
const client = new DecisionClient('http://localhost:8768', 'fixture', async () => response);
|
||||
await expect(
|
||||
client.plan(obs, '搬运', SKILLS, new AbortController().signal),
|
||||
).rejects.not.toThrow(/private|sk-example/);
|
||||
expect(client.receipts).toHaveLength(0);
|
||||
}
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,227 @@
|
||||
import type { RequestStamp, Stamped } from './RequestGate';
|
||||
import {
|
||||
validateContract,
|
||||
validateJevDecision,
|
||||
validatePlan,
|
||||
type JevDecision,
|
||||
type Observation,
|
||||
type Plan,
|
||||
type Skill,
|
||||
} from './protocol';
|
||||
|
||||
export interface DecisionProvider {
|
||||
readonly mode: 'api' | 'mock';
|
||||
plan(
|
||||
obs: Observation,
|
||||
instruction: string,
|
||||
remaining: readonly Skill[],
|
||||
signal: AbortSignal,
|
||||
): Promise<Stamped<Plan>>;
|
||||
decide(
|
||||
obs: Observation,
|
||||
candidates: readonly (Skill | 'stop')[],
|
||||
failure: string,
|
||||
signal: AbortSignal,
|
||||
): Promise<Stamped<JevDecision>>;
|
||||
}
|
||||
export interface DecisionReceipt {
|
||||
kind: 'plan' | 'decide';
|
||||
stamp: RequestStamp;
|
||||
model: string;
|
||||
elapsedMs?: number;
|
||||
provider: string;
|
||||
usage: Record<string, number>;
|
||||
}
|
||||
const PREFIX = '/api/decision/v1';
|
||||
/** Local bridge only. Neither the service token nor API keys enter browser storage. */
|
||||
export class DecisionClient implements DecisionProvider {
|
||||
readonly mode = 'api' as const;
|
||||
readonly receipts: DecisionReceipt[] = [];
|
||||
readonly origin: string;
|
||||
readonly websiteMode: boolean;
|
||||
private csrfToken = '';
|
||||
private configVersion = 0;
|
||||
|
||||
static async website(signal?: AbortSignal, fetcher: typeof fetch = fetch) {
|
||||
const client = new DecisionClient(location.origin, '', fetcher, { website: true });
|
||||
const value = (await client.call('/session', 'POST', {}, signal)) as {
|
||||
csrfToken?: unknown;
|
||||
configVersion?: unknown;
|
||||
};
|
||||
if (typeof value.csrfToken !== 'string' || !Number.isSafeInteger(value.configVersion))
|
||||
throw new Error('网站会话响应无效');
|
||||
client.csrfToken = value.csrfToken;
|
||||
client.configVersion = value.configVersion as number;
|
||||
return client;
|
||||
}
|
||||
|
||||
constructor(
|
||||
address: string,
|
||||
private readonly token: string,
|
||||
private readonly fetcher: typeof fetch = fetch,
|
||||
options: { website?: boolean } = {},
|
||||
) {
|
||||
const url = new URL(address);
|
||||
this.websiteMode = options.website === true;
|
||||
if (
|
||||
!['http:', 'https:'].includes(url.protocol) ||
|
||||
(this.websiteMode
|
||||
? url.origin !== location.origin
|
||||
: !['localhost', '127.0.0.1', '[::1]'].includes(url.hostname)) ||
|
||||
url.username ||
|
||||
url.password ||
|
||||
url.search ||
|
||||
url.hash ||
|
||||
!['', '/'].includes(url.pathname)
|
||||
)
|
||||
throw new Error('模型服务必须是本机地址,不可包含凭据或路径');
|
||||
this.origin = url.origin;
|
||||
}
|
||||
async call(path: string, method = 'GET', body?: unknown, signal?: AbortSignal): Promise<unknown> {
|
||||
const response = await this.fetcher.call(globalThis, this.origin + PREFIX + path, {
|
||||
method,
|
||||
headers: {
|
||||
...(this.websiteMode
|
||||
? { 'X-CSRF-Token': this.csrfToken, 'X-Config-Version': String(this.configVersion) }
|
||||
: { Authorization: `Bearer ${this.token}` }),
|
||||
...(body === undefined ? {} : { 'Content-Type': 'application/json' }),
|
||||
},
|
||||
body: body === undefined ? undefined : JSON.stringify(body),
|
||||
signal: signal ?? AbortSignal.timeout(65000),
|
||||
redirect: 'error',
|
||||
cache: 'no-store',
|
||||
credentials: this.websiteMode ? 'same-origin' : 'omit',
|
||||
});
|
||||
if (
|
||||
this.websiteMode &&
|
||||
method !== 'GET' &&
|
||||
(path === '/configuration' || path.startsWith('/codex/'))
|
||||
) {
|
||||
const version = response.headers.get('X-Config-Version');
|
||||
if (version && /^\d+$/.test(version)) this.configVersion = Number(version);
|
||||
}
|
||||
const reader = response.body?.getReader();
|
||||
if (!reader) throw new Error('模型服务没有响应正文');
|
||||
const chunks: Uint8Array[] = [];
|
||||
let size = 0;
|
||||
try {
|
||||
for (;;) {
|
||||
const { done, value } = await reader.read();
|
||||
if (done) break;
|
||||
size += value.byteLength;
|
||||
if (size > 131072) throw new Error('模型服务响应过大');
|
||||
chunks.push(value);
|
||||
}
|
||||
} finally {
|
||||
await reader.cancel();
|
||||
reader.releaseLock();
|
||||
}
|
||||
const bytes = new Uint8Array(size);
|
||||
let offset = 0;
|
||||
for (const chunk of chunks) {
|
||||
bytes.set(chunk, offset);
|
||||
offset += chunk.length;
|
||||
}
|
||||
let value: unknown;
|
||||
try {
|
||||
value = JSON.parse(new TextDecoder('utf-8', { fatal: true }).decode(bytes));
|
||||
} catch {
|
||||
throw new Error('模型服务 JSON 非法');
|
||||
}
|
||||
if (!response.ok) {
|
||||
const code = (value as { error?: unknown })?.error;
|
||||
throw new Error(
|
||||
typeof code === 'string' && /^[a-z0-9_]{1,100}$/.test(code)
|
||||
? code
|
||||
: `模型服务 HTTP ${response.status}`,
|
||||
);
|
||||
}
|
||||
return value;
|
||||
}
|
||||
private async request(
|
||||
kind: 'plan' | 'decide',
|
||||
obs: Observation,
|
||||
body: object,
|
||||
signal: AbortSignal,
|
||||
) {
|
||||
const cancel = () => {
|
||||
void this.call(
|
||||
'/cancel',
|
||||
'POST',
|
||||
{ runId: obs.stamp.runId, requestId: obs.stamp.requestId },
|
||||
AbortSignal.timeout(2000),
|
||||
).catch(() => undefined);
|
||||
};
|
||||
if (signal.aborted) throw new Error('模型请求已取消');
|
||||
signal.addEventListener('abort', cancel, { once: true });
|
||||
try {
|
||||
const result = (await this.call(
|
||||
'/' + kind,
|
||||
'POST',
|
||||
{ observation: obs, ...body },
|
||||
signal,
|
||||
)) as {
|
||||
stamp: unknown;
|
||||
value: unknown;
|
||||
model?: unknown;
|
||||
provider?: unknown;
|
||||
usage?: unknown;
|
||||
elapsedMs?: unknown;
|
||||
};
|
||||
const stamp = validateContract<RequestStamp>('Stamp', result.stamp);
|
||||
const usage: Record<string, number> = {};
|
||||
if (result.usage && typeof result.usage === 'object') {
|
||||
for (const [key, value] of Object.entries(result.usage))
|
||||
if (
|
||||
[
|
||||
'input_tokens',
|
||||
'output_tokens',
|
||||
'total_tokens',
|
||||
'prompt_tokens',
|
||||
'completion_tokens',
|
||||
'cost',
|
||||
].includes(key) &&
|
||||
typeof value === 'number' &&
|
||||
Number.isFinite(value) &&
|
||||
value >= 0
|
||||
)
|
||||
usage[key] = value;
|
||||
}
|
||||
this.receipts.push({
|
||||
kind,
|
||||
stamp,
|
||||
usage,
|
||||
elapsedMs:
|
||||
typeof result.elapsedMs === 'number' &&
|
||||
Number.isFinite(result.elapsedMs) &&
|
||||
result.elapsedMs >= 0
|
||||
? result.elapsedMs
|
||||
: undefined,
|
||||
model: typeof result.model === 'string' ? result.model.slice(0, 128) : 'unknown',
|
||||
provider: typeof result.provider === 'string' ? result.provider.slice(0, 64) : 'unknown',
|
||||
});
|
||||
if (this.receipts.length > 128) this.receipts.shift();
|
||||
return { stamp, value: result.value };
|
||||
} finally {
|
||||
signal.removeEventListener('abort', cancel);
|
||||
}
|
||||
}
|
||||
async plan(
|
||||
obs: Observation,
|
||||
instruction: string,
|
||||
remaining: readonly Skill[],
|
||||
signal: AbortSignal,
|
||||
): Promise<Stamped<Plan>> {
|
||||
const result = await this.request('plan', obs, { instruction, remaining }, signal);
|
||||
return { stamp: result.stamp, value: validatePlan(result.value, remaining) };
|
||||
}
|
||||
async decide(
|
||||
obs: Observation,
|
||||
candidates: readonly (Skill | 'stop')[],
|
||||
failure: string,
|
||||
signal: AbortSignal,
|
||||
): Promise<Stamped<JevDecision>> {
|
||||
const result = await this.request('decide', obs, { candidates, failure }, signal);
|
||||
return { stamp: result.stamp, value: validateJevDecision(result.value, candidates) };
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,350 @@
|
||||
import type { SimulationSession } from '../../simulation/SimulationSession';
|
||||
import { ModelBindings } from '../../simulation/ModelBindings';
|
||||
import { LeKiwiIK } from './LeKiwiIK';
|
||||
import { PhysicalSampler, PickPlaceEvaluator, type PhysicalSample } from './PhysicalEvidence';
|
||||
import { PICK_PLACE as P, PICK_PLACE_ROBOT as R, seedPositions } from './PickPlaceScene';
|
||||
|
||||
const phases = [
|
||||
'open',
|
||||
'pregrasp',
|
||||
'descend',
|
||||
'close',
|
||||
'verify',
|
||||
'carry',
|
||||
'stop-base',
|
||||
'place',
|
||||
'release',
|
||||
'retreat',
|
||||
'settle',
|
||||
] as const;
|
||||
export type BaselinePhase = (typeof phases)[number];
|
||||
const durations = [2, 6, 3, 3, 4, 22, 1, 4, 3, 4, 1];
|
||||
const clamp = (v: number, lo: number, hi: number) => Math.max(lo, Math.min(hi, v));
|
||||
|
||||
/** No network, RL reward, object attachment or independent physics loop.
|
||||
* The caller exclusively owns the existing simulation clock and validates writes.
|
||||
* Ungated mode is the frozen Stage-5 reference. Gated mode waits at each verified
|
||||
* skill boundary; only a current synchronous owner callback may authorize/recover. */
|
||||
export class DeterministicBaseline {
|
||||
readonly evaluator: PickPlaceEvaluator;
|
||||
readonly trace: (PhysicalSample & {
|
||||
phase: BaselinePhase;
|
||||
secure: boolean;
|
||||
transported: number;
|
||||
})[] = [];
|
||||
readonly ik: LeKiwiIK;
|
||||
private readonly sampler: PhysicalSampler;
|
||||
private readonly arm;
|
||||
private readonly gripper;
|
||||
private readonly wheels;
|
||||
private readonly positions;
|
||||
private graspSource: number[];
|
||||
private readonly baseStart: number[];
|
||||
private readonly servo: number[];
|
||||
private readonly velocities = Array<number>(5).fill(0);
|
||||
private readonly baseVelocity = [0, 0, 0];
|
||||
private readonly translation: readonly number[];
|
||||
private opening: number;
|
||||
private phaseIndex = 0;
|
||||
private phaseStart: number;
|
||||
private readonly start: number;
|
||||
private nextControl: number;
|
||||
private lostGraspTime = 0;
|
||||
private readonly armStallTime = Array<number>(5).fill(0);
|
||||
private disposed = false;
|
||||
private completedPhase = false;
|
||||
waiting = false;
|
||||
sample: PhysicalSample;
|
||||
status: 'running' | 'succeeded' | 'failed' = 'running';
|
||||
error = '';
|
||||
maxJointSpeed = 0;
|
||||
constructor(
|
||||
private readonly session: SimulationSession,
|
||||
seed: number,
|
||||
private readonly writeServo: (actuator: number, value: number) => void,
|
||||
private readonly gated = false,
|
||||
goal?: readonly number[],
|
||||
) {
|
||||
const b = new ModelBindings(session.model);
|
||||
this.positions = seedPositions(seed, goal);
|
||||
this.translation = goal
|
||||
? this.positions.goal.slice(0, 2).map((v, i) => v - this.positions.object[i])
|
||||
: P.baseTranslation;
|
||||
this.graspSource = [...this.positions.object];
|
||||
this.waiting = gated;
|
||||
this.arm = R.armJoints.map((spec, i) =>
|
||||
b.scalarActuator(R.armActuators[i], spec.name, 'position'),
|
||||
);
|
||||
this.gripper = b.scalarActuator(R.gripperActuators[0].name, R.gripperJoint, 'position');
|
||||
this.wheels = R.baseActuators.map((name, i) =>
|
||||
b.scalarActuator(name, R.baseJoints[i], 'velocity'),
|
||||
);
|
||||
this.sampler = new PhysicalSampler(session);
|
||||
let ik: LeKiwiIK | undefined;
|
||||
try {
|
||||
this.ik = ik = new LeKiwiIK(session);
|
||||
this.sample = this.sampler.sample();
|
||||
} catch (error) {
|
||||
ik?.dispose();
|
||||
this.sampler.dispose();
|
||||
throw error;
|
||||
}
|
||||
this.baseStart = [...this.sample.base];
|
||||
this.servo = [...this.sample.joints];
|
||||
this.opening = this.sample.opening;
|
||||
this.start = this.phaseStart = this.nextControl = this.sample.time;
|
||||
this.evaluator = new PickPlaceEvaluator(this.positions.object, this.positions.goal);
|
||||
}
|
||||
get phase(): BaselinePhase {
|
||||
return phases[this.phaseIndex];
|
||||
}
|
||||
get goal(): readonly number[] {
|
||||
return this.positions.goal;
|
||||
}
|
||||
get candidate(): BaselinePhase {
|
||||
return phases[this.phaseIndex + (this.completedPhase ? 1 : 0)] ?? 'settle';
|
||||
}
|
||||
/** Re-check facts even if both models recommend continuation. No model can grant this gate. */
|
||||
authorize(skill: BaselinePhase): void {
|
||||
if (!this.gated || !this.waiting || this.status !== 'running' || skill !== this.candidate)
|
||||
throw new Error('非法技能切换');
|
||||
const s = this.sample;
|
||||
if (skill === 'pregrasp' && s.baseSpeed > 0.01) throw new Error('base_not_stopped');
|
||||
if (skill === 'descend' && s.tcp[2] < s.object[2] + 0.04) throw new Error('tcp_not_above');
|
||||
if (skill === 'close' && Math.hypot(...s.tcp.map((v, i) => v - s.object[i])) > 0.012)
|
||||
throw new Error('alignment_timeout');
|
||||
if (skill === 'verify' && s.fingerForces.some((v) => v < P.fingerForceMin))
|
||||
throw new Error('empty_grasp');
|
||||
if (['carry', 'stop-base'].includes(skill) && !this.evaluator.secure)
|
||||
throw new Error('grasp_unverified');
|
||||
if (skill === 'place' && (this.evaluator.transported < P.transportMin || s.baseSpeed > 0.01))
|
||||
throw new Error('transport_unverified');
|
||||
if (skill === 'release' && !s.onGoalSupport) throw new Error('placement_unsupported');
|
||||
if (
|
||||
skill === 'retreat' &&
|
||||
(s.opening < 0.65 || s.fingerForces.some((v) => v >= P.fingerForceMin))
|
||||
)
|
||||
throw new Error('release_blocked');
|
||||
if (this.completedPhase) this.phaseIndex++;
|
||||
this.completedPhase = false;
|
||||
this.waiting = false;
|
||||
this.phaseStart = this.nextControl = s.time;
|
||||
}
|
||||
canRecover(): boolean {
|
||||
const s = this.sample;
|
||||
return (
|
||||
this.gated &&
|
||||
this.status === 'failed' &&
|
||||
this.phaseIndex <= phases.indexOf('verify') &&
|
||||
['alignment_timeout', 'empty_grasp', 'grasp_unverified'].includes(this.error) &&
|
||||
s.supported &&
|
||||
s.objectSpeed < P.placementSpeedMax &&
|
||||
s.baseSpeed < 0.01 &&
|
||||
this.evaluator.transported === 0 &&
|
||||
Math.hypot(...s.object.map((v, i) => v - this.positions.object[i])) < 0.012
|
||||
);
|
||||
}
|
||||
/** Only a supported, stationary object still at the source may be realigned/regrasped. */
|
||||
recover(): void {
|
||||
if (!this.canRecover()) throw new Error('unsafe_recovery');
|
||||
this.hold();
|
||||
// An empty/blocked attempt may not have reached its requested gripper position.
|
||||
// Re-seed targets from measured pose so a released fault cannot cause a servo jump.
|
||||
this.opening = this.sample.opening;
|
||||
this.servo.splice(0, this.servo.length, ...this.sample.joints);
|
||||
this.armStallTime.fill(0);
|
||||
this.graspSource = [...this.sample.object];
|
||||
this.phaseIndex = 0;
|
||||
this.phaseStart = this.nextControl = this.sample.time;
|
||||
this.completedPhase = false;
|
||||
this.waiting = true;
|
||||
this.status = 'running';
|
||||
this.error = '';
|
||||
// Do not reset the episode deadline or fabricate/reset the evaluator's transport history.
|
||||
}
|
||||
private target(): number[] | undefined {
|
||||
const source = this.graspSource,
|
||||
goal = this.positions.goal;
|
||||
switch (this.phase) {
|
||||
case 'pregrasp':
|
||||
return [source[0], source[1], P.pregraspHeight];
|
||||
case 'descend':
|
||||
case 'close':
|
||||
return [...source];
|
||||
case 'verify':
|
||||
return [source[0], source[1], P.carryHeight];
|
||||
case 'place':
|
||||
case 'release':
|
||||
return [...goal];
|
||||
case 'retreat':
|
||||
case 'settle':
|
||||
return [goal[0], goal[1], P.retreatHeight];
|
||||
default:
|
||||
return undefined;
|
||||
}
|
||||
}
|
||||
beforeStep(): void {
|
||||
if (
|
||||
this.disposed ||
|
||||
this.waiting ||
|
||||
this.status !== 'running' ||
|
||||
this.sample.time + 1e-9 < this.nextControl
|
||||
)
|
||||
return;
|
||||
this.nextControl = this.sample.time + P.controlDt;
|
||||
try {
|
||||
const target = this.target();
|
||||
if (target) {
|
||||
const solution = this.ik.solve(target);
|
||||
if (solution.residual > 0.004) throw new Error('unreachable');
|
||||
this.arm.forEach((joint, i) => {
|
||||
const desired = clamp(5 * (solution.targets[i] - this.servo[i]), -P.armSpeed, P.armSpeed);
|
||||
this.velocities[i] += clamp(
|
||||
desired - this.velocities[i],
|
||||
-P.armAcceleration * P.controlDt,
|
||||
P.armAcceleration * P.controlDt,
|
||||
);
|
||||
const old = this.servo[i],
|
||||
spec = R.armJoints[i];
|
||||
const next = clamp(
|
||||
clamp(
|
||||
old + this.velocities[i] * P.controlDt,
|
||||
this.sample.joints[i] - P.armTrackingError,
|
||||
this.sample.joints[i] + P.armTrackingError,
|
||||
),
|
||||
old - P.armSpeed * P.controlDt,
|
||||
old + P.armSpeed * P.controlDt,
|
||||
);
|
||||
this.servo[i] = clamp(next, spec.min, spec.max);
|
||||
this.velocities[i] = (this.servo[i] - old) / P.controlDt;
|
||||
});
|
||||
}
|
||||
this.arm.forEach((joint, i) => this.writeServo(joint.actuatorId, this.servo[i]));
|
||||
const close = ['close', 'verify', 'carry', 'stop-base', 'place'].includes(this.phase);
|
||||
this.opening += clamp(
|
||||
(close ? 0 : 1) - this.opening,
|
||||
-P.gripperOpeningRate * P.controlDt,
|
||||
P.gripperOpeningRate * P.controlDt,
|
||||
);
|
||||
this.writeServo(
|
||||
this.gripper.actuatorId,
|
||||
R.gripperClosed + this.opening * (R.gripperOpen - R.gripperClosed),
|
||||
);
|
||||
const desired = [0, 0, 0];
|
||||
if (this.phase === 'carry') {
|
||||
const dx = this.baseStart[0] + this.translation[0] - this.sample.base[0];
|
||||
const dy = this.baseStart[1] + this.translation[1] - this.sample.base[1];
|
||||
const scale = Math.min(1, P.baseSpeed / Math.max(1e-12, Math.hypot(dx, dy)));
|
||||
const c = Math.cos(this.sample.yaw),
|
||||
s = Math.sin(this.sample.yaw);
|
||||
desired[0] = scale * (c * dx + s * dy);
|
||||
desired[1] = scale * (-s * dx + c * dy);
|
||||
desired[2] = clamp(-1.5 * this.sample.yaw, -P.yawSpeed, P.yawSpeed);
|
||||
}
|
||||
for (let i = 0; i < 3; i++) {
|
||||
const bound = (i === 2 ? P.yawAcceleration : P.baseAcceleration) * P.controlDt;
|
||||
this.baseVelocity[i] += clamp(desired[i] - this.baseVelocity[i], -bound, bound);
|
||||
}
|
||||
const wheelSpeeds = R.baseMix.map((row) =>
|
||||
row.reduce((sum, v, i) => sum + v * this.baseVelocity[i], 0),
|
||||
);
|
||||
const wheelScale = R.wheelLimit / Math.max(R.wheelLimit, ...wheelSpeeds.map(Math.abs));
|
||||
this.wheels.forEach((wheel, i) =>
|
||||
this.writeServo(wheel.actuatorId, wheelSpeeds[i] * wheelScale),
|
||||
);
|
||||
this.trace.push({
|
||||
...this.sample,
|
||||
phase: this.phase,
|
||||
secure: this.evaluator.secure,
|
||||
transported: this.evaluator.transported,
|
||||
});
|
||||
} catch (error) {
|
||||
this.fail(error instanceof Error ? error.message : String(error));
|
||||
}
|
||||
}
|
||||
afterStep(dt: number): void {
|
||||
if (this.disposed || this.status !== 'running') return;
|
||||
try {
|
||||
const previous = this.sample;
|
||||
this.sample = this.sampler.sample();
|
||||
this.evaluator.update(this.sample, dt);
|
||||
// Only arm joints: gripper contact intentionally blocks a closing servo.
|
||||
// Frozen model waits do not call afterStep and cannot accumulate stall time.
|
||||
for (let i = 0; i < this.arm.length; i++) {
|
||||
const speed = Math.abs(this.sample.joints[i] - previous.joints[i]) / dt;
|
||||
const blocked =
|
||||
Math.abs(this.servo[i] - this.sample.joints[i]) >= P.armStallError &&
|
||||
speed < P.armStallSpeed;
|
||||
this.armStallTime[i] = blocked ? this.armStallTime[i] + dt : 0;
|
||||
if (this.armStallTime[i] >= P.armStallDuration) throw new Error('joint_stall');
|
||||
}
|
||||
this.maxJointSpeed = Math.max(this.maxJointSpeed, this.sample.peakJointSpeed);
|
||||
if (this.sample.peakJointSpeed > P.jointSpeedStop) throw new Error('joint_velocity');
|
||||
if (this.sample.base[2] < 0.03 || this.sample.base[2] > 0.1 || this.sample.tilt > 0.35)
|
||||
throw new Error('base_unstable');
|
||||
if (this.sample.time - this.start > P.episodeTimeout) throw new Error('episode_timeout');
|
||||
if (this.phase === 'carry' || this.phase === 'stop-base') {
|
||||
this.lostGraspTime = this.evaluator.secure ? 0 : this.lostGraspTime + dt;
|
||||
if (this.lostGraspTime > 0.2) throw new Error('lost_grasp');
|
||||
}
|
||||
let ready = this.sample.time - this.phaseStart >= durations[this.phaseIndex];
|
||||
if (this.phase === 'carry') {
|
||||
const distance = Math.hypot(
|
||||
...this.sample.base
|
||||
.slice(0, 2)
|
||||
.map((v, i) => v - this.baseStart[i] - this.translation[i]),
|
||||
);
|
||||
if (distance < 0.007) ready = true;
|
||||
else if (ready) throw new Error('navigation_timeout');
|
||||
}
|
||||
if (!ready) return;
|
||||
const target = this.target();
|
||||
if (target && Math.hypot(...target.map((v, i) => v - this.sample.tcp[i])) > 0.012)
|
||||
throw new Error('alignment_timeout');
|
||||
if (this.phase === 'close' && this.sample.fingerForces.some((f) => f < P.fingerForceMin))
|
||||
throw new Error('empty_grasp');
|
||||
if (this.phase === 'verify' && !this.evaluator.secure) throw new Error('grasp_unverified');
|
||||
if (this.phase === 'carry' && this.evaluator.transported < P.transportMin)
|
||||
throw new Error('transport_unverified');
|
||||
if (this.phase === 'place' && !this.sample.onGoalSupport)
|
||||
throw new Error('placement_unsupported');
|
||||
if (
|
||||
this.phase === 'release' &&
|
||||
(this.sample.opening < 0.65 || this.sample.fingerForces.some((f) => f >= P.fingerForceMin))
|
||||
)
|
||||
throw new Error('release_blocked');
|
||||
if (this.phase === 'settle') {
|
||||
if (!this.evaluator.succeeded) throw new Error('placement_unverified');
|
||||
this.status = 'succeeded';
|
||||
this.hold();
|
||||
} else if (this.gated) {
|
||||
this.completedPhase = true;
|
||||
this.waiting = true;
|
||||
this.hold();
|
||||
} else {
|
||||
this.phaseIndex++;
|
||||
this.phaseStart = this.sample.time;
|
||||
}
|
||||
} catch (error) {
|
||||
this.fail(error instanceof Error ? error.message : String(error));
|
||||
}
|
||||
}
|
||||
/** Called only through the session's current owner write capability. */
|
||||
hold(): void {
|
||||
this.baseVelocity.fill(0);
|
||||
this.velocities.fill(0);
|
||||
this.wheels.forEach((wheel) => this.writeServo(wheel.actuatorId, 0));
|
||||
// Keep arm/gripper's LAST safe servo targets; never zero loaded position servos.
|
||||
}
|
||||
private fail(reason: string): void {
|
||||
this.status = 'failed';
|
||||
this.error = reason;
|
||||
this.hold();
|
||||
}
|
||||
dispose(): void {
|
||||
if (this.disposed) return;
|
||||
this.disposed = true;
|
||||
this.sampler.dispose();
|
||||
this.ik.dispose();
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,134 @@
|
||||
import type { DoubleBuffer, MjData } from '@mujoco/mujoco';
|
||||
import type { SimulationSession } from '../../simulation/SimulationSession';
|
||||
import { ModelBindings } from '../../simulation/ModelBindings';
|
||||
import { PICK_PLACE, PICK_PLACE_ROBOT } from './PickPlaceScene';
|
||||
|
||||
/** Small pivoted solve; DLS adds a positive diagonal before calling it. */
|
||||
export function solveLinear(matrix: number[][], rhs: number[]): number[] {
|
||||
const n = rhs.length;
|
||||
const a = matrix.map((row, i) => [...row, rhs[i]]);
|
||||
for (let k = 0; k < n; k++) {
|
||||
let pivot = k;
|
||||
for (let r = k + 1; r < n; r++) if (Math.abs(a[r][k]) > Math.abs(a[pivot][k])) pivot = r;
|
||||
if (!Number.isFinite(a[pivot][k]) || Math.abs(a[pivot][k]) < 1e-12)
|
||||
throw new Error('IK 矩阵不可解');
|
||||
[a[k], a[pivot]] = [a[pivot], a[k]];
|
||||
const divisor = a[k][k];
|
||||
for (let c = k; c <= n; c++) a[k][c] /= divisor;
|
||||
for (let r = 0; r < n; r++)
|
||||
if (r !== k) {
|
||||
const f = a[r][k];
|
||||
for (let c = k; c <= n; c++) a[r][c] -= f * a[k][c];
|
||||
}
|
||||
}
|
||||
return a.map((row) => row[n]);
|
||||
}
|
||||
|
||||
/** Five-axis position + upright approach IK, not a seven-axis/full-pose solver.
|
||||
* Kinematics run exclusively in scratch MjData; never teleport the live robot. */
|
||||
export class LeKiwiIK {
|
||||
private readonly scratch: MjData;
|
||||
private readonly positionJacobian: DoubleBuffer;
|
||||
private readonly rotationJacobian: DoubleBuffer;
|
||||
private readonly joints;
|
||||
readonly site: number;
|
||||
private disposed = false;
|
||||
constructor(private readonly session: SimulationSession) {
|
||||
const { module, model } = session,
|
||||
b = new ModelBindings(model);
|
||||
this.joints = PICK_PLACE_ROBOT.armJoints.map((j) => b.scalarJoint(j.name));
|
||||
this.site = module.mj_name2id(model, 6, PICK_PLACE.tcpSite);
|
||||
if (this.site < 0 || this.joints.length !== 5) throw new Error('抓放 IK 缺少 A 模型绑定');
|
||||
this.scratch = new module.MjData(model);
|
||||
let positionBuffer: DoubleBuffer | undefined;
|
||||
try {
|
||||
this.positionJacobian = positionBuffer = new module.DoubleBuffer(3 * model.nv);
|
||||
this.rotationJacobian = new module.DoubleBuffer(3 * model.nv);
|
||||
} catch (error) {
|
||||
positionBuffer?.delete();
|
||||
this.scratch.delete();
|
||||
throw error;
|
||||
}
|
||||
}
|
||||
solve(target: readonly number[]): { targets: number[]; residual: number; iterations: number } {
|
||||
if (this.disposed) throw new Error('IK 已释放');
|
||||
if (target.length !== 3 || target.some((v) => !Number.isFinite(v) || Math.abs(v) > 2))
|
||||
throw new Error('IK 目标超出标定范围');
|
||||
const { module, model, data } = this.session;
|
||||
// Refresh views at each call; compiling another candidate can grow WASM memory.
|
||||
const qpos: Float64Array = this.scratch.qpos;
|
||||
qpos.set(data.qpos);
|
||||
const q = this.joints.map((j) => qpos[j.qposAddress]);
|
||||
const jp: Float64Array = this.positionJacobian.GetView(),
|
||||
jr: Float64Array = this.rotationJacobian.GetView();
|
||||
let residual = Infinity,
|
||||
iterations = 0;
|
||||
for (; iterations < 80; iterations++) {
|
||||
this.joints.forEach((j, i) => {
|
||||
qpos[j.qposAddress] = q[i];
|
||||
});
|
||||
module.mj_kinematics(model, this.scratch);
|
||||
module.mj_comPos(model, this.scratch);
|
||||
const pos: Float64Array = this.scratch.site_xpos,
|
||||
mat: Float64Array = this.scratch.site_xmat;
|
||||
const error = [
|
||||
...target.map((v, i) => v - pos[this.site * 3 + i]),
|
||||
0.07 * mat[this.site * 9 + 5],
|
||||
-0.07 * mat[this.site * 9 + 2],
|
||||
];
|
||||
residual = Math.hypot(...error);
|
||||
if (residual < 0.0001) break;
|
||||
module.mj_jacSite(
|
||||
model,
|
||||
this.scratch,
|
||||
this.positionJacobian,
|
||||
this.rotationJacobian,
|
||||
this.site,
|
||||
);
|
||||
const jac = Array.from({ length: 5 }, (_, row) =>
|
||||
this.joints.map((joint) =>
|
||||
row < 3
|
||||
? jp[row * model.nv + joint.qvelAddress]
|
||||
: 0.07 * jr[(row - 3) * model.nv + joint.qvelAddress],
|
||||
),
|
||||
);
|
||||
const gram = jac.map((row, i) =>
|
||||
jac.map(
|
||||
(other, k) => row.reduce((sum, v, j) => sum + v * other[j], 0) + (i === k ? 0.00002 : 0),
|
||||
),
|
||||
);
|
||||
const inverseError = solveLinear(gram, error);
|
||||
for (let j = 0; j < q.length; j++) {
|
||||
const update = jac.reduce((sum, row, i) => sum + row[j] * inverseError[i], 0);
|
||||
const spec = PICK_PLACE_ROBOT.armJoints[j];
|
||||
q[j] = Math.max(
|
||||
spec.min + 0.015,
|
||||
Math.min(spec.max - 0.015, q[j] + Math.max(-0.07, Math.min(0.07, update))),
|
||||
);
|
||||
}
|
||||
}
|
||||
if (iterations === 80) {
|
||||
this.joints.forEach((j, i) => {
|
||||
qpos[j.qposAddress] = q[i];
|
||||
});
|
||||
module.mj_kinematics(model, this.scratch);
|
||||
const pos: Float64Array = this.scratch.site_xpos,
|
||||
mat: Float64Array = this.scratch.site_xmat;
|
||||
residual = Math.hypot(
|
||||
...target.map((v, i) => v - pos[this.site * 3 + i]),
|
||||
0.07 * mat[this.site * 9 + 5],
|
||||
-0.07 * mat[this.site * 9 + 2],
|
||||
);
|
||||
}
|
||||
if (!Number.isFinite(residual) || q.some((v) => !Number.isFinite(v)))
|
||||
throw new Error('IK 非有限结果');
|
||||
return { targets: q, residual, iterations };
|
||||
}
|
||||
dispose(): void {
|
||||
if (this.disposed) return;
|
||||
this.disposed = true;
|
||||
this.rotationJacobian.delete();
|
||||
this.positionJacobian.delete();
|
||||
this.scratch.delete();
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,59 @@
|
||||
import type { DecisionProvider } from './DecisionClient';
|
||||
import type { Stamped } from './RequestGate';
|
||||
import {
|
||||
AGENT_VERSION,
|
||||
PRECONDITIONS,
|
||||
type Observation,
|
||||
type Plan,
|
||||
type Skill,
|
||||
type JevDecision,
|
||||
} from './protocol';
|
||||
|
||||
/** Explicit no-network fixture. Never selected as a fallback by DecisionClient. */
|
||||
export class MockDecisionProvider implements DecisionProvider {
|
||||
readonly mode = 'mock' as const;
|
||||
async plan(
|
||||
obs: Observation,
|
||||
_instruction: string,
|
||||
remaining: readonly Skill[],
|
||||
signal: AbortSignal,
|
||||
): Promise<Stamped<Plan>> {
|
||||
signal.throwIfAborted();
|
||||
return {
|
||||
stamp: { ...obs.stamp },
|
||||
value: {
|
||||
version: AGENT_VERSION,
|
||||
objectId: 'block',
|
||||
goalId: 'placement',
|
||||
summary: 'MOCK:固定规范顺序,成功仅由物理评估决定',
|
||||
steps: remaining.map((skill) => ({
|
||||
skill,
|
||||
precondition: PRECONDITIONS[skill],
|
||||
onFailure: 'retry',
|
||||
})),
|
||||
},
|
||||
};
|
||||
}
|
||||
async decide(
|
||||
obs: Observation,
|
||||
candidates: readonly (Skill | 'stop')[],
|
||||
failure: string,
|
||||
signal: AbortSignal,
|
||||
): Promise<Stamped<JevDecision>> {
|
||||
signal.throwIfAborted();
|
||||
return {
|
||||
stamp: { ...obs.stamp },
|
||||
value: {
|
||||
version: AGENT_VERSION,
|
||||
choice: candidates[0],
|
||||
grasp: obs.evidence.secure
|
||||
? 'secure'
|
||||
: obs.evidence.fingerForces.some((v) => v > 0.2)
|
||||
? 'uncertain'
|
||||
: 'empty',
|
||||
diagnosis: failure === 'none' ? 'none' : failure === 'empty_grasp' ? 'empty' : 'misaligned',
|
||||
recovery: failure === 'none' ? 'continue' : 'retry',
|
||||
},
|
||||
};
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,152 @@
|
||||
import { cleanup, fireEvent, render, screen, waitFor } from '@testing-library/react';
|
||||
import { afterEach, describe, expect, it, vi } from 'vitest';
|
||||
import { ModelConnectionPanel } from './ModelConnectionPanel';
|
||||
|
||||
afterEach(() => {
|
||||
cleanup();
|
||||
vi.unstubAllGlobals();
|
||||
vi.restoreAllMocks();
|
||||
});
|
||||
const status = {
|
||||
version: 'lekiwi-agent-v1',
|
||||
connections: {
|
||||
llm: {
|
||||
protocol: 'responses',
|
||||
baseUrl: 'https://api.deepseek.com',
|
||||
model: 'deepseek-flash',
|
||||
hasKey: true,
|
||||
},
|
||||
jev: {
|
||||
protocol: 'openrouter-decisions',
|
||||
baseUrl: 'https://openrouter.ai/api/alpha/decisions',
|
||||
model: 'typesafe/jev-1.13',
|
||||
hasKey: true,
|
||||
},
|
||||
},
|
||||
};
|
||||
async function connect() {
|
||||
fireEvent.change(screen.getByLabelText('本机服务令牌'), {
|
||||
target: { value: 'fixture-service-token' },
|
||||
});
|
||||
fireEvent.click(screen.getByRole('button', { name: '连接本机模型服务' }));
|
||||
await screen.findByText(/本机桥已连接/);
|
||||
}
|
||||
describe('模型连接面板(HTTP 为 mock,不调用云端)', () => {
|
||||
it('读取脱敏状态不推理、不持久化;换地址清空草稿 key,保存后也清空', async () => {
|
||||
const fetcher = vi.fn<typeof fetch>(async () => Response.json(status));
|
||||
vi.stubGlobal('fetch', fetcher);
|
||||
const storage = vi.spyOn(Storage.prototype, 'setItem');
|
||||
render(<ModelConnectionPanel onClient={vi.fn()} onInvalidate={vi.fn()} />);
|
||||
await connect();
|
||||
expect(fetcher).toHaveBeenCalledOnce();
|
||||
expect(screen.getByLabelText('LLM 型号')).toHaveValue('deepseek-flash');
|
||||
fireEvent.change(screen.getByLabelText('LLM API Key'), {
|
||||
target: { value: 'test-key-not-real' },
|
||||
});
|
||||
fireEvent.change(screen.getByLabelText('LLM Base URL'), {
|
||||
target: { value: 'https://example.test/v1' },
|
||||
});
|
||||
expect(screen.getByLabelText('LLM API Key')).toHaveValue('');
|
||||
fireEvent.change(screen.getByLabelText('LLM API Key'), {
|
||||
target: { value: 'test-key-not-real' },
|
||||
});
|
||||
fireEvent.click(screen.getByRole('button', { name: '保存 LLM 连接' }));
|
||||
await screen.findByText(/LLM 配置已保存/);
|
||||
expect(screen.getByLabelText('LLM API Key')).toHaveValue('');
|
||||
expect(storage).not.toHaveBeenCalled();
|
||||
expect(fetcher.mock.calls.some(([url]) => /\/(plan|decide)$/.test(String(url)))).toBe(false);
|
||||
});
|
||||
it('未保存的型号草稿禁止测试/任务,不能仍用旧付费配置', async () => {
|
||||
const fetcher = vi
|
||||
.fn<typeof fetch>()
|
||||
.mockResolvedValueOnce(Response.json(status))
|
||||
.mockResolvedValueOnce(Response.json({ ok: true }))
|
||||
.mockResolvedValueOnce(
|
||||
Response.json({
|
||||
...status,
|
||||
connections: {
|
||||
...status.connections,
|
||||
llm: { ...status.connections.llm, model: 'new-model' },
|
||||
},
|
||||
}),
|
||||
);
|
||||
vi.stubGlobal('fetch', fetcher);
|
||||
const onClient = vi.fn();
|
||||
render(<ModelConnectionPanel onClient={onClient} onInvalidate={vi.fn()} />);
|
||||
await connect();
|
||||
fireEvent.change(screen.getByLabelText('LLM 型号'), { target: { value: 'new-model' } });
|
||||
expect(onClient).toHaveBeenLastCalledWith(undefined);
|
||||
expect(screen.getByRole('button', { name: '测试 LLM(可能计费)' })).toBeDisabled();
|
||||
fireEvent.click(screen.getByRole('button', { name: '测试 LLM(可能计费)' }));
|
||||
expect(fetcher).toHaveBeenCalledOnce();
|
||||
fireEvent.change(screen.getByLabelText('LLM API Key'), { target: { value: 'fixture-key' } });
|
||||
fireEvent.click(screen.getByRole('button', { name: '保存 LLM 连接' }));
|
||||
await screen.findByText(/LLM 配置已保存/);
|
||||
expect(screen.getByLabelText('LLM 型号')).toHaveValue('new-model');
|
||||
expect(screen.getByRole('button', { name: '测试 LLM(可能计费)' })).toBeEnabled();
|
||||
expect(onClient.mock.lastCall?.[0]).toBeDefined();
|
||||
});
|
||||
it('断开后迟到的连接状态不会重新启用客户端', async () => {
|
||||
let resolve!: (response: Response) => void;
|
||||
vi.stubGlobal(
|
||||
'fetch',
|
||||
vi.fn(
|
||||
() =>
|
||||
new Promise<Response>((r) => {
|
||||
resolve = r;
|
||||
}),
|
||||
),
|
||||
);
|
||||
const onClient = vi.fn();
|
||||
render(<ModelConnectionPanel onClient={onClient} onInvalidate={vi.fn()} />);
|
||||
fireEvent.change(screen.getByLabelText('本机服务令牌'), { target: { value: 'fixture-token' } });
|
||||
fireEvent.click(screen.getByRole('button', { name: '连接本机模型服务' }));
|
||||
fireEvent.click(screen.getByRole('button', { name: '断开模型服务' }));
|
||||
resolve(Response.json(status));
|
||||
await waitFor(() => expect(screen.getByText(/本机桥已断开/)).toBeInTheDocument());
|
||||
expect(screen.queryByLabelText('LLM 协议')).not.toBeInTheDocument();
|
||||
expect(onClient.mock.calls.every(([client]) => client === undefined)).toBe(true);
|
||||
});
|
||||
it('账号只读查询不取消物理任务;拒绝伪官方登录地址', async () => {
|
||||
vi.stubGlobal(
|
||||
'fetch',
|
||||
vi.fn<typeof fetch>(async (url) => {
|
||||
if (String(url).endsWith('/codex/status')) return Response.json({ loggedIn: false });
|
||||
if (String(url).endsWith('/login'))
|
||||
return Response.json({ authUrl: 'https://auth.openai.com.attacker.test/' });
|
||||
return Response.json(status);
|
||||
}),
|
||||
);
|
||||
const invalidate = vi.fn();
|
||||
render(<ModelConnectionPanel onClient={vi.fn()} onInvalidate={invalidate} />);
|
||||
await connect();
|
||||
invalidate.mockClear();
|
||||
fireEvent.click(screen.getByText(/ChatGPT 订阅 · 官方/));
|
||||
fireEvent.click(screen.getByRole('button', { name: '刷新账号' }));
|
||||
await screen.findByText('ChatGPT 未登录');
|
||||
expect(invalidate).not.toHaveBeenCalled();
|
||||
fireEvent.click(screen.getByRole('button', { name: '发起官方登录' }));
|
||||
await screen.findByText('官方登录地址校验失败');
|
||||
expect(screen.queryByRole('link', { name: '打开 OpenAI 官方登录页' })).not.toBeInTheDocument();
|
||||
});
|
||||
it('官方登录只展示 OpenAI 链接,取消清理;不保存/导出完整地址', async () => {
|
||||
vi.stubGlobal(
|
||||
'fetch',
|
||||
vi.fn<typeof fetch>(async (url) => {
|
||||
if (String(url).endsWith('/login'))
|
||||
return Response.json({ authUrl: 'https://auth.openai.com/oauth?state=fixture-only' });
|
||||
if (String(url).endsWith('/cancel')) return Response.json({ cancelled: true });
|
||||
return Response.json(status);
|
||||
}),
|
||||
);
|
||||
render(<ModelConnectionPanel onClient={vi.fn()} onInvalidate={vi.fn()} />);
|
||||
await connect();
|
||||
fireEvent.click(screen.getByText(/ChatGPT 订阅 · 官方/));
|
||||
fireEvent.click(screen.getByRole('button', { name: '发起官方登录' }));
|
||||
const link = await screen.findByRole('link', { name: '打开 OpenAI 官方登录页' });
|
||||
expect(link).toHaveAttribute('rel', 'noopener noreferrer');
|
||||
fireEvent.click(screen.getByRole('button', { name: '取消官方登录' }));
|
||||
await screen.findByText('登录已取消');
|
||||
expect(screen.queryByRole('link', { name: '打开 OpenAI 官方登录页' })).not.toBeInTheDocument();
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,385 @@
|
||||
import { useEffect, useRef, useState, type ComponentProps } from 'react';
|
||||
import { WEBSITE_MODE } from '../../app/websiteConfig';
|
||||
import { WebsiteModelSettings } from './WebsiteModelSettings';
|
||||
import { Button } from '../../components/ui';
|
||||
import { DecisionClient } from './DecisionClient';
|
||||
|
||||
type Role = 'llm' | 'jev';
|
||||
interface Connection {
|
||||
protocol: string;
|
||||
baseUrl: string;
|
||||
model: string;
|
||||
hasKey?: boolean;
|
||||
apiKey: string;
|
||||
}
|
||||
const defaults: Record<Role, Connection> = {
|
||||
llm: { protocol: 'responses', baseUrl: 'https://api.openai.com/v1', model: '', apiKey: '' },
|
||||
jev: {
|
||||
protocol: 'typesafe',
|
||||
baseUrl: 'https://api.typesafe.ai/v1/systemone',
|
||||
model: 'jev-latest',
|
||||
apiKey: '',
|
||||
},
|
||||
};
|
||||
function protocolDefaults(value: string): Partial<Connection> {
|
||||
const next =
|
||||
value === 'codex'
|
||||
? { baseUrl: '', model: '' }
|
||||
: value === 'openrouter-decisions'
|
||||
? { baseUrl: 'https://openrouter.ai/api/alpha/decisions', model: 'typesafe/jev-1.13' }
|
||||
: value === 'typesafe'
|
||||
? { baseUrl: defaults.jev.baseUrl, model: defaults.jev.model }
|
||||
: { baseUrl: defaults.llm.baseUrl, model: '' };
|
||||
return { protocol: value, ...next };
|
||||
}
|
||||
const inputClass = 'w-full rounded border border-border bg-surface px-2 py-1 text-xs';
|
||||
export function ModelConnectionPanel(props: ComponentProps<typeof LocalModelConnectionPanel>) {
|
||||
return WEBSITE_MODE ? (
|
||||
<WebsiteModelSettings {...props} />
|
||||
) : (
|
||||
<LocalModelConnectionPanel {...props} />
|
||||
);
|
||||
}
|
||||
|
||||
function LocalModelConnectionPanel({
|
||||
onClient,
|
||||
onInvalidate,
|
||||
}: {
|
||||
onClient(client: DecisionClient | undefined): void;
|
||||
onInvalidate(): void;
|
||||
}) {
|
||||
const [address, setAddress] = useState('http://127.0.0.1:8768');
|
||||
const [token, setToken] = useState('');
|
||||
const [client, setClient] = useState<DecisionClient>();
|
||||
const [drafts, setDrafts] = useState(defaults);
|
||||
const [busy, setBusy] = useState(false);
|
||||
const [dirty, setDirty] = useState(false);
|
||||
const [message, setMessage] = useState(
|
||||
'启动 npm run decision-server,再输入终端显示的本机令牌。',
|
||||
);
|
||||
const [authUrl, setAuthUrl] = useState('');
|
||||
const [account, setAccount] = useState('订阅尚未检查');
|
||||
const [models, setModels] = useState<{ id: string; name: string }[]>([]);
|
||||
const [limits, setLimits] = useState('尚未读取官方配额');
|
||||
const epoch = useRef(0),
|
||||
pending = useRef<AbortController | undefined>(undefined);
|
||||
useEffect(
|
||||
() => () => {
|
||||
epoch.current++;
|
||||
pending.current?.abort();
|
||||
},
|
||||
[],
|
||||
);
|
||||
function invalidate(stopTask = true) {
|
||||
epoch.current++;
|
||||
pending.current?.abort();
|
||||
pending.current = undefined;
|
||||
setBusy(false);
|
||||
if (stopTask) onInvalidate();
|
||||
}
|
||||
function disconnect() {
|
||||
invalidate();
|
||||
if (client)
|
||||
void client
|
||||
.call('/codex/cancel', 'POST', {}, AbortSignal.timeout(3000))
|
||||
.catch(() => undefined);
|
||||
setAuthUrl('');
|
||||
setAccount('订阅尚未检查');
|
||||
setModels([]);
|
||||
setLimits('尚未读取官方配额');
|
||||
setDirty(false);
|
||||
setClient(undefined);
|
||||
onClient(undefined);
|
||||
setToken('');
|
||||
setDrafts(defaults);
|
||||
setMessage('本机桥已断开;未修改任何全局 Codex 登录。');
|
||||
}
|
||||
async function run(
|
||||
operation: (signal: AbortSignal, generation: number) => Promise<void>,
|
||||
stopTask = true,
|
||||
) {
|
||||
invalidate(stopTask);
|
||||
const generation = epoch.current,
|
||||
abort = new AbortController();
|
||||
pending.current = abort;
|
||||
setBusy(true);
|
||||
try {
|
||||
await operation(abort.signal, generation);
|
||||
} catch (error) {
|
||||
if (generation === epoch.current)
|
||||
setMessage(error instanceof Error ? error.message : '连接失败');
|
||||
} finally {
|
||||
if (generation === epoch.current) {
|
||||
setBusy(false);
|
||||
pending.current = undefined;
|
||||
}
|
||||
}
|
||||
}
|
||||
async function refresh(c: DecisionClient, signal: AbortSignal, generation: number) {
|
||||
const status = (await c.call('/status', 'GET', undefined, signal)) as {
|
||||
version?: string;
|
||||
connections?: Partial<Record<Role, Connection>>;
|
||||
};
|
||||
if (status.version !== 'lekiwi-agent-v1') throw new Error('模型服务版本不匹配');
|
||||
if (generation !== epoch.current) return;
|
||||
setDrafts({
|
||||
llm: { ...defaults.llm, ...status.connections?.llm, apiKey: '' },
|
||||
jev: { ...defaults.jev, ...status.connections?.jev, apiKey: '' },
|
||||
});
|
||||
setDirty(false);
|
||||
setClient(c);
|
||||
onClient(c);
|
||||
}
|
||||
function markDirty() {
|
||||
invalidate();
|
||||
setDirty(true);
|
||||
onClient(undefined); // Do not run against the old paid connection behind an unsaved draft.
|
||||
setMessage('配置草稿尚未保存;真实任务/测试已禁用。保存会刷新两角色的服务配置。');
|
||||
}
|
||||
function patch(role: Role, value: Partial<Connection>) {
|
||||
markDirty();
|
||||
setDrafts((previous) => ({
|
||||
...previous,
|
||||
[role]: {
|
||||
...previous[role],
|
||||
...value,
|
||||
...('baseUrl' in value || 'protocol' in value ? { apiKey: '', hasKey: false } : {}),
|
||||
},
|
||||
}));
|
||||
}
|
||||
async function saveConnection(role: Role) {
|
||||
if (!client) return;
|
||||
await run(async (signal, generation) => {
|
||||
const { protocol, baseUrl, model, apiKey } = drafts[role];
|
||||
await client.call('/connections', 'PUT', { role, protocol, baseUrl, model, apiKey }, signal);
|
||||
await refresh(client, signal, generation);
|
||||
if (generation === epoch.current)
|
||||
setMessage(`${role.toUpperCase()} 配置已保存;密钥不落盘。`);
|
||||
});
|
||||
}
|
||||
async function testConnection(role: Role) {
|
||||
if (!client || dirty) return;
|
||||
await run(async (signal, generation) => {
|
||||
const result = (await client.call('/test', 'POST', { role }, signal)) as { usage?: unknown };
|
||||
if (generation === epoch.current)
|
||||
setMessage(
|
||||
drafts[role].protocol === 'codex'
|
||||
? 'Codex 离线禁工具门禁通过;不代表订阅真实推理通过。'
|
||||
: `连接测试通过(真实请求);服务报告 usage:${JSON.stringify(result.usage ?? {})}`,
|
||||
);
|
||||
});
|
||||
}
|
||||
async function codex(operation: 'status' | 'models' | 'limits' | 'login' | 'cancel' | 'logout') {
|
||||
if (!client) return;
|
||||
await run(async (signal, generation) => {
|
||||
const post = ['login', 'cancel', 'logout'].includes(operation);
|
||||
const value = (await client.call(
|
||||
'/codex/' + operation,
|
||||
post ? 'POST' : 'GET',
|
||||
post ? {} : undefined,
|
||||
signal,
|
||||
)) as Record<string, unknown>;
|
||||
if (generation !== epoch.current) return;
|
||||
if (operation === 'login') {
|
||||
const url = new URL(String(value.authUrl));
|
||||
if (
|
||||
url.protocol !== 'https:' ||
|
||||
url.hostname !== 'auth.openai.com' ||
|
||||
url.username ||
|
||||
url.password ||
|
||||
(url.port && url.port !== '443')
|
||||
)
|
||||
throw new Error('官方登录地址校验失败');
|
||||
setAuthUrl(url.href);
|
||||
setMessage('请打开官方登录页;完成后点击刷新账号。认证地址不进入任务导出。');
|
||||
} else if (operation === 'models') setModels(value.models as { id: string; name: string }[]);
|
||||
else if (operation === 'limits') setLimits(JSON.stringify(value));
|
||||
else if (operation === 'status') {
|
||||
setAccount(value.loggedIn ? 'ChatGPT 已登录(仅本次服务会话)' : 'ChatGPT 未登录');
|
||||
if (value.loggedIn) setAuthUrl('');
|
||||
} else {
|
||||
setAuthUrl('');
|
||||
setAccount(operation === 'logout' ? '已退出本次隔离会话' : '登录已取消');
|
||||
}
|
||||
}, !['status', 'models', 'limits'].includes(operation));
|
||||
}
|
||||
return (
|
||||
<div className="space-y-3 text-xs" aria-label="模型连接管理">
|
||||
<p>密钥仅服务内存;本机令牌仅页面内存。无隐式协议、型号或付费来源切换。</p>
|
||||
<label className="block">
|
||||
本机模型服务
|
||||
<input
|
||||
className={inputClass}
|
||||
value={address}
|
||||
onChange={(e) => {
|
||||
disconnect();
|
||||
setAddress(e.target.value);
|
||||
}}
|
||||
/>
|
||||
</label>
|
||||
<label className="block">
|
||||
本机服务令牌
|
||||
<input
|
||||
className={inputClass}
|
||||
type="password"
|
||||
autoComplete="off"
|
||||
value={token}
|
||||
onChange={(e) => {
|
||||
disconnect();
|
||||
setToken(e.target.value);
|
||||
}}
|
||||
/>
|
||||
</label>
|
||||
<div className="flex flex-wrap gap-2">
|
||||
<Button
|
||||
disabled={busy || !token.trim()}
|
||||
onClick={() =>
|
||||
void run(async (signal, generation) => {
|
||||
const c = new DecisionClient(address, token);
|
||||
await refresh(c, signal, generation);
|
||||
if (generation === epoch.current)
|
||||
setMessage('本机桥已连接;未自动发送任何模型推理。');
|
||||
})
|
||||
}
|
||||
>
|
||||
连接本机模型服务
|
||||
</Button>
|
||||
<Button onClick={disconnect}>断开模型服务</Button>
|
||||
</div>
|
||||
{client &&
|
||||
(['llm', 'jev'] as const).map((role) => {
|
||||
const draft = drafts[role];
|
||||
return (
|
||||
<fieldset
|
||||
key={role}
|
||||
disabled={busy}
|
||||
className="space-y-2 rounded border border-border p-2"
|
||||
>
|
||||
<legend>{role === 'llm' ? 'LLM 规划' : 'Jev 决策'}</legend>
|
||||
<label className="block">
|
||||
{role.toUpperCase()} 协议
|
||||
<select
|
||||
className={inputClass}
|
||||
value={draft.protocol}
|
||||
aria-label={`${role.toUpperCase()} 协议`}
|
||||
onChange={(e) => {
|
||||
markDirty();
|
||||
const config = protocolDefaults(e.target.value);
|
||||
setDrafts((previous) => ({
|
||||
...previous,
|
||||
[role]: { ...previous[role], ...config, apiKey: '', hasKey: false },
|
||||
}));
|
||||
}}
|
||||
>
|
||||
{(role === 'llm'
|
||||
? ['responses', 'chat-completions', 'codex']
|
||||
: ['typesafe', 'openrouter-decisions']
|
||||
).map((p) => (
|
||||
<option key={p} value={p}>
|
||||
{p}
|
||||
</option>
|
||||
))}
|
||||
</select>
|
||||
</label>
|
||||
{draft.protocol !== 'codex' && (
|
||||
<>
|
||||
<label className="block">
|
||||
{role.toUpperCase()} Base URL
|
||||
<input
|
||||
className={inputClass}
|
||||
value={draft.baseUrl}
|
||||
onChange={(e) => patch(role, { baseUrl: e.target.value })}
|
||||
/>
|
||||
</label>
|
||||
<label className="block">
|
||||
{role.toUpperCase()} API Key
|
||||
<input
|
||||
className={inputClass}
|
||||
type="password"
|
||||
autoComplete="off"
|
||||
value={draft.apiKey}
|
||||
onChange={(e) => patch(role, { apiKey: e.target.value })}
|
||||
/>
|
||||
</label>
|
||||
<p>
|
||||
{draft.hasKey ? '服务已有内存密钥;修改配置必须重新输入。' : '尚未载入密钥。'}
|
||||
</p>
|
||||
</>
|
||||
)}
|
||||
<label className="block">
|
||||
{role.toUpperCase()} 型号
|
||||
{draft.protocol === 'codex' ? (
|
||||
<select
|
||||
className={inputClass}
|
||||
value={draft.model}
|
||||
aria-label={`${role.toUpperCase()} 型号`}
|
||||
onChange={(e) => patch(role, { model: e.target.value })}
|
||||
>
|
||||
<option value="">先查询并选择官方型号</option>
|
||||
{models.map((m) => (
|
||||
<option key={m.id} value={m.id}>
|
||||
{m.name} · {m.id}
|
||||
</option>
|
||||
))}
|
||||
</select>
|
||||
) : (
|
||||
<input
|
||||
className={inputClass}
|
||||
value={draft.model}
|
||||
onChange={(e) => patch(role, { model: e.target.value })}
|
||||
/>
|
||||
)}
|
||||
</label>
|
||||
<div className="flex flex-wrap gap-2">
|
||||
<Button onClick={() => void saveConnection(role)}>
|
||||
保存 {role.toUpperCase()} 连接
|
||||
</Button>
|
||||
<Button disabled={dirty} onClick={() => void testConnection(role)}>
|
||||
测试 {role.toUpperCase()}(可能计费)
|
||||
</Button>
|
||||
</div>
|
||||
</fieldset>
|
||||
);
|
||||
})}
|
||||
{client && (
|
||||
<details>
|
||||
<summary>ChatGPT 订阅 · 官方 Codex 0.147.0(实验性)</summary>
|
||||
<p>
|
||||
隔离临时 HOME,仅会话登录,不读取现有全局登录。只做规划,禁止工具。CLI
|
||||
缺失/版本不匹配不会转用 API。
|
||||
</p>
|
||||
<div className="flex flex-wrap gap-2">
|
||||
<Button disabled={busy} onClick={() => void codex('login')}>
|
||||
发起官方登录
|
||||
</Button>
|
||||
<Button disabled={busy} onClick={() => void codex('cancel')}>
|
||||
取消官方登录
|
||||
</Button>
|
||||
<Button disabled={busy} onClick={() => void codex('logout')}>
|
||||
退出隔离账号
|
||||
</Button>
|
||||
<Button disabled={busy} onClick={() => void codex('status')}>
|
||||
刷新账号
|
||||
</Button>
|
||||
<Button disabled={busy} onClick={() => void codex('models')}>
|
||||
查询官方型号
|
||||
</Button>
|
||||
<Button disabled={busy} onClick={() => void codex('limits')}>
|
||||
读取官方配额
|
||||
</Button>
|
||||
</div>
|
||||
{authUrl && (
|
||||
<a href={authUrl} target="_blank" rel="noopener noreferrer" className="underline">
|
||||
打开 OpenAI 官方登录页
|
||||
</a>
|
||||
)}
|
||||
<p>{account}</p>
|
||||
<p className="break-all">{limits}</p>
|
||||
</details>
|
||||
)}
|
||||
<p role="status" className="break-words">
|
||||
{busy ? '请求进行中…' : message}
|
||||
</p>
|
||||
</div>
|
||||
);
|
||||
}
|
||||
@@ -0,0 +1,115 @@
|
||||
import { PickPlaceEvaluator, type PhysicalSample } from './PhysicalEvidence';
|
||||
import { PICK_PLACE as P } from './PickPlaceScene';
|
||||
|
||||
function fixture() {
|
||||
const evaluator = new PickPlaceEvaluator(P.object.position, P.goal.position);
|
||||
let time = 0;
|
||||
const sample: PhysicalSample = {
|
||||
time: 0,
|
||||
base: [0, 0, 0.05],
|
||||
baseSpeed: 0,
|
||||
yaw: 0,
|
||||
tilt: 0,
|
||||
joints: [0, 0, 0, 0, 0],
|
||||
peakJointSpeed: 0,
|
||||
opening: 0.4,
|
||||
tcp: [0.257, 0.015, 0.19],
|
||||
object: [0.257, 0.015, 0.19],
|
||||
objectSpeed: 0,
|
||||
fingerForces: [1, 1],
|
||||
supported: false,
|
||||
onGoalSupport: false,
|
||||
};
|
||||
const tick = (count: number) => {
|
||||
for (let i = 0; i < count; i++) {
|
||||
sample.time = time += 0.02;
|
||||
evaluator.update(sample, 0.02);
|
||||
}
|
||||
};
|
||||
const carry = () => {
|
||||
tick(10);
|
||||
sample.base[1] = 0.6;
|
||||
sample.object[1] += 0.6;
|
||||
sample.tcp[1] += 0.6;
|
||||
tick(2);
|
||||
};
|
||||
const release = () => {
|
||||
Object.assign(sample, {
|
||||
opening: 1,
|
||||
object: [...P.goal.position],
|
||||
tcp: [P.goal.position[0], P.goal.position[1], P.goal.position[2] + 0.1],
|
||||
fingerForces: [0, 0],
|
||||
supported: true,
|
||||
onGoalSupport: true,
|
||||
});
|
||||
};
|
||||
return { evaluator, sample, tick, carry, release };
|
||||
}
|
||||
it('requires actual lift, BOTH fingers, held translation, release, retreat and continuous supported stability', () => {
|
||||
const f = fixture();
|
||||
f.carry();
|
||||
expect(f.evaluator.transported).toBeCloseTo(0.6);
|
||||
f.release();
|
||||
f.tick(24);
|
||||
expect(f.evaluator.succeeded).toBe(false);
|
||||
f.tick(2);
|
||||
expect(f.evaluator.succeeded).toBe(true);
|
||||
});
|
||||
it.each([
|
||||
[10, 0],
|
||||
[0, 10],
|
||||
[0, 0],
|
||||
])('many contacts on one side/empty cannot establish grasp: %j', (a, b) => {
|
||||
const f = fixture();
|
||||
f.sample.fingerForces = [a, b];
|
||||
f.carry();
|
||||
expect(f.evaluator.hasGrasped).toBe(false);
|
||||
f.release();
|
||||
f.tick(100);
|
||||
expect(f.evaluator.succeeded).toBe(false);
|
||||
});
|
||||
it('an object initially at the goal has no transport history', () => {
|
||||
const f = fixture();
|
||||
f.release();
|
||||
f.tick(100);
|
||||
expect(f.evaluator.succeeded).toBe(false);
|
||||
});
|
||||
it('contact on a support is not verified lifting', () => {
|
||||
const f = fixture();
|
||||
f.sample.supported = true;
|
||||
f.carry();
|
||||
expect(f.evaluator.secure).toBe(false);
|
||||
});
|
||||
it('slip invalidates history; empty base travel is not credited after a regrasp', () => {
|
||||
const f = fixture();
|
||||
f.carry();
|
||||
f.sample.fingerForces = [0, 0];
|
||||
f.tick(10);
|
||||
expect(f.evaluator.transported).toBe(0);
|
||||
f.sample.base[1] = 1.2;
|
||||
f.sample.fingerForces = [1, 1];
|
||||
f.tick(20);
|
||||
expect(f.evaluator.secure).toBe(true);
|
||||
expect(f.evaluator.transported).toBe(0);
|
||||
});
|
||||
it('duplicate/frozen observations never accumulate stability and time jumps fail closed', () => {
|
||||
const f = fixture();
|
||||
f.carry();
|
||||
f.release();
|
||||
f.tick(1);
|
||||
for (let i = 0; i < 100; i++) f.evaluator.update(f.sample, 0.02);
|
||||
expect(f.evaluator.succeeded).toBe(false);
|
||||
expect(() => f.evaluator.update({ ...f.sample, time: f.sample.time + 1 }, 0.02)).toThrow(/时钟/);
|
||||
});
|
||||
it('ongoing grasp, high velocity or missing retreat cannot produce placement success', () => {
|
||||
for (const fault of ['grasp', 'speed', 'retreat'] as const) {
|
||||
const f = fixture();
|
||||
f.carry();
|
||||
f.release();
|
||||
if (fault === 'grasp') f.sample.fingerForces = [1, 1];
|
||||
if (fault === 'speed') f.sample.objectSpeed = 0.1;
|
||||
if (fault === 'retreat') f.sample.tcp = [...f.sample.object];
|
||||
f.tick(100);
|
||||
expect(f.evaluator.succeeded).toBe(false);
|
||||
}
|
||||
});
|
||||
@@ -0,0 +1,206 @@
|
||||
import type { DoubleBuffer } from '@mujoco/mujoco';
|
||||
import type { SimulationSession } from '../../simulation/SimulationSession';
|
||||
import { ModelBindings } from '../../simulation/ModelBindings';
|
||||
import { PICK_PLACE as P, PICK_PLACE_ROBOT as R } from './PickPlaceScene';
|
||||
|
||||
export interface PhysicalSample {
|
||||
time: number;
|
||||
base: number[];
|
||||
baseSpeed: number;
|
||||
yaw: number;
|
||||
tilt: number;
|
||||
joints: number[];
|
||||
peakJointSpeed: number;
|
||||
opening: number;
|
||||
tcp: number[];
|
||||
object: number[];
|
||||
objectSpeed: number;
|
||||
fingerForces: [number, number];
|
||||
supported: boolean;
|
||||
onGoalSupport: boolean;
|
||||
}
|
||||
|
||||
/** Geom-to-finger aggregation: many hull contacts on ONE finger never count as two. */
|
||||
export class PhysicalSampler {
|
||||
private readonly force: DoubleBuffer;
|
||||
private readonly arm;
|
||||
private readonly gripper;
|
||||
private readonly objectQ: number;
|
||||
private readonly objectV: number;
|
||||
private readonly baseQ: number;
|
||||
private readonly baseV: number;
|
||||
private readonly objectBody: number;
|
||||
private readonly fingerBodies: number[];
|
||||
private readonly goalSupport: number;
|
||||
private readonly site: number;
|
||||
private disposed = false;
|
||||
constructor(private readonly session: SimulationSession) {
|
||||
const { model, module } = session,
|
||||
b = new ModelBindings(model);
|
||||
this.arm = R.armJoints.map((joint) => b.scalarJoint(joint.name));
|
||||
this.gripper = b.scalarJoint(R.gripperJoint);
|
||||
const objectJoint = b.resolve(b.joints, '__mm_object_joint');
|
||||
this.objectQ = Number(model.jnt_qposadr[objectJoint]);
|
||||
this.objectV = Number(model.jnt_dofadr[objectJoint]);
|
||||
const baseJoint = b.resolve(b.joints, R.baseJointName);
|
||||
this.baseQ = Number(model.jnt_qposadr[baseJoint]);
|
||||
this.baseV = Number(model.jnt_dofadr[baseJoint]);
|
||||
this.objectBody = b.resolve(b.bodies, '__mm_object');
|
||||
this.fingerBodies = ['Wrist_Roll_08c-v1', 'Moving_Jaw_08d-v1'].map((name) =>
|
||||
Number(model.body_weldid[b.resolve(b.bodies, name)]),
|
||||
);
|
||||
this.goalSupport = module.mj_name2id(model, 5, '__agent_support_1');
|
||||
this.site = module.mj_name2id(model, 6, P.tcpSite);
|
||||
if (this.site < 0 || this.goalSupport < 0) throw new Error('缺少抓放场景');
|
||||
this.force = new module.DoubleBuffer(6);
|
||||
}
|
||||
sample(): PhysicalSample {
|
||||
if (this.disposed) throw new Error('物理采样器已释放');
|
||||
const { model: m, data: d, module } = this.session;
|
||||
const forces: [number, number] = [0, 0];
|
||||
let supported = false,
|
||||
onGoalSupport = false;
|
||||
const contacts = d.contact;
|
||||
try {
|
||||
for (let i = 0; i < d.ncon; i++) {
|
||||
const contact = contacts.get(i)!;
|
||||
try {
|
||||
const a = Number(m.geom_bodyid[contact.geom1]),
|
||||
b = Number(m.geom_bodyid[contact.geom2]);
|
||||
if (a !== this.objectBody && b !== this.objectBody) continue;
|
||||
const other = a === this.objectBody ? b : a;
|
||||
const geom = a === this.objectBody ? contact.geom2 : contact.geom1;
|
||||
module.mj_contactForce(m, d, i, this.force);
|
||||
const normal = Math.max(0, Number(this.force.GetView()[0]));
|
||||
if (normal <= 0) continue;
|
||||
if (Number(m.body_weldid[other]) === 0) supported = true;
|
||||
if (geom === this.goalSupport) onGoalSupport = true;
|
||||
const finger = this.fingerBodies.indexOf(Number(m.body_weldid[other]));
|
||||
if (finger >= 0) forces[finger] += normal;
|
||||
} finally {
|
||||
contact.delete();
|
||||
}
|
||||
}
|
||||
} finally {
|
||||
contacts.delete();
|
||||
}
|
||||
const q: Float64Array = d.qpos,
|
||||
v: Float64Array = d.qvel;
|
||||
const [w, x, y, z] = Array.from(q.subarray(this.baseQ + 3, this.baseQ + 7));
|
||||
const result: PhysicalSample = {
|
||||
time: Number(d.time),
|
||||
base: Array.from(q.subarray(this.baseQ, this.baseQ + 3)),
|
||||
baseSpeed: Math.hypot(...v.subarray(this.baseV, this.baseV + 3)),
|
||||
yaw: Math.atan2(2 * (w * z + x * y), 1 - 2 * (y * y + z * z)),
|
||||
tilt: Math.acos(Math.max(-1, Math.min(1, 1 - 2 * (x * x + y * y)))),
|
||||
joints: this.arm.map((joint) => q[joint.qposAddress]),
|
||||
peakJointSpeed: Math.max(
|
||||
...[...this.arm, this.gripper].map((joint) => Math.abs(v[joint.qvelAddress])),
|
||||
),
|
||||
opening: Math.max(
|
||||
0,
|
||||
Math.min(
|
||||
1,
|
||||
(q[this.gripper.qposAddress] - R.gripperClosed) / (R.gripperOpen - R.gripperClosed),
|
||||
),
|
||||
),
|
||||
tcp: Array.from(d.site_xpos.slice(this.site * 3, this.site * 3 + 3), Number),
|
||||
object: Array.from(q.subarray(this.objectQ, this.objectQ + 3)),
|
||||
objectSpeed: Math.hypot(...v.subarray(this.objectV, this.objectV + 3)),
|
||||
fingerForces: forces,
|
||||
supported,
|
||||
onGoalSupport,
|
||||
};
|
||||
if (
|
||||
[
|
||||
result.time,
|
||||
result.baseSpeed,
|
||||
result.yaw,
|
||||
result.tilt,
|
||||
result.opening,
|
||||
result.peakJointSpeed,
|
||||
result.objectSpeed,
|
||||
...result.base,
|
||||
...result.joints,
|
||||
...result.tcp,
|
||||
...result.object,
|
||||
...forces,
|
||||
].some((value) => !Number.isFinite(value))
|
||||
)
|
||||
throw new Error('物理状态非有限');
|
||||
return result;
|
||||
}
|
||||
dispose(): void {
|
||||
if (!this.disposed) {
|
||||
this.disposed = true;
|
||||
this.force.delete();
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
export class PickPlaceEvaluator {
|
||||
private graspTime = 0;
|
||||
private stableTime = 0;
|
||||
private reference?: number[];
|
||||
private lastSecureBase?: number[];
|
||||
private readonly transportVector = [0, 0];
|
||||
private lostTime = 0;
|
||||
private lastTime?: number;
|
||||
secure = false;
|
||||
hasGrasped = false;
|
||||
transported = 0;
|
||||
succeeded = false;
|
||||
constructor(
|
||||
readonly initialObject: readonly number[],
|
||||
readonly goal: readonly number[],
|
||||
) {}
|
||||
update(sample: PhysicalSample, dt: number): void {
|
||||
if (!Number.isFinite(dt) || dt <= 0 || dt > P.controlDt) throw new Error('非法物理采样步长');
|
||||
if (this.lastTime !== undefined) {
|
||||
const elapsed = sample.time - this.lastTime;
|
||||
if (Math.abs(elapsed) < 1e-10) return; // Frozen/repeated observation is not stability evidence.
|
||||
if (Math.abs(elapsed - dt) > 1e-6) throw new Error('物理证据时钟不连续');
|
||||
}
|
||||
this.lastTime = sample.time;
|
||||
const offset = sample.object.map((v, i) => v - sample.tcp[i]);
|
||||
const fingers = sample.fingerForces.every((force) => force >= P.fingerForceMin);
|
||||
if (!fingers) this.reference = undefined;
|
||||
else this.reference ??= offset;
|
||||
const drift = this.reference
|
||||
? Math.hypot(...offset.map((v, i) => v - this.reference![i]))
|
||||
: Infinity;
|
||||
const lifted =
|
||||
fingers &&
|
||||
!sample.supported &&
|
||||
sample.object[2] - this.initialObject[2] >= P.liftMin &&
|
||||
drift <= P.relativeDriftMax;
|
||||
this.graspTime = lifted ? this.graspTime + dt : 0;
|
||||
this.secure = this.graspTime >= P.graspStableTime;
|
||||
if (this.secure) {
|
||||
this.hasGrasped = true;
|
||||
if (this.lastSecureBase)
|
||||
for (let i = 0; i < 2; i++)
|
||||
this.transportVector[i] += sample.base[i] - this.lastSecureBase[i];
|
||||
this.transported = Math.max(this.transported, Math.hypot(...this.transportVector));
|
||||
}
|
||||
this.lastSecureBase = this.secure ? [...sample.base] : undefined;
|
||||
this.lostTime = !fingers && !sample.onGoalSupport ? this.lostTime + dt : 0;
|
||||
if (this.lostTime >= 0.1) {
|
||||
this.hasGrasped = false;
|
||||
this.transported = 0;
|
||||
this.transportVector.fill(0);
|
||||
}
|
||||
const released =
|
||||
sample.opening >= 0.65 && sample.fingerForces.every((force) => force < P.fingerForceMin);
|
||||
const placed =
|
||||
this.hasGrasped &&
|
||||
this.transported >= P.transportMin &&
|
||||
released &&
|
||||
sample.onGoalSupport &&
|
||||
Math.hypot(...sample.object.map((v, i) => v - this.goal[i])) < P.placementTolerance &&
|
||||
sample.objectSpeed < P.placementSpeedMax &&
|
||||
Math.hypot(...sample.object.map((v, i) => v - sample.tcp[i])) >= P.retreatMin;
|
||||
this.stableTime = placed ? this.stableTime + dt : 0;
|
||||
this.succeeded = this.stableTime >= P.placementStableTime;
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,37 @@
|
||||
import { describe, expect, it } from 'vitest';
|
||||
import { composePickPlaceScene, PICK_PLACE_ROBOT, seedPositions } from './PickPlaceScene';
|
||||
describe('受限任务场景与可见支撑台', () => {
|
||||
it('目标编辑保留源位置,拒绝范围外/非有限/高度改动', () => {
|
||||
expect(seedPositions(0, [0.237, 0.565, 0.128])).toEqual({
|
||||
object: [0.257, 0.015, 0.128],
|
||||
goal: [0.237, 0.565, 0.128],
|
||||
});
|
||||
for (const goal of [
|
||||
[NaN, 0.615, 0.128],
|
||||
[0.8, 0.6, 0.128],
|
||||
[0.257, 0.1, 0.128],
|
||||
[0.257, 0.615, 0.13],
|
||||
[0.257, 0.615],
|
||||
])
|
||||
expect(() => seedPositions(0, goal)).toThrow('目标');
|
||||
});
|
||||
it('两个实体支撑台进入视觉组,仍保留物理碰撞;不覆盖用户模型', () => {
|
||||
const input = new TextEncoder().encode(
|
||||
`<mujoco><worldbody><body name="__mm_object"><geom type="box" size=".018 .018 .018"/></body><body name="__mm_goal"/><body name="${PICK_PLACE_ROBOT.eefBodyName}"/></worldbody></mujoco>`,
|
||||
);
|
||||
const result = composePickPlaceScene(input, 0, true, [0.257, 0.617, 0.128]);
|
||||
const doc = new DOMParser().parseFromString(
|
||||
new TextDecoder().decode(result),
|
||||
'application/xml',
|
||||
);
|
||||
expect(doc.querySelector('geom[name="__agent_support_1"]')?.getAttribute('group')).toBe('1');
|
||||
expect(doc.querySelector('geom[name="__agent_support_1"]')?.getAttribute('pos')).toBe(
|
||||
'0.257 0.617 0.055',
|
||||
);
|
||||
expect(doc.querySelector('geom[name="__agent_support_1"]')?.getAttribute('contype')).not.toBe(
|
||||
'0',
|
||||
);
|
||||
expect(new TextDecoder().decode(input)).not.toContain('__agent_');
|
||||
expect(() => composePickPlaceScene(result, 0, true)).toThrow('命名空间');
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,73 @@
|
||||
import preset from '../../../../contracts/lekiwi-pick-place-v1.json';
|
||||
import { MOBILE_ROBOTS } from '../RobotDescriptor';
|
||||
import { composeMobileScene } from '../SceneComposer';
|
||||
|
||||
export const PICK_PLACE = preset;
|
||||
export const PICK_PLACE_ROBOT = MOBILE_ROBOTS.find((robot) => robot.id === preset.robotId)!;
|
||||
export const TASK_GOAL_LIMITS = { min: [0.237, 0.565, 0.128], max: [0.277, 0.665, 0.128] } as const;
|
||||
export interface PickPlaceTaskConfig {
|
||||
seed: number;
|
||||
goal?: readonly number[];
|
||||
}
|
||||
export function seedPositions(
|
||||
seed: number,
|
||||
goal?: readonly number[],
|
||||
): { object: number[]; goal: number[] } {
|
||||
if (
|
||||
goal &&
|
||||
(goal.length !== 3 ||
|
||||
goal.some(
|
||||
(v, i) => !Number.isFinite(v) || v < TASK_GOAL_LIMITS.min[i] || v > TASK_GOAL_LIMITS.max[i],
|
||||
))
|
||||
)
|
||||
throw new Error('目标超出首版受限区域;支撑高度不可更改');
|
||||
const item = preset.frozenSeeds.find((candidate) => candidate.id === seed);
|
||||
if (!item) throw new Error('未知抓放标定种子');
|
||||
const offset = [...item.offset, 0];
|
||||
return {
|
||||
object: preset.object.position.map((v, i) => v + offset[i]),
|
||||
goal: goal ? [...goal] : preset.goal.position.map((v, i) => v + offset[i]),
|
||||
};
|
||||
}
|
||||
|
||||
/** Task-only scene, never rewrites a user's map or the RL scene contract. */
|
||||
export function composePickPlaceScene(
|
||||
source: Uint8Array,
|
||||
seed = 0,
|
||||
alreadyMobile = false,
|
||||
goal?: readonly number[],
|
||||
): Uint8Array {
|
||||
const doc = new DOMParser().parseFromString(
|
||||
new TextDecoder().decode(alreadyMobile ? source : composeMobileScene(source, PICK_PLACE_ROBOT)),
|
||||
'application/xml',
|
||||
);
|
||||
if (doc.querySelector('parsererror')) throw new Error('抓放场景 XML 非法');
|
||||
if (doc.querySelector('[name="__agent_tcp"], [name="__agent_support_0"]'))
|
||||
throw new Error('抓放任务命名空间冲突');
|
||||
const positions = seedPositions(seed, goal);
|
||||
const object = doc.querySelector('body[name="__mm_object"]')!;
|
||||
object.setAttribute('pos', positions.object.join(' '));
|
||||
object.querySelector('geom')!.setAttribute('mass', String(preset.object.mass));
|
||||
doc.querySelector('body[name="__mm_goal"]')!.setAttribute('pos', positions.goal.join(' '));
|
||||
const world = doc.querySelector('mujoco > worldbody')!;
|
||||
for (const [i, position] of [positions.object, positions.goal].entries()) {
|
||||
const geom = doc.createElement('geom');
|
||||
for (const [name, value] of Object.entries({
|
||||
name: `__agent_support_${i}`,
|
||||
type: 'box',
|
||||
group: '1', // Simple support solids are both visible and collidable.
|
||||
pos: `${position[0]} ${position[1]} ${preset.supportHeight / 2}`,
|
||||
size: preset.supportHalfSize.join(' '),
|
||||
friction: '1 .005 .0001',
|
||||
rgba: '.25 .35 .45 1',
|
||||
}))
|
||||
geom.setAttribute(name, value);
|
||||
world.append(geom);
|
||||
}
|
||||
const site = doc.createElement('site');
|
||||
site.setAttribute('name', preset.tcpSite);
|
||||
site.setAttribute('pos', preset.tcpOffset.join(' '));
|
||||
site.setAttribute('size', '.002');
|
||||
doc.querySelector(`body[name="${PICK_PLACE_ROBOT.eefBodyName}"]`)!.append(site);
|
||||
return new TextEncoder().encode(new XMLSerializer().serializeToString(doc));
|
||||
}
|
||||
@@ -0,0 +1,67 @@
|
||||
import { RequestGate, type RequestStamp, type Stamped } from './RequestGate';
|
||||
const stamp: RequestStamp = {
|
||||
runId: 'run',
|
||||
sceneRevision: 1,
|
||||
sequence: 1,
|
||||
planRevision: 1,
|
||||
requestId: 'request',
|
||||
};
|
||||
function deferred<T>() {
|
||||
let resolve!: (v: T) => void;
|
||||
const promise = new Promise<T>((r) => {
|
||||
resolve = r;
|
||||
});
|
||||
return { promise, resolve };
|
||||
}
|
||||
it('accepts only the matching request identity', async () => {
|
||||
const gate = new RequestGate();
|
||||
await expect(gate.request(stamp, 1000, async () => ({ stamp, value: 1 }))).resolves.toBe(1);
|
||||
for (const key of Object.keys(stamp) as (keyof RequestStamp)[])
|
||||
await expect(
|
||||
gate.request(stamp, 1000, async () => ({
|
||||
stamp: { ...stamp, [key]: 'different' },
|
||||
value: 1,
|
||||
})),
|
||||
).rejects.toThrow(/过期/);
|
||||
});
|
||||
it('cancellation rejects immediately even when the provider ignores AbortSignal', async () => {
|
||||
const gate = new RequestGate(),
|
||||
work = deferred<Stamped<number>>();
|
||||
const result = gate.request(stamp, 1000, () => work.promise);
|
||||
const rejected = expect(result).rejects.toThrow('pause');
|
||||
gate.invalidate('pause');
|
||||
await rejected;
|
||||
work.resolve({ stamp, value: 99 });
|
||||
await Promise.resolve();
|
||||
});
|
||||
it('replacement isolates a delayed old result from a new run', async () => {
|
||||
const gate = new RequestGate(),
|
||||
work = deferred<Stamped<number>>();
|
||||
const old = gate.request(stamp, 1000, () => work.promise);
|
||||
const rejected = expect(old).rejects.toThrow('替换');
|
||||
const next = await gate.request({ ...stamp, requestId: 'new' }, 1000, async () => ({
|
||||
stamp: { ...stamp, requestId: 'new' },
|
||||
value: 2,
|
||||
}));
|
||||
await rejected;
|
||||
work.resolve({ stamp, value: 1 });
|
||||
expect(next).toBe(2);
|
||||
});
|
||||
it('timeout aborts the operation, does not fall back, and permits a new request', async () => {
|
||||
vi.useFakeTimers();
|
||||
try {
|
||||
const gate = new RequestGate();
|
||||
let signal: AbortSignal | undefined;
|
||||
const result = gate.request(stamp, 10, async (s) => {
|
||||
signal = s;
|
||||
return new Promise<never>(() => {});
|
||||
});
|
||||
const rejected = expect(result).rejects.toThrow('超时');
|
||||
await vi.advanceTimersByTimeAsync(11);
|
||||
await rejected;
|
||||
expect(signal?.aborted).toBe(true);
|
||||
await expect(gate.request(stamp, 10, async () => ({ stamp, value: 3 }))).resolves.toBe(3);
|
||||
} finally {
|
||||
vi.useRealTimers();
|
||||
}
|
||||
});
|
||||
@@ -0,0 +1,57 @@
|
||||
export interface RequestStamp {
|
||||
runId: string;
|
||||
sceneRevision: number;
|
||||
sequence: number;
|
||||
planRevision: number;
|
||||
requestId: string;
|
||||
}
|
||||
export interface Stamped<T> {
|
||||
stamp: RequestStamp;
|
||||
value: T;
|
||||
}
|
||||
const sameStamp = (a: RequestStamp, b: RequestStamp) =>
|
||||
a.runId === b.runId &&
|
||||
a.sceneRevision === b.sceneRevision &&
|
||||
a.sequence === b.sequence &&
|
||||
a.planRevision === b.planRevision &&
|
||||
a.requestId === b.requestId;
|
||||
|
||||
/** One outstanding decision, explicit cancellation, and rejection of providers
|
||||
* that ignore AbortSignal or reply against a different run/scene/observation. */
|
||||
export class RequestGate {
|
||||
private epoch = 0;
|
||||
private controller?: AbortController;
|
||||
invalidate(reason = '请求已取消'): void {
|
||||
this.epoch++;
|
||||
this.controller?.abort(reason);
|
||||
this.controller = undefined;
|
||||
}
|
||||
async request<T>(
|
||||
stamp: RequestStamp,
|
||||
timeoutMs: number,
|
||||
operation: (signal: AbortSignal) => Promise<Stamped<T>>,
|
||||
): Promise<T> {
|
||||
this.invalidate('请求已替换');
|
||||
const epoch = this.epoch,
|
||||
expected = { ...stamp },
|
||||
controller = new AbortController();
|
||||
this.controller = controller;
|
||||
let rejectAbort!: (error: Error) => void;
|
||||
const aborted = new Promise<never>((_, reject) => {
|
||||
rejectAbort = reject;
|
||||
});
|
||||
const listener = () => rejectAbort(new Error(String(controller.signal.reason ?? '请求已取消')));
|
||||
controller.signal.addEventListener('abort', listener, { once: true });
|
||||
const timer = setTimeout(() => controller.abort('模型请求超时'), timeoutMs);
|
||||
try {
|
||||
const response = await Promise.race([operation(controller.signal), aborted]);
|
||||
if (epoch !== this.epoch || controller.signal.aborted || !sameStamp(expected, response.stamp))
|
||||
throw new Error('过期或不匹配的模型响应');
|
||||
return response.value;
|
||||
} finally {
|
||||
clearTimeout(timer);
|
||||
controller.signal.removeEventListener('abort', listener);
|
||||
if (epoch === this.epoch) this.controller = undefined;
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,95 @@
|
||||
import { cleanup, fireEvent, render, screen, waitFor } from '@testing-library/react';
|
||||
import { afterEach, expect, it, vi } from 'vitest';
|
||||
import { WebsiteModelSettings } from './WebsiteModelSettings';
|
||||
import { DecisionClient } from './DecisionClient';
|
||||
|
||||
afterEach(() => {
|
||||
cleanup();
|
||||
vi.restoreAllMocks();
|
||||
});
|
||||
const status = {
|
||||
ready: false,
|
||||
configuration: {
|
||||
llm: { provider: 'deepseek', model: 'deepseek-flash', hasKey: false },
|
||||
jev: { hasKey: false },
|
||||
},
|
||||
};
|
||||
function fixture() {
|
||||
const call = vi.fn(async (path: string) => {
|
||||
if (path === '/models')
|
||||
return {
|
||||
openrouterAvailable: true,
|
||||
models: [
|
||||
{ provider: 'deepseek', id: 'deepseek-flash', name: 'DeepSeek' },
|
||||
{ provider: 'openrouter', id: 'vendor/model', name: 'Router Model' },
|
||||
],
|
||||
};
|
||||
if (path === '/configuration') return { ...status, ready: true };
|
||||
if (path === '/codex/login') throw new Error('codex_rpc_timeout');
|
||||
return status;
|
||||
});
|
||||
const client = { call, receipts: [], mode: 'api' } as unknown as DecisionClient;
|
||||
vi.spyOn(DecisionClient, 'website').mockResolvedValue(client);
|
||||
return { call, client };
|
||||
}
|
||||
it('只有模型与两角色密钥,原子保存,不持久化/自动测试', async () => {
|
||||
const { call, client } = fixture();
|
||||
const onClient = vi.fn();
|
||||
const storage = vi.spyOn(Storage.prototype, 'setItem');
|
||||
render(<WebsiteModelSettings onClient={onClient} onInvalidate={vi.fn()} />);
|
||||
await waitFor(() => expect(screen.getByRole('button', { name: '保存设置' })).not.toBeDisabled());
|
||||
expect(screen.queryByLabelText('本机服务令牌')).not.toBeInTheDocument();
|
||||
expect(screen.queryByLabelText('LLM Base URL')).not.toBeInTheDocument();
|
||||
fireEvent.change(screen.getByLabelText('LLM API_KEY'), { target: { value: 'fixture-llm' } });
|
||||
fireEvent.change(screen.getByLabelText('Jev API_KEY(OpenRouter)'), {
|
||||
target: { value: 'fixture-jev' },
|
||||
});
|
||||
fireEvent.click(screen.getByRole('button', { name: '保存设置' }));
|
||||
await screen.findByText(/设置已保存/);
|
||||
expect(call).toHaveBeenCalledWith(
|
||||
'/configuration',
|
||||
'PUT',
|
||||
{
|
||||
llm: { provider: 'deepseek', model: 'deepseek-flash', apiKey: 'fixture-llm' },
|
||||
jev: { apiKey: 'fixture-jev' },
|
||||
},
|
||||
expect.any(AbortSignal),
|
||||
);
|
||||
expect(screen.getByLabelText('LLM API_KEY')).toHaveValue('');
|
||||
expect(screen.getByLabelText('Jev API_KEY(OpenRouter)')).toHaveValue('');
|
||||
expect(onClient).toHaveBeenLastCalledWith(client);
|
||||
expect(storage).not.toHaveBeenCalled();
|
||||
expect(call.mock.calls.some(([path]) => ['/plan', '/test', '/decide'].includes(path))).toBe(
|
||||
false,
|
||||
);
|
||||
});
|
||||
it('换提供方清空密钥,保留订阅入口并明确网络不可用', async () => {
|
||||
const { call } = fixture();
|
||||
render(<WebsiteModelSettings onClient={vi.fn()} onInvalidate={vi.fn()} />);
|
||||
await waitFor(() => expect(screen.getByRole('button', { name: '保存设置' })).not.toBeDisabled());
|
||||
fireEvent.change(screen.getByLabelText('LLM API_KEY'), { target: { value: 'fixture-old' } });
|
||||
fireEvent.change(screen.getByLabelText('模型选择'), {
|
||||
target: { value: 'openrouter:vendor/model' },
|
||||
});
|
||||
expect(screen.getByLabelText('LLM API_KEY')).toHaveValue('');
|
||||
fireEvent.click(screen.getByRole('button', { name: 'ChatGPT 订阅' }));
|
||||
fireEvent.click(screen.getByRole('button', { name: '登录 ChatGPT 订阅' }));
|
||||
await screen.findByText(/官方订阅服务连接超时/);
|
||||
expect(call.mock.calls.filter(([path]) => path === '/configuration')).toHaveLength(0);
|
||||
});
|
||||
it('同源客户端自动携带 Cookie/CSRF,但不给原本机模式放开任意远端', async () => {
|
||||
const fetcher = vi.fn<typeof fetch>(async (_url, opts) => {
|
||||
if (String(_url).endsWith('/session'))
|
||||
return Response.json({ csrfToken: 'csrf-fixture', configVersion: 2 });
|
||||
expect(opts?.credentials).toBe('same-origin');
|
||||
expect(opts?.headers).toMatchObject({
|
||||
'X-CSRF-Token': 'csrf-fixture',
|
||||
'X-Config-Version': '2',
|
||||
});
|
||||
expect(opts?.headers).not.toHaveProperty('Authorization');
|
||||
return Response.json({});
|
||||
});
|
||||
const c = await DecisionClient.website(undefined, fetcher);
|
||||
await c.call('/configuration', 'PUT', {});
|
||||
expect(() => new DecisionClient('https://evil.test', 'token')).toThrow();
|
||||
});
|
||||
@@ -0,0 +1,436 @@
|
||||
import { useEffect, useRef, useState } from 'react';
|
||||
import { Button } from '../../components/ui';
|
||||
import { DecisionClient } from './DecisionClient';
|
||||
|
||||
interface Props {
|
||||
onClient(client: DecisionClient | undefined): void;
|
||||
onInvalidate(): void;
|
||||
}
|
||||
interface Configuration {
|
||||
llm: { provider: string; model: string; hasKey: boolean };
|
||||
jev: { hasKey: boolean };
|
||||
}
|
||||
interface Status {
|
||||
ready: boolean;
|
||||
configuration: Configuration;
|
||||
}
|
||||
interface Model {
|
||||
provider: string;
|
||||
id: string;
|
||||
name: string;
|
||||
}
|
||||
const inputClass = 'w-full rounded border border-border bg-surface px-2 py-1.5 text-xs';
|
||||
const errors: Record<string, string> = {
|
||||
session_expired: '本次会话已过期,请重新打开模型设置并填写凭据。',
|
||||
configuration_changed: '配置已在其他页面变更,请重新打开设置,不会使用旧配置继续任务。',
|
||||
api_key_required: '请填写对应的 API_KEY。切换提供方后需要重新填写。',
|
||||
server_busy: '服务器当前繁忙,请稍后手动重试。',
|
||||
ip_rate_limit: '请求次数已达上限,请稍后再试。',
|
||||
subscription_capacity: '订阅登录名额暂满,请稍后再试;不会自动切换付费 API。',
|
||||
codex_chatgpt_login_required: '请先完成官方订阅登录。',
|
||||
codex_rpc_failed: '当前订阅服务不可用,可能是网络或账号限制;不会切换付费 API。',
|
||||
codex_rpc_timeout: '官方订阅服务连接超时;API 模式仍可手动使用。',
|
||||
codex_not_installed: '当前服务器未启用订阅服务。',
|
||||
model_unavailable: '当前模型不可用,请重新选择;不会自动替换。',
|
||||
csrf_required: '会话验证失效,请重新打开模型设置。',
|
||||
};
|
||||
function message(error: unknown) {
|
||||
const code = error instanceof Error ? error.message : '请求失败';
|
||||
return errors[code] ?? `请求未完成:${code}`;
|
||||
}
|
||||
|
||||
export function WebsiteModelSettings({ onClient, onInvalidate }: Props) {
|
||||
const callbacks = useRef({ onClient, onInvalidate });
|
||||
useEffect(() => {
|
||||
callbacks.current = { onClient, onInvalidate };
|
||||
}, [onClient, onInvalidate]);
|
||||
const [client, setClient] = useState<DecisionClient>();
|
||||
const [config, setConfig] = useState<Configuration>();
|
||||
const [mode, setMode] = useState<'api' | 'codex'>('api');
|
||||
const [selection, setSelection] = useState('deepseek:deepseek-flash');
|
||||
const [apiKey, setApiKey] = useState('');
|
||||
const [jevKey, setJevKey] = useState('');
|
||||
const [models, setModels] = useState<Model[]>([]);
|
||||
const [search, setSearch] = useState('');
|
||||
const [codexModels, setCodexModels] = useState<{ id: string; name: string }[]>([]);
|
||||
const [codexModel, setCodexModel] = useState('');
|
||||
const [loggedIn, setLoggedIn] = useState(false);
|
||||
const [device, setDevice] = useState<{ verificationUrl: string; userCode: string }>();
|
||||
const [busy, setBusy] = useState(true);
|
||||
const [dirty, setDirty] = useState(false);
|
||||
const [notice, setNotice] = useState('正在连接网站模型服务…');
|
||||
const generation = useRef(0);
|
||||
const pending = useRef<AbortController | undefined>(undefined);
|
||||
|
||||
useEffect(() => {
|
||||
const abort = new AbortController();
|
||||
pending.current = abort;
|
||||
let alive = true;
|
||||
void (async () => {
|
||||
try {
|
||||
const c = await DecisionClient.website(abort.signal);
|
||||
const status = (await c.call('/status', 'GET', undefined, abort.signal)) as Status;
|
||||
if (!alive) return;
|
||||
setClient(c);
|
||||
setConfig(status.configuration);
|
||||
const llm = status.configuration.llm;
|
||||
setMode(llm.provider === 'codex' ? 'codex' : 'api');
|
||||
if (llm.provider === 'codex') setCodexModel(llm.model);
|
||||
else setSelection(`${llm.provider}:${llm.model}`);
|
||||
callbacks.current.onClient(status.ready ? c : undefined);
|
||||
setNotice('会话已就绪。保存不会调用模型;开始真实任务或测试连接可能计费。');
|
||||
const list = (await c.call('/models', 'GET', undefined, abort.signal)) as {
|
||||
models: Model[];
|
||||
openrouterAvailable: boolean;
|
||||
};
|
||||
if (!alive) return;
|
||||
setModels(list.models);
|
||||
if (!list.openrouterAvailable)
|
||||
setNotice('OpenRouter 模型目录暂不可用;已保存的型号不会自动替换。');
|
||||
} catch (error) {
|
||||
if (alive) setNotice(message(error));
|
||||
} finally {
|
||||
if (alive) setBusy(false);
|
||||
}
|
||||
})();
|
||||
const cancelPending = () => {
|
||||
generation.current++;
|
||||
pending.current?.abort();
|
||||
};
|
||||
return () => {
|
||||
alive = false;
|
||||
cancelPending();
|
||||
abort.abort();
|
||||
};
|
||||
}, []);
|
||||
|
||||
useEffect(() => {
|
||||
if (!device || !client) return;
|
||||
const abort = new AbortController();
|
||||
const deadline = Date.now() + 600_000;
|
||||
let timer: ReturnType<typeof setTimeout>;
|
||||
const poll = async () => {
|
||||
if (Date.now() >= deadline) {
|
||||
setDevice(undefined);
|
||||
setNotice('设备码已过期,请重新发起登录。');
|
||||
return;
|
||||
}
|
||||
try {
|
||||
const status = (await client.call('/codex/status', 'GET', undefined, abort.signal)) as {
|
||||
loggedIn?: boolean;
|
||||
};
|
||||
if (abort.signal.aborted) return;
|
||||
if (status.loggedIn) {
|
||||
const list = (await client.call('/codex/models', 'GET', undefined, abort.signal)) as {
|
||||
models: { id: string; name: string }[];
|
||||
};
|
||||
if (abort.signal.aborted) return;
|
||||
setLoggedIn(true);
|
||||
setCodexModels(list.models);
|
||||
setDevice(undefined);
|
||||
setNotice('官方登录完成;请选择订阅模型并保存。');
|
||||
return;
|
||||
}
|
||||
} catch (error) {
|
||||
if (abort.signal.aborted) return;
|
||||
setDevice(undefined);
|
||||
setNotice(message(error));
|
||||
return;
|
||||
}
|
||||
timer = setTimeout(() => void poll(), 3000);
|
||||
};
|
||||
timer = setTimeout(() => void poll(), 3000);
|
||||
return () => {
|
||||
clearTimeout(timer);
|
||||
abort.abort();
|
||||
};
|
||||
}, [device, client]);
|
||||
|
||||
function invalidate() {
|
||||
callbacks.current.onInvalidate();
|
||||
callbacks.current.onClient(undefined);
|
||||
setDirty(true);
|
||||
}
|
||||
async function run(operation: (c: DecisionClient, signal: AbortSignal) => Promise<void>) {
|
||||
if (!client || busy) return;
|
||||
const current = ++generation.current;
|
||||
const abort = new AbortController();
|
||||
pending.current = abort;
|
||||
setBusy(true);
|
||||
try {
|
||||
await operation(client, abort.signal);
|
||||
} catch (error) {
|
||||
if (current === generation.current) {
|
||||
setNotice(message(error));
|
||||
callbacks.current.onClient(undefined);
|
||||
}
|
||||
} finally {
|
||||
if (current === generation.current) setBusy(false);
|
||||
}
|
||||
}
|
||||
async function save() {
|
||||
await run(async (c, signal) => {
|
||||
invalidate();
|
||||
const split = selection.indexOf(':');
|
||||
const provider = mode === 'codex' ? 'codex' : selection.slice(0, split);
|
||||
const model = mode === 'codex' ? codexModel : selection.slice(split + 1);
|
||||
const result = (await c.call(
|
||||
'/configuration',
|
||||
'PUT',
|
||||
{
|
||||
llm: { provider, model, ...(mode === 'api' && apiKey ? { apiKey } : {}) },
|
||||
jev: { ...(jevKey ? { apiKey: jevKey } : {}) },
|
||||
},
|
||||
signal,
|
||||
)) as Status;
|
||||
if (signal.aborted) return;
|
||||
setConfig(result.configuration);
|
||||
setApiKey('');
|
||||
setJevKey('');
|
||||
setDirty(false);
|
||||
callbacks.current.onClient(result.ready ? c : undefined);
|
||||
setNotice('设置已保存;输入框已清空。密钥仅在本次服务会话内存中。');
|
||||
});
|
||||
}
|
||||
async function account(operation: 'login' | 'status' | 'cancel' | 'logout') {
|
||||
await run(async (c, signal) => {
|
||||
const post = operation !== 'status';
|
||||
if (post) invalidate();
|
||||
const result = (await c.call(
|
||||
`/codex/${operation}`,
|
||||
post ? 'POST' : 'GET',
|
||||
post ? {} : undefined,
|
||||
signal,
|
||||
)) as {
|
||||
loggedIn?: boolean;
|
||||
verificationUrl?: string;
|
||||
userCode?: string;
|
||||
};
|
||||
if (signal.aborted) return;
|
||||
if (operation === 'login') {
|
||||
const url = new URL(result.verificationUrl ?? '');
|
||||
if (
|
||||
url.protocol !== 'https:' ||
|
||||
url.hostname !== 'auth.openai.com' ||
|
||||
url.username ||
|
||||
url.password ||
|
||||
(url.port && url.port !== '443')
|
||||
)
|
||||
throw new Error('官方登录地址校验失败');
|
||||
setDevice({ verificationUrl: url.href, userCode: result.userCode ?? '' });
|
||||
setNotice('请在官方页面输入一次性码。账户可能需要先开启设备码登录;完成后刷新登录状态。');
|
||||
} else {
|
||||
setLoggedIn(Boolean(result.loggedIn));
|
||||
if (operation !== 'status' || result.loggedIn) setDevice(undefined);
|
||||
if (result.loggedIn) {
|
||||
const list = (await c.call('/codex/models', 'GET', undefined, signal)) as {
|
||||
models: { id: string; name: string }[];
|
||||
};
|
||||
if (signal.aborted) return;
|
||||
setCodexModels(list.models);
|
||||
setNotice('官方登录已完成;选择模型并保存。登录不代表已通过真实推理。');
|
||||
} else {
|
||||
setCodexModels([]);
|
||||
setNotice(
|
||||
operation === 'status' ? '尚未完成官方登录;当前网络可能不可用。' : '订阅凭据已清除。',
|
||||
);
|
||||
}
|
||||
}
|
||||
});
|
||||
}
|
||||
const matching = models.filter((m) =>
|
||||
`${m.provider} ${m.id} ${m.name}`.toLowerCase().includes(search.toLowerCase()),
|
||||
);
|
||||
return (
|
||||
<div className="space-y-3 text-xs" aria-label="网站模型设置">
|
||||
<p className="text-text-secondary">
|
||||
凭据经 HTTPS 发送到网站后端,仅存会话内存;空闲 30 分钟过期。模型文件仍在浏览器内。
|
||||
</p>
|
||||
<div className="flex gap-2" role="group" aria-label="LLM 连接方式">
|
||||
{(['api', 'codex'] as const).map((value) => (
|
||||
<Button
|
||||
key={value}
|
||||
disabled={busy}
|
||||
aria-pressed={mode === value}
|
||||
onClick={() => {
|
||||
if (mode !== value) {
|
||||
invalidate();
|
||||
setMode(value);
|
||||
setApiKey('');
|
||||
setNotice('连接方式已修改,请完成设置并保存。');
|
||||
}
|
||||
}}
|
||||
>
|
||||
{value === 'api' ? 'API_KEY' : 'ChatGPT 订阅'}
|
||||
</Button>
|
||||
))}
|
||||
</div>
|
||||
<fieldset disabled={busy || !client} className="space-y-2">
|
||||
{mode === 'api' ? (
|
||||
<>
|
||||
<label className="block">
|
||||
LLM API_KEY
|
||||
<input
|
||||
className={inputClass}
|
||||
type="password"
|
||||
autoComplete="off"
|
||||
value={apiKey}
|
||||
placeholder={
|
||||
config?.llm.hasKey ? '已设置;留空保持同提供方密钥' : '填写自己的 API_KEY'
|
||||
}
|
||||
onChange={(e) => {
|
||||
invalidate();
|
||||
setApiKey(e.target.value);
|
||||
}}
|
||||
/>
|
||||
</label>
|
||||
<label className="block">
|
||||
搜索模型
|
||||
<input
|
||||
className={inputClass}
|
||||
type="search"
|
||||
value={search}
|
||||
onChange={(e) => setSearch(e.target.value)}
|
||||
placeholder="DeepSeek / OpenRouter"
|
||||
/>
|
||||
</label>
|
||||
<label className="block">
|
||||
模型选择
|
||||
<select
|
||||
className={inputClass}
|
||||
value={selection}
|
||||
onChange={(e) => {
|
||||
invalidate();
|
||||
if (e.target.value.split(':')[0] !== selection.split(':')[0]) setApiKey('');
|
||||
setSelection(e.target.value);
|
||||
}}
|
||||
>
|
||||
{!matching.some((m) => `${m.provider}:${m.id}` === selection) && (
|
||||
<option value={selection}>{selection}</option>
|
||||
)}
|
||||
{['deepseek', 'openrouter'].map((provider) => (
|
||||
<optgroup
|
||||
key={provider}
|
||||
label={provider === 'deepseek' ? 'DeepSeek' : 'OpenRouter'}
|
||||
>
|
||||
{matching
|
||||
.filter((m) => m.provider === provider)
|
||||
.map((m) => (
|
||||
<option key={m.id} value={`${m.provider}:${m.id}`}>
|
||||
{m.name} · {m.id}
|
||||
</option>
|
||||
))}
|
||||
</optgroup>
|
||||
))}
|
||||
</select>
|
||||
</label>
|
||||
</>
|
||||
) : (
|
||||
<div className="space-y-2">
|
||||
<p>官方订阅登录(实验性)。当前服务器可能无法连接官方服务;不会自动转付费 API。</p>
|
||||
<div className="flex flex-wrap gap-2">
|
||||
<Button onClick={() => void account('login')}>登录 ChatGPT 订阅</Button>
|
||||
<Button onClick={() => void account('status')}>刷新登录状态</Button>
|
||||
<Button onClick={() => void account('cancel')}>取消登录</Button>
|
||||
<Button onClick={() => void account('logout')}>退出订阅</Button>
|
||||
</div>
|
||||
{device && (
|
||||
<div className="space-y-1">
|
||||
<a
|
||||
className="underline"
|
||||
href={device.verificationUrl}
|
||||
target="_blank"
|
||||
rel="noopener noreferrer"
|
||||
>
|
||||
打开 OpenAI 官方验证页
|
||||
</a>
|
||||
<p>
|
||||
一次性码:<strong>{device.userCode}</strong>(10 分钟内完成,过期请重新登录)
|
||||
</p>
|
||||
</div>
|
||||
)}
|
||||
<p>{loggedIn ? '订阅已登录' : '订阅尚未确认登录'}</p>
|
||||
<label className="block">
|
||||
订阅模型
|
||||
<select
|
||||
className={inputClass}
|
||||
value={codexModel}
|
||||
onChange={(e) => {
|
||||
invalidate();
|
||||
setCodexModel(e.target.value);
|
||||
}}
|
||||
>
|
||||
<option value="">登录后选择官方可用模型</option>
|
||||
{codexModels.map((m) => (
|
||||
<option key={m.id} value={m.id}>
|
||||
{m.name}
|
||||
</option>
|
||||
))}
|
||||
</select>
|
||||
</label>
|
||||
</div>
|
||||
)}
|
||||
<label className="block">
|
||||
Jev API_KEY(OpenRouter)
|
||||
<input
|
||||
className={inputClass}
|
||||
type="password"
|
||||
autoComplete="off"
|
||||
value={jevKey}
|
||||
placeholder={config?.jev.hasKey ? '已设置;留空保持' : '填写 Jev 的 OpenRouter API_KEY'}
|
||||
onChange={(e) => {
|
||||
invalidate();
|
||||
setJevKey(e.target.value);
|
||||
}}
|
||||
/>
|
||||
</label>
|
||||
<Button onClick={() => void save()}>保存设置</Button>
|
||||
<details>
|
||||
<summary className="cursor-pointer">连接测试与凭据管理</summary>
|
||||
<div className="mt-2 flex flex-wrap gap-2">
|
||||
{(['llm', 'jev'] as const).map((role) => (
|
||||
<Button
|
||||
key={role}
|
||||
disabled={dirty || !config?.jev.hasKey}
|
||||
onClick={() =>
|
||||
void run(async (c, signal) => {
|
||||
callbacks.current.onInvalidate();
|
||||
await c.call('/test', 'POST', { role }, signal);
|
||||
if (!signal.aborted)
|
||||
setNotice(
|
||||
`${role.toUpperCase()} 连接测试通过;真实 API 可能计费,订阅测试仅验证工具门禁。`,
|
||||
);
|
||||
})
|
||||
}
|
||||
>
|
||||
测试 {role.toUpperCase()}(可能计费)
|
||||
</Button>
|
||||
))}
|
||||
<Button
|
||||
onClick={() =>
|
||||
void run(async (c, signal) => {
|
||||
invalidate();
|
||||
await c.call('/session', 'DELETE', {}, signal);
|
||||
if (signal.aborted) return;
|
||||
setClient(undefined);
|
||||
setConfig(undefined);
|
||||
setDevice(undefined);
|
||||
setApiKey('');
|
||||
setJevKey('');
|
||||
setLoggedIn(false);
|
||||
setNotice('本次凭据已清除。关闭并重新打开模型设置可创建新会话。');
|
||||
})
|
||||
}
|
||||
>
|
||||
清除本次凭据
|
||||
</Button>
|
||||
</div>
|
||||
</details>
|
||||
</fieldset>
|
||||
<p role="status" className="break-words">
|
||||
{busy ? '正在连接…' : notice}
|
||||
</p>
|
||||
{dirty && <p className="text-warning">设置未保存,真实任务暂不可用。</p>}
|
||||
</div>
|
||||
);
|
||||
}
|
||||
@@ -0,0 +1,93 @@
|
||||
import schema from '../../../../contracts/lekiwi-agent-v1.schema.json';
|
||||
import {
|
||||
AGENT_VERSION,
|
||||
PRECONDITIONS,
|
||||
SKILLS,
|
||||
validatePlan,
|
||||
validateJevDecision,
|
||||
validateContract,
|
||||
observation,
|
||||
} from './protocol';
|
||||
import { PickPlaceEvaluator, type PhysicalSample } from './PhysicalEvidence';
|
||||
import { PICK_PLACE as P } from './PickPlaceScene';
|
||||
const plan = () => ({
|
||||
version: AGENT_VERSION,
|
||||
objectId: 'block',
|
||||
goalId: 'placement',
|
||||
summary: '抓取、搬运和放下',
|
||||
steps: SKILLS.map((skill) => ({ skill, precondition: PRECONDITIONS[skill], onFailure: 'stop' })),
|
||||
});
|
||||
it('shares skill vocabulary with the wire schema and validates a bounded plan without executing text', () => {
|
||||
expect(schema.$defs.Plan.properties.steps.items.properties.skill.enum).toEqual(SKILLS);
|
||||
const input = plan();
|
||||
input.summary = 'rm -rf / 不会被作为命令执行';
|
||||
expect(validatePlan(input).summary).toBe(input.summary);
|
||||
});
|
||||
it.each(['object', 'skill', 'loop', 'order', 'precondition', 'extra', 'version', 'long'])(
|
||||
'rejects invalid planning: %s',
|
||||
(fault) => {
|
||||
const input = plan();
|
||||
if (fault === 'object') input.objectId = 'unknown';
|
||||
if (fault === 'skill') Object.assign(input.steps[0], { skill: 'teleport' });
|
||||
if (fault === 'loop') input.steps.push(input.steps[0]);
|
||||
if (fault === 'order') input.steps.reverse();
|
||||
if (fault === 'precondition') input.steps[0].precondition = 'verified-grasp';
|
||||
if (fault === 'extra') Object.assign(input, { command: 'execute' });
|
||||
if (fault === 'version') Object.assign(input, { version: 'other' });
|
||||
if (fault === 'long') input.summary = 'x'.repeat(513);
|
||||
expect(() => validatePlan(input)).toThrow();
|
||||
},
|
||||
);
|
||||
it('accepts only the local remaining sequence on replan, never skipping verification', () => {
|
||||
const input = plan();
|
||||
input.steps = input.steps.slice(4);
|
||||
expect(validatePlan(input, SKILLS.slice(4)).steps).toHaveLength(7);
|
||||
input.steps.shift();
|
||||
expect(() => validatePlan(input, SKILLS.slice(4))).toThrow();
|
||||
});
|
||||
it('Jev classification remains advisory; unknown or out-of-context candidates are rejected', () => {
|
||||
const value = {
|
||||
version: AGENT_VERSION,
|
||||
choice: 'carry',
|
||||
grasp: 'secure',
|
||||
diagnosis: 'none',
|
||||
recovery: 'continue',
|
||||
};
|
||||
expect(validateJevDecision(value, ['carry', 'stop']).grasp).toBe('secure');
|
||||
expect(() => validateJevDecision(value, ['close', 'stop'])).toThrow(/候选/);
|
||||
expect(() => validateJevDecision({ ...value, recovery: 'shell' }, ['carry', 'stop'])).toThrow();
|
||||
});
|
||||
it('named ground-truth observations reject nonfinite values, unknown fields/units and invalid identities', () => {
|
||||
const sample: PhysicalSample = {
|
||||
time: 0,
|
||||
base: [0, 0, 0.05],
|
||||
baseSpeed: 0,
|
||||
yaw: 0,
|
||||
tilt: 0,
|
||||
joints: [0, 0, 0, 0, 0],
|
||||
opening: 1,
|
||||
tcp: [0.2, 0, 0.2],
|
||||
object: [...P.object.position],
|
||||
objectSpeed: 0,
|
||||
peakJointSpeed: 0,
|
||||
fingerForces: [0, 0],
|
||||
supported: true,
|
||||
onGoalSupport: false,
|
||||
};
|
||||
const value = observation(
|
||||
sample,
|
||||
new PickPlaceEvaluator(P.object.position, P.goal.position),
|
||||
P.goal.position,
|
||||
{ runId: 'x', requestId: 'r', sceneRevision: 1, sequence: 0, planRevision: 0 },
|
||||
'open',
|
||||
);
|
||||
expect(value.source).toBe('mujoco-ground-truth');
|
||||
for (const invalid of [
|
||||
{ ...value, time: NaN },
|
||||
{ ...value, units: 'mm' },
|
||||
{ ...value, command: 'x' },
|
||||
{ ...value, stamp: { ...value.stamp, sequence: -1 } },
|
||||
{ ...value, tcp: [3, 0, 0] },
|
||||
])
|
||||
expect(() => validateContract('Observation', invalid)).toThrow();
|
||||
});
|
||||
@@ -0,0 +1,196 @@
|
||||
import schema from '../../../../contracts/lekiwi-agent-v1.schema.json';
|
||||
import type { BaselinePhase } from './DeterministicBaseline';
|
||||
import type { PhysicalSample, PickPlaceEvaluator } from './PhysicalEvidence';
|
||||
import type { RequestStamp } from './RequestGate';
|
||||
|
||||
export const AGENT_VERSION = 'lekiwi-agent-v1' as const;
|
||||
export type Skill = BaselinePhase;
|
||||
export const PRECONDITIONS = {
|
||||
open: 'scene-ready',
|
||||
pregrasp: 'base-stopped',
|
||||
descend: 'tcp-above',
|
||||
close: 'aligned',
|
||||
verify: 'dual-contact',
|
||||
carry: 'verified-grasp',
|
||||
'stop-base': 'verified-grasp',
|
||||
place: 'transported',
|
||||
release: 'supported',
|
||||
retreat: 'released',
|
||||
settle: 'retreat',
|
||||
} as const satisfies Record<Skill, string>;
|
||||
export const SKILLS = Object.keys(PRECONDITIONS) as Skill[];
|
||||
export interface Plan {
|
||||
version: typeof AGENT_VERSION;
|
||||
objectId: 'block';
|
||||
goalId: 'placement';
|
||||
summary: string;
|
||||
steps: { skill: Skill; precondition: string; onFailure: 'retry' | 'replan' | 'stop' }[];
|
||||
}
|
||||
export interface JevDecision {
|
||||
version: typeof AGENT_VERSION;
|
||||
choice: Skill | 'stop';
|
||||
grasp: 'secure' | 'empty' | 'slipping' | 'uncertain';
|
||||
diagnosis: 'none' | 'empty' | 'slipping' | 'misaligned' | 'unreachable' | 'stalled' | 'uncertain';
|
||||
recovery: 'continue' | 'retry' | 'replan' | 'stop';
|
||||
}
|
||||
export interface Observation {
|
||||
version: typeof AGENT_VERSION;
|
||||
stamp: RequestStamp;
|
||||
source: 'mujoco-ground-truth';
|
||||
units: 'SI';
|
||||
frame: 'world-z-up';
|
||||
time: number;
|
||||
phase: Skill;
|
||||
base: { position: number[]; yaw: number };
|
||||
joints: number[];
|
||||
opening: number;
|
||||
tcp: number[];
|
||||
object: { id: 'block'; position: number[]; speed: number };
|
||||
goal: { id: 'placement'; position: number[] };
|
||||
evidence: {
|
||||
fingerForces: number[];
|
||||
supported: boolean;
|
||||
onGoalSupport: boolean;
|
||||
secure: boolean;
|
||||
transported: number;
|
||||
};
|
||||
safety: string[];
|
||||
}
|
||||
export interface SkillResult {
|
||||
version: typeof AGENT_VERSION;
|
||||
skill: Skill;
|
||||
status: 'running' | 'completed' | 'failed';
|
||||
code: string;
|
||||
evidenceSequence: number;
|
||||
}
|
||||
|
||||
type Node = {
|
||||
$ref?: string;
|
||||
type?: string;
|
||||
const?: unknown;
|
||||
enum?: unknown[];
|
||||
properties?: Record<string, Node>;
|
||||
required?: string[];
|
||||
additionalProperties?: boolean;
|
||||
items?: Node;
|
||||
minItems?: number;
|
||||
maxItems?: number;
|
||||
uniqueItems?: boolean;
|
||||
minimum?: number;
|
||||
maximum?: number;
|
||||
minLength?: number;
|
||||
maxLength?: number;
|
||||
};
|
||||
type ContractName = keyof typeof schema.$defs;
|
||||
const definitions = schema.$defs as unknown as Record<ContractName, Node>;
|
||||
/** The intentionally small JSON Schema subset used by this version; fail closed
|
||||
* on data, not an eval/template engine. Python validates this same schema. */
|
||||
function check(node: Node, value: unknown, path: string): void {
|
||||
const fail = () => {
|
||||
throw new Error(`${path}: 智能任务契约不匹配`);
|
||||
};
|
||||
if (node.$ref) {
|
||||
const name = node.$ref.replace('#/$defs/', '') as ContractName;
|
||||
if (!Object.hasOwn(definitions, name)) fail();
|
||||
return check(definitions[name], value, path);
|
||||
}
|
||||
if (Object.hasOwn(node, 'const') && value !== node.const) fail();
|
||||
if (node.enum && !node.enum.includes(value)) fail();
|
||||
if (node.type === 'number' || node.type === 'integer') {
|
||||
if (
|
||||
typeof value !== 'number' ||
|
||||
!Number.isFinite(value) ||
|
||||
(node.type === 'integer' && !Number.isInteger(value))
|
||||
)
|
||||
fail();
|
||||
if (
|
||||
(value as number) < (node.minimum ?? -Infinity) ||
|
||||
(value as number) > (node.maximum ?? Infinity)
|
||||
)
|
||||
fail();
|
||||
} else if (node.type === 'string') {
|
||||
if (typeof value !== 'string') fail();
|
||||
const length = Array.from(value as string).length;
|
||||
if (length < (node.minLength ?? 0) || length > (node.maxLength ?? Infinity)) fail();
|
||||
} else if (node.type === 'boolean') {
|
||||
if (typeof value !== 'boolean') fail();
|
||||
} else if (node.type === 'array') {
|
||||
if (!Array.isArray(value)) fail();
|
||||
const values = value as unknown[];
|
||||
if (values.length < (node.minItems ?? 0) || values.length > (node.maxItems ?? Infinity)) fail();
|
||||
if (node.uniqueItems && new Set(values.map((v) => JSON.stringify(v))).size !== values.length)
|
||||
fail();
|
||||
if (node.items) values.forEach((v, i) => check(node.items!, v, `${path}[${i}]`));
|
||||
} else if (node.type === 'object') {
|
||||
if (value === null || typeof value !== 'object' || Array.isArray(value)) fail();
|
||||
const obj = value as Record<string, unknown>;
|
||||
for (const key of node.required ?? []) if (!Object.hasOwn(obj, key)) fail();
|
||||
for (const key of Object.keys(obj)) {
|
||||
if (!Object.hasOwn(node.properties ?? {}, key)) {
|
||||
if (node.additionalProperties === false) fail();
|
||||
} else check(node.properties![key], obj[key], `${path}.${key}`);
|
||||
}
|
||||
}
|
||||
}
|
||||
export function validateContract<T>(name: ContractName, value: unknown): T {
|
||||
if ((JSON.stringify(value)?.length ?? 0) > 65536) throw new Error('智能任务消息超长');
|
||||
check(definitions[name], value, name);
|
||||
return structuredClone(value) as T;
|
||||
}
|
||||
export function validatePlan(value: unknown, remaining: readonly Skill[] = SKILLS): Plan {
|
||||
const plan = validateContract<Plan>('Plan', value);
|
||||
if (
|
||||
plan.steps.length !== remaining.length ||
|
||||
plan.steps.some(
|
||||
(step, i) => step.skill !== remaining[i] || step.precondition !== PRECONDITIONS[step.skill],
|
||||
)
|
||||
)
|
||||
throw new Error('计划缺少物理前置阶段、顺序非法或存在循环');
|
||||
return plan;
|
||||
}
|
||||
export function validateJevDecision(
|
||||
value: unknown,
|
||||
candidates: readonly (Skill | 'stop')[],
|
||||
): JevDecision {
|
||||
const decision = validateContract<JevDecision>('JevDecision', value);
|
||||
if (!candidates.includes(decision.choice)) throw new Error('Jev 选择不在当前候选中');
|
||||
return decision;
|
||||
}
|
||||
export function observation(
|
||||
sample: PhysicalSample,
|
||||
evaluator: PickPlaceEvaluator,
|
||||
goal: readonly number[],
|
||||
stamp: RequestStamp,
|
||||
phase: Skill,
|
||||
safety: string[] = [],
|
||||
): Observation {
|
||||
return validateContract<Observation>('Observation', {
|
||||
version: AGENT_VERSION,
|
||||
stamp,
|
||||
source: 'mujoco-ground-truth',
|
||||
units: 'SI',
|
||||
frame: 'world-z-up',
|
||||
time: sample.time,
|
||||
phase,
|
||||
base: { position: sample.base, yaw: sample.yaw },
|
||||
joints: sample.joints,
|
||||
opening: sample.opening,
|
||||
tcp: sample.tcp,
|
||||
object: { id: 'block', position: sample.object, speed: sample.objectSpeed },
|
||||
goal: { id: 'placement', position: [...goal] },
|
||||
evidence: {
|
||||
fingerForces: sample.fingerForces,
|
||||
supported: sample.supported,
|
||||
onGoalSupport: sample.onGoalSupport,
|
||||
secure: evaluator.secure,
|
||||
transported: evaluator.transported,
|
||||
},
|
||||
safety,
|
||||
});
|
||||
}
|
||||
export const PLAN_OUTPUT_SCHEMA = schema.$defs.Plan;
|
||||
export function skillErrorCode(error: string): string {
|
||||
return (schema.$defs.SkillResult.properties.code.enum as readonly string[]).includes(error)
|
||||
? error
|
||||
: 'controller_error';
|
||||
}
|
||||
@@ -0,0 +1,87 @@
|
||||
import type { ProjectManifest } from '../project/types';
|
||||
import type { MainThreadPhysicsAdapter } from '../simulation/PhysicsAdapter';
|
||||
import { MOBILE_ROBOTS, MOBILE_TASK, type RobotConfig } from './RobotDescriptor';
|
||||
import type { TrainingStage } from './TaskKernel';
|
||||
import type { MobilePolicyMetadata } from './ONNXPolicyRunner';
|
||||
|
||||
export const MOBILE_TRAINING_TASKS: Record<string, string> = {
|
||||
'MobileManipulator-LeKiwi-v1': 'lekiwi-v1',
|
||||
'MobileManipulator-LeKiwi-Bundle': 'lekiwi-bundle',
|
||||
};
|
||||
export const isMobileTrainingTask = (id: string) => Object.hasOwn(MOBILE_TRAINING_TASKS, id);
|
||||
export function mobileTrainingRobot(taskId: string): RobotConfig {
|
||||
const robot = MOBILE_ROBOTS.find((r) => r.id === MOBILE_TRAINING_TASKS[taskId]);
|
||||
if (!robot) throw new Error('未知移动操作任务');
|
||||
return robot;
|
||||
}
|
||||
export interface MobileTrainingParams {
|
||||
rolloutSteps: number;
|
||||
stage?: TrainingStage;
|
||||
sourceJobId?: string;
|
||||
positionJitter?: number;
|
||||
evaluationEpisodes?: number;
|
||||
navigationBootstrapSteps?: number;
|
||||
objectPosition: number[];
|
||||
goalPosition: number[];
|
||||
}
|
||||
export interface MobileDeployment extends MobilePolicyMetadata {
|
||||
version: 1;
|
||||
evaluation?: {
|
||||
episodes: number;
|
||||
successRate: number;
|
||||
safetyStops: number;
|
||||
maxJointVelocity: number;
|
||||
meanNavigationDistance: number;
|
||||
seed: number;
|
||||
};
|
||||
trainedTimesteps?: number;
|
||||
trainingParams?: MobileTrainingParams;
|
||||
initialization?: string;
|
||||
browserCompatible: boolean;
|
||||
trainingTaskId: string;
|
||||
sceneSha256: string;
|
||||
resetOptions: { object: number[]; goal: number[] };
|
||||
input: { name: string; dtype: string; shape: number[] };
|
||||
output: { name: string; dtype: string; shape: number[] };
|
||||
terrain?: never;
|
||||
}
|
||||
export interface MobileTrainingBridge {
|
||||
robotId?: string;
|
||||
prepare(taskId: string): Promise<File>;
|
||||
importPolicy(file: File, deployment: MobileDeployment): Promise<void>;
|
||||
}
|
||||
|
||||
/** Internal transport only: the user never downloads or unpacks a training ZIP. */
|
||||
export async function mobileSceneSnapshot(
|
||||
adapter: MainThreadPhysicsAdapter,
|
||||
project: ProjectManifest,
|
||||
entryPath: string,
|
||||
config: RobotConfig,
|
||||
): Promise<File> {
|
||||
const session = adapter.session;
|
||||
if (!session) throw new Error('请先加载机器人');
|
||||
const { zipSync } = await import('fflate');
|
||||
if (session !== adapter.session) throw new Error('模型已切换,场景同步取消');
|
||||
const scene = `${entryPath}.training.xml`;
|
||||
const files: Record<string, Uint8Array> = {};
|
||||
for (const f of [...project.files, ...adapter.cachedSupportFiles()]) files[f.path] = f.data;
|
||||
const encoder = new TextEncoder();
|
||||
files[scene] = adapter.exportMjcf();
|
||||
files['robot.json'] = encoder.encode(JSON.stringify(config));
|
||||
files['task.json'] = encoder.encode(JSON.stringify(MOBILE_TASK));
|
||||
files['environment.json'] = encoder.encode(
|
||||
JSON.stringify({
|
||||
scene,
|
||||
mujoco: session.module.mj_versionString(),
|
||||
taskId: MOBILE_TASK.id,
|
||||
robotId: config.id,
|
||||
}),
|
||||
);
|
||||
return new File(
|
||||
[zipSync(files, { level: 1, mtime: new Date(1980, 0, 1) }).slice().buffer],
|
||||
'scene.zip',
|
||||
{
|
||||
type: 'application/zip',
|
||||
},
|
||||
);
|
||||
}
|
||||
@@ -98,7 +98,7 @@ export function downloadBytes(
|
||||
setTimeout(() => URL.revokeObjectURL(url), 0);
|
||||
}
|
||||
|
||||
export function exportedFileName(projectName: string, extension: 'urdf' | 'xml'): string {
|
||||
export function exportedFileName(projectName: string, extension: 'urdf' | 'xml' | 'zip'): string {
|
||||
const stem =
|
||||
projectName.replace(/\.(?:zip|xml|urdf)$/i, '').replace(/[^\p{L}\p{N}._-]+/gu, '_') || 'model';
|
||||
return `${stem}.${extension}`;
|
||||
|
||||
@@ -0,0 +1,31 @@
|
||||
import { unzipSync } from 'fflate';
|
||||
import { compiledProjectPackage } from './exportPackage';
|
||||
import type { ProjectManifest } from './types';
|
||||
|
||||
it('exports original and generated resources next to a unique compiled entry', () => {
|
||||
const data = new TextEncoder().encode('original');
|
||||
const manifest: ProjectManifest = {
|
||||
id: 'x',
|
||||
name: 'x',
|
||||
maps: [],
|
||||
entries: [],
|
||||
totalBytes: data.length,
|
||||
files: [
|
||||
{ path: 'robot/compiled-scene.xml', data, size: data.length, source: 'file', mimeType: '' },
|
||||
],
|
||||
};
|
||||
const xml = new TextEncoder().encode('<mujoco/>');
|
||||
const resource = {
|
||||
...manifest.files[0],
|
||||
path: 'robot/full.obj',
|
||||
data: new Uint8Array([1, 2, 3]),
|
||||
size: 3,
|
||||
};
|
||||
const result = compiledProjectPackage(manifest, 'robot/original.urdf', xml, [resource]);
|
||||
expect(result.entryPath).toBe('robot/compiled-scene-2.xml');
|
||||
const files = unzipSync(result.data);
|
||||
expect(files['robot/full.obj']).toEqual(resource.data);
|
||||
expect(Array.from(files['robot/compiled-scene.xml'])).toEqual(Array.from(data));
|
||||
expect(Array.from(files[result.entryPath])).toEqual(Array.from(xml));
|
||||
expect(() => compiledProjectPackage(manifest, '../evil.xml', xml, [])).toThrow(/越界/);
|
||||
});
|
||||
@@ -0,0 +1,23 @@
|
||||
import { zipSync } from 'fflate';
|
||||
import { normalizeProjectPath } from './importer';
|
||||
import type { ProjectFile, ProjectManifest } from './types';
|
||||
|
||||
/** Portable companion to XML-only export: preserve original sources/provenance
|
||||
* and include every derived mesh, without overwriting any user's entry file. */
|
||||
export function compiledProjectPackage(
|
||||
manifest: ProjectManifest,
|
||||
sourceEntry: string,
|
||||
xml: Uint8Array,
|
||||
supportFiles: readonly ProjectFile[],
|
||||
): { data: Uint8Array; entryPath: string } {
|
||||
const files: Record<string, Uint8Array> = Object.create(null);
|
||||
for (const file of [...manifest.files, ...supportFiles])
|
||||
files[normalizeProjectPath(file.path)] = file.data;
|
||||
const entry = normalizeProjectPath(sourceEntry);
|
||||
const directory = entry.slice(0, entry.lastIndexOf('/') + 1);
|
||||
let entryPath = `${directory}compiled-scene.xml`;
|
||||
for (let i = 2; Object.hasOwn(files, entryPath); i++)
|
||||
entryPath = `${directory}compiled-scene-${i}.xml`;
|
||||
files[entryPath] = xml;
|
||||
return { data: zipSync(files, { level: 1, mtime: new Date(1980, 0, 1) }), entryPath };
|
||||
}
|
||||
@@ -158,6 +158,64 @@ describe('project importer', () => {
|
||||
expect(text).toContain('fusestatic="false"');
|
||||
expect(prepared.warnings).toHaveLength(2);
|
||||
});
|
||||
it.each(['package://机械臂/meshes/Link1.STL', 'meshes/package://机械臂/meshes/Link1.STL'])(
|
||||
'修复重新打包的 ROS URI,保留原始工程:%s',
|
||||
async (filename) => {
|
||||
const urdf = projectFile(
|
||||
'bundle/robot.urdf',
|
||||
`<robot><link name="Link1"><collision><geometry><mesh filename="${filename}"/></geometry></collision></link></robot>`,
|
||||
);
|
||||
const original = new TextDecoder().decode(urdf.data);
|
||||
const manifest = {
|
||||
id: 'bundle',
|
||||
name: 'bundle',
|
||||
maps: [],
|
||||
totalBytes: urdf.size,
|
||||
entries: [{ path: urdf.path, format: 'urdf' as const, label: 'robot' }],
|
||||
files: [urdf, projectFile('bundle/meshes/Link1.STL', 'mesh')],
|
||||
};
|
||||
const result = await prepareProjectForMujoco(manifest, urdf.path);
|
||||
expect(new TextDecoder().decode(result.manifest.files[0].data)).toContain(
|
||||
'filename="meshes/Link1.STL"',
|
||||
);
|
||||
expect(new TextDecoder().decode(result.manifest.files[0].data)).toContain(
|
||||
'strippath="false"',
|
||||
);
|
||||
expect(new TextDecoder().decode(urdf.data)).toBe(original);
|
||||
expect(result.warnings.join(' ')).toContain('1 个 package://');
|
||||
},
|
||||
);
|
||||
it('不按同名 STL 猜测资源,不自动选择歧义包,也不放行路径穿越', async () => {
|
||||
const prepare = (filename: string, paths: string[]) => {
|
||||
const urdf = projectFile(
|
||||
'robot.urdf',
|
||||
`<robot><link><visual><geometry><mesh filename="${filename}"/></geometry></visual></link></robot>`,
|
||||
);
|
||||
return prepareProjectForMujoco(
|
||||
{
|
||||
id: 'x',
|
||||
name: 'x',
|
||||
maps: [],
|
||||
totalBytes: urdf.size,
|
||||
entries: [{ path: urdf.path, format: 'urdf', label: 'x' }],
|
||||
files: [urdf, ...paths.map((path) => projectFile(path, 'mesh'))],
|
||||
},
|
||||
urdf.path,
|
||||
);
|
||||
};
|
||||
await expect(
|
||||
prepare('package://arm/meshes/a.stl', ['a/arm/meshes/a.stl', 'b/arm/meshes/a.stl']),
|
||||
).rejects.toThrow(/不唯一/);
|
||||
await expect(
|
||||
prepare('meshes/package://arm/meshes/a.stl', ['a/meshes/a.stl', 'b/meshes/a.stl']),
|
||||
).rejects.toThrow(/不唯一/);
|
||||
await expect(prepare('meshes/package://arm/../a.stl', ['a.stl'])).rejects.toThrow(/越界/);
|
||||
const missing = await prepare('meshes/package://arm/meshes/a.stl', ['other/a.stl']);
|
||||
expect(missing.warnings.join(' ')).toContain('未在工程中找到');
|
||||
expect(new TextDecoder().decode(missing.manifest.files[0].data)).toContain(
|
||||
'meshes/package://arm/meshes/a.stl',
|
||||
);
|
||||
});
|
||||
it('将 DAE mesh 转换为 MuJoCo 可读取的 OBJ,并复用于 visual/collision', async () => {
|
||||
const urdf = projectFile(
|
||||
'robot/robot.urdf',
|
||||
|
||||
@@ -154,6 +154,7 @@ export async function prepareProjectForMujoco(
|
||||
}
|
||||
compiler.setAttribute('discardvisual', 'false');
|
||||
compiler.setAttribute('fusestatic', 'false');
|
||||
compiler.setAttribute('strippath', 'false');
|
||||
|
||||
let removedMaterials = 0;
|
||||
for (const visual of Array.from(document.querySelectorAll('visual'))) {
|
||||
@@ -173,16 +174,29 @@ export async function prepareProjectForMujoco(
|
||||
const unresolved: string[] = [];
|
||||
for (const element of Array.from(document.querySelectorAll('[filename]'))) {
|
||||
const value = element.getAttribute('filename');
|
||||
if (!value?.startsWith('package://')) continue;
|
||||
const packagePath = normalizeProjectPath(value.slice('package://'.length));
|
||||
const target =
|
||||
paths.find((path) => path === packagePath) ??
|
||||
paths.find((path) => path.endsWith(`/${packagePath}`));
|
||||
if (!target) {
|
||||
// Some CAD/URDF editors bundle a ROS resource but accidentally prepend
|
||||
// "meshes/" to its package URI. Only this observed spelling is repaired;
|
||||
// do not guess by basename or resolve URLs/files outside the imported project.
|
||||
if (!value || !/^(?:meshes\/)?package:\/\//.test(value)) continue;
|
||||
const packagePath = normalizeProjectPath(value.replace(/^(?:meshes\/)?package:\/\//, ''));
|
||||
let targets = paths.filter((path) => path === packagePath || path.endsWith(`/${packagePath}`));
|
||||
if (!targets.length) {
|
||||
const slash = packagePath.indexOf('/');
|
||||
const bundledPath = slash < 0 ? '' : packagePath.slice(slash + 1);
|
||||
if (bundledPath) {
|
||||
const local = resolveProjectReference(entryPath, bundledPath);
|
||||
targets = paths.includes(local ?? '')
|
||||
? [local!]
|
||||
: paths.filter((path) => path === bundledPath || path.endsWith(`/${bundledPath}`));
|
||||
}
|
||||
}
|
||||
if (targets.length > 1)
|
||||
throw new ProjectImportError(`package:// 资源路径不唯一,不能自动选择:${value}`, entryPath);
|
||||
if (!targets.length) {
|
||||
unresolved.push(value);
|
||||
continue;
|
||||
}
|
||||
element.setAttribute('filename', relativeProjectPath(entryPath, target));
|
||||
element.setAttribute('filename', relativeProjectPath(entryPath, targets[0]));
|
||||
rewrittenUris += 1;
|
||||
}
|
||||
if (rewrittenUris)
|
||||
|
||||
@@ -0,0 +1,146 @@
|
||||
import { webcrypto } from 'node:crypto';
|
||||
import { prepareLargeMeshes } from './meshCompatibility';
|
||||
import { binaryStl } from '../test/stlFixture';
|
||||
import type { ProjectManifest } from './types';
|
||||
|
||||
const encoder = new TextEncoder(),
|
||||
decoder = new TextDecoder();
|
||||
function project(
|
||||
entries: Record<string, string | Uint8Array>,
|
||||
entry = 'robot/main.xml',
|
||||
): ProjectManifest {
|
||||
const files = Object.entries(entries).map(([path, value]) => {
|
||||
const data = typeof value === 'string' ? encoder.encode(value) : value;
|
||||
return { path, data, size: data.length, source: 'file' as const, mimeType: '' };
|
||||
});
|
||||
return {
|
||||
id: 'test',
|
||||
name: 'test',
|
||||
files,
|
||||
entries: [{ path: entry, format: entry.endsWith('urdf') ? 'urdf' : 'mjcf', label: 'test' }],
|
||||
maps: [],
|
||||
totalBytes: files.reduce((sum, f) => sum + f.size, 0),
|
||||
};
|
||||
}
|
||||
beforeEach(() => vi.stubGlobal('crypto', webcrypto));
|
||||
afterEach(() => vi.unstubAllGlobals());
|
||||
|
||||
it('converts shared URDF resources once, preserving scale, origin and input bytes', async () => {
|
||||
const source =
|
||||
'<robot><link name="x"><visual><origin xyz="1 2 3"/><geometry><mesh filename="meshes/w.stl" scale=".001 .001 .001"/></geometry></visual><collision><geometry><mesh filename="meshes/w.stl"/></geometry></collision></link></robot>';
|
||||
const p = project(
|
||||
{ 'robot/main.urdf': source, 'robot/meshes/w.stl': binaryStl(200001) },
|
||||
'robot/main.urdf',
|
||||
);
|
||||
const first = await prepareLargeMeshes(p, 'robot/main.urdf');
|
||||
expect(first.manifest.files.filter((f) => f.path.endsWith('.obj'))).toHaveLength(1);
|
||||
const xml = decoder.decode(first.manifest.files[0].data);
|
||||
expect(xml).toContain('xyz="1 2 3"');
|
||||
expect(xml).toContain('scale=".001 .001 .001"');
|
||||
expect(xml.match(/filename="stl-obj-v1-/g)).toHaveLength(2);
|
||||
expect(decoder.decode(p.files[0].data)).toBe(source);
|
||||
expect(first.warnings).toHaveLength(1);
|
||||
const again = await prepareLargeMeshes(p, 'robot/main.urdf');
|
||||
expect(again.manifest.files.at(-1)!.data).toBe(first.manifest.files.at(-1)!.data);
|
||||
const cooked = await prepareLargeMeshes(first.manifest, 'robot/main.urdf');
|
||||
expect(cooked.manifest).toBe(first.manifest);
|
||||
});
|
||||
|
||||
it('honors meshdir/content_type and include fallback; does not touch unrelated models', async () => {
|
||||
const p = project({
|
||||
'robot/main.xml':
|
||||
'<mujoco><compiler meshdir="meshes"/><include file="sub/assets.xml"/><worldbody/></mujoco>',
|
||||
'robot/sub/assets.xml':
|
||||
'<mujocoinclude><asset><mesh name="wheel" content_type="model/stl" file="wheel.stl" scale=".001 .001 .001"/></asset></mujocoinclude>',
|
||||
'robot/sub/meshes/wheel.stl': binaryStl(200001),
|
||||
'other.xml': '<mujoco><asset><mesh file="not-existing.stl"/></asset></mujoco>',
|
||||
});
|
||||
const result = await prepareLargeMeshes(p, 'robot/main.xml');
|
||||
const generated = result.manifest.files.find((f) => f.path.endsWith('.obj'))!;
|
||||
expect(generated.path).toMatch(/^robot\/meshes\/stl-obj-v1-/);
|
||||
const xml = decoder.decode(
|
||||
result.manifest.files.find((f) => f.path.endsWith('assets.xml'))!.data,
|
||||
);
|
||||
expect(xml).toContain('content_type="model/obj"');
|
||||
expect(xml).toContain('scale=".001 .001 .001"');
|
||||
expect(decoder.decode(result.manifest.files.find((f) => f.path === 'other.xml')!.data)).toContain(
|
||||
'not-existing.stl',
|
||||
);
|
||||
});
|
||||
|
||||
it('uses main-directory asset before include-directory asset, as MuJoCo does', async () => {
|
||||
const p = project({
|
||||
'robot/main.xml': '<mujoco><include file="sub/assets.xml"/></mujoco>',
|
||||
'robot/sub/assets.xml': '<mujocoinclude><asset><mesh file="w.stl"/></asset></mujocoinclude>',
|
||||
'robot/w.stl': binaryStl(1),
|
||||
'robot/sub/w.stl': binaryStl(200001),
|
||||
});
|
||||
const result = await prepareLargeMeshes(p, 'robot/main.xml');
|
||||
expect(result.manifest).toBe(p);
|
||||
});
|
||||
|
||||
it('honors assetdir/strippath and never overwrites a user-supplied derived filename', async () => {
|
||||
const p = project({
|
||||
'robot/main.xml':
|
||||
'<mujoco><compiler assetdir="meshes" strippath="true"/><asset><mesh file="discarded/path/w.stl"/></asset></mujoco>',
|
||||
'robot/meshes/w.stl': binaryStl(200001),
|
||||
});
|
||||
const first = await prepareLargeMeshes(p, 'robot/main.xml');
|
||||
const generated = first.manifest.files.at(-1)!;
|
||||
const fake = { ...generated, data: encoder.encode('user content'), size: 12 };
|
||||
const result = await prepareLargeMeshes({ ...p, files: [...p.files, fake] }, 'robot/main.xml');
|
||||
expect(decoder.decode(result.manifest.files.find((f) => f.path === fake.path)!.data)).toBe(
|
||||
'user content',
|
||||
);
|
||||
expect(result.manifest.files.at(-1)!.path).toMatch(/-2\.obj$/);
|
||||
});
|
||||
|
||||
it('new source bytes invalidate the conversion cache and preserve old resources', async () => {
|
||||
const data = binaryStl(200001);
|
||||
const p = project({
|
||||
'robot/main.xml': '<mujoco><asset><mesh file="w.stl"/></asset></mujoco>',
|
||||
'robot/w.stl': data,
|
||||
});
|
||||
const before = await prepareLargeMeshes(p, 'robot/main.xml');
|
||||
const changed = data.slice();
|
||||
new DataView(changed.buffer).setFloat32(96, 0.25, true);
|
||||
const result = await prepareLargeMeshes(
|
||||
project({ 'robot/main.xml': decoder.decode(p.files[0].data), 'robot/w.stl': changed }),
|
||||
'robot/main.xml',
|
||||
);
|
||||
expect(result.manifest.files.at(-1)!.path).not.toBe(before.manifest.files.at(-1)!.path);
|
||||
});
|
||||
|
||||
it.each(['../../escape.stl', 'https://evil.test/w.stl', '/host/w.stl'])(
|
||||
'rejects unsafe paths: %s',
|
||||
async (path) => {
|
||||
await expect(
|
||||
prepareLargeMeshes(
|
||||
project({ 'robot/main.xml': `<mujoco><asset><mesh file="${path}"/></asset></mujoco>` }),
|
||||
'robot/main.xml',
|
||||
),
|
||||
).rejects.toThrow(/路径/);
|
||||
},
|
||||
);
|
||||
it('rejects cyclic includes and aborts cooperatively', async () => {
|
||||
await expect(
|
||||
prepareLargeMeshes(
|
||||
project({ 'robot/main.xml': '<mujoco><include file="main.xml"/></mujoco>' }),
|
||||
'robot/main.xml',
|
||||
),
|
||||
).rejects.toThrow(/循环/);
|
||||
const p = project({
|
||||
'robot/main.xml': '<mujoco><asset><mesh file="w.stl"/></asset></mujoco>',
|
||||
'robot/w.stl': binaryStl(200001),
|
||||
});
|
||||
let cancelled = false;
|
||||
await expect(
|
||||
prepareLargeMeshes(p, 'robot/main.xml', {
|
||||
shouldCancel: () => cancelled,
|
||||
onMeshProgress: () => {
|
||||
cancelled = true;
|
||||
},
|
||||
}),
|
||||
).rejects.toThrow(/取消/);
|
||||
expect(p.files).toHaveLength(2);
|
||||
});
|
||||
@@ -0,0 +1,166 @@
|
||||
import { relativeAssetPath, resolveProjectAssetPath } from '../map/mapPaths';
|
||||
import { sha256 } from '../robot/registry';
|
||||
import { DEFAULT_IMPORT_LIMITS, type ProjectFile, type ProjectManifest } from './types';
|
||||
import {
|
||||
binaryStlFaceCount,
|
||||
convertStlToObj,
|
||||
MUJOCO_STL_FACE_LIMIT,
|
||||
STL_CONVERTER_VERSION,
|
||||
type MeshConversionOptions,
|
||||
} from './stlConverter';
|
||||
|
||||
const decoder = new TextDecoder(),
|
||||
encoder = new TextEncoder();
|
||||
// Successful conversions only; weak values never pin meshes from a closed project.
|
||||
// Content keys also share the three byte-identical LeKiwi wheel source files.
|
||||
const cache = new Map<string, WeakRef<Uint8Array>>();
|
||||
export interface MeshPreparationOptions extends MeshConversionOptions {
|
||||
onMeshProgress?: (path: string, completed: number, total: number) => void;
|
||||
}
|
||||
|
||||
/** Prepare only the selected entry/include graph. Source assets remain untouched.
|
||||
* MuJoCo resolves assets in the main XML directory first, then the including file's
|
||||
* directory. Do not use basename guessing: two same-named CAD parts can differ.
|
||||
*/
|
||||
export async function prepareLargeMeshes(
|
||||
manifest: ProjectManifest,
|
||||
entryPath: string,
|
||||
options: MeshPreparationOptions = {},
|
||||
): Promise<{ manifest: ProjectManifest; warnings: string[] }> {
|
||||
const files = new Map(manifest.files.map((file) => [file.path, file]));
|
||||
const documents = new Map<string, Document>();
|
||||
const active = new Set<string>();
|
||||
const compiler = new Map<string, string>();
|
||||
const check = () => {
|
||||
if (options.shouldCancel?.()) throw new Error('网格准备已取消');
|
||||
};
|
||||
const resolveExisting = (from: string, reference: string) => {
|
||||
const main = resolveProjectAssetPath(entryPath, reference);
|
||||
if (files.has(main)) return main;
|
||||
return resolveProjectAssetPath(from, reference);
|
||||
};
|
||||
const visit = (path: string, depth = 0): void => {
|
||||
check();
|
||||
if (active.has(path) || depth > 32) throw new Error(`模型 include 循环或嵌套过深:${path}`);
|
||||
if (documents.has(path)) return;
|
||||
const file = files.get(path);
|
||||
if (!file) throw new Error(`模型引用的 XML 不存在:${path}`);
|
||||
const doc = new DOMParser().parseFromString(decoder.decode(file.data), 'application/xml');
|
||||
if (doc.querySelector('parsererror') || doc.doctype) throw new Error(`模型 XML 无效:${path}`);
|
||||
documents.set(path, doc);
|
||||
active.add(path);
|
||||
for (const node of doc.querySelectorAll('compiler, include[file]')) {
|
||||
if (node.tagName === 'compiler') {
|
||||
for (const attribute of Array.from(node.attributes))
|
||||
compiler.set(attribute.name, attribute.value);
|
||||
} else {
|
||||
visit(resolveExisting(path, node.getAttribute('file')!), depth + 1);
|
||||
}
|
||||
}
|
||||
active.delete(path);
|
||||
};
|
||||
visit(entryPath);
|
||||
const meshdir = compiler.get('meshdir') ?? compiler.get('assetdir') ?? '';
|
||||
const strip = compiler.get('strippath') === 'true';
|
||||
const assetAnchor = resolveProjectAssetPath(
|
||||
entryPath,
|
||||
`${meshdir ? meshdir + '/' : ''}.__asset__`,
|
||||
);
|
||||
const warnings: string[] = [];
|
||||
const generated = new Map<string, ProjectFile>();
|
||||
const changed = new Set<string>();
|
||||
let totalBytes = manifest.files.reduce((sum, file) => sum + file.data.byteLength, 0);
|
||||
for (const [path, doc] of documents) {
|
||||
const urdf = doc.documentElement.tagName === 'robot';
|
||||
for (const mesh of doc.querySelectorAll(urdf ? 'mesh[filename]' : 'mesh[file]')) {
|
||||
check();
|
||||
const attr = urdf ? 'filename' : 'file';
|
||||
let reference = mesh.getAttribute(attr)!;
|
||||
const type = mesh.getAttribute('content_type');
|
||||
if (type && type !== 'model/stl') continue;
|
||||
if (type !== 'model/stl' && !/\.stl$/i.test(reference)) continue;
|
||||
// Validate before strippath, too: imported resources never grant host access.
|
||||
resolveProjectAssetPath(path, reference);
|
||||
if (strip) reference = reference.replaceAll('\\', '/').split('/').at(-1)!;
|
||||
const sourcePath = resolveExisting(path, `${meshdir ? meshdir + '/' : ''}${reference}`);
|
||||
const source = files.get(sourcePath);
|
||||
if (!source) throw new Error(`STL 网格不存在:${sourcePath}`);
|
||||
const faces = binaryStlFaceCount(source.data);
|
||||
if (faces <= MUJOCO_STL_FACE_LIMIT) continue;
|
||||
let file = generated.get(sourcePath);
|
||||
if (!file) {
|
||||
const hash = await sha256(source.data);
|
||||
check();
|
||||
let output = cache.get(hash)?.deref();
|
||||
if (!output) {
|
||||
const result = await convertStlToObj(source.data, {
|
||||
...options,
|
||||
maxOutputBytes: Math.min(
|
||||
options.maxOutputBytes ?? DEFAULT_IMPORT_LIMITS.maxFileBytes,
|
||||
DEFAULT_IMPORT_LIMITS.maxFileBytes,
|
||||
DEFAULT_IMPORT_LIMITS.maxTotalBytes - totalBytes,
|
||||
),
|
||||
onProgress: (done, count) => options.onMeshProgress?.(sourcePath, done, count),
|
||||
});
|
||||
output = result.data;
|
||||
if (cache.size >= 64) cache.delete(cache.keys().next().value!);
|
||||
cache.set(hash, new WeakRef(output));
|
||||
}
|
||||
check();
|
||||
if (output.byteLength > (options.maxOutputBytes ?? DEFAULT_IMPORT_LIMITS.maxFileBytes))
|
||||
throw new Error('派生 OBJ 超过文件预算');
|
||||
options.onMeshProgress?.(sourcePath, faces, faces);
|
||||
const converted = { data: output };
|
||||
const stem = `${STL_CONVERTER_VERSION}-${hash}`;
|
||||
let name = `${stem}.obj`,
|
||||
suffix = 1;
|
||||
let target = resolveProjectAssetPath(assetAnchor, name);
|
||||
// Reuse only byte-identical generated data. Never trust a user's filename.
|
||||
while (files.has(target) && !sameBytes(files.get(target)!.data, converted.data)) {
|
||||
name = `${stem}-${++suffix}.obj`;
|
||||
target = resolveProjectAssetPath(assetAnchor, name);
|
||||
}
|
||||
if (!files.has(target)) {
|
||||
totalBytes += converted.data.byteLength;
|
||||
if (
|
||||
converted.data.byteLength > DEFAULT_IMPORT_LIMITS.maxFileBytes ||
|
||||
totalBytes > DEFAULT_IMPORT_LIMITS.maxTotalBytes ||
|
||||
files.size >= DEFAULT_IMPORT_LIMITS.maxFiles
|
||||
)
|
||||
throw new Error('高面数网格派生资源超过工程导入预算');
|
||||
file = {
|
||||
...source,
|
||||
path: target,
|
||||
data: converted.data,
|
||||
size: converted.data.byteLength,
|
||||
mimeType: 'text/plain',
|
||||
};
|
||||
files.set(target, file);
|
||||
} else file = files.get(target)!;
|
||||
generated.set(sourcePath, file);
|
||||
warnings.push(
|
||||
`${sourcePath}:${faces.toLocaleString()} 面已无减面转换为 OBJ;原始 STL 保留`,
|
||||
);
|
||||
}
|
||||
// Generated data always lives under the main compiler's meshdir. Thus this
|
||||
// reference is unambiguous even when the original was in a nested include.
|
||||
mesh.setAttribute(attr, relativeAssetPath(assetAnchor, file.path));
|
||||
if (type) mesh.setAttribute('content_type', 'model/obj');
|
||||
changed.add(path);
|
||||
}
|
||||
}
|
||||
check();
|
||||
if (!changed.size) return { manifest, warnings };
|
||||
for (const path of changed) {
|
||||
const file = files.get(path)!;
|
||||
const data = encoder.encode(new XMLSerializer().serializeToString(documents.get(path)!));
|
||||
totalBytes += data.byteLength - file.data.byteLength;
|
||||
files.set(path, { ...file, data, size: data.byteLength });
|
||||
}
|
||||
if (totalBytes > DEFAULT_IMPORT_LIMITS.maxTotalBytes) throw new Error('派生工程超过总大小预算');
|
||||
return { manifest: { ...manifest, files: [...files.values()], totalBytes }, warnings };
|
||||
}
|
||||
|
||||
function sameBytes(a: Uint8Array, b: Uint8Array): boolean {
|
||||
return a === b || (a.length === b.length && a.every((value, i) => value === b[i]));
|
||||
}
|
||||
@@ -2,20 +2,26 @@ import type { ProjectManifest } from '../types';
|
||||
import { LEKIWI_PROFILE } from '../../robot/profiles/lekiwi';
|
||||
import { sha256, validateLeKiwiSource } from '../../robot/registry';
|
||||
import { RobotError } from '../../robot/types';
|
||||
import sourceMeshes from '../../../../robot_profiles/lekiwi-source-meshes.json';
|
||||
import {
|
||||
LEKIWI_VISUAL_REVISION,
|
||||
LEKIWI_COLLISION_REVISION,
|
||||
LEKIWI_COLLISION_RECIPE_HASH,
|
||||
LEKIWI_COLLISION_SOURCE_HASHES,
|
||||
prepareLeKiwiUrdf,
|
||||
} from './lekiwi';
|
||||
|
||||
async function validateCollisionSources(manifest: ProjectManifest, entryPath: string) {
|
||||
async function validateMeshSources(
|
||||
manifest: ProjectManifest,
|
||||
entryPath: string,
|
||||
hashes: Record<string, string>,
|
||||
) {
|
||||
const directory = entryPath.slice(0, entryPath.lastIndexOf('/') + 1);
|
||||
await Promise.all(
|
||||
Object.entries(LEKIWI_COLLISION_SOURCE_HASHES).map(async ([path, expected]) => {
|
||||
Object.entries(hashes).map(async ([path, expected]) => {
|
||||
const file = manifest.files.find((candidate) => candidate.path === directory + path);
|
||||
if (!file || (await sha256(file.data)) !== expected)
|
||||
throw new RobotError('INCOMPATIBLE_MODEL', `碰撞体源网格缺失或 SHA-256 不匹配:${path}`);
|
||||
throw new RobotError('INCOMPATIBLE_MODEL', `模型源网格缺失或 SHA-256 不匹配:${path}`);
|
||||
}),
|
||||
);
|
||||
}
|
||||
@@ -58,12 +64,24 @@ export async function prepareRobotProject(
|
||||
'INCOMPATIBLE_MODEL',
|
||||
'LeKiwi 碰撞配方已更新;请从原始 URDF 重新转换,旧 MJCF 不会自动升级',
|
||||
);
|
||||
if (
|
||||
xml
|
||||
.querySelector('custom text[name="platform_lekiwi_visual_revision"]')
|
||||
?.getAttribute('data') !== String(LEKIWI_VISUAL_REVISION)
|
||||
)
|
||||
throw new RobotError(
|
||||
'INCOMPATIBLE_MODEL',
|
||||
'LeKiwi 完整轮视觉配方已更新;请从原始 URDF 重新转换,不能复用旧缺轮 MJCF',
|
||||
);
|
||||
// Never enhance twice. Compiled topology/actuators are checked by the backend.
|
||||
await validateCollisionSources(manifest, entryPath);
|
||||
await validateMeshSources(manifest, entryPath, LEKIWI_COLLISION_SOURCE_HASHES);
|
||||
await validateMeshSources(manifest, entryPath, sourceMeshes);
|
||||
return manifest;
|
||||
}
|
||||
await validateLeKiwiSource(source.data);
|
||||
await validateCollisionSources(manifest, entryPath);
|
||||
// Equivalent URDF text is not permission to silently replace its mechanical
|
||||
// assets. The fixed recipe also simplifies the chassis and all three wheels.
|
||||
await validateMeshSources(manifest, entryPath, sourceMeshes);
|
||||
const data = prepareLeKiwiUrdf(source.data);
|
||||
const files = manifest.files.map((f) =>
|
||||
f === source ? { ...f, data, size: data.byteLength } : f,
|
||||
|
||||
@@ -1,6 +1,7 @@
|
||||
import { webcrypto } from 'node:crypto';
|
||||
import {
|
||||
enhanceLeKiwiMjcf,
|
||||
LEKIWI_VISUAL_REVISION,
|
||||
LEKIWI_COLLISION_REVISION,
|
||||
LEKIWI_COLLISION_RECIPE_HASH,
|
||||
type LeKiwiJointGeometry,
|
||||
@@ -33,8 +34,20 @@ function source() {
|
||||
<inertial mass="1" pos="0 0 0" diaginertia="1 1 1"/>
|
||||
<joint name="${p.arm[i].joint}" axis="${i === 5 ? '0 -0.906308 -0.422618' : '1 0 0'}"/>
|
||||
${visuals(p.arm[i].joint)}${arm(i + 1)}</body>`;
|
||||
geometry.push(
|
||||
...p.wheels.map((name) => ({
|
||||
name,
|
||||
body: name,
|
||||
position: [0, 0, 0],
|
||||
wheelCenter: [0, 0, 0],
|
||||
rotation: [1, 0, 0, 0, 1, 0, 0, 0, 1],
|
||||
})),
|
||||
);
|
||||
const wheels = p.wheels
|
||||
.map((name) => `<body name="${name}"><joint name="${name}"/></body>`)
|
||||
.map(
|
||||
(name) =>
|
||||
`<body name="${name}"><joint name="${name}"/><geom name="${name}_visual" type="mesh" mesh="wheel" group="1"/></body>`,
|
||||
)
|
||||
.join('');
|
||||
return {
|
||||
data: encode(`<mujoco><asset/><worldbody><body name="${p.baseBody}">
|
||||
@@ -152,6 +165,7 @@ function manifest(revision?: number, hash = LEKIWI_COLLISION_RECIPE_HASH): Proje
|
||||
const data = encode(`<mujoco><custom>
|
||||
<text name="platform_robot_profile" data="${p.id}"/>
|
||||
<text name="platform_robot_profile_version" data="${p.version}"/>
|
||||
<text name="platform_lekiwi_visual_revision" data="${LEKIWI_VISUAL_REVISION}"/>
|
||||
<text name="platform_robot_source_sha256" data="${p.source.urdfSha256}"/>
|
||||
<text name="platform_lekiwi_collision_recipe_sha256" data="${hash}"/>
|
||||
${revision === undefined ? '' : `<text name="platform_lekiwi_collision_revision" data="${revision}"/>`}
|
||||
|
||||
@@ -5,16 +5,16 @@ import type { ProjectManifest } from '../types';
|
||||
|
||||
const encode = (xml: string) => new TextEncoder().encode(xml);
|
||||
describe('explicit LeKiwi profile preparation', () => {
|
||||
it('replaces only oversized wheels and preserves source bytes', () => {
|
||||
it('preserves complete wheel visuals and source bytes', () => {
|
||||
const xml =
|
||||
'<robot><link name="wheel"><visual><geometry><mesh filename="meshes/4-Omni-Directional-Wheel_Single_Body-v1.stl"/></geometry></visual></link><link name="arm"><visual><geometry><mesh filename="meshes/arm.stl"/></geometry></visual></link></robot>';
|
||||
const data = encode(xml),
|
||||
result = new TextDecoder().decode(prepareLeKiwiUrdf(data));
|
||||
expect(new TextDecoder().decode(data)).toBe(xml);
|
||||
expect(result).toContain('strippath="false"');
|
||||
expect(result).toContain('<sphere');
|
||||
expect(result).not.toContain('<sphere');
|
||||
expect(result).toContain('meshes/arm.stl');
|
||||
expect(result).not.toContain('Omni-Directional');
|
||||
expect(result).toContain('4-Omni-Directional-Wheel_Single_Body-v1.stl');
|
||||
});
|
||||
it('rejects invalid topology and repeated enhancement', () => {
|
||||
expect(() => enhanceLeKiwiMjcf(encode('<mujoco><worldbody/></mujoco>'), [])).toThrow(/基座/);
|
||||
@@ -43,7 +43,7 @@ describe('explicit LeKiwi profile preparation', () => {
|
||||
/未注册/,
|
||||
);
|
||||
await expect(prepareRobotProject(manifest, 'LeKiwi.urdf', 'lekiwi-v1')).rejects.toThrow(
|
||||
/版本不符/,
|
||||
/控制拓扑不匹配/,
|
||||
);
|
||||
} finally {
|
||||
vi.unstubAllGlobals();
|
||||
|
||||
@@ -1,7 +1,9 @@
|
||||
import { LEKIWI_PROFILE as p, WHEEL_TANGENTS } from '../../robot/profiles/lekiwi';
|
||||
import { RobotError } from '../../robot/types';
|
||||
import collision from '../../../../robot_profiles/lekiwi-full-collision.json';
|
||||
import { Matrix4, Quaternion, Vector3 } from 'three';
|
||||
|
||||
export const LEKIWI_VISUAL_REVISION = 1;
|
||||
export const LEKIWI_COLLISION_REVISION = collision.revision;
|
||||
export const LEKIWI_COLLISION_RECIPE_HASH = collision.recipeSha256;
|
||||
export const LEKIWI_COLLISION_SOURCE_HASHES = collision.source.meshes;
|
||||
@@ -13,6 +15,8 @@ export interface LeKiwiJointGeometry {
|
||||
body: string;
|
||||
position: number[];
|
||||
rotation: number[];
|
||||
/** World center of the CAD wheel visual, from MuJoCo's compiled mesh frame. */
|
||||
wheelCenter?: number[];
|
||||
}
|
||||
function parse(data: Uint8Array): Document {
|
||||
const doc = new DOMParser().parseFromString(decoder.decode(data), 'application/xml');
|
||||
@@ -28,9 +32,9 @@ function element(doc: Document, tag: string, attributes: Record<string, string |
|
||||
for (const [name, value] of Object.entries(attributes)) node.setAttribute(name, String(value));
|
||||
return node;
|
||||
}
|
||||
/** The CAD wheel STL has >300k faces, above MuJoCo's STL limit. This explicit,
|
||||
* source-hash-gated preprocessing removes only the oversized wheel visual/collision.
|
||||
* The original source files remain intact; the simulation recipe replaces wheels.
|
||||
/** Preserve full CAD visuals. The generic mesh compatibility pass converts large
|
||||
* STL references to lossless OBJ after source validation. Only temporary wheel
|
||||
* collision is simplified; the final recipe still installs passive rollers.
|
||||
*/
|
||||
export function prepareLeKiwiUrdf(data: Uint8Array): Uint8Array {
|
||||
const doc = parse(data),
|
||||
@@ -46,7 +50,7 @@ export function prepareLeKiwiUrdf(data: Uint8Array): Uint8Array {
|
||||
mujoco.append(compiler);
|
||||
}
|
||||
compiler.setAttribute('strippath', 'false');
|
||||
for (const mesh of doc.querySelectorAll('mesh[filename]')) {
|
||||
for (const mesh of doc.querySelectorAll('collision mesh[filename]')) {
|
||||
if (mesh.getAttribute('filename')!.includes('4-Omni-Directional-Wheel_Single_Body'))
|
||||
mesh.replaceWith(element(doc, 'sphere', { radius: p.wheelRadius }));
|
||||
}
|
||||
@@ -143,6 +147,7 @@ function addArmCollisions(doc: Document, cad: Element, contact: Element): void {
|
||||
export function enhanceLeKiwiMjcf(
|
||||
data: Uint8Array,
|
||||
geometry: readonly LeKiwiJointGeometry[],
|
||||
inputSha256 = p.source.urdfSha256,
|
||||
): Uint8Array {
|
||||
const doc = parse(data),
|
||||
root = doc.documentElement;
|
||||
@@ -176,7 +181,17 @@ export function enhanceLeKiwiMjcf(
|
||||
);
|
||||
for (const inertial of cad.querySelectorAll('inertial')) inertial.remove();
|
||||
for (const joint of cad.querySelectorAll(':scope > freejoint, :scope > joint')) joint.remove();
|
||||
for (const wheel of p.wheels) joints.get(wheel)!.parentElement!.remove();
|
||||
const wheelVisuals = p.wheels.map((name) => {
|
||||
const joint = joints.get(name)!;
|
||||
const body = joint.parentElement!;
|
||||
const frame = geometry.find((g) => g.name === name && g.body === body.getAttribute('name'));
|
||||
if (!frame?.wheelCenter || !body.querySelector('geom[group="1"][type="mesh"]'))
|
||||
throw new RobotError('INCOMPATIBLE_MODEL', `缺少完整轮视觉/运动学:${name}`);
|
||||
const axis = (joint.getAttribute('axis') ?? '0 0 1').split(/\s+/).map(Number);
|
||||
joint.remove();
|
||||
body.remove();
|
||||
return { body, frame, axis };
|
||||
});
|
||||
root.querySelector('actuator')?.remove();
|
||||
const actuators = element(doc, 'actuator', {});
|
||||
root.append(actuators);
|
||||
@@ -307,6 +322,36 @@ export function enhanceLeKiwiMjcf(
|
||||
diaginertia: '.00004 .00004 .00004',
|
||||
}),
|
||||
);
|
||||
// Map the original moving CAD subtree to the ideal wheel's axle. Keep every
|
||||
// visual vertex/scale and fixed child transform; change neither inertia nor
|
||||
// collision. The wheel center (not motor-mount origin) sits on the new axle.
|
||||
const { body: visual, frame, axis } = wheelVisuals[i];
|
||||
const r = frame.rotation;
|
||||
const sourceRotation = new Quaternion().setFromRotationMatrix(
|
||||
new Matrix4().set(r[0], r[1], r[2], 0, r[3], r[4], r[5], 0, r[6], r[7], r[8], 0, 0, 0, 0, 1),
|
||||
);
|
||||
const canonicalRotation = new Quaternion().setFromAxisAngle(
|
||||
new Vector3(0, 0, 1),
|
||||
p.mapping.cadYawRad,
|
||||
);
|
||||
const rotation = canonicalRotation.multiply(sourceRotation);
|
||||
const sourceAxis = new Vector3(...(axis as [number, number, number]))
|
||||
.normalize()
|
||||
.applyQuaternion(rotation);
|
||||
rotation.premultiply(
|
||||
new Quaternion().setFromUnitVectors(
|
||||
sourceAxis,
|
||||
new Vector3(...(axle as [number, number, number])).normalize(),
|
||||
),
|
||||
);
|
||||
const center = new Vector3(...(frame.wheelCenter! as [number, number, number])).sub(
|
||||
new Vector3(...(frame.position as [number, number, number])),
|
||||
);
|
||||
// SourceRotation was not mutated by canonicalRotation.multiply().
|
||||
center.applyQuaternion(sourceRotation.clone().invert()).applyQuaternion(rotation).negate();
|
||||
visual.setAttribute('pos', center.toArray().join(' '));
|
||||
visual.setAttribute('quat', [rotation.w, rotation.x, rotation.y, rotation.z].join(' '));
|
||||
wheel.append(visual);
|
||||
wheel.append(
|
||||
element(doc, 'geom', {
|
||||
name: `__lekiwi_${name}_hub`,
|
||||
@@ -315,7 +360,8 @@ export function enhanceLeKiwiMjcf(
|
||||
zaxis: axle.join(' '),
|
||||
contype: 0,
|
||||
conaffinity: 0,
|
||||
group: 1,
|
||||
// Diagnostic proxy only; full CAD supplies ordinary wheel rendering.
|
||||
group: 3,
|
||||
mass: 0,
|
||||
rgba: '.12 .15 .18 1',
|
||||
}),
|
||||
@@ -346,7 +392,7 @@ export function enhanceLeKiwiMjcf(
|
||||
size: '.009 .006',
|
||||
zaxis: axis.join(' '),
|
||||
mass: p.simulation.rollerMass,
|
||||
group: 0,
|
||||
group: 3,
|
||||
contype: 4,
|
||||
conaffinity: 1,
|
||||
friction: '1 .001 .0001',
|
||||
@@ -374,6 +420,12 @@ export function enhanceLeKiwiMjcf(
|
||||
const custom = root.querySelector('custom') ?? element(doc, 'custom', {});
|
||||
if (!custom.parentElement) root.append(custom);
|
||||
custom.append(element(doc, 'text', { name: 'platform_robot_profile', data: p.id }));
|
||||
custom.append(
|
||||
element(doc, 'text', {
|
||||
name: 'platform_lekiwi_visual_revision',
|
||||
data: LEKIWI_VISUAL_REVISION,
|
||||
}),
|
||||
);
|
||||
custom.append(
|
||||
element(doc, 'text', {
|
||||
name: 'platform_lekiwi_collision_revision',
|
||||
@@ -389,8 +441,10 @@ export function enhanceLeKiwiMjcf(
|
||||
custom.append(
|
||||
element(doc, 'text', { name: 'platform_robot_profile_version', data: String(p.version) }),
|
||||
);
|
||||
// Keep the reviewed recipe's source separate from the actual imported bytes.
|
||||
custom.append(
|
||||
element(doc, 'text', { name: 'platform_robot_source_sha256', data: p.source.urdfSha256 }),
|
||||
element(doc, 'text', { name: 'platform_robot_input_sha256', data: inputSha256 }),
|
||||
);
|
||||
return serialize(doc);
|
||||
}
|
||||
|
||||
@@ -0,0 +1,53 @@
|
||||
import { binaryStlFaceCount, convertStlToObj, MUJOCO_STL_FACE_LIMIT } from './stlConverter';
|
||||
|
||||
import { binaryStl } from '../test/stlFixture';
|
||||
|
||||
it.each([200_000, 200_001])('preserves all %i faces and original bytes', async (count) => {
|
||||
const input = binaryStl(count);
|
||||
const before = input.slice();
|
||||
const result = await convertStlToObj(input);
|
||||
const text = new TextDecoder().decode(result.data);
|
||||
expect(binaryStlFaceCount(input)).toBe(count);
|
||||
expect(result.faces).toBe(count);
|
||||
expect(result.vertices).toBe(3);
|
||||
expect(text.match(/^f /gm)).toHaveLength(count);
|
||||
expect(text.split('\n').slice(0, 4)).toEqual(['v 0 0 0', 'v 1 0 0', 'v 0 1 0', 'f 1 2 3']);
|
||||
expect(Buffer.from(input).equals(Buffer.from(before))).toBe(true);
|
||||
expect(MUJOCO_STL_FACE_LIMIT).toBe(200_000);
|
||||
});
|
||||
|
||||
it('does not quantize almost-identical float32 vertices or reverse winding', async () => {
|
||||
const data = binaryStl(2);
|
||||
const v = new DataView(data.buffer);
|
||||
v.setFloat32(84 + 50 + 12 + 12, 1.0000001192092896, true);
|
||||
const result = await convertStlToObj(data);
|
||||
expect(result.vertices).toBe(4);
|
||||
expect(new TextDecoder().decode(result.data)).toContain('v 1.0000001192092896 0 0');
|
||||
expect(new TextDecoder().decode(result.data)).toContain('f 1 4 3');
|
||||
});
|
||||
|
||||
it('rejects truncated/invalid declarations, non-finite vertices and bounded budgets', async () => {
|
||||
expect(() => binaryStlFaceCount(new Uint8Array(80))).toThrow(/头/);
|
||||
expect(() => binaryStlFaceCount(binaryStl(1).slice(0, -1))).toThrow(/长度/);
|
||||
expect(() => binaryStlFaceCount(binaryStl(0))).toThrow(/面数/);
|
||||
const bad = binaryStl(1);
|
||||
new DataView(bad.buffer).setFloat32(96, NaN, true);
|
||||
await expect(convertStlToObj(bad)).rejects.toThrow(/非法/);
|
||||
await expect(convertStlToObj(binaryStl(1), { maxOutputBytes: 10 })).rejects.toThrow(/预算/);
|
||||
await expect(convertStlToObj(binaryStl(1), { maxWorkingBytes: 10 })).rejects.toThrow(/预算/);
|
||||
});
|
||||
|
||||
it('supports byte offsets and cooperatively cancels without returning partial data', async () => {
|
||||
const data = binaryStl(5000);
|
||||
const buffer = new Uint8Array(data.length + 100);
|
||||
buffer.set(data, 50);
|
||||
let cancelled = false;
|
||||
await expect(
|
||||
convertStlToObj(buffer.subarray(50, 50 + data.length), {
|
||||
shouldCancel: () => cancelled,
|
||||
onProgress: () => {
|
||||
cancelled = true;
|
||||
},
|
||||
}),
|
||||
).rejects.toThrow(/取消/);
|
||||
});
|
||||
@@ -0,0 +1,87 @@
|
||||
/** MuJoCo 3.11's STL decoder has a format-specific 200,000 face limit.
|
||||
* Convert geometry, not fidelity: exact float32 vertices, original winding/all faces.
|
||||
* No quantization, welding tolerance, decimation, filesystem or network access.
|
||||
*/
|
||||
export const MUJOCO_STL_FACE_LIMIT = 200_000;
|
||||
export const STL_CONVERTER_VERSION = 'stl-obj-v1';
|
||||
const encoder = new TextEncoder();
|
||||
const BATCH_FACES = 4096;
|
||||
|
||||
export interface MeshConversionOptions {
|
||||
maxOutputBytes?: number;
|
||||
maxWorkingBytes?: number;
|
||||
shouldCancel?: () => boolean;
|
||||
onProgress?: (completed: number, total: number) => void;
|
||||
}
|
||||
|
||||
export function binaryStlFaceCount(data: Uint8Array): number {
|
||||
if (data.byteLength < 84) throw new Error('STL 二进制头不完整(不支持 ASCII STL)');
|
||||
const count = new DataView(data.buffer, data.byteOffset, data.byteLength).getUint32(80, true);
|
||||
if (!count || 84 + count * 50 !== data.byteLength)
|
||||
throw new Error('STL 声明面数与文件长度不符(需要有效二进制 STL)');
|
||||
return count;
|
||||
}
|
||||
|
||||
export async function convertStlToObj(
|
||||
data: Uint8Array,
|
||||
options: MeshConversionOptions = {},
|
||||
): Promise<{ data: Uint8Array; faces: number; vertices: number }> {
|
||||
const faces = binaryStlFaceCount(data);
|
||||
// Bound worst-case JS map/string/typed-array intermediates as well as output.
|
||||
// This is an admission estimate, not a promise about a browser's heap use.
|
||||
if (faces * 600 + data.byteLength > (options.maxWorkingBytes ?? 512 * 1024 * 1024))
|
||||
throw new Error('STL 转换峰值内存估算超过预算,请拆分网格或显式降低视觉精度');
|
||||
const limit = options.maxOutputBytes ?? 128 * 1024 * 1024;
|
||||
const view = new DataView(data.buffer, data.byteOffset, data.byteLength);
|
||||
const seen = new Map<string, number>();
|
||||
const vertexChunks: Uint8Array[] = [],
|
||||
faceChunks: Uint8Array[] = [];
|
||||
let bytes = 0;
|
||||
const append = (target: Uint8Array[], lines: string[]) => {
|
||||
const chunk = encoder.encode(lines.join('\n') + '\n');
|
||||
bytes += chunk.length;
|
||||
if (bytes > limit) throw new Error('派生 OBJ 超过文件或工程总大小预算');
|
||||
target.push(chunk);
|
||||
};
|
||||
const check = () => {
|
||||
if (options.shouldCancel?.()) throw new Error('网格转换已取消');
|
||||
};
|
||||
check();
|
||||
for (let first = 0; first < faces; first += BATCH_FACES) {
|
||||
check();
|
||||
const vertices: string[] = [],
|
||||
triangles: string[] = [];
|
||||
const end = Math.min(faces, first + BATCH_FACES);
|
||||
for (let i = first; i < end; i++) {
|
||||
const ids: number[] = [];
|
||||
for (let j = 0; j < 3; j++) {
|
||||
const offset = 84 + 50 * i + 12 + j * 12;
|
||||
const xyz = [0, 4, 8].map((k) => view.getFloat32(offset + k, true));
|
||||
if (xyz.some((value) => !Number.isFinite(value) || Math.abs(value) > 2 ** 30))
|
||||
throw new Error(`STL 第 ${i + 1} 面含非法/超范围顶点`);
|
||||
const key = [0, 4, 8].map((k) => view.getUint32(offset + k, true)).join(',');
|
||||
let id = seen.get(key);
|
||||
if (id === undefined) {
|
||||
id = seen.size + 1;
|
||||
seen.set(key, id);
|
||||
vertices.push(`v ${xyz.map((n) => (Object.is(n, -0) ? '-0' : String(n))).join(' ')}`);
|
||||
}
|
||||
ids.push(id);
|
||||
}
|
||||
triangles.push(`f ${ids.join(' ')}`);
|
||||
}
|
||||
if (vertices.length) append(vertexChunks, vertices);
|
||||
append(faceChunks, triangles);
|
||||
options.onProgress?.(end, faces);
|
||||
// Allow rendering, cancellation and disposal between bounded batches.
|
||||
if (end < faces) await new Promise<void>((resolve) => setTimeout(resolve, 0));
|
||||
}
|
||||
check();
|
||||
const output = new Uint8Array(bytes);
|
||||
let offset = 0;
|
||||
for (const chunk of [...vertexChunks, ...faceChunks]) {
|
||||
output.set(chunk, offset);
|
||||
offset += chunk.length;
|
||||
}
|
||||
return { data: output, faces, vertices: seen.size };
|
||||
}
|
||||
@@ -70,4 +70,13 @@ it('仅避障策略显示实时导航状态,支持暂停时设定/复位', ()
|
||||
/>,
|
||||
);
|
||||
expect(screen.queryByText('复位目标点')).not.toBeInTheDocument();
|
||||
view.rerender(
|
||||
<RLPolicyPanel
|
||||
{...props}
|
||||
status={{ ...status, taskId: 'MobileManipulator-LeKiwi-v1', observationSize: 68 }}
|
||||
/>,
|
||||
);
|
||||
expect(screen.getByText('分阶段移动操作')).toBeVisible();
|
||||
expect(screen.queryByText('速度指令')).not.toBeInTheDocument();
|
||||
expect(screen.getByText('68 / 12')).toBeVisible();
|
||||
});
|
||||
|
||||
@@ -1,3 +1,4 @@
|
||||
import { isMobileTrainingTask } from '../mobile/training';
|
||||
import { useRef, type ChangeEvent } from 'react';
|
||||
import { BrainCircuit, FileUp, Power, RotateCw, Trash2 } from 'lucide-react';
|
||||
import type { RLCommand, RLPolicyStatus } from './types';
|
||||
@@ -147,7 +148,14 @@ export function RLPolicyPanel({
|
||||
</div>
|
||||
</div>
|
||||
)}
|
||||
{status.observationSize === 81 || status.observationSize === 97 ? (
|
||||
{isMobileTrainingTask(status.taskId) ? (
|
||||
<div className="mt-2">
|
||||
<Badge tone="accent">分阶段移动操作</Badge>
|
||||
<p className="mt-1 text-xs text-warning">
|
||||
按部署阶段运行;底盘接近阶段保持机械臂。成功、超时或安全终止后暂停,请重置再继续。策略及元数据请从强化学习任务面板一起导入。
|
||||
</p>
|
||||
</div>
|
||||
) : status.observationSize === 81 || status.observationSize === 97 ? (
|
||||
<div
|
||||
className="mt-2"
|
||||
title="评测在超时、跌倒或越界时停止;水平射线存在矮障碍与跌落盲区"
|
||||
|
||||
@@ -31,9 +31,13 @@ export function ExternalControlPanel({
|
||||
const [profileId, setProfileId] = useState(snapshot.robot?.profileId ?? '');
|
||||
const external = snapshot.externalControl,
|
||||
ready = bridge.status.phase === 'ready';
|
||||
const owner = { manual: '手动', python: 'Python', policy: 'ONNX', external: '外部' }[
|
||||
snapshot.controlOwner ?? 'manual'
|
||||
];
|
||||
const owner = {
|
||||
manual: '手动',
|
||||
python: 'Python',
|
||||
policy: 'ONNX',
|
||||
external: '外部',
|
||||
agent: '智能任务',
|
||||
}[snapshot.controlOwner ?? 'manual'];
|
||||
return (
|
||||
<section aria-label="开源机器人控制" className="space-y-3 text-xs">
|
||||
<label className="block">
|
||||
|
||||
@@ -69,6 +69,6 @@ export function lekiwiDescriptor(modelFingerprint: string): RobotDescriptor {
|
||||
: 'm',
|
||||
})),
|
||||
],
|
||||
capabilities: { reset: true, lockstep: false, cameras: false, training: false },
|
||||
capabilities: { reset: true, lockstep: false, cameras: false, training: true },
|
||||
};
|
||||
}
|
||||
|
||||
@@ -0,0 +1,125 @@
|
||||
import { webcrypto } from 'node:crypto';
|
||||
import reference from '../../../../robot_profiles/lekiwi-reference.urdf?raw';
|
||||
import meshHashes from '../../../../robot_profiles/lekiwi-source-meshes.json';
|
||||
import collision from '../../../../robot_profiles/lekiwi-full-collision.json';
|
||||
import { lekiwiSourceMismatch } from './lekiwiSource';
|
||||
import { LEKIWI_PROFILE as p } from './lekiwi';
|
||||
import { sha256, validateLeKiwiSource } from '../registry';
|
||||
|
||||
const encode = (xml: string) => new TextEncoder().encode(xml);
|
||||
function changed(edit: (doc: Document) => void): Uint8Array {
|
||||
const doc = new DOMParser().parseFromString(reference, 'application/xml');
|
||||
edit(doc);
|
||||
return encode(new XMLSerializer().serializeToString(doc));
|
||||
}
|
||||
const joint = (doc: Document, name = 'arm_elbow_flex') =>
|
||||
doc.querySelector(`joint[name="${name}"]`)!;
|
||||
|
||||
beforeEach(() => vi.stubGlobal('crypto', webcrypto));
|
||||
afterEach(() => vi.unstubAllGlobals());
|
||||
|
||||
it('binds the checked-in reference and mesh inventory to the reviewed collision/profile source', async () => {
|
||||
expect(await sha256(encode(reference))).toBe(p.source.urdfSha256);
|
||||
expect(lekiwiSourceMismatch(encode(reference))).toBeUndefined();
|
||||
const doc = new DOMParser().parseFromString(reference, 'application/xml');
|
||||
expect(Object.keys(meshHashes).sort()).toEqual(
|
||||
[
|
||||
...new Set(Array.from(doc.querySelectorAll('mesh')).map((m) => m.getAttribute('filename'))),
|
||||
].sort(),
|
||||
);
|
||||
for (const [path, hash] of Object.entries(collision.source.meshes))
|
||||
expect(meshHashes[path as keyof typeof meshHashes]).toBe(hash);
|
||||
});
|
||||
|
||||
it('accepts display name, colors, whitespace, attribute/root order and numeric spellings only', async () => {
|
||||
const data = changed((doc) => {
|
||||
doc.documentElement.setAttribute('name', 'LeKiwi 换色版');
|
||||
doc.documentElement.append(doc.createComment('local export'));
|
||||
const visual = doc.querySelector('visual')!;
|
||||
const material = doc.createElement('material');
|
||||
material.setAttribute('name', 'red');
|
||||
const color = doc.createElement('color');
|
||||
color.setAttribute('rgba', '1 0 0 1');
|
||||
material.append(color);
|
||||
visual.append(material);
|
||||
const axis = joint(doc).querySelector('axis')!;
|
||||
axis.setAttribute('xyz', '1e0 -0.000 0');
|
||||
// Attribute order does not matter, but joint and geometry names still do.
|
||||
const node = joint(doc);
|
||||
node.removeAttribute('name');
|
||||
node.setAttribute('name', 'arm_elbow_flex');
|
||||
doc.documentElement.prepend(node);
|
||||
});
|
||||
expect(await sha256(data)).not.toBe(p.source.urdfSha256);
|
||||
await expect(validateLeKiwiSource(data)).resolves.toBeUndefined();
|
||||
});
|
||||
|
||||
it('compares equivalent Euler rotations as matrices, including gimbal-lock re-exports', () => {
|
||||
const data = changed((doc) => {
|
||||
joint(doc, 'base_back_wheel')
|
||||
.querySelector('origin')!
|
||||
.setAttribute('rpy', `0 ${Math.PI / 2} ${Math.PI / 2}`);
|
||||
doc.querySelector('visual origin')!.removeAttribute('rpy');
|
||||
doc.querySelector('collision origin')!.remove();
|
||||
});
|
||||
expect(lekiwiSourceMismatch(data)).toBeUndefined();
|
||||
});
|
||||
|
||||
it.each([
|
||||
['axis', (doc: Document) => joint(doc).querySelector('axis')!.setAttribute('xyz', '-1 0 0')],
|
||||
['origin', (doc: Document) => joint(doc).querySelector('origin')!.setAttribute('xyz', '0 .01 0')],
|
||||
[
|
||||
'mesh scale',
|
||||
(doc: Document) => doc.querySelector('mesh')!.setAttribute('scale', '.002 .001 .001'),
|
||||
],
|
||||
[
|
||||
'mesh path',
|
||||
(doc: Document) => doc.querySelector('mesh')!.setAttribute('filename', 'other.stl'),
|
||||
],
|
||||
['inertia', (doc: Document) => doc.querySelector('inertia')!.setAttribute('ixx', '.1')],
|
||||
['NaN', (doc: Document) => doc.querySelector('mass')!.setAttribute('value', 'NaN')],
|
||||
['mimic', (doc: Document) => joint(doc).append(doc.createElement('mimic'))],
|
||||
['limits', (doc: Document) => joint(doc).append(doc.createElement('limit'))],
|
||||
['compiler', (doc: Document) => doc.documentElement.append(doc.createElement('mujoco'))],
|
||||
['visual name', (doc: Document) => doc.querySelector('visual')!.setAttribute('name', 'new')],
|
||||
])('rejects same-topology physical/recipe changes: %s', async (_label, edit) => {
|
||||
await expect(validateLeKiwiSource(changed(edit))).rejects.toThrow(/不能自动复用/);
|
||||
});
|
||||
|
||||
it('diagnoses changed parents and joint types even with exactly the expected nine names', () => {
|
||||
expect(
|
||||
lekiwiSourceMismatch(
|
||||
changed((doc) => joint(doc).querySelector('parent')!.setAttribute('link', p.baseBody)),
|
||||
),
|
||||
).toContain('运动链连接变化');
|
||||
expect(
|
||||
lekiwiSourceMismatch(changed((doc) => joint(doc).setAttribute('type', 'prismatic'))),
|
||||
).toContain('关节类型变化');
|
||||
expect(
|
||||
lekiwiSourceMismatch(changed((doc) => doc.documentElement.append(joint(doc).cloneNode(true)))),
|
||||
).toContain('实际 10 个');
|
||||
});
|
||||
|
||||
it('does not infer Link1…Link7 as five arm channels and a normalized gripper', async () => {
|
||||
const data = changed((doc) => {
|
||||
p.arm.forEach((spec, index) => {
|
||||
const node = joint(doc, spec.joint);
|
||||
node.setAttribute('name', `Link${index + 1}`);
|
||||
node.setAttribute('type', 'revolute');
|
||||
});
|
||||
const seventh = doc.createElement('joint');
|
||||
seventh.setAttribute('name', 'Link7');
|
||||
seventh.setAttribute('type', 'revolute');
|
||||
doc.documentElement.append(seventh);
|
||||
});
|
||||
await expect(validateLeKiwiSource(data)).rejects.toThrow(/需要 9 个活动关节,实际 10 个/);
|
||||
expect(lekiwiSourceMismatch(data)).toContain('arm_gripper');
|
||||
expect(lekiwiSourceMismatch(data)).toContain('Link7');
|
||||
});
|
||||
|
||||
it.each(['<robot/>', '<mujoco/>', '<robot>', '<!DOCTYPE robot><robot/>'])(
|
||||
'fails closed on incomplete/invalid inputs: %s',
|
||||
async (xml) => {
|
||||
await expect(validateLeKiwiSource(encode(xml))).rejects.toThrow(/不兼容/);
|
||||
},
|
||||
);
|
||||
@@ -0,0 +1,142 @@
|
||||
import reference from '../../../../robot_profiles/lekiwi-reference.urdf?raw';
|
||||
|
||||
/** A deliberately narrow compatibility gate, NOT a generic robot/FK mapper.
|
||||
* Ignore presentation only. Geometry, inertia, limits, extensions and the entire
|
||||
* fixed/moving tree must still match the reviewed source. Mesh bytes are checked
|
||||
* separately by the project loader, including meshes outside the arm recipe.
|
||||
*/
|
||||
const numeric: Record<string, readonly string[]> = {
|
||||
origin: ['xyz', 'rpy'],
|
||||
axis: ['xyz'],
|
||||
mass: ['value'],
|
||||
inertia: ['ixx', 'iyy', 'izz', 'ixy', 'ixz', 'iyz'],
|
||||
mesh: ['scale'],
|
||||
limit: ['lower', 'upper', 'effort', 'velocity'],
|
||||
dynamics: ['damping', 'friction'],
|
||||
box: ['size'],
|
||||
sphere: ['radius'],
|
||||
cylinder: ['radius', 'length'],
|
||||
};
|
||||
const parse = (xml: string) => new DOMParser().parseFromString(xml, 'application/xml');
|
||||
function children(node: Element): Element[] {
|
||||
const result = Array.from(node.children).filter(
|
||||
(child) => !(child.tagName === 'material' && ['robot', 'visual'].includes(node.tagName)),
|
||||
);
|
||||
// An omitted origin is the identity, not a topology change. Compare a virtual
|
||||
// default without mutating either the imported XML or the reference.
|
||||
if (
|
||||
['joint', 'inertial', 'visual', 'collision'].includes(node.tagName) &&
|
||||
!result.some((child) => child.tagName === 'origin')
|
||||
)
|
||||
result.push(node.ownerDocument.createElement('origin'));
|
||||
return result.sort((a, b) =>
|
||||
`${a.tagName}:${a.getAttribute('name') ?? ''}`.localeCompare(
|
||||
`${b.tagName}:${b.getAttribute('name') ?? ''}`,
|
||||
),
|
||||
);
|
||||
}
|
||||
const vector = (text: string) => (text.trim() ? text.trim().split(/\s+/).map(Number) : [NaN]);
|
||||
// Compare Rz(yaw) Ry(pitch) Rx(roll), not ambiguous Euler angle spellings.
|
||||
function rotation(values: number[]): number[] {
|
||||
if (values.length !== 3) return [NaN];
|
||||
const [r, p, y] = values,
|
||||
[sr, sp, sy] = values.map(Math.sin),
|
||||
[cr, cp, cy] = [r, p, y].map(Math.cos);
|
||||
return [
|
||||
cy * cp,
|
||||
cy * sp * sr - sy * cr,
|
||||
cy * sp * cr + sy * sr,
|
||||
sy * cp,
|
||||
sy * sp * sr + cy * cr,
|
||||
sy * sp * cr - cy * sr,
|
||||
-sp,
|
||||
cp * sr,
|
||||
cp * cr,
|
||||
];
|
||||
}
|
||||
function attributes(node: Element): Map<string, string> {
|
||||
const result = new Map(Array.from(node.attributes).map((a) => [a.name, a.value]));
|
||||
if (node.tagName === 'robot') result.delete('name');
|
||||
if (node.tagName === 'origin') {
|
||||
if (!result.has('xyz')) result.set('xyz', '0 0 0');
|
||||
if (!result.has('rpy')) result.set('rpy', '0 0 0');
|
||||
}
|
||||
if (node.tagName === 'mesh' && !result.has('scale')) result.set('scale', '1 1 1');
|
||||
return result;
|
||||
}
|
||||
function difference(actual: Element, expected: Element, parent = ''): string | undefined {
|
||||
const path = `${parent}/${expected.tagName}${expected.hasAttribute('name') ? `[${expected.getAttribute('name')}]` : ''}`;
|
||||
if (actual.tagName !== expected.tagName) return `${path} 元素类型变化`;
|
||||
// URDF has no character data in physical elements. Do not ignore extensions
|
||||
// or injected text while ignoring ordinary whitespace/comments.
|
||||
if (
|
||||
Array.from(actual.childNodes).some(
|
||||
(node) => (node.nodeType === 3 || node.nodeType === 4) && node.textContent?.trim(),
|
||||
)
|
||||
)
|
||||
return `${path} 包含不支持的文本`;
|
||||
const a = attributes(actual),
|
||||
b = attributes(expected);
|
||||
for (const key of new Set([...a.keys(), ...b.keys()])) {
|
||||
if (!a.has(key) || !b.has(key)) return `${path}/@${key} 属性变化`;
|
||||
if (numeric[expected.tagName]?.includes(key)) {
|
||||
let av = vector(a.get(key)!),
|
||||
bv = vector(b.get(key)!);
|
||||
if (expected.tagName === 'origin' && key === 'rpy') {
|
||||
av = rotation(av);
|
||||
bv = rotation(bv);
|
||||
}
|
||||
if (
|
||||
av.length !== bv.length ||
|
||||
av.some(
|
||||
(v, i) => !Number.isFinite(v) || Math.abs(v - bv[i]) > 1e-12 + 1e-9 * Math.abs(bv[i]),
|
||||
)
|
||||
)
|
||||
return `${path}/@${key} 数值变化`;
|
||||
} else if (a.get(key) !== b.get(key)) return `${path}/@${key} 变化`;
|
||||
}
|
||||
const ac = children(actual),
|
||||
bc = children(expected);
|
||||
if (ac.length !== bc.length) return `${path} 子元素数量变化`;
|
||||
for (let i = 0; i < bc.length; i++) {
|
||||
const reason = difference(ac[i], bc[i], path);
|
||||
if (reason) return reason;
|
||||
}
|
||||
}
|
||||
|
||||
export function lekiwiSourceMismatch(data: Uint8Array): string | undefined {
|
||||
const doc = parse(new TextDecoder().decode(data));
|
||||
if (doc.querySelector('parsererror') || doc.documentElement.tagName !== 'robot' || doc.doctype)
|
||||
return 'URDF XML 无效或包含不支持的 DOCTYPE';
|
||||
const expected = parse(reference).documentElement,
|
||||
actual = doc.documentElement;
|
||||
const joints = (root: Element) =>
|
||||
Array.from(root.children).filter((node) => node.tagName === 'joint');
|
||||
const movable = (root: Element) =>
|
||||
joints(root).filter((node) => node.getAttribute('type') !== 'fixed');
|
||||
const oldNames = movable(expected).map((node) => node.getAttribute('name')!),
|
||||
newNames = movable(actual).map((node) => node.getAttribute('name')!);
|
||||
const missing = oldNames.filter((name) => !newNames.includes(name)),
|
||||
extra = newNames.filter((name) => !oldNames.includes(name));
|
||||
if (missing.length || extra.length || oldNames.length !== newNames.length)
|
||||
return `控制拓扑不匹配:需要 ${oldNames.length} 个活动关节,实际 ${newNames.length} 个;缺少 [${missing.join(', ')}];新增 [${extra.join(', ')}]。不能按顺序重命名或把新关节当作原夹爪`;
|
||||
for (const joint of joints(expected)) {
|
||||
const candidates = joints(actual).filter(
|
||||
(node) => node.getAttribute('name') === joint.getAttribute('name'),
|
||||
);
|
||||
if (candidates.length !== 1) return `运动链缺少或重复关节:${joint.getAttribute('name')}`;
|
||||
const incoming = candidates[0];
|
||||
for (const tag of ['parent', 'child'])
|
||||
if (
|
||||
incoming.querySelector(`:scope > ${tag}`)?.getAttribute('link') !==
|
||||
joint.querySelector(`:scope > ${tag}`)?.getAttribute('link')
|
||||
)
|
||||
return `运动链连接变化:${joint.getAttribute('name')}/${tag}`;
|
||||
if (incoming.getAttribute('type') !== joint.getAttribute('type'))
|
||||
return `关节类型变化:${joint.getAttribute('name')}`;
|
||||
}
|
||||
const reason = difference(actual, expected);
|
||||
return reason
|
||||
? `${reason};现有碰撞/控制配方不能自动复用,需重新审核模型参数与碰撞数据`
|
||||
: undefined;
|
||||
}
|
||||
@@ -17,9 +17,12 @@ export async function sha256(data: Uint8Array): Promise<string> {
|
||||
);
|
||||
}
|
||||
export async function validateLeKiwiSource(data: Uint8Array): Promise<void> {
|
||||
if ((await sha256(data)) !== LEKIWI_PROFILE.source.urdfSha256)
|
||||
if ((await sha256(data)) === LEKIWI_PROFILE.source.urdfSha256) return;
|
||||
const { lekiwiSourceMismatch } = await import('./profiles/lekiwiSource');
|
||||
const reason = lekiwiSourceMismatch(data);
|
||||
if (reason)
|
||||
throw new RobotError(
|
||||
'INCOMPATIBLE_MODEL',
|
||||
`LeKiwi URDF 与受支持的 ${LEKIWI_PROFILE.source.revision.slice(0, 8)} 版本不符;请使用原始 URDF,不能仅凭关节数量套用 profile`,
|
||||
`LeKiwi URDF 不兼容:${reason}。参见 docs/lekiwi-urdf-compatibility.md`,
|
||||
);
|
||||
}
|
||||
|
||||
@@ -17,7 +17,7 @@ export interface RobotDescriptor {
|
||||
frame: 'x-forward-y-left-z-up';
|
||||
actionChannels: RobotChannel[];
|
||||
observationChannels: RobotChannel[];
|
||||
capabilities: { reset: boolean; lockstep: false; cameras: false; training: false };
|
||||
capabilities: { reset: boolean; lockstep: false; cameras: false; training: boolean };
|
||||
}
|
||||
export interface RobotIdentity {
|
||||
sessionId: string;
|
||||
|
||||
@@ -81,9 +81,9 @@ export function validateDescriptor(value: unknown): RobotDescriptor {
|
||||
typeof caps.reset !== 'boolean' ||
|
||||
caps.lockstep !== false ||
|
||||
caps.cameras !== false ||
|
||||
caps.training !== false
|
||||
typeof caps.training !== 'boolean'
|
||||
)
|
||||
throw new RobotError('UNSUPPORTED', 'V1 不支持相机、锁步或训练');
|
||||
throw new RobotError('UNSUPPORTED', 'V1 不支持相机或锁步;training 必须是布尔值');
|
||||
return {
|
||||
protocolVersion: 1,
|
||||
profileId: identifier(d.profileId),
|
||||
@@ -92,7 +92,7 @@ export function validateDescriptor(value: unknown): RobotDescriptor {
|
||||
frame: 'x-forward-y-left-z-up',
|
||||
actionChannels: channels(d.actionChannels),
|
||||
observationChannels: channels(d.observationChannels),
|
||||
capabilities: { reset: caps.reset, lockstep: false, cameras: false, training: false },
|
||||
capabilities: { reset: caps.reset, lockstep: false, cameras: false, training: caps.training },
|
||||
};
|
||||
}
|
||||
/** Complete channel set; reject non-numeric input rather than coercing strings/bools. */
|
||||
|
||||
@@ -0,0 +1,10 @@
|
||||
/** Only the session physics clock may call beforeStep/afterStep. Network handlers
|
||||
* must enqueue validated decisions, never invoke the supplied write capability. */
|
||||
export interface AgentDriver {
|
||||
beforeStep(): boolean;
|
||||
afterStep(dt: number): void;
|
||||
hold(): void;
|
||||
cancel(reason: string): void;
|
||||
dispose(): void;
|
||||
}
|
||||
export type AgentFactory = (writeServo: (actuatorId: number, value: number) => void) => AgentDriver;
|
||||
@@ -1,4 +1,4 @@
|
||||
export type ControlOwner = 'manual' | 'python' | 'policy' | 'external';
|
||||
export type ControlOwner = 'manual' | 'python' | 'policy' | 'external' | 'agent';
|
||||
export interface ControlTicket {
|
||||
readonly owner: ControlOwner;
|
||||
readonly generation: number;
|
||||
|
||||
@@ -1,3 +1,5 @@
|
||||
import type { RobotConfig } from '../mobile/RobotDescriptor';
|
||||
import { composeMobileScene, prepareBundle } from '../mobile/SceneComposer';
|
||||
import {
|
||||
composeTrainingMap,
|
||||
trainingTerrainFromCompiledScene,
|
||||
@@ -8,6 +10,7 @@ import type { PolicyDeployment, TrainingTerrain } from '../rl/deployment';
|
||||
import type { MainModule } from '@mujoco/mujoco';
|
||||
import type { ProjectFile, ProjectManifest } from '../project/types';
|
||||
import { prepareProjectForMujoco } from '../project/importer';
|
||||
import { prepareLargeMeshes } from '../project/meshCompatibility';
|
||||
import { prepareRobotProject } from '../project/robotProfiles';
|
||||
import { sha256 } from '../robot/registry';
|
||||
import type { ExternalControlStatus } from '../robot/RobotRuntime';
|
||||
@@ -56,6 +59,13 @@ export interface PhysicsLoadProgress {
|
||||
export interface PhysicsLoadOptions {
|
||||
/** Explicit built-in profile; never inferred from actuator count. */
|
||||
robotProfileId?: string;
|
||||
/** false: reuse model recipe without installing the external-control runtime. */
|
||||
configureRobotRuntime?: boolean;
|
||||
/** Optional caller-owned free VFS slot; bounds native asset-cache path identities. */
|
||||
workspaceSlot?: string;
|
||||
/** Trusted application hook, after URDF/map conversion and include expansion. */
|
||||
sceneComposer?: (mjcf: Uint8Array) => Uint8Array;
|
||||
mobileRobot?: RobotConfig;
|
||||
trainingDeployment?: PolicyDeployment;
|
||||
/** Candidate policy is initialized/validated before replacing the active session. */
|
||||
trainingPolicy?: { data: Uint8Array; path: string };
|
||||
@@ -158,6 +168,8 @@ export class MainThreadPhysicsAdapter implements PhysicsAdapter {
|
||||
}
|
||||
>();
|
||||
private loadGeneration = 0;
|
||||
private readonly workspacePrefix = `adapter_${crypto.randomUUID().replaceAll('-', '_')}`;
|
||||
private readonly candidateRoots = new Set<string>();
|
||||
private disposed = false;
|
||||
private retiredSession: SimulationSession | null = null;
|
||||
private retiredWorkspace: MemfsWorkspace | null = null;
|
||||
@@ -169,6 +181,7 @@ export class MainThreadPhysicsAdapter implements PhysicsAdapter {
|
||||
options: PhysicsLoadOptions = {},
|
||||
): Promise<SimulationSnapshot> {
|
||||
if (this.disposed) throw new Error('物理适配器已释放');
|
||||
this.session?.stopAgent?.('模型或地图正在重载');
|
||||
if (this.session?.describeRobot?.())
|
||||
this.session.stopExternalControl('模型正在重载,请重新授权');
|
||||
const generation = ++this.loadGeneration;
|
||||
@@ -190,19 +203,76 @@ export class MainThreadPhysicsAdapter implements PhysicsAdapter {
|
||||
const module = await getMujocoModule();
|
||||
if (this.disposed || generation !== this.loadGeneration) throw new Error('模型加载已取消');
|
||||
report(0.32, '校验并准备模型资源');
|
||||
const workspace = new MemfsWorkspace(module, `${manifest.id}_stage_${generation}`);
|
||||
if (
|
||||
options.robotProfileId &&
|
||||
(urdfMode !== 'mjcf' || baseMode !== 'floating' || options.trainingDeployment)
|
||||
)
|
||||
throw new Error('机器人 profile 需要浮动基座 URDF 转换,不能与 Go2 部署混用');
|
||||
const mobileRobot = options.mobileRobot;
|
||||
if (
|
||||
mobileRobot &&
|
||||
(options.trainingDeployment || urdfMode !== 'mjcf' || baseMode !== 'floating')
|
||||
)
|
||||
throw new Error('移动操作需要浮动基座 MJCF,不能与 Go2 部署混用');
|
||||
const sourceManifest =
|
||||
mobileRobot?.recipe === 'lekiwi-bundle'
|
||||
? prepareBundle(manifest, entryPath, mobileRobot)
|
||||
: manifest;
|
||||
const profileManifest = options.robotProfileId
|
||||
? await prepareRobotProject(manifest, entryPath, options.robotProfileId)
|
||||
: manifest;
|
||||
const prepared = await prepareProjectForMujoco(profileManifest, entryPath);
|
||||
? await prepareRobotProject(sourceManifest, entryPath, options.robotProfileId)
|
||||
: sourceManifest;
|
||||
const normalized = await prepareProjectForMujoco(profileManifest, entryPath);
|
||||
const prepared = await prepareLargeMeshes(normalized.manifest, entryPath, {
|
||||
shouldCancel: () => this.disposed || generation !== this.loadGeneration,
|
||||
onMeshProgress: (path, done, total) =>
|
||||
report(
|
||||
0.34 + (0.1 * done) / total,
|
||||
`兼容高面数网格 ${path}(${done}/${total} 面,保留全部三角形)`,
|
||||
),
|
||||
});
|
||||
prepared.warnings.unshift(...normalized.warnings);
|
||||
// Map files may not be in the robot's include graph. Prepare the selected
|
||||
// physics assets before MapComposer rebases them into the resulting scene.
|
||||
const mapPaths = new Set<string>();
|
||||
for (const selection of placedMapAssets?.map((asset) => asset.selection) ?? [mapSelection]) {
|
||||
if (selection.kind === 'project') {
|
||||
const map = resolveProjectMap(prepared.manifest, selection.descriptorPath);
|
||||
if (map.physicsPath) mapPaths.add(map.physicsPath);
|
||||
}
|
||||
}
|
||||
for (const path of mapPaths) {
|
||||
const next = await prepareLargeMeshes(prepared.manifest, path, {
|
||||
shouldCancel: () => this.disposed || generation !== this.loadGeneration,
|
||||
});
|
||||
prepared.manifest = next.manifest;
|
||||
prepared.warnings.push(...next.warnings);
|
||||
}
|
||||
if (this.disposed || generation !== this.loadGeneration) throw new Error('模型加载已取消');
|
||||
const supportFiles = prepared.manifest.files.filter(
|
||||
(file) => !manifest.files.some((original) => original.path === file.path),
|
||||
);
|
||||
// Bounded path identities stop the native mesh cache accumulating a new
|
||||
// copy on every reload. Three slots permit active + rollback + candidate;
|
||||
// callers with explicit two-slot viewer transactions retain their contract.
|
||||
const freeSlot = [0, 1, 2]
|
||||
.map((i) => `${this.workspacePrefix}_${i}`)
|
||||
.find((slot) => {
|
||||
const root = `/workspace/${slot}`;
|
||||
return (
|
||||
root !== this.workspace?.root &&
|
||||
root !== this.retiredWorkspace?.root &&
|
||||
!this.candidateRoots.has(root)
|
||||
);
|
||||
});
|
||||
if (!options.workspaceSlot && !freeSlot) throw new Error('候选 VFS 槽已满,请等待当前加载结束');
|
||||
const workspace = new MemfsWorkspace(module, options.workspaceSlot ?? freeSlot!);
|
||||
if (
|
||||
workspace.root === this.workspace?.root ||
|
||||
workspace.root === this.retiredWorkspace?.root ||
|
||||
this.candidateRoots.has(workspace.root)
|
||||
)
|
||||
throw new Error('候选 VFS 槽仍被活动/待释放模型占用');
|
||||
this.candidateRoots.add(workspace.root);
|
||||
let nextSession: SimulationSession | null = null;
|
||||
try {
|
||||
report(0.46, '写入浏览器内存文件系统');
|
||||
@@ -237,8 +307,33 @@ export class MainThreadPhysicsAdapter implements PhysicsAdapter {
|
||||
const bodyId = Number(intermediate.model.jnt_bodyid[id]);
|
||||
const body = intermediate.model.body(bodyId);
|
||||
try {
|
||||
let wheelCenter: number[] | undefined;
|
||||
if (joint.name.startsWith('base_') && joint.name.endsWith('_wheel')) {
|
||||
for (let geomId = 0; geomId < intermediate.model.ngeom; geomId++) {
|
||||
const geom = intermediate.model.geom(geomId);
|
||||
try {
|
||||
if (
|
||||
!geom.name.startsWith('4-Omni-Directional-Wheel_Single_Body') ||
|
||||
Number(intermediate.model.geom_group[geomId]) !== 1
|
||||
)
|
||||
continue;
|
||||
let ancestor = Number(intermediate.model.geom_bodyid[geomId]);
|
||||
while (ancestor && ancestor !== bodyId)
|
||||
ancestor = Number(intermediate.model.body_parentid[ancestor]);
|
||||
if (ancestor === bodyId) {
|
||||
wheelCenter = Array.from(
|
||||
intermediate.data.geom_xpos.slice(geomId * 3, geomId * 3 + 3),
|
||||
);
|
||||
break;
|
||||
}
|
||||
} finally {
|
||||
geom.delete();
|
||||
}
|
||||
}
|
||||
}
|
||||
geometry.push({
|
||||
name: joint.name,
|
||||
wheelCenter,
|
||||
body: body.name,
|
||||
position: Array.from(
|
||||
intermediate.data.xpos.slice(bodyId * 3, bodyId * 3 + 3),
|
||||
@@ -254,9 +349,12 @@ export class MainThreadPhysicsAdapter implements PhysicsAdapter {
|
||||
body.delete();
|
||||
}
|
||||
}
|
||||
convertedData = enhanceLeKiwiMjcf(convertedData, geometry);
|
||||
const inputHash = await sha256(
|
||||
manifest.files.find((file) => file.path === entryPath)!.data,
|
||||
);
|
||||
convertedData = enhanceLeKiwiMjcf(convertedData, geometry, inputHash);
|
||||
warnings.unshift(
|
||||
'LeKiwi v1:仿真估计参数,简化被动滚子/碰撞与九路伺服;超大轮 STL 已替换为简化视觉,不代表实机标定',
|
||||
'LeKiwi v1:完整 CAD 轮视觉;碰撞仍使用被动滚子/整臂凸分解与九路伺服,轮距/动力学为仿真估计,不代表实机标定',
|
||||
);
|
||||
}
|
||||
workspace.writeGenerated(convertedPath, convertedData);
|
||||
@@ -342,10 +440,36 @@ export class MainThreadPhysicsAdapter implements PhysicsAdapter {
|
||||
}
|
||||
warnings.push('已用策略配套训练布局替换场景地形;编辑器地图未更改。Go2-W动力学不等同Go2。');
|
||||
}
|
||||
const sceneComposer = mobileRobot
|
||||
? (xml: Uint8Array) => {
|
||||
const mobileScene = composeMobileScene(xml, mobileRobot);
|
||||
return options.sceneComposer ? options.sceneComposer(mobileScene) : mobileScene;
|
||||
}
|
||||
: options.sceneComposer;
|
||||
if (sceneComposer) {
|
||||
if (entry?.format === 'urdf' && urdfMode === 'native')
|
||||
throw new Error('任务场景组合需要 MJCF 转换模式');
|
||||
const intermediate = new SimulationSession(module, modelPath);
|
||||
try {
|
||||
const scenePath = `${modelRelativePath}.task.xml`;
|
||||
// Register before native save so failed composition is cleaned up too.
|
||||
workspace.writeGenerated(scenePath, new Uint8Array());
|
||||
if (!module.mj_saveLastXML(workspace.path(scenePath), intermediate.model))
|
||||
throw new Error('无法展开任务场景');
|
||||
workspace.writeGenerated(
|
||||
scenePath,
|
||||
sceneComposer(new TextEncoder().encode(workspace.readText(scenePath))),
|
||||
);
|
||||
modelRelativePath = scenePath;
|
||||
modelPath = workspace.path(scenePath);
|
||||
} finally {
|
||||
intermediate.dispose();
|
||||
}
|
||||
}
|
||||
report(0.84, '编译模型与物理数据');
|
||||
console.info('[MuJoCo] 编译模型', modelPath);
|
||||
nextSession = new SimulationSession(module, modelPath, warnings);
|
||||
if (options.robotProfileId) {
|
||||
if (options.robotProfileId && options.configureRobotRuntime !== false && !mobileRobot) {
|
||||
const fingerprint = await sha256(
|
||||
new TextEncoder().encode(workspace.readText(modelRelativePath)),
|
||||
);
|
||||
@@ -411,6 +535,7 @@ export class MainThreadPhysicsAdapter implements PhysicsAdapter {
|
||||
});
|
||||
}
|
||||
}
|
||||
if (mobileRobot) nextSession.configureMobile(mobileRobot);
|
||||
if (options.trainingPolicy) {
|
||||
if (!options.trainingDeployment?.terrain) throw new Error('事务策略加载需要配套训练地图');
|
||||
report(0.9, '校验候选场景的 ONNX 策略');
|
||||
@@ -445,6 +570,10 @@ export class MainThreadPhysicsAdapter implements PhysicsAdapter {
|
||||
`模型编译失败(${entryPath}):${error instanceof Error ? error.message : String(error)}`,
|
||||
{ cause: error },
|
||||
);
|
||||
} finally {
|
||||
this.candidateRoots.delete(workspace.root);
|
||||
this.session?.mobile?.env.refreshViews();
|
||||
this.retiredSession?.mobile?.env.refreshViews();
|
||||
}
|
||||
}
|
||||
exportTrainingTerrain(
|
||||
|
||||
@@ -0,0 +1,165 @@
|
||||
import { SimulationSession } from './SimulationSession';
|
||||
import { ControlArbiter } from './ControlArbiter';
|
||||
import type { AgentDriver } from './AgentDriver';
|
||||
|
||||
function fixture() {
|
||||
const data = {
|
||||
ctrl: new Float64Array([0.4]),
|
||||
qpos: new Float64Array([0.3]),
|
||||
xfrc_applied: new Float64Array(6),
|
||||
time: 0,
|
||||
};
|
||||
const arbiter = new ControlArbiter();
|
||||
const session = Object.assign(Object.create(SimulationSession.prototype), {
|
||||
disposed: false,
|
||||
paused: false,
|
||||
speed: 1,
|
||||
forceBody: -1,
|
||||
perturb: { active: 0 },
|
||||
frameBudgetMs: 8,
|
||||
maxCatchUpSteps: 100,
|
||||
accumulator: 0,
|
||||
controlArbiter: arbiter,
|
||||
data,
|
||||
model: {
|
||||
nactuator: 1,
|
||||
nu: 1,
|
||||
actuator_ctrladr: [0],
|
||||
opt: { timestep: 0.001, delete: vi.fn() },
|
||||
actuator: () => ({ ctrllimited: true, ctrlrange: [-1, 1], delete: vi.fn() }),
|
||||
},
|
||||
robotAdapter: { safeStop: vi.fn() },
|
||||
robotRuntime: {
|
||||
revoke: vi.fn(),
|
||||
capture: vi.fn(),
|
||||
reset: vi.fn(),
|
||||
afterStep: vi.fn(),
|
||||
authorize: vi.fn(),
|
||||
},
|
||||
module: {
|
||||
mj_step: () => {
|
||||
data.time += 0.001;
|
||||
},
|
||||
mj_forward: vi.fn(),
|
||||
mj_resetData: vi.fn(),
|
||||
},
|
||||
dataRecorder: { capture: vi.fn(), simulationReset: vi.fn() },
|
||||
startExternalClock: vi.fn(),
|
||||
}) as SimulationSession;
|
||||
return { session, arbiter, data };
|
||||
}
|
||||
function driver(overrides: Partial<AgentDriver> = {}): AgentDriver {
|
||||
return {
|
||||
beforeStep: () => true,
|
||||
afterStep: vi.fn(),
|
||||
hold: vi.fn(),
|
||||
cancel: vi.fn(),
|
||||
dispose: vi.fn(),
|
||||
...overrides,
|
||||
};
|
||||
}
|
||||
it('external-force interaction revokes an agent and pauses before any further step', () => {
|
||||
const f = fixture(),
|
||||
agent = driver();
|
||||
let write!: (id: number, value: number) => void;
|
||||
f.session.startAgent((cap) => {
|
||||
write = cap;
|
||||
return agent;
|
||||
});
|
||||
f.session.setExternalForce(1, [1, 0, 0]);
|
||||
expect(f.session.controlOwner).toBe('manual');
|
||||
expect(f.session.paused).toBe(true);
|
||||
expect(agent.cancel).toHaveBeenCalledWith('外力交互使旧计划失效');
|
||||
expect(() => write(0, 0.8)).toThrow();
|
||||
});
|
||||
it('only synchronous physics hooks can use agent writes; manual and asynchronous writes are rejected', () => {
|
||||
const f = fixture();
|
||||
let write!: (id: number, value: number) => void;
|
||||
const agent = driver({
|
||||
beforeStep: () => {
|
||||
write(0, 0.7);
|
||||
return true;
|
||||
},
|
||||
});
|
||||
f.session.startAgent((cap) => {
|
||||
write = cap;
|
||||
return agent;
|
||||
});
|
||||
expect(f.arbiter.current().owner).toBe('agent');
|
||||
expect(() => write(0, 0.9)).toThrow(/物理步/);
|
||||
expect(() => f.session.setActuator(0, 0.9)).toThrow(/停止当前/);
|
||||
f.session.singleStep();
|
||||
expect(f.data.ctrl[0]).toBe(0.7);
|
||||
expect(agent.afterStep).toHaveBeenCalledWith(0.001);
|
||||
});
|
||||
it.each(['pause', 'reset', 'external'] as const)(
|
||||
'%s revokes the old ticket, cancels the request, and preserves arm hold',
|
||||
(kind) => {
|
||||
const f = fixture();
|
||||
let write!: (id: number, value: number) => void;
|
||||
const agent = driver({ hold: () => write(0, 0.4) });
|
||||
f.session.startAgent((cap) => {
|
||||
write = cap;
|
||||
return agent;
|
||||
});
|
||||
if (kind === 'pause') f.session.setPaused(true);
|
||||
if (kind === 'reset') f.session.reset();
|
||||
if (kind === 'external') f.session.setExternalControlEnabled(true);
|
||||
expect(agent.cancel).toHaveBeenCalled();
|
||||
expect(() => write(0, 0.9)).toThrow(/票据/);
|
||||
if (kind !== 'reset') expect(f.data.ctrl[0]).toBe(0.4);
|
||||
expect(f.data.qpos[0]).toBe(0.3);
|
||||
},
|
||||
);
|
||||
it('a replaced callback cannot piggyback on the next agent physics write window', () => {
|
||||
const f = fixture();
|
||||
let old!: (id: number, value: number) => void;
|
||||
const first = driver();
|
||||
f.session.startAgent((cap) => {
|
||||
old = cap;
|
||||
return first;
|
||||
});
|
||||
f.session.startAgent(() =>
|
||||
driver({
|
||||
beforeStep: () => {
|
||||
expect(() => old(0, 0.9)).toThrow(/票据/);
|
||||
return true;
|
||||
},
|
||||
}),
|
||||
);
|
||||
f.session.singleStep();
|
||||
expect(first.dispose).toHaveBeenCalledOnce();
|
||||
expect(f.data.ctrl[0]).toBe(0.4);
|
||||
});
|
||||
it('model waits freeze the one clock, and resumed frames never catch up the wait', () => {
|
||||
const f = fixture();
|
||||
let ready = false;
|
||||
f.session.startAgent(() => driver({ beforeStep: () => ready }));
|
||||
f.session.advance(0);
|
||||
f.session.advance(1000);
|
||||
expect(f.data.time).toBe(0);
|
||||
ready = true;
|
||||
f.session.advance(200000);
|
||||
expect(f.data.time).toBe(0);
|
||||
f.session.advance(200001);
|
||||
expect(f.data.time).toBeCloseTo(0.001);
|
||||
});
|
||||
it('stopping from a post-step hook prevents remaining frame substeps', () => {
|
||||
const f = fixture();
|
||||
f.session.startAgent(() => driver({ afterStep: () => f.session.stopAgent('done') }));
|
||||
f.session.advance(0);
|
||||
f.session.advance(50);
|
||||
expect(f.data.time).toBeCloseTo(0.001);
|
||||
expect(f.session.paused).toBe(true);
|
||||
expect(f.arbiter.current().owner).toBe('manual');
|
||||
});
|
||||
it('factory failure fails paused/manual instead of retaining an orphan agent owner', () => {
|
||||
const f = fixture();
|
||||
expect(() =>
|
||||
f.session.startAgent(() => {
|
||||
throw new Error('binding');
|
||||
}),
|
||||
).toThrow('binding');
|
||||
expect(f.session.paused).toBe(true);
|
||||
expect(f.arbiter.current().owner).toBe('manual');
|
||||
});
|
||||
@@ -1,3 +1,6 @@
|
||||
import { MobilePolicyController } from '../mobile/MobilePolicyController';
|
||||
import { MOBILE_TASK, type RobotConfig } from '../mobile/RobotDescriptor';
|
||||
import type { MobileDeployment } from '../mobile/training';
|
||||
import { Go2ObstacleAvoidanceBindings } from '../rl/runtime/Go2ObstacleAvoidanceBindings';
|
||||
import {
|
||||
OBSTACLE_TASK_ID,
|
||||
@@ -10,6 +13,7 @@ import { meshIdFromSceneDataId } from './geometry';
|
||||
import { PythonControllerRuntime } from '../controller/PythonControllerRuntime';
|
||||
import { ModelBindings } from './ModelBindings';
|
||||
import { ControlArbiter, type ControlOwner, type ControlTicket } from './ControlArbiter';
|
||||
import type { AgentDriver, AgentFactory } from './AgentDriver';
|
||||
import { MuJoCoRobotAdapter } from './MuJoCoRobotAdapter';
|
||||
import { RobotRuntime, type ExternalControlStatus } from '../robot/RobotRuntime';
|
||||
import {
|
||||
@@ -96,6 +100,7 @@ export interface SimulationSnapshot {
|
||||
warnings: string[];
|
||||
controller?: ControllerStatus;
|
||||
rlPolicy?: RLPolicyStatus;
|
||||
mobileRobotId?: string;
|
||||
controlOwner?: ControlOwner;
|
||||
robot?: RobotDescriptor;
|
||||
externalControl?: ExternalControlStatus;
|
||||
@@ -135,12 +140,30 @@ export class SimulationSession {
|
||||
private pythonController?: PythonControllerRuntime;
|
||||
private controllerLoadGeneration = 0;
|
||||
private rlPolicy?: OnnxPolicyRuntime;
|
||||
mobile?: MobilePolicyController;
|
||||
configureMobile(config: RobotConfig): void {
|
||||
this.mobile = new MobilePolicyController(this, config);
|
||||
}
|
||||
async loadMobilePolicy(
|
||||
model: Uint8Array,
|
||||
path: string,
|
||||
deployment: MobileDeployment,
|
||||
): Promise<RLPolicyStatus> {
|
||||
if (!this.mobile) throw new Error('请先加载移动操作任务场景');
|
||||
this.setPaused(true);
|
||||
await this.mobile.load(model, path, deployment);
|
||||
this.setRLPolicyEnabled(true);
|
||||
return this.mobile.status()!;
|
||||
}
|
||||
private obstacleBindings?: Go2ObstacleAvoidanceBindings;
|
||||
private deployment?: PolicyDeployment;
|
||||
private deploymentInitialQpos?: Float64Array;
|
||||
private rlPolicyLoadGeneration = 0;
|
||||
private dataRecorder!: DataRecorder;
|
||||
private controlArbiter?: ControlArbiter;
|
||||
private agentDriver?: AgentDriver;
|
||||
private agentScope?: ControllerWriteScope;
|
||||
private agentWritingScope?: ControllerWriteScope;
|
||||
private pythonScope?: ControllerWriteScope;
|
||||
private policyScope?: ControllerWriteScope;
|
||||
private robotAdapter?: RobotAdapter;
|
||||
@@ -174,6 +197,9 @@ export class SimulationSession {
|
||||
}
|
||||
}, 1000 / 120);
|
||||
}
|
||||
get controlOwner(): ControlOwner {
|
||||
return this.arbiter.current().owner;
|
||||
}
|
||||
private get arbiter(): ControlArbiter {
|
||||
return (this.controlArbiter ??= new ControlArbiter());
|
||||
}
|
||||
@@ -207,18 +233,99 @@ export class SimulationSession {
|
||||
return this.robotRuntime?.latest();
|
||||
}
|
||||
private clearControlTargets(): void {
|
||||
if (this.robotAdapter) this.robotAdapter.safeStop();
|
||||
// Zero ctrl is a full position jump for mobile arm servos, not a safe stop.
|
||||
if (this.mobile) this.mobile.env.hold();
|
||||
else if (this.robotAdapter) this.robotAdapter.safeStop();
|
||||
else this.data.ctrl.fill(0);
|
||||
}
|
||||
private selectControl(owner: ControlOwner): ControlTicket {
|
||||
private selectControl(owner: ControlOwner, reason = '控制权已切换'): ControlTicket {
|
||||
this.invalidateAgent(reason);
|
||||
this.stopExternalClock();
|
||||
const ticket = this.arbiter.claim(owner);
|
||||
this.pythonController?.setEnabled(false, Number(this.data.time));
|
||||
this.rlPolicy?.setEnabled(false, Number(this.data.time));
|
||||
this.mobile?.setEnabled(false);
|
||||
this.robotRuntime?.revoke('控制权已切换', false);
|
||||
this.clearControlTargets();
|
||||
return ticket;
|
||||
}
|
||||
startAgent(factory: AgentFactory): void {
|
||||
if (this.disposed || (!this.robotAdapter && !this.mobile))
|
||||
throw new Error('智能任务需要已配置的机器人环境');
|
||||
this.invalidateAgent('智能任务已替换');
|
||||
this.agentDriver?.dispose();
|
||||
this.agentDriver = undefined;
|
||||
const scope: ControllerWriteScope = { ticket: this.selectControl('agent') };
|
||||
this.agentScope = scope;
|
||||
this.clearExternalForce();
|
||||
try {
|
||||
this.agentDriver = factory((id, value) => {
|
||||
if (this.disposed || !this.arbiter.owns(scope.ticket) || this.agentWritingScope !== scope)
|
||||
throw new Error('智能任务写入票据已失效或不在物理步内');
|
||||
this.writeActuator(id, value);
|
||||
});
|
||||
this.mobile?.env.refreshViews();
|
||||
this.withAgentWrite(() => this.agentDriver!.hold());
|
||||
this.accumulator = 0;
|
||||
this.lastNow = undefined;
|
||||
} catch (error) {
|
||||
this.mobile?.env.refreshViews();
|
||||
this.invalidateAgent('智能任务初始化失败');
|
||||
this.agentDriver?.dispose();
|
||||
this.agentDriver = undefined;
|
||||
this.arbiter.claim('manual');
|
||||
this.paused = true;
|
||||
this.clearControlTargets();
|
||||
throw error;
|
||||
}
|
||||
}
|
||||
stopAgent(reason = '智能任务已停止'): void {
|
||||
if (this.arbiter.current().owner === 'agent') {
|
||||
this.selectControl('manual', reason);
|
||||
this.paused = true;
|
||||
this.accumulator = 0;
|
||||
this.lastNow = undefined;
|
||||
} else this.invalidateAgent(reason);
|
||||
}
|
||||
private withAgentWrite<T>(action: () => T): T {
|
||||
const scope = this.agentScope;
|
||||
if (!scope || !this.arbiter.owns(scope.ticket)) throw new Error('智能任务控制权失效');
|
||||
this.agentWritingScope = scope;
|
||||
try {
|
||||
return action();
|
||||
} finally {
|
||||
this.agentWritingScope = undefined;
|
||||
}
|
||||
}
|
||||
private invalidateAgent(reason: string): void {
|
||||
if (!this.agentDriver) return;
|
||||
try {
|
||||
if (!this.disposed && this.arbiter.owns(this.agentScope?.ticket))
|
||||
this.withAgentWrite(() => this.agentDriver!.hold());
|
||||
} catch {
|
||||
if (!this.disposed) this.clearControlTargets();
|
||||
console.warn('[MuJoCo] 智能任务保持失败,已使用机器人安全保持');
|
||||
} finally {
|
||||
if (this.agentScope) this.agentScope.ticket = undefined;
|
||||
try {
|
||||
this.agentDriver.cancel(reason);
|
||||
} catch {
|
||||
console.warn('[MuJoCo] 智能任务取消回调失败;写入票据已撤销');
|
||||
}
|
||||
}
|
||||
}
|
||||
private afterAgentStep(): void {
|
||||
if (this.arbiter.current().owner !== 'agent' || !this.agentDriver) return;
|
||||
const opt = this.model.opt,
|
||||
dt = Number(opt.timestep);
|
||||
opt.delete();
|
||||
try {
|
||||
this.withAgentWrite(() => this.agentDriver!.afterStep(dt));
|
||||
} catch (error) {
|
||||
this.stopAgent('智能任务物理回调失败');
|
||||
throw error;
|
||||
}
|
||||
}
|
||||
setExternalControlEnabled(enabled: boolean): void {
|
||||
if (!enabled) {
|
||||
this.stopExternalControl();
|
||||
@@ -324,9 +431,11 @@ export class SimulationSession {
|
||||
}
|
||||
|
||||
setPaused(paused: boolean): void {
|
||||
if (paused && this.arbiter.current().owner === 'agent') this.stopAgent('智能任务已暂停');
|
||||
if (paused && this.arbiter.current().owner === 'external')
|
||||
this.stopExternalControl('仿真已暂停,请重新授权');
|
||||
this.paused = paused;
|
||||
this.mobile?.setPaused(paused);
|
||||
this.accumulator = 0;
|
||||
this.lastNow = undefined;
|
||||
this.robotRuntime?.capture(Number(this.data.time));
|
||||
@@ -346,17 +455,22 @@ export class SimulationSession {
|
||||
this.pythonController?.reset(Number(this.data.time));
|
||||
this.rlPolicy?.reset(Number(this.data.time));
|
||||
this.robotRuntime?.reset();
|
||||
this.mobile?.reset();
|
||||
this.module.mj_forward(this.model, this.data);
|
||||
this.robotRuntime?.capture(Number(this.data.time));
|
||||
this.dataRecorder.simulationReset();
|
||||
}
|
||||
singleStep(): void {
|
||||
if (this.arbiter.current().owner === 'agent' && this.paused)
|
||||
throw new RobotError('PAUSED', '智能任务不能在暂停时单步');
|
||||
if (this.mobile && this.arbiter.current().owner === 'policy') return;
|
||||
if (this.arbiter.current().owner === 'external' && this.paused)
|
||||
throw new RobotError('PAUSED', '外部控制不能在暂停时单步');
|
||||
if (!this.runController()) return;
|
||||
this.applyForce();
|
||||
this.module.mj_step(this.model, this.data);
|
||||
this.robotRuntime?.afterStep(Number(this.data.time));
|
||||
this.afterAgentStep();
|
||||
this.dataRecorder.capture();
|
||||
}
|
||||
|
||||
@@ -377,20 +491,40 @@ export class SimulationSession {
|
||||
this.lastNow = now;
|
||||
if (this.paused) return { steps: 0, stepMs: 0, overBudget: false };
|
||||
this.accumulator += elapsed * this.speed;
|
||||
const dt = Number(this.model.opt.timestep) || 0.002;
|
||||
const mobile = this.arbiter.current().owner === 'policy' ? this.mobile : undefined;
|
||||
const dt = mobile ? MOBILE_TASK.controlDt : Number(this.model.opt.timestep) || 0.002;
|
||||
if (mobile) this.accumulator = Math.min(this.accumulator, 0.1);
|
||||
const started = performance.now();
|
||||
let steps = 0;
|
||||
while (
|
||||
!this.paused &&
|
||||
this.accumulator >= dt &&
|
||||
steps < this.maxCatchUpSteps &&
|
||||
performance.now() - started < this.frameBudgetMs
|
||||
) {
|
||||
if (!this.runController()) break;
|
||||
this.applyForce();
|
||||
this.module.mj_step(this.model, this.data);
|
||||
this.robotRuntime?.afterStep(Number(this.data.time));
|
||||
if (mobile) {
|
||||
try {
|
||||
if (!mobile.tick()) {
|
||||
if (mobile.error) this.setPaused(true);
|
||||
break;
|
||||
}
|
||||
if (mobile.ended) {
|
||||
this.setPaused(true);
|
||||
break;
|
||||
}
|
||||
} catch (error) {
|
||||
this.setPaused(true);
|
||||
throw error;
|
||||
}
|
||||
} else {
|
||||
if (!this.runController()) break;
|
||||
this.applyForce();
|
||||
this.module.mj_step(this.model, this.data);
|
||||
this.robotRuntime?.afterStep(Number(this.data.time));
|
||||
this.afterAgentStep();
|
||||
}
|
||||
this.dataRecorder.capture();
|
||||
this.accumulator -= dt;
|
||||
this.accumulator = Math.max(0, this.accumulator - dt);
|
||||
steps++;
|
||||
}
|
||||
const overBudget = this.accumulator >= dt;
|
||||
@@ -527,6 +661,7 @@ export class SimulationSession {
|
||||
path: string,
|
||||
deployment?: PolicyDeployment,
|
||||
): Promise<RLPolicyStatus> {
|
||||
if (this.mobile) throw new Error('移动操作场景需要匹配的 ONNX + deployment 元数据');
|
||||
const generation = ++this.rlPolicyLoadGeneration;
|
||||
if (deployment) deployment = validatePolicyDeployment(deployment);
|
||||
if (
|
||||
@@ -565,6 +700,13 @@ export class SimulationSession {
|
||||
}
|
||||
|
||||
setRLPolicyEnabled(enabled: boolean): void {
|
||||
if (this.mobile) {
|
||||
if (enabled && this.mobile.status()) {
|
||||
this.selectControl('policy');
|
||||
this.mobile.setEnabled(true);
|
||||
} else if (this.arbiter.current().owner === 'policy') this.selectControl('manual');
|
||||
return;
|
||||
}
|
||||
if (enabled && this.rlPolicy && this.policyScope) {
|
||||
this.policyScope.ticket = this.selectControl('policy');
|
||||
this.rlPolicy.setEnabled(true, Number(this.data.time));
|
||||
@@ -601,6 +743,7 @@ export class SimulationSession {
|
||||
this.rlPolicyLoadGeneration += 1;
|
||||
if (this.arbiter.current().owner === 'policy') this.selectControl('manual');
|
||||
this.policyScope = undefined;
|
||||
this.mobile?.remove();
|
||||
this.rlPolicy?.dispose();
|
||||
this.rlPolicy = undefined;
|
||||
this.obstacleBindings = undefined;
|
||||
@@ -621,6 +764,15 @@ export class SimulationSession {
|
||||
private runController(): boolean {
|
||||
try {
|
||||
const owner = this.arbiter.current().owner;
|
||||
if (
|
||||
owner === 'agent' &&
|
||||
(!this.agentDriver || !this.withAgentWrite(() => this.agentDriver!.beforeStep()))
|
||||
) {
|
||||
// Model waits freeze physics and discard accumulated wall time.
|
||||
this.accumulator = 0;
|
||||
this.lastNow = undefined;
|
||||
return false;
|
||||
}
|
||||
if (owner === 'external' && !this.robotRuntime?.beforeStep()) return false;
|
||||
if (owner === 'python') this.pythonController?.stepIfDue(Number(this.data.time));
|
||||
if (owner === 'policy') this.rlPolicy?.step(Number(this.data.time));
|
||||
@@ -733,8 +885,11 @@ export class SimulationSession {
|
||||
}
|
||||
}
|
||||
|
||||
private get robotMechanicsLocked(): boolean {
|
||||
return Boolean(this.robotAdapter || this.mobile);
|
||||
}
|
||||
setActuatorParameters(id: number, parameters: ActuatorParameters): boolean {
|
||||
if (this.arbiter.current().owner !== 'manual' || this.robotAdapter) return false;
|
||||
if (this.arbiter.current().owner !== 'manual' || this.robotMechanicsLocked) return false;
|
||||
if (id < 0 || id >= this.model.nactuator) return false;
|
||||
const finite = (value: number, fallback: number) => (Number.isFinite(value) ? value : fallback);
|
||||
const ordered = (
|
||||
@@ -813,7 +968,7 @@ export class SimulationSession {
|
||||
// Robot profiles use physical drives. Direct qpos writes teleport through
|
||||
// colliders, even with perfect collision meshes. Keep pose editing for generic
|
||||
// models; a configured robot is controlled via its associated actuators.
|
||||
if (this.robotAdapter) return false;
|
||||
if (this.robotMechanicsLocked) return false;
|
||||
if (id < 0 || id >= this.model.njnt) return false;
|
||||
const joint = this.model.jnt(id);
|
||||
try {
|
||||
@@ -837,7 +992,7 @@ export class SimulationSession {
|
||||
resetJoints(): void {
|
||||
if (this.arbiter.current().owner !== 'manual')
|
||||
throw new RobotError('CONFLICT', '请先停止当前控制器');
|
||||
if (this.robotAdapter)
|
||||
if (this.robotMechanicsLocked)
|
||||
throw new RobotError('CONFLICT', '机器人 profile 不允许关节瞬移;请使用驱动器或完整仿真重置');
|
||||
this.setPaused(true);
|
||||
for (let id = 0; id < this.model.njnt; id += 1) {
|
||||
@@ -858,7 +1013,7 @@ export class SimulationSession {
|
||||
setIgnoreJointLimits(ignore: boolean): void {
|
||||
if (this.arbiter.current().owner !== 'manual')
|
||||
throw new RobotError('CONFLICT', '请先停止当前控制器');
|
||||
if (this.robotAdapter)
|
||||
if (this.robotMechanicsLocked)
|
||||
throw new RobotError('CONFLICT', '机器人 profile 的关节限位不可修改;请使用物理驱动器控制');
|
||||
this.ignoreJointLimits = ignore;
|
||||
for (let id = 0; id < this.model.njnt; id += 1) {
|
||||
@@ -882,6 +1037,7 @@ export class SimulationSession {
|
||||
}
|
||||
|
||||
setExternalForce(bodyId: number, force: [number, number, number]): void {
|
||||
if (this.arbiter.current().owner === 'agent') this.stopAgent('外力交互使旧计划失效');
|
||||
this.forceBody = bodyId > 0 && bodyId < this.model.nbody ? bodyId : -1;
|
||||
this.force = force;
|
||||
}
|
||||
@@ -892,6 +1048,7 @@ export class SimulationSession {
|
||||
this.perturb.active = 0;
|
||||
}
|
||||
initializePerturb(scene: MjvScene, bodyId: number): void {
|
||||
if (this.arbiter.current().owner === 'agent') this.stopAgent('外力拖动使旧计划失效');
|
||||
this.perturb.select = bodyId;
|
||||
this.module.mjv_initPerturb(this.model, this.data, scene, this.perturb);
|
||||
}
|
||||
@@ -1176,7 +1333,7 @@ export class SimulationSession {
|
||||
limited,
|
||||
limitsIgnored: this.ignoreJointLimits,
|
||||
editable:
|
||||
!this.robotAdapter &&
|
||||
!this.robotMechanicsLocked &&
|
||||
(type === 2 || type === 3) &&
|
||||
this.arbiter.current().owner === 'manual',
|
||||
bodyId: Number(joint.bodyid),
|
||||
@@ -1213,7 +1370,8 @@ export class SimulationSession {
|
||||
bodies,
|
||||
warnings: this.warnings,
|
||||
controller: this.pythonController?.status(),
|
||||
rlPolicy: this.rlPolicy?.status(),
|
||||
rlPolicy: this.mobile?.status() ?? this.rlPolicy?.status(),
|
||||
mobileRobotId: this.mobile?.env.config.id,
|
||||
controlOwner: this.arbiter.current().owner,
|
||||
robot: this.describeRobot(),
|
||||
externalControl: this.robotRuntime?.status(),
|
||||
@@ -1242,9 +1400,15 @@ export class SimulationSession {
|
||||
};
|
||||
this.stopExternalClock();
|
||||
this.arbiter.claim('manual');
|
||||
cleanup('智能任务', () => {
|
||||
this.invalidateAgent('仿真会话已释放');
|
||||
this.agentDriver?.dispose();
|
||||
this.agentDriver = undefined;
|
||||
});
|
||||
cleanup('机器人接口', () => this.robotRuntime?.dispose());
|
||||
cleanup('Python 控制器', () => this.removeController());
|
||||
cleanup('RL 策略', () => this.removeRLPolicy());
|
||||
cleanup('移动操作环境', () => this.mobile?.dispose());
|
||||
cleanup('外力状态', () => this.clearExternalForce());
|
||||
cleanup('扰动对象', () => this.perturb.delete());
|
||||
cleanup('仿真数据', () => this.data.delete());
|
||||
|
||||
@@ -0,0 +1,11 @@
|
||||
/** Deliberately repeated triangles: parser tests must preserve every face. */
|
||||
export function binaryStl(faces: number): Uint8Array {
|
||||
const data = new Uint8Array(84 + 50 * faces);
|
||||
const v = new DataView(data.buffer);
|
||||
v.setUint32(80, faces, true);
|
||||
for (let i = 0; i < faces; i++) {
|
||||
v.setFloat32(84 + 50 * i + 24, 1, true);
|
||||
v.setFloat32(84 + 50 * i + 40, 1, true);
|
||||
}
|
||||
return data;
|
||||
}
|
||||
@@ -4,6 +4,21 @@ import { LocalTrainingClient } from './LocalTrainingClient';
|
||||
afterEach(() => vi.unstubAllGlobals());
|
||||
|
||||
describe('LocalTrainingClient', () => {
|
||||
it('移动场景二进制上传与部署元数据均携带认证,作业请求只传服务器快照ID', async () => {
|
||||
const fetchMock = vi.fn().mockImplementation(() => Promise.resolve(new Response('{}')));
|
||||
vi.stubGlobal('fetch', fetchMock);
|
||||
const client = new LocalTrainingClient('http://localhost:8765', 'secret');
|
||||
const file = new File(['scene'], 'scene.zip');
|
||||
await client.uploadMobileScene(file);
|
||||
await client.downloadMobileDeployment('a'.repeat(32));
|
||||
expect(fetchMock.mock.calls[0][0]).toContain('/api/training/mobile-packages');
|
||||
const options = fetchMock.mock.calls[0][1] as RequestInit;
|
||||
expect(options.body).toBe(file);
|
||||
expect(new Headers(options.headers).get('Content-Type')).toBe('application/zip');
|
||||
expect(new Headers(options.headers).get('Authorization')).toBe('Bearer secret');
|
||||
expect(fetchMock.mock.calls[1][0]).toContain('/artifacts/deployment.json');
|
||||
expect(() => client.uploadMobileScene(new File([], 'empty.zip'))).toThrow('128 MiB');
|
||||
});
|
||||
it('规范化服务地址并提交受类型约束的 JSON 请求', async () => {
|
||||
const fetchMock = vi.fn().mockResolvedValue(
|
||||
new Response(JSON.stringify({ id: 'a'.repeat(32), state: 'queued' }), {
|
||||
|
||||
@@ -1,3 +1,4 @@
|
||||
import type { MobileDeployment } from '../mobile/training';
|
||||
import type {
|
||||
ParameterConstraint,
|
||||
PretrainedSource,
|
||||
@@ -79,6 +80,20 @@ export class LocalTrainingClient {
|
||||
});
|
||||
}
|
||||
start(request: TrainingRequest): Promise<TrainingJob> {
|
||||
return this.startJob(request);
|
||||
}
|
||||
uploadMobileScene(file: File): Promise<{ id: string; robotId: string; sceneSha256: string }> {
|
||||
if (!file.size || file.size > 128 * 1024 ** 2) throw new Error('场景快照上限 128 MiB');
|
||||
return this.json('/api/training/mobile-packages', {
|
||||
method: 'POST',
|
||||
headers: { 'Content-Type': 'application/zip' },
|
||||
body: file,
|
||||
});
|
||||
}
|
||||
downloadMobileDeployment(id: string): Promise<MobileDeployment> {
|
||||
return this.json(`/api/training/jobs/${encodeURIComponent(id)}/artifacts/deployment.json`);
|
||||
}
|
||||
startJob(request: TrainingRequest): Promise<TrainingJob> {
|
||||
return this.json('/api/training/jobs', {
|
||||
method: 'POST',
|
||||
headers: { 'Content-Type': 'application/json' },
|
||||
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user