Files
2026-11-03 19:46:08 +08:00

6.5 KiB

name, description
name description
rl-engineer Engineer humanoid reward revisions, bounded mjlab training, measured evaluation, replay and research-only ONNX packaging in Universal Humanoid.

RL engineer

1. Establish a contract

Use this as the cross-stage coordinator; consult setup-env, new-task, train, controls and deploy through the manual index in AGENTS.md for stage-specific work.

Read AGENTS.md, README.md, and the target experiment. Confirm the robot/task, desired behavior, resource budget and acceptance metrics. A request to build architecture authorizes short smoke validation, not days of training or physical robot operation. Use the current coding agent directly; do not launch subagents without operator authorization.

Run ./scripts/uh doctor and list. Inspect the factory's source and relevant mjlab reward functions before editing. Follow docs/SETUP.md to verify the installed vendored backends, locked dependencies and bundled assets. Use verify-vendor to check the source inventory. Do not install a second mjlab/rsl-rl-lib distribution. Record actual dependency provenance.

2. Baseline and reward hypothesis

Copy a baseline experiment into a NEW JSON file with an explicit task and seed. Write the hypothesis next to the experiment or in a run note. Inspect uh rewards --config FILE.

  • Weights/parameters: edit JSON; misspelled term names must fail.
  • New mechanisms: add vectorized functions to src/universal_humanoid/rewards.py and the FUNCTIONS allowlist; add tests for shape [num_envs], finiteness, sign and limiting cases.
  • Do not change observations/control/mechanics while claiming a reward-only comparison.
  • Never eval code supplied in experiment JSON.

Run baseline evaluation/export before editing Python. Source hashes are strict; archived sources may be restored only into a separate trusted workspace, never over a shared reference checkout.

3. Cheap checks before expensive runs

Run ./scripts/check, uh plan, uh rewards, and uh check-robot --config FILE --output NEW_REPORT, then uh smoke with 4 environments and 16 steps. For each new robot, verify free root, DoF, mass/inertia, mesh resolution, contacts, foot sites, IMU frame, torque limits, gains, action ordering, reset posture and observation terms. Model compilation alone cannot validate these semantics. Zero-action success is not balance training.

4. Bounded training

Use uh train --config FILE --output NEW_DIRECTORY, or pipeline for a smoke-sized end-to-end run. Read docs/TRAINING_OPERATIONS.md for lifecycle and statistics semantics. Use train --background for a requested long-running Linux job, status RUN to query progress, checkpoints RUN to inspect committed checkpoints, and stop RUN for a guarded graceful stop. Resume with train --resume CHECKPOINT --iterations ADDITIONAL --output NEW_DIRECTORY; optimizer continuation is not bitwise simulator/RNG replay. Never guess checkpoints by mtime.

Keep TensorBoard local; no W&B authentication or automatic upload. Use curves RUN --output NEW_DIR to inspect tracking, action distribution std and episode-length trends DURING training. Report missing tags, non-finite samples and short curves. Rising total reward alone is not learning. Inspect run.json, progress.json, resolved_env.yaml, resolved_agent.json and sources.zip.

Long runs require an explicit operator-approved budget. Do not automatically increase parallelism past measured VRAM capacity. This backend requires CUDA; do not pretend Warp CPU is fast CPU RL.

5. Evaluate and replay

Use benchmark --checkpoints ... --seeds ... --profiles randomized stress --output NEW_DIR for stand (0,0,0), forward (0.3,0,0), lateral (0,0.2,0), turn (0,0,0.3). Keep identical seeds, profiles, world counts and horizons for candidates. These are evaluation seeds, NOT independent training seeds; separately train multiple seeds when investigating training variance.

Default evaluation randomizes each world's reset state. stress also enables actor observation noise and per-world interval velocity kicks; inspect actual initial states and push counts. nominal intentionally repeats fixed initial states. Startup mass/friction randomization is not part of these profiles. None constitutes full sim-to-real stress coverage.

Fall/time-limit terminations remain active; curriculum is disabled. Report RMSE, completed-only failure fraction, failure events per simulated world-second, first-episode failures AND censoring. No completed episodes means completed-only means/fractions are null. Variance is across seed-level reports, not repeated worlds, and is null for fewer than two values. Prefer horizons longer than the episode duration. compare supports paired matrices and refuses protocol mismatches. Use select-checkpoint with an explicit metric, inspect trade-offs, then evaluate held-out seeds; never equate a selection-set winner with generalization or automatically promote it to hardware.

Record MP4 with evaluate --video FILE. Use play --measure FILE.csv --steps N for bounded headless joint/velocity/actuator-torque measurement before terminal-world reset, or play --viewer viser for interactive inspection. CSV samples are policy-rate endpoints, not substep torque peaks. Refer to the CSV sidecar for units and completion status. Look for reward hacking: standing instead of tracking, sliding, knee collapse, asymmetric contacts, large actions, unrealistic high-frequency gait. Videos are complementary to metrics, not proof.

6. Package, never auto-enable hardware

uh export --checkpoint ... --evaluation ... --output NEW_DIRECTORY checks report identity and ONNX parity on real observations. uh verify-bundle DIRECTORY --inference rechecks checksums and the saved reference. Output is a research bundle even when rollout succeeds. Inspect the resolved action joint order, offsets/scales, observation ordering and simulation motor gains.

Physical deployment requires a separate validated adapter for transport, estimator, bus order, calibration, bounds, timeout watchdog, torque ramp, E-stop and operator authorization. There is no motor-enable command in this repository. Never change hardware_approved to bypass this.

7. Handoff

Report checkpoint, backend, env/iteration counts, tracking/std/episode curves, evaluation matrix, CSV/video paths and measured evidence—not only wall time or command exit codes. State whether learning was actually demonstrated. List limitations, failed attempts and next validation requirements. Avoid claiming general morphology support from the fact that two task adapters share a CLI.