6.5 KiB
name, description
| name | description |
|---|---|
| rl-engineer | Engineer humanoid reward revisions, bounded mjlab training, measured evaluation, replay and research-only ONNX packaging in Universal Humanoid. |
RL engineer
1. Establish a contract
Use this as the cross-stage coordinator; consult setup-env, new-task, train, controls and deploy
through the manual index in AGENTS.md for stage-specific work.
Read AGENTS.md, README.md, and the target experiment. Confirm the robot/task, desired
behavior, resource budget and acceptance metrics. A request to build architecture authorizes
short smoke validation, not days of training or physical robot operation. Use the current coding
agent directly; do not launch subagents without operator authorization.
Run ./scripts/uh doctor and list. Inspect the factory's source and relevant mjlab reward
functions before editing. Follow docs/SETUP.md to verify the installed vendored backends,
locked dependencies and bundled assets. Use verify-vendor to check the source inventory.
Do not install a second mjlab/rsl-rl-lib distribution. Record actual dependency provenance.
2. Baseline and reward hypothesis
Copy a baseline experiment into a NEW JSON file with an explicit task and seed. Write the
hypothesis next to the experiment or in a run note. Inspect uh rewards --config FILE.
- Weights/parameters: edit JSON; misspelled term names must fail.
- New mechanisms: add vectorized functions to
src/universal_humanoid/rewards.pyand theFUNCTIONSallowlist; add tests for shape[num_envs], finiteness, sign and limiting cases. - Do not change observations/control/mechanics while claiming a reward-only comparison.
- Never
evalcode supplied in experiment JSON.
Run baseline evaluation/export before editing Python. Source hashes are strict; archived sources may be restored only into a separate trusted workspace, never over a shared reference checkout.
3. Cheap checks before expensive runs
Run ./scripts/check, uh plan, uh rewards, and uh check-robot --config FILE --output NEW_REPORT,
then uh smoke with 4 environments and 16 steps.
For each new robot, verify free root, DoF, mass/inertia, mesh resolution, contacts, foot sites,
IMU frame, torque limits, gains, action ordering, reset posture and observation terms. Model
compilation alone cannot validate these semantics. Zero-action success is not balance training.
4. Bounded training
Use uh train --config FILE --output NEW_DIRECTORY, or pipeline for a smoke-sized end-to-end
run. Read docs/TRAINING_OPERATIONS.md for lifecycle and statistics semantics. Use
train --background for a requested long-running Linux job, status RUN to query progress,
checkpoints RUN to inspect committed checkpoints, and stop RUN for a guarded graceful stop.
Resume with train --resume CHECKPOINT --iterations ADDITIONAL --output NEW_DIRECTORY;
optimizer continuation is not bitwise simulator/RNG replay. Never guess checkpoints by mtime.
Keep TensorBoard local; no W&B authentication or automatic upload. Use curves RUN --output NEW_DIR
to inspect tracking, action distribution std and episode-length trends DURING training. Report
missing tags, non-finite samples and short curves. Rising total reward alone is not learning.
Inspect run.json, progress.json, resolved_env.yaml, resolved_agent.json and sources.zip.
Long runs require an explicit operator-approved budget. Do not automatically increase parallelism past measured VRAM capacity. This backend requires CUDA; do not pretend Warp CPU is fast CPU RL.
5. Evaluate and replay
Use benchmark --checkpoints ... --seeds ... --profiles randomized stress --output NEW_DIR
for stand (0,0,0), forward (0.3,0,0), lateral (0,0.2,0), turn (0,0,0.3). Keep identical
seeds, profiles, world counts and horizons for candidates. These are evaluation seeds, NOT
independent training seeds; separately train multiple seeds when investigating training variance.
Default evaluation randomizes each world's reset state. stress also enables actor observation
noise and per-world interval velocity kicks; inspect actual initial states and push counts.
nominal intentionally repeats fixed initial states. Startup mass/friction randomization is not
part of these profiles. None constitutes full sim-to-real stress coverage.
Fall/time-limit terminations remain active; curriculum is disabled. Report RMSE, completed-only
failure fraction, failure events per simulated world-second, first-episode failures AND censoring.
No completed episodes means completed-only means/fractions are null. Variance is across seed-level
reports, not repeated worlds, and is null for fewer than two values. Prefer horizons longer than
the episode duration. compare supports paired matrices and refuses protocol mismatches.
Use select-checkpoint with an explicit metric, inspect trade-offs, then evaluate held-out seeds;
never equate a selection-set winner with generalization or automatically promote it to hardware.
Record MP4 with evaluate --video FILE. Use play --measure FILE.csv --steps N for bounded
headless joint/velocity/actuator-torque measurement before terminal-world reset, or
play --viewer viser for interactive inspection. CSV samples are policy-rate endpoints, not
substep torque peaks. Refer to the CSV sidecar for units and completion status.
Look for reward hacking: standing instead of tracking, sliding, knee collapse, asymmetric contacts,
large actions, unrealistic high-frequency gait. Videos are complementary to metrics, not proof.
6. Package, never auto-enable hardware
uh export --checkpoint ... --evaluation ... --output NEW_DIRECTORY checks report identity and
ONNX parity on real observations. uh verify-bundle DIRECTORY --inference rechecks checksums
and the saved reference. Output is a research bundle even when rollout succeeds. Inspect the
resolved action joint order, offsets/scales, observation ordering and simulation motor gains.
Physical deployment requires a separate validated adapter for transport, estimator, bus order,
calibration, bounds, timeout watchdog, torque ramp, E-stop and operator authorization. There is
no motor-enable command in this repository. Never change hardware_approved to bypass this.
7. Handoff
Report checkpoint, backend, env/iteration counts, tracking/std/episode curves, evaluation matrix, CSV/video paths and measured evidence—not only wall time or command exit codes. State whether learning was actually demonstrated. List limitations, failed attempts and next validation requirements. Avoid claiming general morphology support from the fact that two task adapters share a CLI.