104 lines
5.6 KiB
Markdown
104 lines
5.6 KiB
Markdown
# Reference-environment validation
|
||
|
||
For current vendored installation, relocated-checkout/wheel tests and automatic robot gates,
|
||
see [Portability validation](PORTABILITY_VALIDATION.md).
|
||
|
||
For the subsequent background/resume, randomized matrix evaluation, TensorBoard and CSV work,
|
||
see [Training-operations validation](OPERATIONS_VALIDATION.md). The sections below preserve the
|
||
initial pipeline validation and its original source/protocol version.
|
||
|
||
This is a historical validation record for one development configuration, not a minimum
|
||
hardware specification or a guarantee of compatibility with every installation. See
|
||
[Setup](SETUP.md) for environment requirements and [Releasing](RELEASING.md) for distribution
|
||
limitations. Paths under `runs/` below identify development artifacts; they are ignored by Git
|
||
and are not bundled with a repository checkout or offered here as downloads.
|
||
|
||
Measured on **2026-11-03**, NVIDIA GeForce **RTX 5080, 16 GB**, in a reference mjlab
|
||
Python 3.13 environment. These results validate engineering plumbing, **not learned locomotion**.
|
||
|
||
## Runtime
|
||
|
||
| Component | Observed version |
|
||
|---|---|
|
||
| mjlab | 1.6.0, customized checkout |
|
||
| mjlab git HEAD | `8bb2bc9e187797689ec82c78c35019afd2fd0376` (dirty/untracked changes retained) |
|
||
| MuJoCo / MuJoCo Warp | 3.16.0 / 3.16.0 |
|
||
| Warp | 1.18.0 |
|
||
| PyTorch | 2.14.0+cu130 |
|
||
| RSL-RL | 5.5.1 |
|
||
|
||
The exact Python/asset hashes and dirty git status are in each `train/run.json`. The reference
|
||
repositories were not modified. `sources.zip` preserves 230 Python/license entries per final run.
|
||
Do not replace the customized dependency with a public version just because its version number matches.
|
||
|
||
## Automated checks
|
||
|
||
`./scripts/check`: **35 tests passed**, Ruff lint and format checks passed. One upstream
|
||
`torch.jit.script` deprecation warning remains. Tests cover strict configs, lightweight discovery,
|
||
independent factories, reward shape/sign, unknown overrides, retained evaluation terminations,
|
||
source/experiment/checkpoint drift, partial-run rejection, bundle tampering/path traversal and
|
||
protocol mismatch.
|
||
|
||
An initial direct pytest invocation inherited ROS's Python 3.12 plugin path and failed on a missing
|
||
`lark` import. The project check script excludes inherited ROS paths and disables unrelated pytest
|
||
plugin autoload; no external Python environment was modified.
|
||
|
||
## Actual CUDA simulation and PPO
|
||
|
||
Zero-action checks ran for G1 and T1 with 4 environments and 8 steps; observations and rewards
|
||
were finite. Shapes: G1 actor/critic/action = **99 / 111 / 29**; T1 = **81 / 93 / 23**.
|
||
|
||
Final end-to-end runs each used **16 environments, 2 PPO iterations, 24 rollout steps per iteration**
|
||
(768 collected environment transitions), then **4 evaluation worlds × 500 steps**, dt 0.02 s,
|
||
5 s episode limit, fixed command `(0.3, 0, 0)`, seed 42.
|
||
|
||
| Run directory | XY tracking RMSE (m/s) | Yaw RMSE (rad/s) | Failure events | Mean completed episode (s) | ONNX max abs error |
|
||
|---|---:|---:|---:|---:|---:|
|
||
| `runs/g1_validation` | 0.709230 | 0.322807 | 24 | 1.500 | 5.59e-8 |
|
||
| `runs/t1_validation` | 0.480955 | 0.047538 | 24 | 1.560 | 5.96e-8 |
|
||
| `runs/g1_reward_validation` | 0.675789 | 0.313275 | 24 | 1.480 | 5.22e-8 |
|
||
|
||
**All three policies fall. None is a deployment candidate.** Failure events count repeated
|
||
episode failures across resets, not a probability. The current nominal evaluation deliberately
|
||
uses identical resets in parallel worlds; those worlds are not independent statistical trials.
|
||
The small candidate RMSE difference is not evidence of an improved learned policy after two updates.
|
||
|
||
Each run has `pipeline.json`, `train/final.pt`, `evaluation.json`, and `bundle/`.
|
||
`verify-bundle --inference` passed for G1 and T1. ONNX parity was checked against the PyTorch actor
|
||
at 16 actual rollout observations for every export. The reward candidate exercises local reward
|
||
function injection, training, evaluation, export and protocol-matched comparison.
|
||
|
||
## Replay evidence
|
||
|
||
`runs/t1_validation/replay.mp4`: H.264, **100 frames / 2 s / 50 FPS / 320×240**;
|
||
`video_evaluation.json` is the corresponding one-world report. Encoding and decoding succeeded,
|
||
and a decoded frame was inspected: the T1 and ground are visible, not an empty render.
|
||
This replay also records an episode failure. Desktop native and interactive Viser sessions were
|
||
not manually exercised; their CLI paths use mjlab's existing viewer classes.
|
||
|
||
## External Unitree asset checks
|
||
|
||
Native MuJoCo compilation succeeded for:
|
||
|
||
- `unitree_ros/robots/g1_description/g1_29dof.xml`: nq=36, nv=35, nu=29.
|
||
- `unitree_ros/robots/h1_description/mjcf/h1.xml`: nq=26, nv=25, nu=19.
|
||
|
||
These are **asset compilation checks only**. They are not registered RL task adapters; the
|
||
validated G1 training run uses mjlab's adapted G1 asset. T1 training uses an external Booster
|
||
T1 asset through the registered adapter.
|
||
|
||
## Failed development runs retained
|
||
|
||
`runs/g1_smoke_v1` trained but evaluation failed because the upstream command rejected
|
||
`heading_command=False` with a non-null heading range. The adapter now clears the heading range
|
||
and regression tests check evaluation configuration. `runs/g1_smoke_v2` passed an earlier short
|
||
pipeline before the final provenance/archive changes. Its old checkpoints intentionally fail
|
||
current source identity checks; use the final `*_validation` runs above.
|
||
|
||
## Remaining validation
|
||
|
||
Long-run convergence, gait quality, domain-randomized/stress evaluation, independent seeds/reset
|
||
conditions, contact realism, raw Unitree asset adaptation, rough terrain, motion imitation,
|
||
sim-to-sim dynamics parity and all physical deployment safety remain unverified/unimplemented.
|
||
No long 3,000-iteration recipe, robot transport or motor-enable operation was executed.
|