Files
UniversalHumanoid/docs/TRAINING_OPERATIONS.md
2026-11-03 19:46:08 +08:00

206 lines
12 KiB
Markdown
Raw Permalink Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# Training operations and experimental evidence
Training duration is not evidence of learning. Read tracking curves, exploration noise and
completed episode lengths during training; then measure the policy under explicit evaluation
conditions. These commands use the same configured runtime as the main README.
## 1. Train in the foreground or background
```bash
./scripts/uh train --config experiments/g1_flat.json --output runs/g1-baseline --background
./scripts/uh status runs/g1-baseline
./scripts/uh checkpoints runs/g1-baseline
./scripts/uh stop runs/g1-baseline
```
Omit `--background` for foreground training. Background execution is a **local Linux process**,
not a remote scheduler or reboot-persistent service. It uses the selected Python environment,
redirects stdout/stderr to `runs/g1-baseline.job/stdout.log`, and disconnects stdin/session from
the launching terminal. No TensorBoard HTTP server is started.
`status` is machine-readable JSON: process liveness, effective state, latest committed checkpoint,
completed/target iterations, session environment steps, iteration duration, learning rate,
action standard deviation, losses, last-update age and an approximate ETA. ETA includes setup
cost and is not an acceptance metric. Progress updates once per completed optimizer iteration;
a long compilation/rollout may be alive without a new progress record. A dead worker whose
manifest still says running is reported as `lost`, not completed.
`stop` sends SIGTERM to the recorded process, guarded by Linux process identity and pidfd to
avoid signaling a reused PID. Poll `status` until termination. Graceful interruption preserves
previous committed checkpoints; an incomplete optimizer update is not saved as a usable model.
SIGKILL, OOM or power loss may leave temporary files; only checkpoints published in the manifest
are usable. Failed runs and job logs are retained.
Set `checkpoint_interval` in an experiment JSON to control periodic saves (default 50 optimizer
iterations). The backend also saves after its first update and at completion. Each checkpoint
is written to a temporary file, renamed, hashed and committed to `run.json`. Published files
are immutable; new experiments and resumes always use **new output directories**.
## 2. Resume a specific checkpoint
```bash
./scripts/uh train --resume runs/g1-baseline/model_49.pt \
--iterations 200 --output runs/g1-continued --background
```
`--iterations 200` means **200 additional updates**, not a total of 200. Choose a filename from
`checkpoints`; do not assume `model_49.pt` exists. `model_N.pt` stores the zero-based last update:
`model_49.pt` represents 50 completed updates, so its continuation starts with update index 50.
The manifest records `completed_iterations` to remove this ambiguity.
Resume restores actor, critic, observation normalization, optimizer state (including learning
rate), and the backend's common-step curriculum counter. Simulator state, pending rollouts,
random-number state and logger history restart: this is **learning continuation, not bitwise
trajectory replay**. The new run records parent checkpoint and experiment hashes. Code, assets,
dependencies and experiment settings must match; only the additional iteration budget and
checkpoint interval may change. Older checkpoints lacking explicit completed-update metadata
are not accepted for continuation.
A committed intermediate checkpoint may be evaluated or resumed even if its original run is
still training or was interrupted. Export still requires a completed training run. Uncommitted
files are never recovered by guessing their modification time.
## 3. Inspect TensorBoard curves without a server
```bash
./scripts/uh curves runs/g1-baseline --output analysis/g1-curves --window 20
# Inspect another set of scalar tags:
./scripts/uh curves runs/g1-baseline --output analysis/g1-losses --match 'Loss/|Metrics/'
```
Outputs:
- `curves.json`: all selected scalar samples, available tags, missing categories, first/last
values, min/max, recent-window mean and slope per iteration.
- `curves.csv`: tag, scalar step, step-axis type, wall-clock timestamp, value.
- `curves.svg`: independently labelled panels; no GUI or plotting dependency required.
By default, select tracking terms (`Episode_Reward/track_*`, velocity-error metrics),
`Policy/mean_std`, `Train/mean_episode_length`, total reward and curriculum terms. The command
excludes RSL-RL's duplicate `/time` tags by default; if explicitly matched, these are labelled
elapsed seconds, not iterations, and their slopes are per second. It reads scalar TensorBoard
events directly in the specified run directory, including while a
training process is active. It does not recursively mix different runs. Duplicate steps use the
latest wall-time sample; non-finite points are counted and excluded, not disguised as zeros.
A missing curve is reported as missing. Episode metrics may not exist until episodes finish.
Interpret together:
- Rising total reward with flat tracking may reflect regularizer exploitation rather than command following.
- Inspect action standard deviation together with learning rate/losses; a plateau alone does not diagnose its cause.
- Check episode length, failure events, command coverage and video—not just a smooth reward curve.
- Tracking rewards are weighted training terms, not physical RMSE or calibrated success probabilities.
No curve slope automatically declares convergence or approves a policy.
## 4. Evaluate independent initial-state draws and disturbances
```bash
./scripts/uh evaluate --checkpoint runs/g1-baseline/final.pt \
--output analysis/g1-stress.json --seed 101 --profile stress \
--command 0.3 0 0 --num-envs 8 --steps 1000
```
Profiles are versioned in the recorded `fixed-twist-v2` protocol:
| Profile | Initial state on every reset | Actor observation noise | Interval disturbances |
|---|---|---|---|
| `nominal` | Fixed pose/joints/velocity | Off | None |
| `randomized` (evaluation default) | Separate uniform samples for each world | Off | None |
| `stress` | Same reset distribution as randomized | Configured task noise enabled | Per-world random velocity kicks |
Randomized reset offsets: root x/y ±0.1 m, z +0.01…0.03 m, roll/pitch ±0.03 rad, yaw ±0.2 rad;
root x/y velocity ±0.05 m/s, yaw rate ±0.1 rad/s; joint position ±0.03 rad and velocity ±0.1 rad/s.
Joint positions are clamped to the adapter's soft limits. Root offsets are relative to the robot's
default root state and world origin. These are modest test distributions, not physical validity
guarantees for future adapters.
Stress kicks add x/y velocity increments within ±0.4 m/s and yaw-rate increments within
±0.5 rad/s, at independently sampled 0.5–1.5 s intervals per world (timers restart on reset).
These are **mass-independent instantaneous velocity changes**, not force-calibrated pushes.
Training startup mass/friction/COM randomization is not enabled by these profiles; this is not a
complete sim-to-real robustness suite.
Reports include all initial states relative to world origins, unique-state counts, exact reset
ranges, actor noise definitions, per-world kick counts, failure counts and first-episode censoring.
A short rollout may experience no kick; inspect the actual counts. Repeating a seed reproduces
initial-state draws, but CUDA trajectories are not guaranteed bitwise deterministic. Distinct world samples share
a policy and are not independent training replicas. Nominal replicas remain intentionally fixed.
## 5. Run a checkpoint × seed × four-command matrix
```bash
./scripts/uh benchmark \
--checkpoints runs/g1-baseline/model_0.pt runs/g1-baseline/final.pt \
--seeds 101 202 303 --profiles randomized stress \
--steps 1000 --num-envs 8 --output analysis/g1-matrix
```
The four commands are standing `(0,0,0)`, forward `(0.3,0,0)`, lateral `(0,0.2,0)`, and turn
`(0,0,0.3)`. This example has 2 × 3 × 2 × 4 = **48 evaluation cells**, executed serially to bound
GPU memory. `summary.json` updates after every cell and records failures; completed cells are
retained, but automatic continuation of an interrupted matrix is not implemented.
Outputs include per-cell reports and aggregate `table.csv` / `table.md`:
- Mean and **sample variance across evaluation seeds**, with the number of defined values.
- XY/yaw RMSE, completed-episode duration and failure metrics.
- **Failure fraction among completed episodes** = failures / completed episodes; null if none ended.
- **Failure incidence** = failure events / simulated world-seconds, including repeated episodes.
- **First-episode failure fraction** = worlds whose first episode failed within the observation
horizon / worlds. Report censoring alongside it; unfinished episodes are not proven successes.
Variance is null with fewer than two values. Never treat repeated world resets as independent
training seeds. If a termination and timeout coincide, failure takes precedence. Means weight
seed-level reports equally; the table also supplies raw completion/failure counts. Long-lived
censored episodes can bias completed-only statistics, so inspect the entire set of metrics.
`--seeds` changes **evaluation randomness**, not training seeds. For training-seed robustness,
train separate experiment JSONs with different `seed` values, then pass their checkpoints to
this command. Results remain separated by checkpoint, with each training seed recorded.
### Select, compare, then validate on held-out seeds
```bash
./scripts/uh select-checkpoint analysis/g1-matrix/summary.json \
--metric failure_events_per_sim_second --output analysis/g1-selection.json
# Each input here must be a completed matrix containing one checkpoint:
./scripts/uh compare analysis/baseline-matrix/summary.json analysis/candidate-matrix/summary.json
```
Selection ranks the **explicit candidate list**, equally weighting command/profile mean values
for a named lower-is-better metric. No undefined cells are accepted. Ties retain input order;
there is no hidden composite quality score or hardware promotion. The selection file references
the checkpoint; it does not overwrite `final.pt`. Evaluate the winner on held-out seeds and
inspect tracking/video, since low failure incidence alone can favor standing still.
Matrix comparison reports paired candidate-minus-baseline means and sample variances by seed,
command and profile. It refuses mismatched protocols; it does not claim statistical significance.
The original two-single-report comparison remains available for quick diagnostics.
## 6. Replay with joint measurements
```bash
./scripts/uh play --checkpoint runs/g1-baseline/final.pt \
--measure analysis/g1-joints.csv --steps 1000 --seed 101 \
--profile stress --command 0.3 0 0 --video analysis/g1-replay.mp4
```
`play --measure` is a **bounded headless replay**, not an interactive viewer session. Without
`--measure`, `play --viewer native|viser` retains the interactive viewer. For multiple worlds,
use `evaluate --measure FILE --num-envs N`.
The long-form CSV has one row per controlled joint × world × control step: global and episode
time, world/episode IDs, termination/timeout flags, joint name, position (rad), velocity (rad/s),
actuator contribution to joint torque (Nm), raw/applied action and actual joint target.
Simulation uses manual resets during evaluation. CSV and video are sampled **after the control
step but before resetting ended worlds**. Torque is `qfrc_actuator` projected into joint DOFs,
not motor-space force, a desired torque, or a decimation-averaged value. Sampling is at policy
rate; substep peaks may be missed. A `.csv.json` sidecar records schema semantics, row count and
completion/failure status; `.evaluation.json` contains task metrics.
All commands remain simulation-only. These measurements do not replace hardware sensing,
transport adapters, safety supervision or sim-to-real validation.