206 lines
12 KiB
Markdown
206 lines
12 KiB
Markdown
# Training operations and experimental evidence
|
||
|
||
Training duration is not evidence of learning. Read tracking curves, exploration noise and
|
||
completed episode lengths during training; then measure the policy under explicit evaluation
|
||
conditions. These commands use the same configured runtime as the main README.
|
||
|
||
## 1. Train in the foreground or background
|
||
|
||
```bash
|
||
./scripts/uh train --config experiments/g1_flat.json --output runs/g1-baseline --background
|
||
./scripts/uh status runs/g1-baseline
|
||
./scripts/uh checkpoints runs/g1-baseline
|
||
./scripts/uh stop runs/g1-baseline
|
||
```
|
||
|
||
Omit `--background` for foreground training. Background execution is a **local Linux process**,
|
||
not a remote scheduler or reboot-persistent service. It uses the selected Python environment,
|
||
redirects stdout/stderr to `runs/g1-baseline.job/stdout.log`, and disconnects stdin/session from
|
||
the launching terminal. No TensorBoard HTTP server is started.
|
||
|
||
`status` is machine-readable JSON: process liveness, effective state, latest committed checkpoint,
|
||
completed/target iterations, session environment steps, iteration duration, learning rate,
|
||
action standard deviation, losses, last-update age and an approximate ETA. ETA includes setup
|
||
cost and is not an acceptance metric. Progress updates once per completed optimizer iteration;
|
||
a long compilation/rollout may be alive without a new progress record. A dead worker whose
|
||
manifest still says running is reported as `lost`, not completed.
|
||
|
||
`stop` sends SIGTERM to the recorded process, guarded by Linux process identity and pidfd to
|
||
avoid signaling a reused PID. Poll `status` until termination. Graceful interruption preserves
|
||
previous committed checkpoints; an incomplete optimizer update is not saved as a usable model.
|
||
SIGKILL, OOM or power loss may leave temporary files; only checkpoints published in the manifest
|
||
are usable. Failed runs and job logs are retained.
|
||
|
||
Set `checkpoint_interval` in an experiment JSON to control periodic saves (default 50 optimizer
|
||
iterations). The backend also saves after its first update and at completion. Each checkpoint
|
||
is written to a temporary file, renamed, hashed and committed to `run.json`. Published files
|
||
are immutable; new experiments and resumes always use **new output directories**.
|
||
|
||
## 2. Resume a specific checkpoint
|
||
|
||
```bash
|
||
./scripts/uh train --resume runs/g1-baseline/model_49.pt \
|
||
--iterations 200 --output runs/g1-continued --background
|
||
```
|
||
|
||
`--iterations 200` means **200 additional updates**, not a total of 200. Choose a filename from
|
||
`checkpoints`; do not assume `model_49.pt` exists. `model_N.pt` stores the zero-based last update:
|
||
`model_49.pt` represents 50 completed updates, so its continuation starts with update index 50.
|
||
The manifest records `completed_iterations` to remove this ambiguity.
|
||
|
||
Resume restores actor, critic, observation normalization, optimizer state (including learning
|
||
rate), and the backend's common-step curriculum counter. Simulator state, pending rollouts,
|
||
random-number state and logger history restart: this is **learning continuation, not bitwise
|
||
trajectory replay**. The new run records parent checkpoint and experiment hashes. Code, assets,
|
||
dependencies and experiment settings must match; only the additional iteration budget and
|
||
checkpoint interval may change. Older checkpoints lacking explicit completed-update metadata
|
||
are not accepted for continuation.
|
||
|
||
A committed intermediate checkpoint may be evaluated or resumed even if its original run is
|
||
still training or was interrupted. Export still requires a completed training run. Uncommitted
|
||
files are never recovered by guessing their modification time.
|
||
|
||
## 3. Inspect TensorBoard curves without a server
|
||
|
||
```bash
|
||
./scripts/uh curves runs/g1-baseline --output analysis/g1-curves --window 20
|
||
# Inspect another set of scalar tags:
|
||
./scripts/uh curves runs/g1-baseline --output analysis/g1-losses --match 'Loss/|Metrics/'
|
||
```
|
||
|
||
Outputs:
|
||
|
||
- `curves.json`: all selected scalar samples, available tags, missing categories, first/last
|
||
values, min/max, recent-window mean and slope per iteration.
|
||
- `curves.csv`: tag, scalar step, step-axis type, wall-clock timestamp, value.
|
||
- `curves.svg`: independently labelled panels; no GUI or plotting dependency required.
|
||
|
||
By default, select tracking terms (`Episode_Reward/track_*`, velocity-error metrics),
|
||
`Policy/mean_std`, `Train/mean_episode_length`, total reward and curriculum terms. The command
|
||
excludes RSL-RL's duplicate `/time` tags by default; if explicitly matched, these are labelled
|
||
elapsed seconds, not iterations, and their slopes are per second. It reads scalar TensorBoard
|
||
events directly in the specified run directory, including while a
|
||
training process is active. It does not recursively mix different runs. Duplicate steps use the
|
||
latest wall-time sample; non-finite points are counted and excluded, not disguised as zeros.
|
||
A missing curve is reported as missing. Episode metrics may not exist until episodes finish.
|
||
|
||
Interpret together:
|
||
|
||
- Rising total reward with flat tracking may reflect regularizer exploitation rather than command following.
|
||
- Inspect action standard deviation together with learning rate/losses; a plateau alone does not diagnose its cause.
|
||
- Check episode length, failure events, command coverage and video—not just a smooth reward curve.
|
||
- Tracking rewards are weighted training terms, not physical RMSE or calibrated success probabilities.
|
||
|
||
No curve slope automatically declares convergence or approves a policy.
|
||
|
||
## 4. Evaluate independent initial-state draws and disturbances
|
||
|
||
```bash
|
||
./scripts/uh evaluate --checkpoint runs/g1-baseline/final.pt \
|
||
--output analysis/g1-stress.json --seed 101 --profile stress \
|
||
--command 0.3 0 0 --num-envs 8 --steps 1000
|
||
```
|
||
|
||
Profiles are versioned in the recorded `fixed-twist-v2` protocol:
|
||
|
||
| Profile | Initial state on every reset | Actor observation noise | Interval disturbances |
|
||
|---|---|---|---|
|
||
| `nominal` | Fixed pose/joints/velocity | Off | None |
|
||
| `randomized` (evaluation default) | Separate uniform samples for each world | Off | None |
|
||
| `stress` | Same reset distribution as randomized | Configured task noise enabled | Per-world random velocity kicks |
|
||
|
||
Randomized reset offsets: root x/y ±0.1 m, z +0.01…0.03 m, roll/pitch ±0.03 rad, yaw ±0.2 rad;
|
||
root x/y velocity ±0.05 m/s, yaw rate ±0.1 rad/s; joint position ±0.03 rad and velocity ±0.1 rad/s.
|
||
Joint positions are clamped to the adapter's soft limits. Root offsets are relative to the robot's
|
||
default root state and world origin. These are modest test distributions, not physical validity
|
||
guarantees for future adapters.
|
||
|
||
Stress kicks add x/y velocity increments within ±0.4 m/s and yaw-rate increments within
|
||
±0.5 rad/s, at independently sampled 0.5–1.5 s intervals per world (timers restart on reset).
|
||
These are **mass-independent instantaneous velocity changes**, not force-calibrated pushes.
|
||
Training startup mass/friction/COM randomization is not enabled by these profiles; this is not a
|
||
complete sim-to-real robustness suite.
|
||
|
||
Reports include all initial states relative to world origins, unique-state counts, exact reset
|
||
ranges, actor noise definitions, per-world kick counts, failure counts and first-episode censoring.
|
||
A short rollout may experience no kick; inspect the actual counts. Repeating a seed reproduces
|
||
initial-state draws, but CUDA trajectories are not guaranteed bitwise deterministic. Distinct world samples share
|
||
a policy and are not independent training replicas. Nominal replicas remain intentionally fixed.
|
||
|
||
## 5. Run a checkpoint × seed × four-command matrix
|
||
|
||
```bash
|
||
./scripts/uh benchmark \
|
||
--checkpoints runs/g1-baseline/model_0.pt runs/g1-baseline/final.pt \
|
||
--seeds 101 202 303 --profiles randomized stress \
|
||
--steps 1000 --num-envs 8 --output analysis/g1-matrix
|
||
```
|
||
|
||
The four commands are standing `(0,0,0)`, forward `(0.3,0,0)`, lateral `(0,0.2,0)`, and turn
|
||
`(0,0,0.3)`. This example has 2 × 3 × 2 × 4 = **48 evaluation cells**, executed serially to bound
|
||
GPU memory. `summary.json` updates after every cell and records failures; completed cells are
|
||
retained, but automatic continuation of an interrupted matrix is not implemented.
|
||
|
||
Outputs include per-cell reports and aggregate `table.csv` / `table.md`:
|
||
|
||
- Mean and **sample variance across evaluation seeds**, with the number of defined values.
|
||
- XY/yaw RMSE, completed-episode duration and failure metrics.
|
||
- **Failure fraction among completed episodes** = failures / completed episodes; null if none ended.
|
||
- **Failure incidence** = failure events / simulated world-seconds, including repeated episodes.
|
||
- **First-episode failure fraction** = worlds whose first episode failed within the observation
|
||
horizon / worlds. Report censoring alongside it; unfinished episodes are not proven successes.
|
||
|
||
Variance is null with fewer than two values. Never treat repeated world resets as independent
|
||
training seeds. If a termination and timeout coincide, failure takes precedence. Means weight
|
||
seed-level reports equally; the table also supplies raw completion/failure counts. Long-lived
|
||
censored episodes can bias completed-only statistics, so inspect the entire set of metrics.
|
||
|
||
`--seeds` changes **evaluation randomness**, not training seeds. For training-seed robustness,
|
||
train separate experiment JSONs with different `seed` values, then pass their checkpoints to
|
||
this command. Results remain separated by checkpoint, with each training seed recorded.
|
||
|
||
### Select, compare, then validate on held-out seeds
|
||
|
||
```bash
|
||
./scripts/uh select-checkpoint analysis/g1-matrix/summary.json \
|
||
--metric failure_events_per_sim_second --output analysis/g1-selection.json
|
||
|
||
# Each input here must be a completed matrix containing one checkpoint:
|
||
./scripts/uh compare analysis/baseline-matrix/summary.json analysis/candidate-matrix/summary.json
|
||
```
|
||
|
||
Selection ranks the **explicit candidate list**, equally weighting command/profile mean values
|
||
for a named lower-is-better metric. No undefined cells are accepted. Ties retain input order;
|
||
there is no hidden composite quality score or hardware promotion. The selection file references
|
||
the checkpoint; it does not overwrite `final.pt`. Evaluate the winner on held-out seeds and
|
||
inspect tracking/video, since low failure incidence alone can favor standing still.
|
||
|
||
Matrix comparison reports paired candidate-minus-baseline means and sample variances by seed,
|
||
command and profile. It refuses mismatched protocols; it does not claim statistical significance.
|
||
The original two-single-report comparison remains available for quick diagnostics.
|
||
|
||
## 6. Replay with joint measurements
|
||
|
||
```bash
|
||
./scripts/uh play --checkpoint runs/g1-baseline/final.pt \
|
||
--measure analysis/g1-joints.csv --steps 1000 --seed 101 \
|
||
--profile stress --command 0.3 0 0 --video analysis/g1-replay.mp4
|
||
```
|
||
|
||
`play --measure` is a **bounded headless replay**, not an interactive viewer session. Without
|
||
`--measure`, `play --viewer native|viser` retains the interactive viewer. For multiple worlds,
|
||
use `evaluate --measure FILE --num-envs N`.
|
||
|
||
The long-form CSV has one row per controlled joint × world × control step: global and episode
|
||
time, world/episode IDs, termination/timeout flags, joint name, position (rad), velocity (rad/s),
|
||
actuator contribution to joint torque (Nm), raw/applied action and actual joint target.
|
||
|
||
Simulation uses manual resets during evaluation. CSV and video are sampled **after the control
|
||
step but before resetting ended worlds**. Torque is `qfrc_actuator` projected into joint DOFs,
|
||
not motor-space force, a desired torque, or a decimation-averaged value. Sampling is at policy
|
||
rate; substep peaks may be missed. A `.csv.json` sidecar records schema semantics, row count and
|
||
completion/failure status; `.evaluation.json` contains task metrics.
|
||
|
||
All commands remain simulation-only. These measurements do not replace hardware sensing,
|
||
transport adapters, safety supervision or sim-to-real validation.
|