Files
UniversalHumanoid/docs/TRAINING_OPERATIONS.md
2026-11-03 19:46:08 +08:00

12 KiB
Raw Permalink Blame History

Training operations and experimental evidence

Training duration is not evidence of learning. Read tracking curves, exploration noise and completed episode lengths during training; then measure the policy under explicit evaluation conditions. These commands use the same configured runtime as the main README.

1. Train in the foreground or background

./scripts/uh train --config experiments/g1_flat.json --output runs/g1-baseline --background
./scripts/uh status runs/g1-baseline
./scripts/uh checkpoints runs/g1-baseline
./scripts/uh stop runs/g1-baseline

Omit --background for foreground training. Background execution is a local Linux process, not a remote scheduler or reboot-persistent service. It uses the selected Python environment, redirects stdout/stderr to runs/g1-baseline.job/stdout.log, and disconnects stdin/session from the launching terminal. No TensorBoard HTTP server is started.

status is machine-readable JSON: process liveness, effective state, latest committed checkpoint, completed/target iterations, session environment steps, iteration duration, learning rate, action standard deviation, losses, last-update age and an approximate ETA. ETA includes setup cost and is not an acceptance metric. Progress updates once per completed optimizer iteration; a long compilation/rollout may be alive without a new progress record. A dead worker whose manifest still says running is reported as lost, not completed.

stop sends SIGTERM to the recorded process, guarded by Linux process identity and pidfd to avoid signaling a reused PID. Poll status until termination. Graceful interruption preserves previous committed checkpoints; an incomplete optimizer update is not saved as a usable model. SIGKILL, OOM or power loss may leave temporary files; only checkpoints published in the manifest are usable. Failed runs and job logs are retained.

Set checkpoint_interval in an experiment JSON to control periodic saves (default 50 optimizer iterations). The backend also saves after its first update and at completion. Each checkpoint is written to a temporary file, renamed, hashed and committed to run.json. Published files are immutable; new experiments and resumes always use new output directories.

2. Resume a specific checkpoint

./scripts/uh train --resume runs/g1-baseline/model_49.pt \
  --iterations 200 --output runs/g1-continued --background

--iterations 200 means 200 additional updates, not a total of 200. Choose a filename from checkpoints; do not assume model_49.pt exists. model_N.pt stores the zero-based last update: model_49.pt represents 50 completed updates, so its continuation starts with update index 50. The manifest records completed_iterations to remove this ambiguity.

Resume restores actor, critic, observation normalization, optimizer state (including learning rate), and the backend's common-step curriculum counter. Simulator state, pending rollouts, random-number state and logger history restart: this is learning continuation, not bitwise trajectory replay. The new run records parent checkpoint and experiment hashes. Code, assets, dependencies and experiment settings must match; only the additional iteration budget and checkpoint interval may change. Older checkpoints lacking explicit completed-update metadata are not accepted for continuation.

A committed intermediate checkpoint may be evaluated or resumed even if its original run is still training or was interrupted. Export still requires a completed training run. Uncommitted files are never recovered by guessing their modification time.

3. Inspect TensorBoard curves without a server

./scripts/uh curves runs/g1-baseline --output analysis/g1-curves --window 20
# Inspect another set of scalar tags:
./scripts/uh curves runs/g1-baseline --output analysis/g1-losses --match 'Loss/|Metrics/'

Outputs:

  • curves.json: all selected scalar samples, available tags, missing categories, first/last values, min/max, recent-window mean and slope per iteration.
  • curves.csv: tag, scalar step, step-axis type, wall-clock timestamp, value.
  • curves.svg: independently labelled panels; no GUI or plotting dependency required.

By default, select tracking terms (Episode_Reward/track_*, velocity-error metrics), Policy/mean_std, Train/mean_episode_length, total reward and curriculum terms. The command excludes RSL-RL's duplicate /time tags by default; if explicitly matched, these are labelled elapsed seconds, not iterations, and their slopes are per second. It reads scalar TensorBoard events directly in the specified run directory, including while a training process is active. It does not recursively mix different runs. Duplicate steps use the latest wall-time sample; non-finite points are counted and excluded, not disguised as zeros. A missing curve is reported as missing. Episode metrics may not exist until episodes finish.

Interpret together:

  • Rising total reward with flat tracking may reflect regularizer exploitation rather than command following.
  • Inspect action standard deviation together with learning rate/losses; a plateau alone does not diagnose its cause.
  • Check episode length, failure events, command coverage and video—not just a smooth reward curve.
  • Tracking rewards are weighted training terms, not physical RMSE or calibrated success probabilities.

No curve slope automatically declares convergence or approves a policy.

4. Evaluate independent initial-state draws and disturbances

./scripts/uh evaluate --checkpoint runs/g1-baseline/final.pt \
  --output analysis/g1-stress.json --seed 101 --profile stress \
  --command 0.3 0 0 --num-envs 8 --steps 1000

Profiles are versioned in the recorded fixed-twist-v2 protocol:

Profile Initial state on every reset Actor observation noise Interval disturbances
nominal Fixed pose/joints/velocity Off None
randomized (evaluation default) Separate uniform samples for each world Off None
stress Same reset distribution as randomized Configured task noise enabled Per-world random velocity kicks

Randomized reset offsets: root x/y ±0.1 m, z +0.01…0.03 m, roll/pitch ±0.03 rad, yaw ±0.2 rad; root x/y velocity ±0.05 m/s, yaw rate ±0.1 rad/s; joint position ±0.03 rad and velocity ±0.1 rad/s. Joint positions are clamped to the adapter's soft limits. Root offsets are relative to the robot's default root state and world origin. These are modest test distributions, not physical validity guarantees for future adapters.

Stress kicks add x/y velocity increments within ±0.4 m/s and yaw-rate increments within ±0.5 rad/s, at independently sampled 0.5–1.5 s intervals per world (timers restart on reset). These are mass-independent instantaneous velocity changes, not force-calibrated pushes. Training startup mass/friction/COM randomization is not enabled by these profiles; this is not a complete sim-to-real robustness suite.

Reports include all initial states relative to world origins, unique-state counts, exact reset ranges, actor noise definitions, per-world kick counts, failure counts and first-episode censoring. A short rollout may experience no kick; inspect the actual counts. Repeating a seed reproduces initial-state draws, but CUDA trajectories are not guaranteed bitwise deterministic. Distinct world samples share a policy and are not independent training replicas. Nominal replicas remain intentionally fixed.

5. Run a checkpoint × seed × four-command matrix

./scripts/uh benchmark \
  --checkpoints runs/g1-baseline/model_0.pt runs/g1-baseline/final.pt \
  --seeds 101 202 303 --profiles randomized stress \
  --steps 1000 --num-envs 8 --output analysis/g1-matrix

The four commands are standing (0,0,0), forward (0.3,0,0), lateral (0,0.2,0), and turn (0,0,0.3). This example has 2 × 3 × 2 × 4 = 48 evaluation cells, executed serially to bound GPU memory. summary.json updates after every cell and records failures; completed cells are retained, but automatic continuation of an interrupted matrix is not implemented.

Outputs include per-cell reports and aggregate table.csv / table.md:

  • Mean and sample variance across evaluation seeds, with the number of defined values.
  • XY/yaw RMSE, completed-episode duration and failure metrics.
  • Failure fraction among completed episodes = failures / completed episodes; null if none ended.
  • Failure incidence = failure events / simulated world-seconds, including repeated episodes.
  • First-episode failure fraction = worlds whose first episode failed within the observation horizon / worlds. Report censoring alongside it; unfinished episodes are not proven successes.

Variance is null with fewer than two values. Never treat repeated world resets as independent training seeds. If a termination and timeout coincide, failure takes precedence. Means weight seed-level reports equally; the table also supplies raw completion/failure counts. Long-lived censored episodes can bias completed-only statistics, so inspect the entire set of metrics.

--seeds changes evaluation randomness, not training seeds. For training-seed robustness, train separate experiment JSONs with different seed values, then pass their checkpoints to this command. Results remain separated by checkpoint, with each training seed recorded.

Select, compare, then validate on held-out seeds

./scripts/uh select-checkpoint analysis/g1-matrix/summary.json \
  --metric failure_events_per_sim_second --output analysis/g1-selection.json

# Each input here must be a completed matrix containing one checkpoint:
./scripts/uh compare analysis/baseline-matrix/summary.json analysis/candidate-matrix/summary.json

Selection ranks the explicit candidate list, equally weighting command/profile mean values for a named lower-is-better metric. No undefined cells are accepted. Ties retain input order; there is no hidden composite quality score or hardware promotion. The selection file references the checkpoint; it does not overwrite final.pt. Evaluate the winner on held-out seeds and inspect tracking/video, since low failure incidence alone can favor standing still.

Matrix comparison reports paired candidate-minus-baseline means and sample variances by seed, command and profile. It refuses mismatched protocols; it does not claim statistical significance. The original two-single-report comparison remains available for quick diagnostics.

6. Replay with joint measurements

./scripts/uh play --checkpoint runs/g1-baseline/final.pt \
  --measure analysis/g1-joints.csv --steps 1000 --seed 101 \
  --profile stress --command 0.3 0 0 --video analysis/g1-replay.mp4

play --measure is a bounded headless replay, not an interactive viewer session. Without --measure, play --viewer native|viser retains the interactive viewer. For multiple worlds, use evaluate --measure FILE --num-envs N.

The long-form CSV has one row per controlled joint × world × control step: global and episode time, world/episode IDs, termination/timeout flags, joint name, position (rad), velocity (rad/s), actuator contribution to joint torque (Nm), raw/applied action and actual joint target.

Simulation uses manual resets during evaluation. CSV and video are sampled after the control step but before resetting ended worlds. Torque is qfrc_actuator projected into joint DOFs, not motor-space force, a desired torque, or a decimation-averaged value. Sampling is at policy rate; substep peaks may be missed. A .csv.json sidecar records schema semantics, row count and completion/failure status; .evaluation.json contains task metrics.

All commands remain simulation-only. These measurements do not replace hardware sensing, transport adapters, safety supervision or sim-to-real validation.