Files
UniversalHumanoid/docs/OPERATIONS_VALIDATION.md
T
2026-11-03 19:46:08 +08:00

5.7 KiB
Raw Blame History

Training-operations validation

Reference-environment checks on 2026-11-03, using the Linux/RTX 5080 backend described in VALIDATION.md. The files below are development evidence under ignored runs/, not artifacts shipped with a fresh repository clone. Usage and statistical definitions are in TRAINING_OPERATIONS.md.

Automated checks

58 tests passed; Ruff lint/format passed. One upstream TorchScript deprecation warning remains. New checks cover seed-level sample variance and null statistics, duplicate/mismatched protocols, paired comparisons, selection coverage, TensorBoard scalar reading and /time axis handling, CSV joint ordering and projected torque, atomic checkpoint publication and update indexing, committed-vs-uncommitted partial runs, Linux PID identity/stop, and both robots' evaluation profiles.

Background training and real interruption/resume

  • runs/ops_g1_validation: 16 environments × 24 steps × 4 updates, launched with train --background; state progressed to completed, with live progress and committed model_0.pt, model_3.pt, final.pt visible through the CLI.
  • runs/ops_interrupt_validation: a bounded 16-update job was deliberately stopped after its first committed checkpoint. status observed it alive/training; stop sent SIGTERM; worker and run both recorded interrupted, preserving model_0.pt.
  • runs/ops_resume_validation: resumed that interrupted checkpoint for 2 additional updates. Final metadata reports 3 completed updates, not 2 and not 4. Direct inspection of these trusted generated checkpoint files confirmed Adam step counters 20 → 60 and the backend common-step counter 24 → 72. The optimizer was continued, not reinitialized.

No long convergence run, scheduler restart/reboot recovery, or multi-GPU orchestration was tested. Process-loss/PID-reuse guards are covered by process-level tests, not a power-loss experiment.

Actual matrix and checkpoint selection

runs/ops_g1_matrix contains 32 completed cells:

  • 2 checkpoints: after update 1 and update 4;
  • 2 evaluation seeds: 101 and 202;
  • 4 commands: standing, forward, lateral, turn;
  • 2 profiles: randomized and stress;
  • each cell: 2 worlds × 150 control steps, dt 0.02 s, episode limit 5 s.

Every cell recorded 2 distinct initial states out of 2 worlds. Stress cells recorded 71 total velocity-kick events. Repeating the same stress seed/command produced exactly matching initial state arrays; XY RMSE differed by about 1.88e-5 m/s, demonstrating why seeded draws are not a promise of bitwise deterministic CUDA trajectories.

table.csv, table.md, and summary.json contain seed-level means/sample variances, failure metrics and censoring. This 3 s observation horizon is intentionally short for engineering checks, not sufficient policy acceptance evidence.

runs/ops_g1_selection.json ranked the earlier checkpoint first by equal-weight mean failure incidence: 0.3333 vs 0.3750 events per simulated world-second. This is not a general quality ranking. Both policies fall; more updates did not automatically produce a better measured result.

runs/ops_g1_baseline_matrix and runs/ops_g1_candidate_matrix are single-checkpoint views copied from the same 32 real cells, not additional independent experiments. compare successfully produced paired-seed differences in runs/ops_g1_comparison.json.

Training curves, not time spent

runs/ops_g1_curves contains JSON, CSV and parseable SVG extracted from actual TensorBoard events:

Scalar First Last Samples
Linear tracking reward 0.01648 0.05844 4
Angular tracking reward 0.004257 0.001208 4
Mean action std 1.00023 1.00013 4
Mean episode length (control steps) 12.0 51.65 4

Linear tracking/episode length increased, angular tracking decreased, and exploration std stayed near 1. Four noisy points, random initial episode lengths and falling replays do not establish learning or convergence. /time companion tags are no longer mislabeled as iteration curves.

Measured replay and T1 coverage

Evidence Sampling CSV rows Failure events Recorded kicks
runs/ops_g1_joints.csv 1 world × 150 steps × 29 joints 4,350 1 2
runs/ops_t1_joints.csv 4 worlds × 150 steps × 23 joints 13,800 6 7

Both used the stress profile and finite projected joint torques. CSV terminal-row counts matched failure events × joint count (29 and 138), confirming terminal samples were retained before reset. T1 recorded 4 distinct initial states out of 4 worlds. Its policy came from a separate two-update run at runs/ops_t1_validation.

runs/ops_g1_replay.mp4 decoded successfully: 150 frames, 50 FPS, 3 seconds. A decoded frame was visually inspected; robot and ground are visible. The matching measured report records a fall. Interactive native/Viser viewing was not manually exercised in this revision.

A separate four-step evaluation, runs/ops_g1_censored.json, had no completed episodes: completed-only failure fraction/return/duration were null, and both worlds were explicitly censored rather than reported as successful episodes.

Export regression and provenance

The updated evaluation report was accepted by ONNX export; runs/ops_g1_bundle passed verify-bundle --inference. All outputs remain hardware_approved: false.

Earlier ops_*_v1 development runs are retained. They predate the TensorBoard time-axis correction and other final source changes and intentionally fail current source-identity checks. Historical g1_validation / t1_validation runs also require their recorded sources for replay. Do not rewrite provenance hashes to make old runs appear current.