Files
UniversalHumanoid/docs/OPERATIONS_VALIDATION.md
2026-11-03 19:46:08 +08:00

103 lines
5.7 KiB
Markdown
Raw Permalink Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# Training-operations validation
Reference-environment checks on **2026-11-03**, using the Linux/RTX 5080 backend described in
[VALIDATION.md](VALIDATION.md). The files below are development evidence under ignored `runs/`,
not artifacts shipped with a fresh repository clone. Usage and statistical definitions are in
[TRAINING_OPERATIONS.md](TRAINING_OPERATIONS.md).
## Automated checks
**58 tests passed**; Ruff lint/format passed. One upstream TorchScript deprecation warning remains.
New checks cover seed-level sample variance and null statistics, duplicate/mismatched protocols,
paired comparisons, selection coverage, TensorBoard scalar reading and `/time` axis handling,
CSV joint ordering and projected torque, atomic checkpoint publication and update indexing,
committed-vs-uncommitted partial runs, Linux PID identity/stop, and both robots' evaluation profiles.
## Background training and real interruption/resume
- `runs/ops_g1_validation`: 16 environments × 24 steps × **4 updates**, launched with
`train --background`; state progressed to completed, with live progress and committed
`model_0.pt`, `model_3.pt`, `final.pt` visible through the CLI.
- `runs/ops_interrupt_validation`: a bounded 16-update job was deliberately stopped after its
first committed checkpoint. `status` observed it alive/training; `stop` sent SIGTERM;
worker and run both recorded **interrupted**, preserving `model_0.pt`.
- `runs/ops_resume_validation`: resumed that interrupted checkpoint for **2 additional updates**.
Final metadata reports **3 completed updates**, not 2 and not 4. Direct inspection of these
trusted generated checkpoint files confirmed Adam step counters **20 → 60** and the backend
common-step counter **24 → 72**. The optimizer was continued, not reinitialized.
No long convergence run, scheduler restart/reboot recovery, or multi-GPU orchestration was tested.
Process-loss/PID-reuse guards are covered by process-level tests, not a power-loss experiment.
## Actual matrix and checkpoint selection
`runs/ops_g1_matrix` contains **32 completed cells**:
- 2 checkpoints: after update 1 and update 4;
- 2 evaluation seeds: 101 and 202;
- 4 commands: standing, forward, lateral, turn;
- 2 profiles: randomized and stress;
- each cell: **2 worlds × 150 control steps**, dt 0.02 s, episode limit 5 s.
Every cell recorded **2 distinct initial states out of 2 worlds**. Stress cells recorded **71 total
velocity-kick events**. Repeating the same stress seed/command produced exactly matching initial
state arrays; XY RMSE differed by about `1.88e-5 m/s`, demonstrating why seeded draws are not a
promise of bitwise deterministic CUDA trajectories.
`table.csv`, `table.md`, and `summary.json` contain seed-level means/sample variances, failure
metrics and censoring. This 3 s observation horizon is intentionally short for engineering checks,
not sufficient policy acceptance evidence.
`runs/ops_g1_selection.json` ranked the earlier checkpoint first by equal-weight mean failure
incidence: **0.3333 vs 0.3750 events per simulated world-second**. This is not a general quality
ranking. Both policies fall; more updates did not automatically produce a better measured result.
`runs/ops_g1_baseline_matrix` and `runs/ops_g1_candidate_matrix` are single-checkpoint views copied
from the same 32 real cells, not additional independent experiments. `compare` successfully
produced paired-seed differences in `runs/ops_g1_comparison.json`.
## Training curves, not time spent
`runs/ops_g1_curves` contains JSON, CSV and parseable SVG extracted from actual TensorBoard events:
| Scalar | First | Last | Samples |
|---|---:|---:|---:|
| Linear tracking reward | 0.01648 | 0.05844 | 4 |
| Angular tracking reward | 0.004257 | 0.001208 | 4 |
| Mean action std | 1.00023 | 1.00013 | 4 |
| Mean episode length (control steps) | 12.0 | 51.65 | 4 |
Linear tracking/episode length increased, angular tracking decreased, and exploration std stayed
near 1. Four noisy points, random initial episode lengths and falling replays do **not** establish
learning or convergence. `/time` companion tags are no longer mislabeled as iteration curves.
## Measured replay and T1 coverage
| Evidence | Sampling | CSV rows | Failure events | Recorded kicks |
|---|---|---:|---:|---:|
| `runs/ops_g1_joints.csv` | 1 world × 150 steps × 29 joints | 4,350 | 1 | 2 |
| `runs/ops_t1_joints.csv` | 4 worlds × 150 steps × 23 joints | 13,800 | 6 | 7 |
Both used the stress profile and finite projected joint torques. CSV terminal-row counts matched
failure events × joint count (29 and 138), confirming terminal samples were retained before reset.
T1 recorded 4 distinct initial states out of 4 worlds. Its policy came from a separate two-update
run at `runs/ops_t1_validation`.
`runs/ops_g1_replay.mp4` decoded successfully: **150 frames, 50 FPS, 3 seconds**. A decoded frame
was visually inspected; robot and ground are visible. The matching measured report records a fall.
Interactive native/Viser viewing was not manually exercised in this revision.
A separate four-step evaluation, `runs/ops_g1_censored.json`, had no completed episodes:
completed-only failure fraction/return/duration were **null**, and both worlds were explicitly
censored rather than reported as successful episodes.
## Export regression and provenance
The updated evaluation report was accepted by ONNX export; `runs/ops_g1_bundle` passed
`verify-bundle --inference`. All outputs remain `hardware_approved: false`.
Earlier `ops_*_v1` development runs are retained. They predate the TensorBoard time-axis correction
and other final source changes and intentionally fail current source-identity checks. Historical
`g1_validation` / `t1_validation` runs also require their recorded sources for replay. Do not
rewrite provenance hashes to make old runs appear current.