103 lines
5.7 KiB
Markdown
103 lines
5.7 KiB
Markdown
# Training-operations validation
|
||
|
||
Reference-environment checks on **2026-11-03**, using the Linux/RTX 5080 backend described in
|
||
[VALIDATION.md](VALIDATION.md). The files below are development evidence under ignored `runs/`,
|
||
not artifacts shipped with a fresh repository clone. Usage and statistical definitions are in
|
||
[TRAINING_OPERATIONS.md](TRAINING_OPERATIONS.md).
|
||
|
||
## Automated checks
|
||
|
||
**58 tests passed**; Ruff lint/format passed. One upstream TorchScript deprecation warning remains.
|
||
New checks cover seed-level sample variance and null statistics, duplicate/mismatched protocols,
|
||
paired comparisons, selection coverage, TensorBoard scalar reading and `/time` axis handling,
|
||
CSV joint ordering and projected torque, atomic checkpoint publication and update indexing,
|
||
committed-vs-uncommitted partial runs, Linux PID identity/stop, and both robots' evaluation profiles.
|
||
|
||
## Background training and real interruption/resume
|
||
|
||
- `runs/ops_g1_validation`: 16 environments × 24 steps × **4 updates**, launched with
|
||
`train --background`; state progressed to completed, with live progress and committed
|
||
`model_0.pt`, `model_3.pt`, `final.pt` visible through the CLI.
|
||
- `runs/ops_interrupt_validation`: a bounded 16-update job was deliberately stopped after its
|
||
first committed checkpoint. `status` observed it alive/training; `stop` sent SIGTERM;
|
||
worker and run both recorded **interrupted**, preserving `model_0.pt`.
|
||
- `runs/ops_resume_validation`: resumed that interrupted checkpoint for **2 additional updates**.
|
||
Final metadata reports **3 completed updates**, not 2 and not 4. Direct inspection of these
|
||
trusted generated checkpoint files confirmed Adam step counters **20 → 60** and the backend
|
||
common-step counter **24 → 72**. The optimizer was continued, not reinitialized.
|
||
|
||
No long convergence run, scheduler restart/reboot recovery, or multi-GPU orchestration was tested.
|
||
Process-loss/PID-reuse guards are covered by process-level tests, not a power-loss experiment.
|
||
|
||
## Actual matrix and checkpoint selection
|
||
|
||
`runs/ops_g1_matrix` contains **32 completed cells**:
|
||
|
||
- 2 checkpoints: after update 1 and update 4;
|
||
- 2 evaluation seeds: 101 and 202;
|
||
- 4 commands: standing, forward, lateral, turn;
|
||
- 2 profiles: randomized and stress;
|
||
- each cell: **2 worlds × 150 control steps**, dt 0.02 s, episode limit 5 s.
|
||
|
||
Every cell recorded **2 distinct initial states out of 2 worlds**. Stress cells recorded **71 total
|
||
velocity-kick events**. Repeating the same stress seed/command produced exactly matching initial
|
||
state arrays; XY RMSE differed by about `1.88e-5 m/s`, demonstrating why seeded draws are not a
|
||
promise of bitwise deterministic CUDA trajectories.
|
||
|
||
`table.csv`, `table.md`, and `summary.json` contain seed-level means/sample variances, failure
|
||
metrics and censoring. This 3 s observation horizon is intentionally short for engineering checks,
|
||
not sufficient policy acceptance evidence.
|
||
|
||
`runs/ops_g1_selection.json` ranked the earlier checkpoint first by equal-weight mean failure
|
||
incidence: **0.3333 vs 0.3750 events per simulated world-second**. This is not a general quality
|
||
ranking. Both policies fall; more updates did not automatically produce a better measured result.
|
||
|
||
`runs/ops_g1_baseline_matrix` and `runs/ops_g1_candidate_matrix` are single-checkpoint views copied
|
||
from the same 32 real cells, not additional independent experiments. `compare` successfully
|
||
produced paired-seed differences in `runs/ops_g1_comparison.json`.
|
||
|
||
## Training curves, not time spent
|
||
|
||
`runs/ops_g1_curves` contains JSON, CSV and parseable SVG extracted from actual TensorBoard events:
|
||
|
||
| Scalar | First | Last | Samples |
|
||
|---|---:|---:|---:|
|
||
| Linear tracking reward | 0.01648 | 0.05844 | 4 |
|
||
| Angular tracking reward | 0.004257 | 0.001208 | 4 |
|
||
| Mean action std | 1.00023 | 1.00013 | 4 |
|
||
| Mean episode length (control steps) | 12.0 | 51.65 | 4 |
|
||
|
||
Linear tracking/episode length increased, angular tracking decreased, and exploration std stayed
|
||
near 1. Four noisy points, random initial episode lengths and falling replays do **not** establish
|
||
learning or convergence. `/time` companion tags are no longer mislabeled as iteration curves.
|
||
|
||
## Measured replay and T1 coverage
|
||
|
||
| Evidence | Sampling | CSV rows | Failure events | Recorded kicks |
|
||
|---|---|---:|---:|---:|
|
||
| `runs/ops_g1_joints.csv` | 1 world × 150 steps × 29 joints | 4,350 | 1 | 2 |
|
||
| `runs/ops_t1_joints.csv` | 4 worlds × 150 steps × 23 joints | 13,800 | 6 | 7 |
|
||
|
||
Both used the stress profile and finite projected joint torques. CSV terminal-row counts matched
|
||
failure events × joint count (29 and 138), confirming terminal samples were retained before reset.
|
||
T1 recorded 4 distinct initial states out of 4 worlds. Its policy came from a separate two-update
|
||
run at `runs/ops_t1_validation`.
|
||
|
||
`runs/ops_g1_replay.mp4` decoded successfully: **150 frames, 50 FPS, 3 seconds**. A decoded frame
|
||
was visually inspected; robot and ground are visible. The matching measured report records a fall.
|
||
Interactive native/Viser viewing was not manually exercised in this revision.
|
||
|
||
A separate four-step evaluation, `runs/ops_g1_censored.json`, had no completed episodes:
|
||
completed-only failure fraction/return/duration were **null**, and both worlds were explicitly
|
||
censored rather than reported as successful episodes.
|
||
|
||
## Export regression and provenance
|
||
|
||
The updated evaluation report was accepted by ONNX export; `runs/ops_g1_bundle` passed
|
||
`verify-bundle --inference`. All outputs remain `hardware_approved: false`.
|
||
|
||
Earlier `ops_*_v1` development runs are retained. They predate the TensorBoard time-axis correction
|
||
and other final source changes and intentionally fail current source-identity checks. Historical
|
||
`g1_validation` / `t1_validation` runs also require their recorded sources for replay. Do not
|
||
rewrite provenance hashes to make old runs appear current.
|