# Training-operations validation Reference-environment checks on **2026-11-03**, using the Linux/RTX 5080 backend described in [VALIDATION.md](VALIDATION.md). The files below are development evidence under ignored `runs/`, not artifacts shipped with a fresh repository clone. Usage and statistical definitions are in [TRAINING_OPERATIONS.md](TRAINING_OPERATIONS.md). ## Automated checks **58 tests passed**; Ruff lint/format passed. One upstream TorchScript deprecation warning remains. New checks cover seed-level sample variance and null statistics, duplicate/mismatched protocols, paired comparisons, selection coverage, TensorBoard scalar reading and `/time` axis handling, CSV joint ordering and projected torque, atomic checkpoint publication and update indexing, committed-vs-uncommitted partial runs, Linux PID identity/stop, and both robots' evaluation profiles. ## Background training and real interruption/resume - `runs/ops_g1_validation`: 16 environments × 24 steps × **4 updates**, launched with `train --background`; state progressed to completed, with live progress and committed `model_0.pt`, `model_3.pt`, `final.pt` visible through the CLI. - `runs/ops_interrupt_validation`: a bounded 16-update job was deliberately stopped after its first committed checkpoint. `status` observed it alive/training; `stop` sent SIGTERM; worker and run both recorded **interrupted**, preserving `model_0.pt`. - `runs/ops_resume_validation`: resumed that interrupted checkpoint for **2 additional updates**. Final metadata reports **3 completed updates**, not 2 and not 4. Direct inspection of these trusted generated checkpoint files confirmed Adam step counters **20 → 60** and the backend common-step counter **24 → 72**. The optimizer was continued, not reinitialized. No long convergence run, scheduler restart/reboot recovery, or multi-GPU orchestration was tested. Process-loss/PID-reuse guards are covered by process-level tests, not a power-loss experiment. ## Actual matrix and checkpoint selection `runs/ops_g1_matrix` contains **32 completed cells**: - 2 checkpoints: after update 1 and update 4; - 2 evaluation seeds: 101 and 202; - 4 commands: standing, forward, lateral, turn; - 2 profiles: randomized and stress; - each cell: **2 worlds × 150 control steps**, dt 0.02 s, episode limit 5 s. Every cell recorded **2 distinct initial states out of 2 worlds**. Stress cells recorded **71 total velocity-kick events**. Repeating the same stress seed/command produced exactly matching initial state arrays; XY RMSE differed by about `1.88e-5 m/s`, demonstrating why seeded draws are not a promise of bitwise deterministic CUDA trajectories. `table.csv`, `table.md`, and `summary.json` contain seed-level means/sample variances, failure metrics and censoring. This 3 s observation horizon is intentionally short for engineering checks, not sufficient policy acceptance evidence. `runs/ops_g1_selection.json` ranked the earlier checkpoint first by equal-weight mean failure incidence: **0.3333 vs 0.3750 events per simulated world-second**. This is not a general quality ranking. Both policies fall; more updates did not automatically produce a better measured result. `runs/ops_g1_baseline_matrix` and `runs/ops_g1_candidate_matrix` are single-checkpoint views copied from the same 32 real cells, not additional independent experiments. `compare` successfully produced paired-seed differences in `runs/ops_g1_comparison.json`. ## Training curves, not time spent `runs/ops_g1_curves` contains JSON, CSV and parseable SVG extracted from actual TensorBoard events: | Scalar | First | Last | Samples | |---|---:|---:|---:| | Linear tracking reward | 0.01648 | 0.05844 | 4 | | Angular tracking reward | 0.004257 | 0.001208 | 4 | | Mean action std | 1.00023 | 1.00013 | 4 | | Mean episode length (control steps) | 12.0 | 51.65 | 4 | Linear tracking/episode length increased, angular tracking decreased, and exploration std stayed near 1. Four noisy points, random initial episode lengths and falling replays do **not** establish learning or convergence. `/time` companion tags are no longer mislabeled as iteration curves. ## Measured replay and T1 coverage | Evidence | Sampling | CSV rows | Failure events | Recorded kicks | |---|---|---:|---:|---:| | `runs/ops_g1_joints.csv` | 1 world × 150 steps × 29 joints | 4,350 | 1 | 2 | | `runs/ops_t1_joints.csv` | 4 worlds × 150 steps × 23 joints | 13,800 | 6 | 7 | Both used the stress profile and finite projected joint torques. CSV terminal-row counts matched failure events × joint count (29 and 138), confirming terminal samples were retained before reset. T1 recorded 4 distinct initial states out of 4 worlds. Its policy came from a separate two-update run at `runs/ops_t1_validation`. `runs/ops_g1_replay.mp4` decoded successfully: **150 frames, 50 FPS, 3 seconds**. A decoded frame was visually inspected; robot and ground are visible. The matching measured report records a fall. Interactive native/Viser viewing was not manually exercised in this revision. A separate four-step evaluation, `runs/ops_g1_censored.json`, had no completed episodes: completed-only failure fraction/return/duration were **null**, and both worlds were explicitly censored rather than reported as successful episodes. ## Export regression and provenance The updated evaluation report was accepted by ONNX export; `runs/ops_g1_bundle` passed `verify-bundle --inference`. All outputs remain `hardware_approved: false`. Earlier `ops_*_v1` development runs are retained. They predate the TensorBoard time-axis correction and other final source changes and intentionally fail current source-identity checks. Historical `g1_validation` / `t1_validation` runs also require their recorded sources for replay. Do not rewrite provenance hashes to make old runs appear current.