Clarify pipeline deployment and remove personal path dependencies

This commit is contained in:
Codex
2026-10-27 10:57:38 +08:00
parent ae28d55f81
commit 0aa2f49ab8
14 changed files with 465 additions and 63 deletions
+82 -36
View File
@@ -1,53 +1,99 @@
# Hand Motion Pipeline:Dyn-HaMR · dex-retargeting · SPIDER
# Hand Motion Pipeline
在 HandFlow 工作区中集成 RGB-D / 单目人手重建(Dyn-HaMR、HandFlow)、L20 左右手重定向、FoundationPose 物体跟踪、手物参考修复、SPIDER 接触优化实验和 HDF5 轨迹导出。
从 D405 RGB-D 视频重建人手动作,将动作重定向到 L20 左右手,估计物体 6D 位姿,再进行手物配准、接触参考修复和 SPIDER 物理优化。
```text
D405 RGB-D 视频 → 静态背景里程计 → Dyn-HaMR 双手世界轨迹(也可 HandFlow / ViPE 单目分支)
→ dex-retargeting + L20 左右手 URDF → 机器人腕部与关节轨迹
→ FoundationPose 物体 6D 跟踪 → 手物配准 → 碰撞修复 / 桌面约束 / 平滑投影 → 运动学参考
→ SPIDER 接触引导物理滚动(MuJoCo Warp)→ 审计 / 回放 / HDF5
RGB-D 视频 + 内参 + 物体 CAD
→ 相机运动估计 + Dyn-HaMR 人手重建
→ dex-retargeting + L20 左右手模型 → 手指关节与腕部轨迹
→ FoundationPose 物体 6D 跟踪 → 手物配准
→ 接触目标 / 碰撞修复 / 桌面约束 / 平滑投影 → 运动学参考
→ SPIDER / MuJoCo Warp → 自由物体物理验证、审计和回放
```
## 从这里开始
当前示例为 `20260915_171525` 的双手红蓝盒子录制。链路各阶段已有代码和执行说明;参考轨迹与物理结果分别验收。现有实验尚未复现稳定夹持、抬升和翻转,也未验证真机。HandFlow / ViPE 单目入口和旧瓶子、HDF5 示例作为独立分支保留。
- **[最新流程(RGB-D 双手 → Dyn-HaMR → L20 重定向 → FoundationPose → 参考修复 → SPIDER)](docs/PIPELINE_LATEST.md)** ← 当前交付链路,按此执行
- [安装依赖、环境与权重下载](docs/SETUP_AND_WEIGHTS.md) ← 部署先看这个
- 克隆本仓库后:`git lfs pull && python3 scripts/large_files.py assemble`(大于 50 MB 的权重是分片存放的,见 SETUP §4.0)
- [环境变量配置模板](configs/project_env.example.sh)
- [集成结构、环境、运行顺序与上传步骤](docs/INTEGRATED_PROJECT.md)
- [重定向方法、验证与限制](docs/retargeting_workflow_and_lessons_zh.md)
- [HDF5 原始接口要求](docs/HDF5_REQUIREMENTS.md):原文为左手;当前导出为明确标记的右手变体。
- [HandFlow 原始 README](docs/HANDFLOW_UPSTREAM_README.md)
## 部署入口
## 当前能力
**完整 RGB-D 流程按 [安装依赖与资源](docs/SETUP_AND_WEIGHTS.md) 配置,再按 [逐阶段执行说明](docs/PIPELINE_LATEST.md) 运行。** 根目录 `requirements.txt`、`setup_env.sh` 和 `setup_vipe_env.sh` 服务旧 HandFlow / ViPE 分支,不会安装完整流程。
- 已有 L20 右手重定向、腕部运动、平滑、模型约束检查、可视化及 HDF5 导出代码。
- 1333 帧完整回放包含运动学接触配准,不能等同于完整 SPIDER 动力学抓取成功。
- SPIDER 实验和完整运动学回放是不同产物;只根据各自的实际评估结果作结论。
- 部分脚本仍使用 `2047635068` / `15886123` 示例路径,需要按数据修改;尚未整理为任意视频的一键通用工具。
依赖分成四个环境,避免 MuJoCo 与上游依赖互相覆盖:
## 目录
| 环境 | 角色 | Python |
|---|---|---|
| `.venv` | FoundationPose、RGB-D 处理、配准、参考修复、渲染 | 3.11 |
| `third_party/Dyn-HaMR/.dynhamr` | Dyn-HaMR 人手重建 | 3.10 |
| `.dex` | L20 重定向和 HDF5 导出 | 3.11 |
| `.spider` | SPIDER 物理优化与验证 | 新部署按上游使用 ≥ 3.12 |
| 目录 | 内容 |
|---|---|
| `scripts/` | 重建导出、L20 适配、SPIDER 场景/运行、回放验证、HDF5 导出 |
| `model/`、`utils/`、`preprocessing/`、`visualization/` | HandFlow 推理代码 |
| `third_party/` | 上游源码;源码交付包使用带来源记录的普通目录快照 |
| `configs/`、`requirements/` | 配置、重定向依赖和本机观测版本 |
| `docs/` | 集成、数据与方法说明 |
| `output/`、权重和本地环境 | 本地运行资源,不进入源码交付 |
需要 NVIDIA GPU、与 Torch 匹配的 CUDA toolkit / nvcc,以及 C++ 编译和离屏渲染依赖。具体版本和安装顺序见安装文档;现有版本记录来自开发机,尚未完成干净机器的从零安装复现。
## 生成源码交付包
在当前完整开发工作区运行:
## 获取资源与准备示例
```bash
python3 scripts/package_source_release.py
git lfs install
GIT_LFS_SKIP_SMUDGE=1 git clone https://gitea.robotquan.com/liyang/hand-motion-pipeline.git
cd hand-motion-pipeline
git lfs pull
python3 scripts/large_files.py assemble
python3 scripts/check_pipeline_resources.py --prepare-example --recording-only
```
输出至 `dist/hand-motion-pipeline/` 和同名 `.tar.gz`。包中保存第三方版本、源码文件哈希、外部资源清单、排除文件及可移植性检查报告。此命令只整理文件,不运行推理,不上传远程仓库。
大于平台单请求限制的权重以分片存储,`assemble` 会按清单校验 SHA-256。资源检查脚本不依赖 GPU 包;`--prepare-example` 为录制中多嵌套的一层创建相对链接,保留原视频与深度文件。
## 来源和授权
| 资源 | 云端状态 | 接收方操作 |
|---|---|---|
| 主流程权重、L20 URDF / 网格、双盒示例录制 | 已通过 Git LFS 管理 | `git lfs pull`,再组装权重 |
| 两个盒子 CAD | 已包含 | 使用 `docs/上半.stl`、`docs/下半.stl` |
| MANO 人手模型 | **未上传** | 按 MANO 官方获取条件自行准备,见下方 |
| Python 环境、CUDA 编译产物、缓存、`output/` | 未包含 | 在目标机器安装、构建和生成 |
| ViPE 单目分支的额外模型缓存 | 未完整包含 | 仅使用该分支时按安装文档准备 |
保留 [HandFlow 原 LICENSE](LICENSE);Dyn-HaMR、HaMeR、ViPE、SPIDER 等使用各自目录的上游 LICENSE。dex-retargeting 作为版本依赖使用。本地代码是组件适配与实验集成,上游算法和资源保持原归属。
### MANO 模型
仓库不分发 `MANO_RIGHT.pkl` / `MANO_LEFT.pkl`。请从 [MANO 官方网站](https://mano.is.tue.mpg.de/) 按其条件取得,放到:
```text
third_party/hamer/_DATA/data/mano/MANO_RIGHT.pkl
third_party/hamer/_DATA/data/mano/MANO_LEFT.pkl # 使用独立左手模型的分支需要
```
Dyn-HaMR 的 `_DATA/data/mano` 已通过相对软链接指向该目录。默认资源检查要求右手模型;需要独立左手模型时加 `--require-left-mano`。`MANO_ROOT` 用于支持该变量的入口,不能替代 Dyn-HaMR 固定目录或其软链接。
```bash
python3 scripts/check_pipeline_resources.py
```
该命令检查输入布局、缺失文件、LFS 指针和部分清单大小,不加载模型或执行推理。运行环境仍须完成安装文档中的 GPU 冒烟检查。
## 路径配置与更换数据
项目可以放在任意目录,无需复制开发者的主目录、下载目录或 Python 环境:
```bash
source configs/project_env.example.sh
# 若 nvcc 不在 PATH 中,先设置自己机器的 CUDA toolkit:
# export CUDA_HOME=/path/to/cuda-toolkit
source scripts/foundationpose_env.sh
```
项目根目录自动定位;CUDA 头文件从当前主 Python 环境查找。`CUDA_HOME`、`HANDFLOW_PYTHON` 和显式设置的 GPU 编译架构可按目标机器覆盖。两个旧视频脚本现在要求调用者提供输入:
```bash
python scripts/run_handflow_15886123.py --input /path/to/source.mp4
python scripts/export_camera_delivery.py --input /path/to/source.mp4
```
主流程仍固定部分示例序列名、帧数、颜色和 CAD 映射。更换视频按 [新数据适配说明](docs/PIPELINE_LATEST.md#3-换一段新视频要改什么) 修改,不能将其当作任意物体的一键工具。上游源码还保留部分示例路径,范围见 [路径与迁移说明](docs/PORTABILITY.md)。
## 其他文档与源码包
- [FoundationPose 构建与验证](docs/FOUNDATIONPOSE_SETUP.md)
- [集成结构与旧瓶子分支](docs/INTEGRATED_PROJECT.md)
- [重定向方法、验证与限制](docs/retargeting_workflow_and_lessons_zh.md)
- [HDF5 接口约定](docs/HDF5_REQUIREMENTS.md)
- [HandFlow 原始说明](docs/HANDFLOW_UPSTREAM_README.md)
`python3 scripts/package_source_release.py` 导出的源码包与当前 Git LFS 仓库不同:源码包不携带权重、L20 资产、用户录制、运行环境和实验输出。`UPSTREAM_SOURCES.json`、`SHA256SUMS`、`EXTERNAL_ASSETS.json` 等为原源码快照的来源与资源记录,后续提交以 Git 历史和 `configs/large_files.json` 为准。
保留 [HandFlow 原 LICENSE](LICENSE) 及各第三方目录 LICENSE;资源使用条件由对应提供方规定。
+5 -4
View File
@@ -1,15 +1,16 @@
#!/usr/bin/env bash
# Source from any directory: source /path/to/project/configs/project_env.example.sh
export HANDFLOW_PROJECT_ROOT="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")/.." && pwd)"
export HANDFLOW_PYTHON="${HANDFLOW_PYTHON:-$HANDFLOW_PROJECT_ROOT/.venv/bin/python}"
export TORCH_HOME="$HANDFLOW_PROJECT_ROOT/.torch_cache"
export HF_HOME="$HANDFLOW_PROJECT_ROOT/.hf_cache"
export WARP_CACHE_PATH="$HANDFLOW_PROJECT_ROOT/.spider_cache/warp"
export TORCHINDUCTOR_CACHE_DIR="$HANDFLOW_PROJECT_ROOT/.spider_cache/torchinductor"
export VIPE_ROOT="$HANDFLOW_PROJECT_ROOT/third_party/vipe"
export VIPE_PYTHON="$HANDFLOW_PROJECT_ROOT/.venv/bin/python"
export HAMER_CKPT="$HANDFLOW_PROJECT_ROOT/third_party/hamer/_DATA/hamer_ckpts/checkpoints/hamer.ckpt"
export MANO_ROOT="$HANDFLOW_PROJECT_ROOT/third_party/hamer/_DATA/data/mano"
export DETECTOR_CKPT="$HANDFLOW_PROJECT_ROOT/weights/detector.pt"
export VIPE_PYTHON="${VIPE_PYTHON:-$HANDFLOW_PYTHON}"
export HAMER_CKPT="${HAMER_CKPT:-$HANDFLOW_PROJECT_ROOT/third_party/hamer/_DATA/hamer_ckpts/checkpoints/hamer.ckpt}"
export MANO_ROOT="${MANO_ROOT:-$HANDFLOW_PROJECT_ROOT/third_party/hamer/_DATA/data/mano}"
export DETECTOR_CKPT="${DETECTOR_CKPT:-$HANDFLOW_PROJECT_ROOT/weights/detector.pt}"
export HANDFLOW_NORMALIZATION_STATS="$HANDFLOW_PROJECT_ROOT/weights/normalization_stats.npz"
export MUJOCO_GL="${MUJOCO_GL:-osmesa}"
# Offline flags intentionally unset here: populate caches before offline inference.
+10
View File
@@ -10,6 +10,16 @@
## 0. 总览
首次克隆先按 [安装与资源说明](SETUP_AND_WEIGHTS.md) 拉取并组装 LFS 资源,准备 MANO。云端录制多嵌套了一层,可运行以下命令创建兼容的相对链接;数据不移动、不覆盖:
```bash
python3 scripts/check_pipeline_resources.py --prepare-example --recording-only
python3 scripts/check_pipeline_resources.py
mkdir -p output
```
资源检查不等于 GPU 或物理验证,后续仍需对应环境和每阶段审计。
```text
docs/<seq>/{color.mp4, depth/*.png, intrinsics.json} docs/上半.stl(红盒) docs/下半.stl(蓝盒)
│
+29
View File
@@ -0,0 +1,29 @@
# 路径配置与部署迁移
完整 RGB-D 流程以 [SETUP_AND_WEIGHTS.md](SETUP_AND_WEIGHTS.md) 为安装入口。项目根目录由脚本位置计算,不要求开发者的主目录、下载目录或共享 Python 环境存在。
## 本机路径如何替换
| 项目 | 当前行为 | 目标机器操作 |
|---|---|---|
| 项目根目录 | `Path(__file__)` 或 Bash 脚本位置定位 | 可以放在任意目录,命令从项目根目录执行 |
| MANO | 默认 `third_party/hamer/_DATA/data/mano/`,Dyn-HaMR 相对链接共用 | 自行取得模型并放入共享目录 |
| CUDA toolkit | 已设 `CUDA_HOME` → 项目内 toolkit → `PATH` 的 nvcc | 安装匹配的 toolkit;必要时显式设置 `CUDA_HOME` |
| NVIDIA 包头文件 | 从主 Python 环境查找 | 安装到自己的主环境,不复制开发者环境 |
| 编译 GPU 架构 | 默认由 PyTorch 检测;保留调用者配置 | 无可见 GPU 编译时按自己的目标设置 `TORCH_CUDA_ARCH_LIST` |
| 历史视频入口 | `run_handflow_15886123.py`、`export_camera_delivery.py` 使用必填 `--input` | 显式传入自己的原视频 |
| 嵌套示例录制 | 资源检查命令建立缺失的相对链接 | 拉取 LFS 后执行 `--prepare-example --recording-only` |
`MANO_ROOT`、`HAMER_CKPT`、`DETECTOR_CKPT`、`VIPE_PYTHON` 的自定义值在加载环境模板时保留。支持环境变量的入口可采用覆盖值,Dyn-HaMR 等上游固定资源目录仍需放置资源或正确链接。
## 示例参数与上游路径
消除个人主目录依赖不等于任意视频自动适配。当前双盒主流程仍有固定序列名、帧数、颜色阈值和 CAD 映射,见 [更换视频](PIPELINE_LATEST.md#3-换一段新视频要改什么)。两个旧视频入口仍对应各自历史数据的输出和验收约定,仅输入路径已参数化。
第三方源码保留了上游示例默认值,例如 Dyn-HaMR 的 `video_vipe.yaml`、可选 METRO 入口 `HMP/fitting_utils.py::run_metro`、HaMeR 评测数据目录,以及 FoundationPose 的调试目录。集成 RGB-D 启动脚本显式传入数据与相机路径,物体跟踪脚本显式传入调试目录。这些可选上游入口需要另行配置,不应直接当成可移植的完整入口。
`PORTABILITY_REPORT.json`、`UPSTREAM_SOURCES.json`、`SHA256SUMS` 是原源码快照的审计与来源记录,可能包含后来已修改的行号、标识或历史路径;当前内容以 Git 提交为准。它们不会要求上传记录中提到的个人文件夹。当前分片权重校验使用 `configs/large_files.json`。
## 验证范围
`scripts/check_pipeline_resources.py` 仅使用 Python 标准库,检查录制布局、模型文件存在性、未拉取的 LFS 指针和部分权重大小;不反序列化 MANO,不加载网络,不验证位姿精度或物理抓取。`large_files.py assemble` 另行校验分片权重 SHA-256。GPU 扩展、网络前向和物理冒烟检查分别按安装文档执行,完整抓取按每段实验结果验收。
+40 -10
View File
@@ -1,5 +1,7 @@
# 安装依赖、环境与权重(2026-09-17)
部署指引更新:2026-10-27。本文是完整 RGB-D 流程的安装入口;根目录 `requirements.txt`、`setup_env.sh`、`setup_vipe_env.sh` 仅用于旧 HandFlow / ViPE 分支。先按第 4.0 节准备资源,再安装四个环境并执行第 6 节检查。
适用:源码快照 `hand-motion-pipeline-20260917`(由 `scripts/package_source_release.py` 生成)。链路为 **RGB-D → Dyn-HaMR → L20 重定向 → FoundationPose → 参考修复 → SPIDER**,执行顺序与每步命令见 [PIPELINE_LATEST.md](PIPELINE_LATEST.md)。本文按本机(2026-09-17)实际环境核对;**没有在干净机器上从零复现安装**,版本表是观测值,不是跨平台 lock。
## 1. 机器与系统依赖
@@ -11,7 +13,7 @@
| apt 包 | `build-essential cmake ninja-build ffmpeg libeigen3-dev libosmesa6-dev libegl1-mesa-dev libgl1-mesa-dev libglfw3-dev` |
| 渲染 | 离屏 `MUJOCO_GL=osmesa`(默认);交互回放 `glfw`;Dyn-HaMR 可视化用 EGL |
`scripts/foundationpose_env.sh` 会清空 ROS 注入的 `PYTHONPATH / LD_LIBRARY_PATH`,把 `.venv`、nvcc、编译缓存(`.cache/`)指到项目内。**所有 `.venv` 脚本都先 `source` 它**。
`scripts/foundationpose_env.sh` 会清空 ROS 注入的 `PYTHONPATH / LD_LIBRARY_PATH`,默认选择项目 `.venv`,从这个环境发现 NVIDIA 包头文件。toolkit 优先使用调用者设置的 `CUDA_HOME`,其次使用项目内 toolkit,再从 `PATH` 的 `nvcc` 定位;缺失时明确报错。GPU 编译架构默认由 PyTorch 检测,显式设置的 `TORCH_CUDA_ARCH_LIST` 会保留。无需复制开发者的 Python 环境。**所有 `.venv` 脚本都先 `source` 它**。
## 2. 环境角色(4 个解释器,不要混装)
@@ -32,7 +34,8 @@
python3.11 -m venv .venv
.venv/bin/python -m pip install torch==2.7.1 torchvision==0.22.1 --index-url https://download.pytorch.org/whl/cu128
.venv/bin/python -m pip install -r requirements/pipeline_venv.txt
source scripts/foundationpose_env.sh # 之后 python == .venv/bin/python,CUDA_HOME 指向 nvcc 12.8
# nvcc 不在 PATH 中时,先 export CUDA_HOME=/path/to/your/cuda-toolkit
source scripts/foundationpose_env.sh # 默认使用 .venv/bin/python,检查本机 nvcc
python -m pip install --no-build-isolation "git+https://github.com/facebookresearch/pytorch3d.git@stable" # 观测 0.7.9
python -m pip install --no-build-isolation --no-deps ./third_party/nvdiffrast # 快照 0.4.0,需 nvcc
# FoundationPose C++ 旋转聚类模块(pybind11),产物 third_party/FoundationPose/mycpp/build/mycpp.cpython-311-*.so
@@ -87,20 +90,24 @@ env PYTHONPATH= LD_LIBRARY_PATH= .spider/bin/python scripts/verify_spider_instal
```bash
git lfs install
git clone https://gitea.robotquan.com/liyang/hand-motion-pipeline.git && cd hand-motion-pipeline
GIT_LFS_SKIP_SMUDGE=1 git clone https://gitea.robotquan.com/liyang/hand-motion-pipeline.git
cd hand-motion-pipeline
git lfs pull # 约 6.3 GB
python3 scripts/large_files.py assemble # 拼回 hamer.ckpt / UniDepth / HandFlow / HMP / FoundationPose / detector,逐个 sha256 校验
python3 scripts/large_files.py assemble --clean # 可选:拼回后删除分片省空间
python3 scripts/check_pipeline_resources.py --prepare-example --recording-only
mkdir -p output
```
`third_party/Dyn-HaMR/_DATA/` 里指向 HaMeR 检查点、MANO、检测器的是相对符号链接,克隆后自动生效。之后只差 MANO(4.2 节)。
资源检查命令会为嵌套录制创建缺失的相对链接,将 `docs/20260915_171525/20260915_171525/{color.mp4,depth/}` 暴露到脚本读取的外层目录;已有文件不覆盖、源数据不移动。标准布局不改动;未拉取的 LFS 指针和缺失深度帧会报错。
源码包只含代码、配置、文档和两个小 CAD(`docs/上半.stl`、`docs/下半.stl`)。下表其余项都要另行准备;哈希与大小见包内 `EXTERNAL_ASSETS.json`。
`third_party/Dyn-HaMR/_DATA/` 保留资源相对软链接,MANO 目标目录尚无模型。LFS 拉取与组装后,还需按第 4.1 节准备 MANO,在目标机器安装环境、构建扩展。`assemble --clean` 仅是可选的本地磁盘整理操作,日常部署可保留分片。
**当前 Git LFS 仓库与源码压缩包是两种交付方式。** 仓库已包含主流程权重、L20、录制与 CAD;`package_source_release.py` 导出的源码包仅含代码、配置、文档和小 CAD,其他资源需另行恢复。下表列资源位置与来源,不表示 Git LFS 接收方必须重复下载。`EXTERNAL_ASSETS.json` 是原源码包资源记录,当前分片校验以 `configs/large_files.json` 为准。
| 资源 | 大小 | 放置位置 | 来源 | 本链路需要 |
|---|---|---|---|---|
| Dyn-HaMR 数据包 `_DATA/`(`hamer_ckpts/`、`data/mano/`、`hmp_model/`、`BMC/`) | 787 MB | `third_party/Dyn-HaMR/_DATA/` | 上游 `scripts/prepare.sh`(Google Drive)+ HaMeR `fetch_demo_data.sh` | 是 |
| `MANO_RIGHT.pkl` | — | `third_party/Dyn-HaMR/_DATA/data/mano/` | [MANO 官网](https://mano.is.tue.mpg.de/) 注册下载,不在任何包里 | 是(双手处理按 Dyn-HaMR 代码检查左手资源) |
| `MANO_RIGHT.pkl`,以及需要独立左手模型时的 `MANO_LEFT.pkl` | — | `third_party/hamer/_DATA/data/mano/`(Dyn-HaMR 软链接共用) | [MANO 官网](https://mano.is.tue.mpg.de/) 自行取得;未上传 | 是(左手文件按分支准备) |
| WiLoR 手检测器 `detector.pt` | 52 MB | `weights/detector.pt` + `third_party/Dyn-HaMR/third-party/hamer/pretrained_models/detector.pt` | [WiLoR HF Space](https://huggingface.co/spaces/rolpotamias/WiLoR/tree/main/pretrained_models) | 是 |
| FoundationPose 权重:`2023-10-28-18-33-37/`(refiner,68 MB)、`2024-01-11-20-02-45/`(scorer,190 MB),各含 `config.yml + model_best.pth` | 247 MB | `third_party/FoundationPose/weights/` | `scripts/download_foundationpose_weights.py`(清单 `configs/foundationpose_weights_manifest.json`,SHA-256 校验) | 是 |
| L20 灵巧手模型:`L20/{LEFT,RIGHT}/linkerhand_g20_{left,right}.urdf` + `meshes/` + 标定 JSON | 36 MB | `third_party/l20_assets/` | 模型提供方交付;左右手各自的 URDF,不是镜像 | 是 |
@@ -113,9 +120,26 @@ python3 scripts/large_files.py assemble --clean # 可选:拼回后删除分
### 4.1 Dyn-HaMR 数据包、MANO、检测器
**Git LFS 部署无需重新运行上游下载脚本覆盖已有资源。** 完成第 4.0 节后,将按 MANO 官方条件取得的模型放入共享目录:
```text
third_party/hamer/_DATA/data/mano/MANO_RIGHT.pkl
third_party/hamer/_DATA/data/mano/MANO_LEFT.pkl # 使用独立左手模型的分支需要
```
`third_party/Dyn-HaMR/_DATA/data/mano` 是指向这里的相对软链接,默认 `MANO_ROOT` 也指向这里。修改 `MANO_ROOT` 只影响支持该变量的入口,不能替代 Dyn-HaMR 固定目录或其软链接。模型不在 Git / LFS 中,无需上传开发者主目录或完整 Python 环境。
```bash
python3 scripts/check_pipeline_resources.py
# 使用独立左手模型的分支:
python3 scripts/check_pipeline_resources.py --require-left-mano
```
以下下载方式只用于不含资源的源码包部署或缺失资源恢复;先保留已有资源,按需执行:
```bash
cd third_party/Dyn-HaMR && bash scripts/prepare.sh && cd ../.. # 需要 gdown;下载后核对 _DATA 内四个子目录都非空
# MANO:官网下载后放 third_party/Dyn-HaMR/_DATA/data/mano/MANO_RIGHT.pkl
# MANO:按上方说明放到 third_party/hamer/_DATA/data/mano/
mkdir -p weights third_party/Dyn-HaMR/third-party/hamer/pretrained_models
curl -fL --retry 3 -C - 'https://huggingface.co/spaces/rolpotamias/WiLoR/resolve/main/pretrained_models/detector.pt' -o weights/detector.pt
cp weights/detector.pt third_party/Dyn-HaMR/third-party/hamer/pretrained_models/detector.pt
@@ -125,6 +149,8 @@ cp weights/detector.pt third_party/Dyn-HaMR/third-party/hamer/pretrained_models/
### 4.2 FoundationPose 权重
Git LFS 拉取并组装成功后可跳过本节下载;以下用于源码包部署或缺失文件恢复。
```bash
source scripts/foundationpose_env.sh
python scripts/download_foundationpose_weights.py
@@ -134,7 +160,7 @@ python scripts/download_foundationpose_weights.py
### 4.3 L20 模型与录制数据
由提供方 / 自己放到上表路径。`third_party/l20_assets` 与 `docs/20260915_171525` 的逐文件 SHA-256 在 `EXTERNAL_ASSETS.json`,可用来核对搬运是否完整。
Git LFS 仓库已包含左右手 URDF、网格、标定文件和示例录制。先拉取 LFS,再执行第 4.0 节示例准备命令。只有源码包部署需要单独恢复这些资源,原资源哈希见 `EXTERNAL_ASSETS.json`。自己的录制使用 `docs/<seq>/{color.mp4,depth/,intrinsics.json}` 布局,并按 PIPELINE 适配序列、帧数和 CAD 映射。
### 4.4 可选:HandFlow 与 ViPE 分支
@@ -155,14 +181,18 @@ Hugging Face 可换 `hf-mirror.com` 测速(第三方镜像)。支持 Range
```bash
source configs/project_env.example.sh # TORCH_HOME / HF_HOME / WARP_CACHE_PATH / HAMER_CKPT / MANO_ROOT / DETECTOR_CKPT / MUJOCO_GL
# nvcc 不在 PATH 中时,先 export CUDA_HOME=/path/to/your/cuda-toolkit
source scripts/foundationpose_env.sh # .venv + nvcc + 清 ROS 干扰;跑 .venv 脚本前必 source
```
流程脚本自己的环境变量(`HF_*`、`SPIDER_*`)只改**输出**目录,序列名与输入路径写死在脚本里;换视频要改哪些常量见 [PIPELINE_LATEST.md §3](PIPELINE_LATEST.md)。
`HANDFLOW_PYTHON` 可指定环境脚本所用主 Python,默认 `.venv/bin/python`;其他显式解释器路径仍需按部署对应。旧 `run_handflow_15886123.py`、`export_camera_delivery.py` 已要求 `--input /path/to/source.mp4`。项目不依赖开发者的主目录或下载目录。上游示例路径及迁移边界见 [PORTABILITY.md](PORTABILITY.md)。
## 6. 启动前检查
```bash
python3 scripts/check_pipeline_resources.py
source scripts/foundationpose_env.sh
python -c "import torch, pytorch3d, nvdiffrast.torch, mujoco, mujoco_warp, warp, trimesh, coacd, open3d, kornia, cv2; print(torch.cuda.is_available(), torch.__version__, mujoco.__version__)"
python scripts/verify_foundationpose_setup.py
@@ -194,7 +224,7 @@ git init -b main && git add . && git commit -m "Import RGB-D hand-object pipelin
git remote add origin <仓库地址> && git push -u origin main
```
包里的 `.gitignore` 已把权重、缓存、录制数据、`output/` 挡在仓库外;大文件走平台附件或独立存储,按第 4 节路径恢复。
本节生成的是源码包,运行资源不在包内。按第 4 节恢复资源,或直接使用当前带 Git LFS 资源的仓库。当前云端 LFS 跟踪规则与源码包排除规则不同,不能用源码包说明判断云端是否有权重。
## 8. 已知限制
+2
View File
@@ -1,4 +1,6 @@
# HandFlow V1 — inference + render deps (no training deps)
# Legacy branch only. For the complete RGB-D/L20/FoundationPose/SPIDER pipeline
# use docs/SETUP_AND_WEIGHTS.md and its four separate environments.
# One-line install: pip install -r requirements.txt
# Or use setup_env.sh to create a conda environment
+135
View File
@@ -0,0 +1,135 @@
#!/usr/bin/env python3
"""Check local RGB-D pipeline assets without importing GPU packages.
Use --prepare-example to expose a nested recording through relative symlinks.
This never downloads models, overwrites existing paths, or loads pickle files.
"""
import argparse
import json
from pathlib import Path
ROOT = Path(__file__).resolve().parents[1]
LFS_HEADER = b'version https://git-lfs.github.com/spec/v1'
def is_pointer(path):
with path.open('rb') as stream:
return stream.read(len(LFS_HEADER)) == LFS_HEADER
def prepare_recording(recording):
"""Make the existing nested layout usable without moving the source data."""
nested = recording / recording.name
planned = []
for name in ('color.mp4', 'depth', 'intrinsics.json'):
target, source = recording / name, nested / name
if target.exists():
continue
if target.is_symlink():
raise ValueError(f'Broken link; inspect it before retrying: {target}')
if not source.exists():
raise ValueError(f'Missing recording asset: {target}; run git lfs pull or provide the recording.')
planned.append((target, source))
# Check all sources first, so a missing input cannot leave partial links.
for target, source in planned:
target.symlink_to(source.relative_to(recording), target_is_directory=source.is_dir())
print(f'LINK {target.name} -> {source.relative_to(recording)}')
def check_recording(recording):
errors = []
color, depth, metadata = [recording / name for name in ('color.mp4', 'depth', 'intrinsics.json')]
for path in (color, metadata):
if not path.is_file():
errors.append(f'Missing {path}; use --prepare-example for the nested bundled layout.')
elif is_pointer(path):
errors.append(f'Git LFS pointer instead of data: {path}; run git lfs pull.')
elif path.stat().st_size == 0:
errors.append(f'Empty file: {path}')
if not depth.is_dir():
errors.append(f'Missing depth directory: {depth}; use --prepare-example if the recording is nested.')
if errors:
return errors
try:
meta = json.loads(metadata.read_text())
frames = int(meta['frames'])
if frames <= 0:
raise ValueError('frames must be positive')
for index in range(frames):
path = depth / f'{index:06d}.png'
if not path.is_file() or path.stat().st_size == 0:
errors.append(f'Missing/empty depth frame: {path}')
break
if is_pointer(path):
errors.append(f'Git LFS pointer instead of depth: {path}; run git lfs pull.')
break
except (ValueError, KeyError, TypeError) as exc:
errors.append(f'Invalid recording metadata {metadata}: {exc}')
return errors
def check_assets(root, require_left=False):
required = [
'third_party/hamer/_DATA/data/mano/MANO_RIGHT.pkl',
'third_party/hamer/_DATA/hamer_ckpts/checkpoints/hamer.ckpt',
'third_party/Dyn-HaMR/third-party/hamer/pretrained_models/detector.pt',
'third_party/Dyn-HaMR/_DATA/hmp_model/results/model/local_encoder.pth',
'third_party/Dyn-HaMR/_DATA/hmp_model/results/model/nemf.pth',
'third_party/Dyn-HaMR/_DATA/BMC/joint_angles.npy',
'docs/上半.stl', 'docs/下半.stl',
]
if require_left:
required.append('third_party/hamer/_DATA/data/mano/MANO_LEFT.pkl')
for side in ('left', 'right'):
required.append(f'third_party/l20_assets/L20/{side.upper()}/linkerhand_g20_{side}.urdf')
for checkpoint in ('2023-10-28-18-33-37', '2024-01-11-20-02-45'):
required.extend(f'third_party/FoundationPose/weights/{checkpoint}/{name}' for name in ('config.yml', 'model_best.pth'))
errors = []
manifest_path = root / 'configs/large_files.json'
manifest = json.loads(manifest_path.read_text()) if manifest_path.is_file() else {'files': []}
expected = {entry['path']: entry['bytes'] for entry in manifest['files']}
for relative in required:
path = root / relative
if not path.is_file() or path.stat().st_size == 0:
action = 'Provide your MANO model; see docs/SETUP_AND_WEIGHTS.md.' if path.suffix == '.pkl' else 'Run git lfs pull and scripts/large_files.py assemble.'
errors.append(f'Missing/empty {relative}. {action}')
elif is_pointer(path):
errors.append(f'Git LFS pointer: {relative}; run git lfs pull and assemble.')
elif relative in expected and path.stat().st_size != expected[relative]:
errors.append(f'Wrong size: {relative}; reassemble and verify SHA-256 with scripts/large_files.py assemble.')
for side in ('LEFT', 'RIGHT'):
directory = root / f'third_party/l20_assets/L20/{side}/meshes'
meshes = list(directory.rglob('*.stl')) + list(directory.rglob('*.STL'))
if not meshes:
errors.append(f'Missing L20 meshes: {directory}')
elif any(is_pointer(path) for path in meshes):
errors.append(f'Git LFS pointers in L20 {side} meshes; run git lfs pull.')
return errors
def main():
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument('--recording', type=Path, default=ROOT / 'docs/20260915_171525')
parser.add_argument('--prepare-example', action='store_true', help='Create missing relative links from the nested recording directory; preserve originals.')
parser.add_argument('--recording-only', action='store_true', help='Check/prepare recording layout without requiring model assets.')
parser.add_argument('--require-left-mano', action='store_true', help='Also require MANO_LEFT.pkl for branches which use a separate left model.')
args = parser.parse_args()
recording = args.recording.expanduser().resolve()
try:
if args.prepare_example:
prepare_recording(recording)
errors = check_recording(recording)
if not args.recording_only:
errors.extend(check_assets(ROOT, args.require_left_mano))
except (OSError, ValueError, KeyError, TypeError) as exc:
errors = [str(exc)]
for error in errors:
print(f'FAIL {error}')
if errors:
return 1
print('PASS recording layout and local asset files; model validity, GPU runtime, pose accuracy and grasp success require separate checks.')
return 0
if __name__ == '__main__':
raise SystemExit(main())
+8 -2
View File
@@ -1,4 +1,5 @@
"""Package the 1333-frame replay cameras, source cameras and frame mapping."""
import argparse
import os
os.environ.setdefault('MUJOCO_GL', 'osmesa')
import csv
@@ -38,9 +39,14 @@ def probe(path):
def main():
OUT.mkdir(exist_ok=True)
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument('--input', type=Path, required=True, help='Original source video for this historical 1333-frame replay.')
args = parser.parse_args()
original = args.input.expanduser().resolve()
if not original.is_file():
parser.error(f'Input video not found: {original}')
OUT.mkdir(parents=True, exist_ok=True)
replay = ROOT/'output/l20_full_replay'
original = Path('/home/timessage/下载/2047635068.mp4')
final = np.load(replay/'motion.npz')
temporal = np.load(ROOT/'output/l20_2047635068/jitter_audit/temporal/motion.npz')
reconstruction = np.load(ROOT/'output/results_2047635068/world_results.npz')
+37 -9
View File
@@ -1,17 +1,45 @@
#!/usr/bin/env bash
# Source this file before building or running FoundationPose.
FP_ROOT="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")/.." && pwd)"
fp_python="${HANDFLOW_PYTHON:-$FP_ROOT/.venv/bin/python}"
if [ ! -x "$fp_python" ]; then
echo "Main Python environment missing: $fp_python (see docs/SETUP_AND_WEIGHTS.md)" >&2
return 1
fi
if [ -z "${CUDA_HOME:-}" ]; then
if [ -x "$FP_ROOT/.cuda/usr/local/cuda-12.8/bin/nvcc" ]; then
CUDA_HOME="$FP_ROOT/.cuda/usr/local/cuda-12.8"
elif command -v nvcc >/dev/null 2>&1; then
CUDA_HOME="$(dirname -- "$(dirname -- "$(readlink -f -- "$(command -v nvcc)")")")"
else
echo "CUDA toolkit missing: install nvcc and set CUDA_HOME before sourcing this file." >&2
return 1
fi
fi
if [ ! -x "$CUDA_HOME/bin/nvcc" ]; then
echo "CUDA_HOME must contain bin/nvcc: $CUDA_HOME" >&2
return 1
fi
unset PYTHONPATH LD_LIBRARY_PATH
export CUDA_HOME="$FP_ROOT/.cuda/usr/local/cuda-12.8"
export PATH="$FP_ROOT/.venv/bin:$CUDA_HOME/bin:$PATH"
export TORCH_CUDA_ARCH_LIST=8.6
export MAX_JOBS=2
export CUDA_HOME
export PATH="$(dirname -- "$fp_python"):$CUDA_HOME/bin:$PATH"
# PyTorch detects the visible GPU by default; preserve explicit build settings.
export MAX_JOBS="${MAX_JOBS:-2}"
export PIP_CACHE_DIR="$FP_ROOT/.cache/pip"
export TORCH_EXTENSIONS_DIR="$FP_ROOT/.cache/torch_extensions"
export XDG_CACHE_HOME="$FP_ROOT/.cache"
export MPLCONFIGDIR="$FP_ROOT/.cache/matplotlib"
export CPATH="$CUDA_HOME/include"
for fp_inc in /home/timessage/python_envs/unitree_rl_mjlab/lib/python3.11/site-packages/nvidia/*/include; do
export CPATH="$CPATH:$fp_inc"
done
unset fp_inc
export CPATH="$CUDA_HOME/include${CPATH:+:$CPATH}"
fp_include_dirs="$("$fp_python" - <<'PY'
import site
from pathlib import Path
roots = [*site.getsitepackages(), site.getusersitepackages()]
for directory in sorted({p for root in roots for p in (Path(root) / 'nvidia').glob('*/include') if p.is_dir()}):
print(directory)
PY
)" || return 1
while IFS= read -r fp_inc; do
[ -z "$fp_inc" ] || export CPATH="$CPATH:$fp_inc"
done <<< "$fp_include_dirs"
unset fp_python fp_include_dirs fp_inc
+8 -1
View File
@@ -1,4 +1,5 @@
"""Resume the requested egocentric video inference with existing local models."""
import argparse
import json
import os
from pathlib import Path
@@ -6,6 +7,12 @@ import subprocess
import time
ROOT = Path(__file__).resolve().parents[1]
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument('--input', type=Path, required=True, help='Source video; supplied by the caller, not bundled with this historical example.')
args = parser.parse_args()
input_video = args.input.expanduser().resolve()
if not input_video.is_file():
parser.error(f'Input video not found: {input_video}')
OUT = ROOT / 'output/15886123_handflow'
OUT.mkdir(parents=True, exist_ok=True)
env = os.environ.copy()
@@ -20,7 +27,7 @@ env.update(PYTHONPATH=str(ROOT), LD_LIBRARY_PATH='', OMP_NUM_THREADS='4',
MANO_ROOT=str(ROOT / 'third_party/hamer/_DATA/data/mano'),
HANDFLOW_NORMALIZATION_STATS=str(ROOT / 'weights/normalization_stats.npz'))
cmd = [str(ROOT / '.venv/bin/python'), '-u', 'scripts/demo.py',
'--input', '/home/timessage/下载/15886123.mp4',
'--input', str(input_video),
'--fm_ckpt', str(ROOT / 'weights/handflow_denoiser.pt'),
'--intrinsics', 'auto', '--side', 'right', '--rotation_smooth_window', '5',
'--output_dir', str(OUT), '--save_npz', str(OUT / 'handflow_results.npz')]
+1
View File
@@ -1,5 +1,6 @@
#!/bin/bash
# HandFlow inference + render environment (training deps excluded)
# Legacy branch only. The full RGB-D pipeline uses docs/SETUP_AND_WEIGHTS.md.
# Usage: bash setup_env.sh [env_name=handflow] [python=3.10]
set -e
+1
View File
@@ -1,5 +1,6 @@
#!/bin/bash
# ViPE SLAM standalone environment (invoked by demo.py via conda run -n vipe when --fix_camera is unset)
# Legacy optional branch. The RGB-D pipeline uses docs/SETUP_AND_WEIGHTS.md.
# Install only if you need the world-view trajectory (demo.py without --fix_camera); --fix_camera mode skips ViPE.
#
# ViPE's CUDA extensions are compiled against torch 2.7.0+cu128, matching the handflow environment.
+102
View File
@@ -0,0 +1,102 @@
"""Deployment checks use synthetic files; they do not run GPU inference."""
import importlib.util
import json
import os
from pathlib import Path
import shutil
import subprocess
import sys
import tempfile
import unittest
ROOT = Path(__file__).resolve().parents[1]
spec = importlib.util.spec_from_file_location('pipeline_resources', ROOT / 'scripts/check_pipeline_resources.py')
resources = importlib.util.module_from_spec(spec)
spec.loader.exec_module(resources)
class RecordingChecks(unittest.TestCase):
def setUp(self):
self.temp = tempfile.TemporaryDirectory(prefix='pipeline resources ')
self.addCleanup(self.temp.cleanup)
self.root = Path(self.temp.name)
self.recording = self.root / 'example recording'
self.recording.mkdir()
def recording_data(self, path):
path.mkdir(parents=True, exist_ok=True)
(path / 'color.mp4').write_bytes(b'synthetic-video')
(path / 'depth').mkdir()
(path / 'depth/000000.png').write_bytes(b'synthetic-depth')
(path / 'intrinsics.json').write_text(json.dumps({'frames': 1}))
def test_standard_recording_is_unchanged(self):
self.recording_data(self.recording)
resources.prepare_recording(self.recording)
self.assertEqual(resources.check_recording(self.recording), [])
self.assertFalse((self.recording / 'color.mp4').is_symlink())
def test_nested_recording_uses_relative_links_and_is_idempotent(self):
nested = self.recording / self.recording.name
self.recording_data(nested)
resources.prepare_recording(self.recording)
resources.prepare_recording(self.recording)
self.assertEqual(resources.check_recording(self.recording), [])
self.assertFalse(os.path.isabs(os.readlink(self.recording / 'color.mp4')))
self.assertEqual((nested / 'color.mp4').read_bytes(), b'synthetic-video')
def test_existing_outer_metadata_is_preserved(self):
self.recording_data(self.recording / self.recording.name)
data = '{"frames": 1, "preserve": true}'
(self.recording / 'intrinsics.json').write_text(data)
resources.prepare_recording(self.recording)
self.assertEqual((self.recording / 'intrinsics.json').read_text(), data)
def test_incomplete_nested_input_creates_no_partial_links(self):
nested = self.recording / self.recording.name
nested.mkdir()
(nested / 'color.mp4').write_bytes(b'video')
with self.assertRaises(ValueError):
resources.prepare_recording(self.recording)
self.assertFalse((self.recording / 'color.mp4').is_symlink())
def test_unpulled_lfs_is_rejected(self):
self.recording_data(self.recording)
(self.recording / 'color.mp4').write_bytes(resources.LFS_HEADER + b'\noid sha256:example\n')
self.assertTrue(any('LFS pointer' in error for error in resources.check_recording(self.recording)))
def test_missing_depth_frame_is_rejected(self):
self.recording_data(self.recording)
(self.recording / 'intrinsics.json').write_text('{"frames": 2}')
self.assertTrue(any('000001.png' in error for error in resources.check_recording(self.recording)))
def test_missing_mano_explains_required_action(self):
errors = resources.check_assets(self.root)
self.assertTrue(any('MANO_RIGHT.pkl' in error and 'Provide your MANO' in error for error in errors))
self.assertFalse(any('MANO_LEFT.pkl' in error for error in errors))
self.assertTrue(any('MANO_LEFT.pkl' in error for error in resources.check_assets(self.root, require_left=True)))
class EnvironmentChecks(unittest.TestCase):
def test_cuda_and_python_paths_with_spaces_and_explicit_settings(self):
with tempfile.TemporaryDirectory(prefix='pipeline env ') as directory:
root = Path(directory)
scripts = root / 'project with spaces/scripts'
scripts.mkdir(parents=True)
script = scripts / 'foundationpose_env.sh'
shutil.copy2(ROOT / 'scripts/foundationpose_env.sh', script)
python = root / 'python environment/bin/python'
python.parent.mkdir(parents=True)
python.symlink_to(sys.executable)
toolkit = root / 'cuda toolkit'
(toolkit / 'bin').mkdir(parents=True)
nvcc = toolkit / 'bin/nvcc'
nvcc.write_text('#!/bin/sh\nexit 0\n')
nvcc.chmod(0o755)
env = dict(os.environ, HANDFLOW_PYTHON=str(python), CUDA_HOME=str(toolkit), TORCH_CUDA_ARCH_LIST='9.0', MAX_JOBS='3', CPATH='/existing/include')
result = subprocess.run(['bash', '-c', 'source "$1" && test "$CUDA_HOME" = "$2" && test "$TORCH_CUDA_ARCH_LIST" = 9.0 && test "$MAX_JOBS" = 3 && [[ "$CPATH" == "$2/include:/existing/include"* ]]', 'check-env', str(script), str(toolkit)], env=env, text=True, capture_output=True)
self.assertEqual(result.returncode, 0, result.stderr)
if __name__ == '__main__':
unittest.main()
+5 -1
View File
@@ -1 +1,5 @@
MANO_RIGHT.pkl / MANO_LEFT.pkl are not redistributed (MANO license). Register at https://mano.is.tue.mpg.de/ and place them here.
MANO model files are not included in Git or Git LFS.
Obtain MANO_RIGHT.pkl (and MANO_LEFT.pkl for branches using a separate left model)
from https://mano.is.tue.mpg.de/ under its terms and place them in this directory.
Dyn-HaMR shares this directory through its relative _DATA/data/mano symlink.
See docs/SETUP_AND_WEIGHTS.md, section 4.1, and scripts/check_pipeline_resources.py.