liyang
ae28d55f81
Source: RGB-D -> Dyn-HaMR -> L20 retargeting -> FoundationPose -> reference repair -> SPIDER, documented in docs/PIPELINE_LATEST.md and docs/SETUP_AND_WEIGHTS.md. Adds FoundationPose and nvdiffrast upstream snapshots, requirements/pipeline_venv.txt and the FoundationPose weight manifest/downloader. Assets (Git LFS): weights/ (WiLoR detector, HandFlow denoiser, UniDepth-L), FoundationPose checkpoints, HaMeR checkpoint, Dyn-HaMR HMP model and BMC constraints, L20 URDF/meshes, the 20260915_171525 D405 recording and the two box CADs. MANO models are not redistributed (third_party/hamer/_DATA/data/mano/README.txt). Environments, caches and run outputs excluded. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2.0 KiB
FoundationPose local setup
Validated 2026-09-16 for model-based RGB-D inference using an existing STL.
- Source:
third_party/FoundationPose, upstream commita1b694b83e633c2cb6115b9063d940a687759392. - Interpreter: existing
.venv/bin/python; reused Torch 2.7.1+cu128 and PyTorch3D. - CUDA toolkit: existing
.cuda/usr/local/cuda-12.8; GPU architecture 8.6. - Installed upstream Python requirements, pybind11, ninja, and locally built nvdiffrast 0.4.0.
- Compiled
third_party/FoundationPose/mycpp/build/mycpp.cpython-311-x86_64-linux-gnu.so. - Weights and mirror provenance:
third_party/FoundationPose/weights/download_manifest.json.
Run the verified smoke check
From the HandFlow root, in a shell with NVIDIA GPU access:
source scripts/foundationpose_env.sh
python scripts/verify_foundationpose_setup.py
The environment script clears ROS-related Python/library overrides and selects the existing project environment/toolkit. It does not install Torch or CUDA.
The check exercises actual CUDA rasterization, loads both pretrained networks and executes their forward passes, and initializes FoundationPose with docs/上半.stl, including compiled C++ rotation clustering. Results: output/foundationpose_setup/runtime_verification.json; installation/build logs are in the same directory.
This does not validate pose accuracy or contact constraints on the recorded video. Optional BundleSDF mycuda/NeRF extensions are not required for the existing-STL path and were not built.
Rebuild compiled modules
source scripts/foundationpose_env.sh
python -m pip install --no-build-isolation --no-deps ./third_party/nvdiffrast
cmake -S third_party/FoundationPose/mycpp -B third_party/FoundationPose/mycpp/build \
-DCMAKE_BUILD_TYPE=Release \
-Dpybind11_DIR="$FP_ROOT/.venv/lib/python3.11/site-packages/pybind11/share/cmake/pybind11" \
-DPYTHON_EXECUTABLE="$FP_ROOT/.venv/bin/python" \
-DPython_EXECUTABLE="$FP_ROOT/.venv/bin/python"
cmake --build third_party/FoundationPose/mycpp/build -j2