hand-motion-pipeline/third_party/Dyn-HaMR/third-party/VIPE_INTEGRATION.md

VIPE Integration for Dyn-HaMR

This document explains how to use VIPE camera estimation instead of DROID-SLAM in the Dyn-HaMR pipeline.

Quick Start (TL;DR)

Step 1: Run VIPE on your video

conda activate vipe
cd /data/home/zy3023/code/hand/Dyn-HaMR/third-party/vipe
vipe infer /data/home/zy3023/code/hand/Dyn-HaMR/test/videos/prod1.mp4

Step 2: Run Dyn-HaMR with VIPE cameras

conda activate slahmr
cd /data/home/zy3023/code/hand/Dyn-HaMR/dyn-hamr
python run_opt.py data=video_vipe data.seq=prod1

Important Notes:

• If VIPE results don't exist, the pipeline automatically falls back to DROID-SLAM
• If you previously ran with DROID-SLAM, use overwrite=True to regenerate cameras:

  python run_opt.py data=video_vipe data.seq=prod1 overwrite=True
  

To verify the integration works:

cd /data/home/zy3023/code/hand/Dyn-HaMR
python3 test_vipe_integration.py

Overview

VIPE (Visual Inertial Pose Estimation) is an alternative camera estimation method that can be used instead of DROID-SLAM. The integration allows seamless switching between the two methods via configuration.

VIPE Output Format

VIPE produces the following outputs:

• pose/{sequence}.npz: Camera-to-world transformation matrices
  • data: (N, 4, 4) c2w matrices
  • inds: (N,) frame indices
• intrinsics/{sequence}.npz: Camera intrinsics
  • data: (N, 4) [fx, fy, cx, cy]
  • inds: (N,) frame indices

How to Use VIPE

Option 1: Using the VIPE Config File

The easiest way is to use the pre-configured VIPE config:

cd dyn-hamr
python run_opt.py data=video_vipe data.seq=prod1

Option 2: Override Config Parameters

You can override any existing config to use VIPE:

cd dyn-hamr
python run_opt.py data=video \
    data.seq=prod1 \
    data.use_vipe=True \
    data.vipe_dir=/path/to/vipe/results

Option 3: Modify Your Config File

Add these lines to your data config YAML file:

# VIPE-specific options
use_vipe: True
vipe_dir: /path/to/vipe/vipe_results

Configuration Parameters

• use_vipe (bool): Set to True to use VIPE instead of DROID-SLAM
• vipe_dir (str): Path to VIPE results directory containing pose/ and intrinsics/ subdirectories

How It Works

When use_vipe=True:

1. The pipeline loads VIPE camera poses and intrinsics from .npz files
2. Converts camera-to-world (c2w) matrices to world-to-camera (w2c) format
3. Saves cameras in DROID-SLAM compatible format (cameras.npz)
4. The rest of the Dyn-HaMR pipeline proceeds normally

The conversion happens automatically in dyn-hamr/data/vidproc.py:

• load_vipe_cameras(): Loads and converts VIPE outputs
• save_vipe_cameras_as_droid(): Saves in DROID-SLAM format
• preprocess_cameras(): Routes to VIPE or DROID-SLAM based on config

Directory Structure

your_project/
├── third-party/
│   └── vipe/
│       └── vipe_results/
│           ├── pose/
│           │   └── prod1.npz
│           ├── intrinsics/
│           │   └── prod1.npz
│           └── rgb/
│               └── prod1.mp4
├── test/
│   ├── images/
│   │   └── prod1/
│   │       └── *.jpg
│   └── dynhamr/
│       └── cameras/
│           └── prod1/
│               └── shot-0/
│                   └── cameras.npz  # Generated from VIPE

Example: Complete Workflow

1. Run VIPE on your video:

   # Activate VIPE environment and run camera estimation
   conda activate vipe
   cd /data/home/zy3023/code/hand/Dyn-HaMR/third-party/vipe
   vipe infer /data/home/zy3023/code/hand/Dyn-HaMR/test/videos/prod1.mp4
   
   # This generates:
   # - vipe_results/pose/prod1.npz
   # - vipe_results/intrinsics/prod1.npz
   # - vipe_results/rgb/prod1.mp4
   

2. Verify VIPE results:

   ls third-party/vipe/vipe_results/pose/prod1.npz
   ls third-party/vipe/vipe_results/intrinsics/prod1.npz
   

3. Run Dyn-HaMR with VIPE:

   conda activate slahmr  # or your Dyn-HaMR environment
   cd dyn-hamr
   python run_opt.py data=video_vipe data.seq=prod1
   

4. The pipeline will:

   • Extract frames from video
   • Run HaMeR for hand tracking
   • Load VIPE cameras (instead of running DROID-SLAM)
   • Optimize hand poses
   • Generate visualizations

Running VIPE on New Videos

To use VIPE with a new video sequence:

# 1. Run VIPE
conda activate vipe
cd /data/home/zy3023/code/hand/Dyn-HaMR/third-party/vipe
vipe infer /data/home/zy3023/code/hand/Dyn-HaMR/test/videos/YOUR_VIDEO.mp4

# 2. Run Dyn-HaMR (it will automatically use VIPE if results exist)
conda activate slahmr
cd /data/home/zy3023/code/hand/Dyn-HaMR/dyn-hamr
python run_opt.py data=video_vipe data.seq=YOUR_VIDEO

Note: If VIPE results don't exist for a sequence, the pipeline will automatically fall back to DROID-SLAM with a warning message

Verification

Run the test suite to verify the integration:

python3 test_vipe_integration.py

Expected output:

✓ All tests passed! VIPE integration is working correctly.

Troubleshooting

Error: "VIPE pose file not found"

• Ensure vipe_dir points to the correct directory
• Check that {sequence}.npz exists in pose/ and intrinsics/ subdirectories
• Verify the sequence name matches between config and VIPE outputs

Error: "No images found"

• The pipeline will try to infer image size from intrinsics if images aren't available yet
• For best results, ensure frames are extracted before camera processing

Cameras look wrong / VIPE not being used

• Most common issue: Old DROID-SLAM cameras still in the directory
• Solution: Delete the cameras directory or run with overwrite=True
• Verify VIPE cameras are loaded by checking the first camera matrix
• VIPE uses camera-to-world (c2w) format, which is automatically converted to world-to-camera (w2c)
• Check that VIPE's coordinate system matches your expectations
• Verify intrinsics are correct (fx, fy, cx, cy)

Switching Back to DROID-SLAM

To use DROID-SLAM instead of VIPE:

cd dyn-hamr
python run_opt.py data=video data.seq=prod1 data.use_vipe=False

Or simply use a config without use_vipe set (defaults to DROID-SLAM).

Technical Details

Coordinate System Conversion

• VIPE output: Camera-to-world (c2w) matrices
• Dyn-HaMR expects: World-to-camera (w2c) matrices
• Conversion: w2c = inv(c2w)

Intrinsics Format

• VIPE: (N, 4) [fx, fy, cx, cy]
• Dyn-HaMR: (N, 6) [fx, fy, cx, cy, W, H]
• Conversion: Image dimensions are added from actual images or inferred from intrinsics

Saved Format

The cameras.npz file contains:

• w2c: (N, 4, 4) world-to-camera matrices
• intrins: (N, 4) [fx, fy, cx, cy]
• focal: float, average of fx and fy
• width: int, image width
• height: int, image height

This format is identical to DROID-SLAM output, ensuring compatibility with the rest of the pipeline.