Initial snapshot: hand motion pipeline (Dyn-HaMR + dex-retargeting + SPIDER)

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-09-15 14:13:49 +08:00
commit 7e4ef6f98b
7002 changed files with 1452249 additions and 0 deletions
@@ -0,0 +1,173 @@
+# Add New Dataset
+
+This guide explains how to add a new dataset to SPIDER for retargeting human motion data to robots.
+
+## Overview
+
+Adding a new dataset involves:
+1. Preparing raw data in standardized format
+2. Creating a dataset processor script
+3. Extracting hand kinematics and object poses
+4. Converting meshes to MuJoCo-compatible format
+5. Testing the integration
+
+## Dataset Requirements
+
+Your dataset should include:
+- **Hand motion data**: Either MANO parameters or joint angles
+- **Object information**: 3D meshes and 6D poses (position + orientation)
+- **Temporal alignment**: Synchronized hand and object trajectories
+
+Supported formats:
+- MANO parameters (shape, pose)
+- Direct joint angles
+- Wrist poses (position + rotation)
+- Object 6D poses
+
+## Data File Structure
+
+SPIDER uses a standardized directory structure:
+
+```
+example_datasets/
+├── raw/                          # Raw data from original dataset
+│   └── my_dataset/
+│       ├── task_name_01.pkl      # Raw motion capture data
+│       ├── task_name_02.pkl
+│       └── meshes/               # Object meshes
+│           ├── cup.obj
+│           └── spoon.obj
+│
+└── processed/                    # Processed data for SPIDER
+    └── my_dataset/
+        ├── dataset_summary.json  # Dataset metadata
+        ├── assets/               # Shared assets
+        │   ├── objects/          # Object meshes
+        │   │   └── cup/
+        │   │       ├── convex/   # Convex decomposition
+        │   │       │   ├── 0.obj
+        │   │       │   ├── 1.obj
+        │   │       │   └── ...
+        │   │       └── visual.obj
+        │   └── robots/           # Robot models
+        │       └── allegro/
+        │           ├── left.xml
+        │           └── right.xml
+        └── mano/                 # Processed MANO data
+            └── bimanual/
+                └── task_name/
+                    └── 0/
+                        ├── trajectory_keypoint.npz
+                        └── info.json
+```
+
+## Step 1: Prepare Raw Data
+
+Place your raw data in the appropriate directory:
+
+```bash
+mkdir -p example_datasets/raw/my_dataset
+# Copy your raw data files here
+```
+
+### Raw Data Format
+
+Your raw data file (`.pkl` or `.npz`) should contain:
+
+```python
+{
+    # Hand data (one of the following):
+    'mano_pose': [...],           # [T, 48] MANO pose parameters
+    'mano_shape': [...],          # [10] or [T, 10] MANO shape parameters
+    # OR
+    'qpos_finger_left': [...],    # [T, n_joints] Left finger joints
+    'qpos_finger_right': [...],   # [T, n_joints] Right finger joints
+    'qpos_wrist_left': [...],     # [T, 7] Left wrist pose (xyz + quat)
+    'qpos_wrist_right': [...],    # [T, 7] Right wrist pose (xyz + quat)
+
+    # Object data:
+    'object_pose_left': [...],    # [T, 7] Object pose (xyz + quat) for left hand
+    'object_pose_right': [...],   # [T, 7] Object pose (xyz + quat) for right hand
+    'object_name_left': 'cup',    # Object identifier
+    'object_name_right': 'spoon',
+
+    # Metadata:
+    'fps': 30.0,                  # Frame rate
+    'task_name': 'pick_cup',      # Task identifier
+}
+```
+
+## Step 2: Create Dataset Processor
+
+Create a processor script at `spider/process_datasets/my_dataset.py`:
+
+## Step 3: Handle Object Meshes
+
+Convert Meshes to OBJ Format
+
+Ensure meshes are in MuJoCo-compatible OBJ format:
+
+```python
+def convert_mesh_to_obj(input_mesh: str, output_mesh: str):
+    """
+    Convert mesh to MuJoCo-compatible OBJ format.
+
+    Supports various input formats (STL, PLY, glb, etc.)
+    """
+    import trimesh
+
+    # Load mesh
+    mesh = trimesh.load(input_mesh)
+
+    # Ensure single mesh (merge if needed)
+    if isinstance(mesh, trimesh.Scene):
+        mesh = mesh.dump(concatenate=True)
+
+    # Export as OBJ
+    mesh.export(output_mesh)
+
+    print(f"Converted {input_mesh} to {output_mesh}")
+```
+
+Place Meshes in Assets
+
+Place meshes in assets folder, for example, for cup:
+
+```bash
+mkdir -p example_datasets/processed/my_dataset/assets/objects/cup
+cp example_datasets/raw/my_dataset/meshes/cup.obj example_datasets/processed/my_dataset/assets/objects/cup/visual.obj
+```
+
+## Step 4: Test Your Dataset Processor
+
+Run Processor
+
+```bash
+# Process a single sample
+uv run spider/process_datasets/my_dataset.py \
+  --task=pick_cup \
+  --embodiment-type=bimanual \
+  --data-id=0
+```
+
+### Verify Output
+
+Check that the output files exist and have correct format.
+
+```python
+import numpy as np
+
+# Load processed data
+data = np.load('example_datasets/processed/my_dataset/mano/bimanual/pick_cup/0/trajectory_keypoint.npz')
+
+print("Keys:", list(data.keys()))
+print("qpos_finger shape:", data['qpos_finger'].shape)
+print("qpos_wrist shape:", data['qpos_wrist'].shape)
+```
+
+## Resources
+
+- [OakInk Dataset](https://oakink.net/)
+- [GigaHand Dataset](https://gigahands.github.io/)
+- [Hot3D Dataset](https://github.com/facebookresearch/hot3d)
+- [MANO Hand Model](https://mano.is.tue.mpg.de/)