liyang/hand-motion-pipeline
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Initial snapshot: hand motion pipeline (Dyn-HaMR + dex-retargeting + SPIDER)

Browse Source 
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
liyang
2026-09-15 14:13:49 +08:00
commit 7e4ef6f98b
7002 changed files with 1452249 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
third_party/spider/docs/development/add-dataset.md
Vendored
+173
View File
@@ -0,0 +1,173 @@
# Add New Dataset
This guide explains how to add a new dataset to SPIDER for retargeting human motion data to robots.
## Overview
Adding a new dataset involves:
1. Preparing raw data in standardized format
2. Creating a dataset processor script
3. Extracting hand kinematics and object poses
4. Converting meshes to MuJoCo-compatible format
5. Testing the integration
## Dataset Requirements
Your dataset should include:
- **Hand motion data**: Either MANO parameters or joint angles
- **Object information**: 3D meshes and 6D poses (position + orientation)
- **Temporal alignment**: Synchronized hand and object trajectories
Supported formats:
- MANO parameters (shape, pose)
- Direct joint angles
- Wrist poses (position + rotation)
- Object 6D poses
## Data File Structure
SPIDER uses a standardized directory structure:
```
example_datasets/
├── raw/ # Raw data from original dataset
│ └── my_dataset/
│ ├── task_name_01.pkl # Raw motion capture data
│ ├── task_name_02.pkl
│ └── meshes/ # Object meshes
│ ├── cup.obj
│ └── spoon.obj
│
└── processed/ # Processed data for SPIDER
└── my_dataset/
├── dataset_summary.json # Dataset metadata
├── assets/ # Shared assets
│ ├── objects/ # Object meshes
│ │ └── cup/
│ │ ├── convex/ # Convex decomposition
│ │ │ ├── 0.obj
│ │ │ ├── 1.obj
│ │ │ └── ...
│ │ └── visual.obj
│ └── robots/ # Robot models
│ └── allegro/
│ ├── left.xml
│ └── right.xml
└── mano/ # Processed MANO data
└── bimanual/
└── task_name/
└── 0/
├── trajectory_keypoint.npz
└── info.json
```
## Step 1: Prepare Raw Data
Place your raw data in the appropriate directory:
```bash
mkdir -p example_datasets/raw/my_dataset
# Copy your raw data files here
```
### Raw Data Format
Your raw data file (`.pkl` or `.npz`) should contain:
```python
{
# Hand data (one of the following):
'mano_pose': [...], # [T, 48] MANO pose parameters
'mano_shape': [...], # [10] or [T, 10] MANO shape parameters
# OR
'qpos_finger_left': [...], # [T, n_joints] Left finger joints
'qpos_finger_right': [...], # [T, n_joints] Right finger joints
'qpos_wrist_left': [...], # [T, 7] Left wrist pose (xyz + quat)
'qpos_wrist_right': [...], # [T, 7] Right wrist pose (xyz + quat)
# Object data:
'object_pose_left': [...], # [T, 7] Object pose (xyz + quat) for left hand
'object_pose_right': [...], # [T, 7] Object pose (xyz + quat) for right hand
'object_name_left': 'cup', # Object identifier
'object_name_right': 'spoon',
# Metadata:
'fps': 30.0, # Frame rate
'task_name': 'pick_cup', # Task identifier
}
```
## Step 2: Create Dataset Processor
Create a processor script at `spider/process_datasets/my_dataset.py`:
## Step 3: Handle Object Meshes
Convert Meshes to OBJ Format
Ensure meshes are in MuJoCo-compatible OBJ format:
```python
def convert_mesh_to_obj(input_mesh: str, output_mesh: str):
"""
Convert mesh to MuJoCo-compatible OBJ format.
Supports various input formats (STL, PLY, glb, etc.)
"""
import trimesh
# Load mesh
mesh = trimesh.load(input_mesh)
# Ensure single mesh (merge if needed)
if isinstance(mesh, trimesh.Scene):
mesh = mesh.dump(concatenate=True)
# Export as OBJ
mesh.export(output_mesh)
print(f"Converted {input_mesh} to {output_mesh}")
```
Place Meshes in Assets
Place meshes in assets folder, for example, for cup:
```bash
mkdir -p example_datasets/processed/my_dataset/assets/objects/cup
cp example_datasets/raw/my_dataset/meshes/cup.obj example_datasets/processed/my_dataset/assets/objects/cup/visual.obj
```
## Step 4: Test Your Dataset Processor
Run Processor
```bash
# Process a single sample
uv run spider/process_datasets/my_dataset.py \
--task=pick_cup \
--embodiment-type=bimanual \
--data-id=0
```
### Verify Output
Check that the output files exist and have correct format.
```python
import numpy as np
# Load processed data
data = np.load('example_datasets/processed/my_dataset/mano/bimanual/pick_cup/0/trajectory_keypoint.npz')
print("Keys:", list(data.keys()))
print("qpos_finger shape:", data['qpos_finger'].shape)
print("qpos_wrist shape:", data['qpos_wrist'].shape)
```
## Resources
- [OakInk Dataset](https://oakink.net/)
- [GigaHand Dataset](https://gigahands.github.io/)
- [Hot3D Dataset](https://github.com/facebookresearch/hot3d)
- [MANO Hand Model](https://mano.is.tue.mpg.de/)
third_party/spider/docs/development/add-robot.md
Vendored
+678
View File
@@ -0,0 +1,678 @@
# Add New Robot
This guide explains how to add a new robot to SPIDER, covering both dexterous hands and humanoid robots.
## Overview
Adding a new robot involves:
1. Converting/preparing robot model (URDF/MJCF)
2. Configuring default settings
3. Setting up collision geometry
4. Adding trace and tracking sites
5. Configuring actuators
6. Testing the integration
## Prerequisites
- Robot URDF or MJCF model
- Basic understanding of MuJoCo XML format
- Robot specifications (joint limits, actuator parameters)
## Add Dexterous Hand
### Step 1: Convert URDF to MJCF (Optional)
If you have a URDF file, convert it to MuJoCo XML:
```xml
<!-- Add this snippet to your URDF before conversion -->
<mujoco>
<compiler meshdir="assets" balanceinertia="true" discardvisual="false"/>
</mujoco>
```
Then use MuJoCo's compiler:
```bash
# Using MuJoCo command-line tool
python -c "import mujoco; mujoco.MjModel.from_xml_path('robot.urdf').save('robot.xml')"
```
### Step 2: Add Default Settings
Add default simulation parameters to your XML:
```xml
<default>
<!-- Disable all contacts by default, use explicit collision pairs -->
<geom density="800" condim="1" contype="0" conaffinity="0" />
<!-- Position controller settings (response time ~1s) -->
<position kp="300" dampratio="1.0" inheritrange="1" />
<!-- Joint dynamics -->
<joint damping="0.0" armature="1.0" frictionloss="0.0" />
<!-- Trace sites for visualization -->
<site size="0.01" type="sphere" rgba="1 0 0 1" group="3" />
</default>
```
**Parameter explanations:**
- `density="800"`: Material density (kg/m³)
- `kp="300"`: Position controller stiffness
- `dampratio="1.0"`: Critical damping
- `armature="1.0"`: Joint inertia
### Step 3: Add Assets
Include textures and materials:
```xml
<asset>
<!-- Skybox texture -->
<texture type="skybox" builtin="gradient"
rgb1="1 1 1" rgb2="0 0 0"
width="512" height="3072" />
<!-- Ground plane texture -->
<texture type="2d" name="left_groundplane" builtin="checker"
mark="edge" rgb1="1 1 1" rgb2="1 1 1"
markrgb="0.8 0.8 0.8" width="300" height="300" />
<!-- Ground material -->
<material name="left_groundplane" texture="left_groundplane"
texuniform="true" texrepeat="5 5" reflectance="0.2" />
</asset>
<!-- Camera for tracking -->
<camera name="left_track" pos="0.868 -0.348 -0.175"
xyaxes="0.037 0.999 -0.000 0.011 -0.000 1.000" />
```
### Step 4: Clean Up Joint Properties
Remove incompatible joint properties from the URDF-converted file:
```xml
<!-- Remove these attributes from joints: -->
<!-- actuatorfrcrange, damping, frictionloss, armature, etc. -->
<!-- Before: -->
<joint name="finger_joint" damping="0.1" armature="0.01" ... />
<!-- After: -->
<joint name="finger_joint" ... />
<!-- Dynamics are now inherited from <default> section -->
```
### Step 5: Rename Bodies and Geoms
Change mesh names to avoid conflicts (especially for bimanual setups):
```xml
<!-- Left hand bodies should start with "left_" -->
<body name="left_palm">
<geom name="left_palm_visual" type="mesh" mesh="left_palm_mesh" />
</body>
<!-- Right hand bodies should start with "right_" -->
<body name="right_palm">
<geom name="right_palm_visual" type="mesh" mesh="right_palm_mesh" />
</body>
```
::: tip Naming Convention
- Bodies: `{side}_{finger}_{part}` (e.g., `left_index_proximal`)
- Visual geoms: `{side}_{name}_visual`
- Collision geoms: `collision_hand_{side}_{finger}_{part}`
:::
### Step 6: Add Collision Geometry
Create simplified collision geometry using primitives:
For each finger, in different parts, the collision index start from 0 and increase in each new body.
```xml
<body name="left_index_tip">
<!-- Visual mesh -->
<geom name="left_index_tip_visual" type="mesh" mesh="index_tip"
contype="0" conaffinity="0" group="0" />
<!-- Collision geometry (capsule primitive) -->
<geom name="collision_hand_left_index_0" type="capsule"
size="0.008 0.015" pos="0.015 0 0" quat="0.707 0 0.707 0"
contype="1" conaffinity="1" group="3" rgba="0 1 0 1" />
</body>
```
**Naming convention for collision geoms:**
```
collision_hand_{side}_{finger}_{part}
```
Where:
- `side`: `left` or `right`
- `finger`: `thumb`, `index`, `middle`, `ring`, `pinky`, `palm`
- `part`: `0`, `1`, `2`, `3`, ... (from tip to base)
**Collision geom guidelines:**
- Use primitives (`box`, `sphere`, `cylinder`, `capsule`) instead of meshes
- Set `group="3"` for collision geoms
- Set `rgba="0 1 0 1"` (green) for visibility
- Keep collision geoms simple for performance
### Step 7: Add Trace Sites
Add trace sites for fingertip visualization:
```xml
<body name="left_thumb_tip">
<!-- Trace site (for trajectory recording) -->
<site name="trace_hand_left_thumb_tip" pos="0.02 0 0"
size="0.005" type="sphere" rgba="1 0 0 1" group="3" />
</body>
<body name="left_index_tip">
<site name="trace_hand_left_index_tip" pos="0.02 0 0"
size="0.005" type="sphere" rgba="1 0 0 1" group="3" />
</body>
<!-- Repeat for middle, ring, pinky -->
```
**Naming convention:** `trace_hand_{side}_{finger}_tip`
### Step 8: Add IK Tracking Sites
Add tracking sites for end-effector control:
```xml
<body name="left_palm">
<site name="left_palm" pos="0 0 0"
euler="0 0 0" size="0.01" type="sphere" rgba="1 1 0 1" group="3" />
</body>
<body name="left_thumb_tip">
<site name="left_thumb_tip" pos="0.02 0 0"
size="0.01" type="sphere" rgba="0 1 1 1" group="3" />
</body>
<!-- Repeat for all fingertips -->
```
**Required sites:**
- `{side}_palm`: Palm center with proper orientation
- `{side}_thumb_tip`, `{side}_index_tip`, `{side}_middle_tip`
- `{side}_ring_tip`, `{side}_pinky_tip`
**Palm site orientation:**
- Z-axis: pointing forward (finger direction)
- X-axis: pointing down (toward ground)
![Base Definition](/figs/base_def.png)
Optional: Add contact tracking sites for contact tracking.
```xml
<body name="left_thumb_tip">
<site name="track_hand_left_thumb_tip" pos="0.02 0 0"
size="0.01" type="sphere" rgba="0 1 1 1" group="3" />
</body>
<!-- Repeat for all fingertips -->
```
### Step 9: Add Actuators
Define position actuators for all joints:
```xml
<actuator>
<!-- Base DOFs (for free-floating hand) -->
<position name="left_pos_x_position" joint="left_pos_x" kp="1000" />
<position name="left_pos_y_position" joint="left_pos_y" kp="1000" />
<position name="left_pos_z_position" joint="left_pos_z" kp="1000" />
<position name="left_rot_x_position" joint="left_rot_x" kp="1000" />
<position name="left_rot_y_position" joint="left_rot_y" kp="1000" />
<position name="left_rot_z_position" joint="left_rot_z" kp="1000" />
<!-- Thumb joints -->
<position name="left_thumb_proximal_yaw_position"
joint="left_thumb_proximal_yaw_joint" />
<position name="left_thumb_proximal_pitch_position"
joint="left_thumb_proximal_pitch_joint" />
<position name="left_thumb_intermediate_position"
joint="left_thumb_intermediate_joint" />
<position name="left_thumb_distal_position"
joint="left_thumb_distal_joint" />
<!-- Index finger joints -->
<position name="left_index_proximal_position"
joint="left_index_proximal_joint" />
<position name="left_index_intermediate_position"
joint="left_index_intermediate_joint" />
<!-- Repeat for middle, ring, pinky -->
</actuator>
```
**Important notes:**
- Base DOFs use higher `kp` (e.g., 1000) for stability
- Finger joints inherit `kp` from `<default>` section
- **Actuator order must match joint order** (SPIDER uses joint positions as control)
### Step 10: Example Complete Hand XML
Here's a minimal example structure:
```xml
<mujoco model="my_hand">
<compiler angle="radian" meshdir="assets" autolimits="true"/>
<option timestep="0.01" iterations="6" solver="CG" jacobian="auto">
<flag eulerdamp="disable" />
</option>
<default>
<geom density="800" condim="1" contype="0" conaffinity="0" />
<position kp="300" dampratio="1.0" inheritrange="1" />
<joint damping="0.0" armature="1.0" frictionloss="0.0" />
<site size="0.01" type="sphere" rgba="1 0 0 1" group="3" />
</default>
<asset>
<!-- Textures and materials -->
<texture type="skybox" builtin="gradient" rgb1="0.3 0.5 0.7" rgb2="0 0 0" width="512" height="3072" />
<material name="metal" rgba="0.7 0.7 0.7 1" />
<!-- Meshes -->
<mesh name="left_palm_mesh" file="meshes/palm.obj" />
<mesh name="left_thumb_tip_mesh" file="meshes/thumb_tip.obj" />
<!-- ... more meshes -->
</asset>
<worldbody>
<geom name="floor" type="plane" size="0 0 0.05" material="groundplane"/>
<light pos="0 0 3" dir="0 0 -1" directional="true"/>
<!-- Hand body -->
<body name="left_hand" pos="0 0 0.2">
<!-- Base free joint for 6-DOF -->
<freejoint name="left_base_joint"/>
<!-- Palm -->
<body name="left_palm">
<geom name="left_palm_visual" type="mesh" mesh="left_palm_mesh" />
<geom name="collision_hand_left_palm_0" type="box" size="0.04 0.03 0.01" />
<site name="left_palm" pos="0 0 0" euler="0 0 0"/>
<!-- Thumb -->
<body name="left_thumb_proximal">
<joint name="left_thumb_proximal_yaw_joint" axis="0 0 1" range="-0.5 0.5"/>
<joint name="left_thumb_proximal_pitch_joint" axis="0 1 0" range="-0.2 1.0"/>
<!-- ... thumb links ... -->
<body name="left_thumb_tip">
<site name="left_thumb_tip" pos="0.02 0 0"/>
<site name="trace_hand_left_thumb_tip" pos="0.02 0 0"/>
</body>
</body>
<!-- Index, middle, ring, pinky similar structure -->
</body>
</body>
</worldbody>
<actuator>
<position name="left_pos_x_position" joint="left_pos_x" kp="1000"/>
<!-- ... all actuators ... -->
</actuator>
<contact>
<!-- Define collision pairs if needed -->
</contact>
</mujoco>
```
## Add Humanoid Robot
Humanoid robots have similar requirements but with additional considerations.
### Step 1: Performance Optimization
Reduce solver iterations for faster simulation:
```xml
<option timestep="0.02" iterations="2" ls_iterations="10">
<flag eulerdamp="disable" />
</option>
<visual>
<global azimuth="135" elevation="-25" offwidth="1920" offheight="1080" />
<quality shadowsize="8192" />
<headlight ambient="0.3 0.3 0.3" diffuse="0.6 0.6 0.6" specular="0 0 0" />
<scale forcewidth="0.25" contactwidth="0.4" contactheight="0.15"
framelength="5" framewidth="0.3" />
<rgba haze="0.15 0.25 0.35 1" force="1 0 0 1" />
</visual>
<custom>
<numeric data="15" name="max_contact_points" />
<numeric data="15" name="max_geom_pairs" />
</custom>
```
### Step 2: Default Settings
```xml
<default>
<!-- Disable all contacts by default -->
<geom contype="0" conaffinity="0" />
<!-- Position controller (stronger for humanoid) -->
<position kp="500" dampratio="1" inheritrange="1" />
<!-- Joint dynamics -->
<joint damping="0.0" armature="1.0" frictionloss="0.0" />
</default>
```
### Step 3: Add Collision Geoms for Feet
Add collision spheres to feet (4 contact points per foot):
```xml
<body name="left_foot">
<!-- Visual mesh -->
<geom name="left_foot_visual" type="mesh" mesh="left_foot_mesh"
contype="0" conaffinity="0"/>
<!-- Collision spheres -->
<geom name="lf0" type="sphere" size="0.02" pos="-0.05 0.025 -0.03"
priority="1" friction="2.0" condim="3" />
<geom name="lf1" type="sphere" size="0.02" pos="-0.05 -0.025 -0.03"
priority="1" friction="2.0" condim="3" />
<geom name="lf2" type="sphere" size="0.02" pos="0.12 0.03 -0.03"
priority="1" friction="2.0" condim="3" />
<geom name="lf3" type="sphere" size="0.02" pos="0.12 -0.03 -0.03"
priority="1" friction="2.0" condim="3" />
</body>
<!-- Similar for right foot: rf0, rf1, rf2, rf3 -->
```
### Step 4: Add Collision Geoms for Hands
```xml
<body name="left_hand">
<!-- Visual mesh -->
<geom name="left_hand_visual" type="mesh" mesh="left_hand_mesh"
contype="0" conaffinity="0"/>
<!-- Collision sphere -->
<geom name="lh" type="sphere" size="0.05" pos="0.1 0.0 0.0" />
</body>
<!-- Similar for right hand: rh -->
```
### Step 5: Add Trace Sites
```xml
<body name="pelvis">
<site name="trace_pelvis" size="0.01" pos="0 0 0" />
</body>
<body name="left_foot">
<site name="trace_left_foot" size="0.01" pos="0 0 0" />
</body>
<body name="right_foot">
<site name="trace_right_foot" size="0.01" pos="0 0 0" />
</body>
<body name="left_hand">
<site name="trace_left_hand" size="0.01" pos="0 0 0" />
</body>
<body name="right_hand">
<site name="trace_right_hand" size="0.01" pos="0 0 0" />
</body>
```
### Step 6: Add Contact Pairs
Define explicit contact pairs:
```xml
<contact>
<!-- Left foot contacts -->
<pair name="left_foot_floor0" geom1="lf0" geom2="floor"
solref="0.008 1" friction="1 1" condim="3" />
<pair name="left_foot_floor1" geom1="lf1" geom2="floor"
solref="0.008 1" friction="1 1" condim="3" />
<pair name="left_foot_floor2" geom1="lf2" geom2="floor"
solref="0.008 1" friction="1 1" condim="3" />
<pair name="left_foot_floor3" geom1="lf3" geom2="floor"
solref="0.008 1" friction="1 1" condim="3" />
<!-- Right foot contacts -->
<pair name="right_foot_floor0" geom1="rf0" geom2="floor"
solref="0.008 1" friction="1 1" condim="3" />
<pair name="right_foot_floor1" geom1="rf1" geom2="floor"
solref="0.008 1" friction="1 1" condim="3" />
<pair name="right_foot_floor2" geom1="rf2" geom2="floor"
solref="0.008 1" friction="1 1" condim="3" />
<pair name="right_foot_floor3" geom1="rf3" geom2="floor"
solref="0.008 1" friction="1 1" condim="3" />
<!-- Hand contacts -->
<pair name="left_hand_floor" geom1="lh" geom2="floor"
solref="0.008 1" friction="1 1" condim="3" />
<pair name="right_hand_floor" geom1="rh" geom2="floor"
solref="0.008 1" friction="1 1" condim="3" />
</contact>
```
### Step 7: Use Position Actuators
```xml
<actuator>
<!-- Legs -->
<position name="left_hip_pitch" joint="left_hip_pitch_joint" />
<position name="left_hip_roll" joint="left_hip_roll_joint" />
<position name="left_hip_yaw" joint="left_hip_yaw_joint" />
<position name="left_knee" joint="left_knee_joint" />
<position name="left_ankle_pitch" joint="left_ankle_pitch_joint" />
<position name="left_ankle_roll" joint="left_ankle_roll_joint" />
<!-- Arms -->
<position name="left_shoulder_pitch" joint="left_shoulder_pitch_joint" />
<position name="left_shoulder_roll" joint="left_shoulder_roll_joint" />
<position name="left_elbow" joint="left_elbow_joint" />
<!-- ... -->
<!-- Repeat for right side -->
</actuator>
```
::: warning Important
Actuator order must match joint order in the model definition!
:::
### Step 8: Setup Scene
Add white background and tracking camera:
```xml
<statistic center="1.0 0.7 1.0" extent="0.8"/>
<asset>
<!-- White skybox -->
<texture type="skybox" builtin="flat" rgb1="1 1 1" rgb2="1 1 1"
width="800" height="800"/>
<!-- White groundplane -->
<texture type="2d" name="groundplane" builtin="checker" mark="edge"
rgb1="1 1 1" rgb2="1 1 1" markrgb="0 0 0"
width="300" height="300"/>
<material name="groundplane" texture="groundplane" texuniform="true"
texrepeat="5 5" reflectance="0"/>
</asset>
<worldbody>
<!-- Floor -->
<geom name="floor" size="0 0 0.01" type="plane" material="groundplane"
contype="1" conaffinity="0" priority="1" friction="0.6" condim="3"/>
<!-- Directional light -->
<light pos="0 0 5" dir="0 0 -1" type="directional"/>
<!-- Tracking camera -->
<camera name="track" pos="1.734 -1.135 .35"
xyaxes="0.552 0.834 -0.000 -0.170 0.112 0.979"
mode="trackcom"/>
</worldbody>
```
## Testing Your Robot
### Step 1: Visualize in MuJoCo
```python
import mujoco
import mujoco.viewer
# Load model
model = mujoco.MjModel.from_xml_path('my_robot.xml')
data = mujoco.MjData(model)
# Launch viewer
mujoco.viewer.launch(model, data)
```
Check:
- [ ] Model loads without errors
- [ ] All joints move freely
- [ ] Collision geometry looks correct (green geoms)
- [ ] Trace sites are visible
- [ ] Tracking sites are positioned correctly
### Step 2: Test with SPIDER
Create a simple test scene:
```bash
# Test IK (for hands)
uv run spider/preprocess/ik.py \
--robot-type=my_hand \
--task=test_task \
--dataset-name=test \
--data-id=0 \
--embodiment-type=right
# Test physics optimization
uv run examples/run_mjwp.py \
robot_type=my_hand \
task=test_task \
max_sim_steps=100
```
### Step 3: Verify Output
Check that:
- [ ] Trajectory is saved successfully
- [ ] Joint positions are within limits
- [ ] No collision warnings
- [ ] Video renders correctly
## Common Issues
### Issue 1: Model Doesn't Load
**Error:** `XML parse error`
**Solutions:**
- Check XML syntax
- Validate with MuJoCo's `simulate` tool
- Remove incompatible URDF attributes
### Issue 2: Unstable Simulation
**Symptoms:** Robot falls through floor, explodes
**Solutions:**
```xml
<!-- Increase solver iterations -->
<option iterations="10" ls_iterations="20"/>
<!-- Add more collision points -->
<!-- Adjust contact parameters -->
<pair solref="0.01 1" friction="2 2"/>
<!-- Check mass distribution -->
<!-- Ensure reasonable inertia values -->
```
### Issue 3: Actuators Don't Work
**Symptoms:** Robot doesn't move
**Solutions:**
- Check actuator names match joint names
- Verify `kp` values are reasonable (100-1000)
- Ensure joints have proper ranges
- Check actuator order matches joint order
### Issue 4: Poor IK Results
**Symptoms:** IK fails or produces bad poses
**Solutions:**
- Check tracking site orientations (especially palm)
- Verify joint limits are correct
- Add more trace sites for debugging
- Tune IK weights in config
## Robot Configuration Files
### Directory Structure
```
spider/assets/robots/
├── my_hand/
│ ├── left.xml # Left hand only
│ ├── right.xml # Right hand only
│ ├── bimanual.xml # Both hands
│ ├── meshes/
│ │ ├── palm.obj
│ │ ├── thumb_tip.obj
│ │ └── ...
│ └── textures/
│ └── hand_texture.png
```
### Register Robot
Add robot to SPIDER's config:
```python
# In spider/config.py or appropriate config file
ROBOT_TYPES = {
"allegro": "spider/assets/robots/allegro",
"inspire": "spider/assets/robots/inspire",
"my_hand": "spider/assets/robots/my_hand", # Add your robot
# ...
}
```
## Next Steps
After adding your robot:
1. **Test thoroughly** with multiple tasks
2. **Document** robot-specific parameters
3. **Tune** IK and optimization parameters
4. **Share** your robot configuration (PR welcome!)
third_party/spider/docs/development/add-simulator.md
Vendored
+337
View File
@@ -0,0 +1,337 @@
# Add New Simulator / Environment
This guide explains how to integrate a new physics simulator with SPIDER, enabling physics-based retargeting with different simulation backends.
## Overview
SPIDER is designed to work with any simulator out of the box. The modular architecture allows you to port new simulators by implementing a standard interface.
**Current simulators:**
- **MJWP**: MuJoCo + Warp (default)
- **DexMachina**: Genesis simulator
- **HDMI**: MjLab (MuJoCo fork)
**This guide covers:**
- Required interface functions
- Implementation steps
- Configuration setup
- Testing and debugging
## Architecture
```mermaid
graph TB
A[SPIDER Optimizer] --> B[Simulator Interface]
B --> C[MuJoCo Warp]
B --> D[Genesis]
B --> E[MjLab]
B --> F[Your Simulator]
```
The simulator interface provides a standardized API that SPIDER's optimizer uses, regardless of the underlying physics engine.
## Required Interface
### Core Functions (Required)
These functions are essential for SPIDER optimization:
#### 1. `setup_env(config, ref_data)`
Initialize and return the environment instance.
```python
def setup_env(config: Config, ref_data: tuple | None) -> Any:
"""
Setup simulation environment.
Args:
config: SPIDER configuration
ref_data: Reference trajectory data (qpos, qvel, ctrl, contact, contact_pos)
or None if environment provides its own reference
Returns:
Environment instance (simulator-specific type)
"""
# Initialize your simulator
env = YourSimulator(
num_envs=config.num_samples,
device=config.device,
dt=config.sim_dt,
)
# Load scene/model
env.load_scene(config.model_path)
# Set initial state
if ref_data is not None:
qpos_ref, qvel_ref, _, _, _ = ref_data
env.set_state(qpos_ref[0], qvel_ref[0])
return env
```
#### 2. `step_env(config, env, ctrl)`
Step the simulator with control inputs.
```python
def step_env(config: Config, env: Any, ctrl: torch.Tensor) -> None:
"""
Step simulation forward with controls.
Args:
config: SPIDER configuration
env: Environment instance
ctrl: Control inputs [num_samples, nu]
"""
# Apply controls
env.set_action(ctrl)
# Step physics
env.step()
```
#### 3. `save_state(env)`
Save the current state of all parallel environments.
```python
def save_state(env: Any) -> dict:
"""
Save current simulation state.
Args:
env: Environment instance
Returns:
State dictionary (simulator-specific format)
"""
return {
'qpos': env.get_qpos().clone(),
'qvel': env.get_qvel().clone(),
# Add any other state needed for perfect restoration
'time': env.time,
}
```
#### 4. `load_state(env, state)`
Load a saved state back into the environments.
```python
def load_state(env: Any, state: dict) -> None:
"""
Restore simulation state.
Args:
env: Environment instance
state: State dictionary from save_state()
"""
env.set_qpos(state['qpos'])
env.set_qvel(state['qvel'])
env.time = state['time']
```
#### 5. `get_reward(config, env, ref)`
Compute reward for current state.
```python
def get_reward(config: Config, env: Any, ref: tuple) -> torch.Tensor:
"""
Compute reward for current state.
Args:
config: SPIDER configuration
env: Environment instance
ref: Reference data (qpos_ref, qvel_ref, ctrl_ref, contact, contact_pos)
Returns:
Rewards [num_samples]
"""
qpos_ref, qvel_ref, ctrl_ref, contact, contact_pos = ref
qpos = env.get_qpos()
qvel = env.get_qvel()
# Tracking reward
pos_error = torch.norm(qpos - qpos_ref, dim=-1)
vel_error = torch.norm(qvel - qvel_ref, dim=-1)
reward = -pos_error * config.joint_rew_scale
reward += -vel_error * config.vel_rew_scale
return reward
```
#### 6. `get_terminal_reward(config, env, ref_slice)`
Compute terminal reward with scaling.
```python
def get_terminal_reward(
config: Config,
env: Any,
ref_slice: tuple
) -> torch.Tensor:
"""
Compute terminal (final state) reward.
Args:
config: SPIDER configuration
env: Environment instance
ref_slice: Reference slice for horizon
Returns:
Terminal rewards [num_samples]
"""
# Often just scaled version of regular reward
reward = get_reward(config, env, ref_slice[-1])
return reward * config.terminal_rew_scale
```
### Helper Functions (Required for Optimizer)
#### 7. `get_trace(config, env)`
Return visualization trace points.
```python
def get_trace(config: Config, env: Any) -> torch.Tensor:
"""
Get trace points for visualization.
Args:
config: SPIDER configuration
env: Environment instance
Returns:
Trace positions [num_samples, num_traces, 3]
"""
# Get positions of trace sites (hands, feet, objects, etc.)
traces = []
for site_id in config.trace_site_ids:
site_pos = env.get_site_position(site_id)
traces.append(site_pos)
return torch.stack(traces, dim=1) # [num_samples, num_traces, 3]
```
#### 8. `save_env_params(config, env)`
Save environment parameters for domain randomization.
```python
def save_env_params(config: Config, env: Any) -> dict:
"""
Save environment parameters (for domain randomization).
Args:
config: SPIDER configuration
env: Environment instance
Returns:
Parameters dictionary
"""
return {
'object_pos': env.get_object_position().clone(),
'object_mass': env.get_object_mass().clone(),
'friction': env.get_friction().clone(),
# Add any randomized parameters
}
```
#### 9. `load_env_params(config, env, env_param)`
Load environment parameters.
```python
def load_env_params(config: Config, env: Any, env_param: dict) -> None:
"""
Load environment parameters.
Args:
config: SPIDER configuration
env: Environment instance
env_param: Parameters from save_env_params()
"""
env.set_object_position(env_param['object_pos'])
env.set_object_mass(env_param['object_mass'])
env.set_friction(env_param['friction'])
```
#### 10. `sync_env(config, env)`
Synchronize state across parallel environments.
```python
def sync_env(config: Config, env: Any) -> None:
"""
Synchronize state across parallel environments.
Typically broadcasts first environment to all others.
Args:
config: SPIDER configuration
env: Environment instance
"""
# Get state from first environment
qpos = env.get_qpos()
qvel = env.get_qvel()
# Broadcast to all environments
qpos_sync = qpos[0:1].repeat(config.num_samples, 1)
qvel_sync = qvel[0:1].repeat(config.num_samples, 1)
env.set_qpos(qpos_sync)
env.set_qvel(qvel_sync)
```
### Optional Functions
#### `copy_sample_state(state, sample_idx)`
Copy state from specific sample (for best trajectory selection).
```python
def copy_sample_state(state: dict, sample_idx: int) -> dict:
"""
Copy state from specific parallel environment.
Args:
state: Full state dictionary
sample_idx: Index of sample to copy
Returns:
State dictionary for single sample
"""
return {
'qpos': state['qpos'][sample_idx:sample_idx+1].clone(),
'qvel': state['qvel'][sample_idx:sample_idx+1].clone(),
'time': state['time'],
}
```
#### `get_terminate(config, env, ref)`
Early termination check (optional optimization).
```python
def get_terminate(config: Config, env: Any, ref: tuple) -> torch.Tensor:
"""
Check if rollout should terminate early.
Args:
config: SPIDER configuration
env: Environment instance
ref: Reference data
Returns:
Boolean mask [num_samples]
"""
# Example: terminate if object falls
object_height = env.get_object_position()[:, 2]
terminate = object_height < 0.1
return terminate
```
third_party/spider/docs/guide/installation.md
Vendored
+114
View File
@@ -0,0 +1,114 @@
# Installation
This guide walks you through installing SPIDER and its dependencies.
## Prerequisites
- **Python**: 3.12 or higher
- **CUDA**: 11.8 or higher (for GPU acceleration)
- **Git**: For cloning repositories
- **Git LFS**: For large file storage (datasets)
## Option 1: Using uv (Recommended)
[uv](https://github.com/astral-sh/uv) is a fast Python package installer that we recommend for development.
### Install uv
```bash
# Linux/macOS
curl -LsSf https://astral.sh/uv/install.sh | sh
# Windows
powershell -c "irm https://astral.sh/uv/install.ps1 | iex"
```
### Install SPIDER
```bash
# Clone the repository
git clone https://github.com/facebookresearch/spider.git
cd spider
uv sync
```
### Clone Example Datasets
```bash
# Install git-lfs
sudo apt install git-lfs # Ubuntu/Debian
# brew install git-lfs # macOS
git lfs install
git clone https://huggingface.co/datasets/retarget/retarget_example example_datasets
```
## Option 2: Using Conda
If you prefer conda for environment management:
```bash
# Clone the repository
git clone https://github.com/facebookresearch/spider.git
cd spider
# Create conda environment
conda create -n spider python=3.12
conda activate spider
# Install dependencies
python -m pip install --upgrade pip
python -m pip install -r requirements.txt
python -m pip install --no-deps -e .
```
## Verify Installation
Test your installation by running a simple example:
```bash
# With uv
uv run examples/run_mjwp.py
# With conda
python examples/run_mjwp.py
```
If everything is installed correctly, you should see the MuJoCo viewer open with a retargeting simulation.
## Optional: Install Additional Backends
### DexMachina (Genesis Simulator)
For RL training with Genesis simulator:
```bash
# Follow DexMachina installation instructions
# https://mandizhao.github.io/dexmachina-docs/0_install.html
conda activate dexmachina
pip install --ignore-requires-python --no-deps -e .
```
### HDMI (Humanoid RL)
For humanoid robot RL training:
```bash
# Clone and install HDMI
git clone https://github.com/lecar-lab/hdmi.git
cd hdmi
# Install SPIDER in HDMI environment
uv pip install --no-deps -e ../spider
```
### IDE Setup
For VSCode, recommended extensions:
- Python
- Pylance
- Ruff
You can load the `.vscode/settings.json` file to your VSCode to get the recommended settings.
third_party/spider/docs/guide/quick-start.md
Vendored
+281
View File
@@ -0,0 +1,281 @@
# Quick Start
This guide will help you run your first SPIDER retargeting example in minutes.
## Prerequisites
Make sure you have completed the [installation](/guide/installation) and cloned the example datasets.
## Run Preprocessed Example
The quickest way to see SPIDER in action is to run a preprocessed example:
```bash
# Run with default config (Xhand bimanual, OakInk dataset)
uv run examples/run_mjwp.py
```
This will:
1. Load preprocessed kinematic trajectory
2. Optimize trajectory with physics constraints
3. Display real-time visualization in MuJoCo viewer
4. Save optimized trajectory and video
## Full Workflow Example For Dexterous Hand
To process a task from scratch, follow these steps:
### 1. Set Task Parameters
```bash
export TASK=p36-tea
export HAND_TYPE=bimanual
export DATA_ID=10
export ROBOT_TYPE=xhand
export DATASET_NAME=gigahand
```
### 2. Process Dataset
Convert raw human motion data to standardized format:
```bash
uv run spider/process_datasets/${DATASET_NAME}.py \
--task=${TASK} \
--embodiment-type=${HAND_TYPE} \
--data-id=${DATA_ID}
```
### 3. Decompose Object Mesh
Create convex decomposition for physics simulation:
```bash
uv run spider/preprocess/decompose_fast.py \
--task=${TASK} \
--dataset-name=${DATASET_NAME} \
--data-id=${DATA_ID} \
--embodiment-type=${HAND_TYPE}
```
### 4. Detect Contacts (Optional)
Identify hand-object contact points:
```bash
uv run spider/preprocess/detect_contact.py \
--task=${TASK} \
--dataset-name=${DATASET_NAME} \
--data-id=${DATA_ID} \
--embodiment-type=${HAND_TYPE}
```
### 5. Generate Scene
Create MuJoCo scene XML:
```bash
uv run spider/preprocess/generate_xml.py \
--task=${TASK} \
--dataset-name=${DATASET_NAME} \
--data-id=${DATA_ID} \
--embodiment-type=${HAND_TYPE} \
--robot-type=${ROBOT_TYPE}
```
### 6. Run Inverse Kinematics
Convert human poses to robot joint angles:
```bash
uv run spider/preprocess/ik.py \
--task=${TASK} \
--dataset-name=${DATASET_NAME} \
--data-id=${DATA_ID} \
--embodiment-type=${HAND_TYPE} \
--robot-type=${ROBOT_TYPE} \
--open-hand
```
### 7. Physics-Based Retargeting
Optimize trajectory with physics constraints:
```bash
uv run examples/run_mjwp.py \
task=${TASK} \
dataset_name=${DATASET_NAME} \
data_id=${DATA_ID} \
robot_type=${ROBOT_TYPE} \
embodiment_type=${HAND_TYPE}
```
## Full Workflow Example For Humanoid Robot
Set environment variables:
```bash
# humanoid only
export TASK=dance
export HAND_TYPE=humanoid
export DATA_ID=0
export ROBOT_TYPE=unitree_g1
export DATASET_NAME=lafan
# humanoid + object
export TASK=move_largebox
export HAND_TYPE=humanoid_object
export DATA_ID=0
export ROBOT_TYPE=unitree_g1
export DATASET_NAME=omomo
```
### Run IK
```bash
# with GMR (remember to generate GMR data trajectoru_gmr.pkl first with their official code. )
uv run spider/process_datasets/gmr.py \
--task=${TASK} \
--dataset-name=${DATASET_NAME} \
--data-id=${DATA_ID} \
--robot-type=${ROBOT_TYPE} \
--embodiment-type=${HAND_TYPE} \
--contact-detection-mode=one
# with locomujoco
uv run spider/process_datasets/locomujoco.py \
--task=${TASK} \
--dataset-name=${DATASET_NAME} \
--data-id=${DATA_ID} \
--robot-type=${ROBOT_TYPE} \
--embodiment-type=${HAND_TYPE}
# for humanoid + object, we don't have a pipeline yet
# you can directly download data from omniretarget
# https://huggingface.co/datasets/omniretarget/OmniRetarget_Dataset
```
### Run Physics-Based Retargeting
```bash
# for humanoid only
uv run examples/run_mjwp.py \
+override=humanoid \
dataset_name=${DATASET_NAME} \
task=${TASK} \
data_id=${DATA_ID} \
robot_type=${ROBOT_TYPE} \
embodiment_type=${HAND_TYPE}
# for humanoid + object, we don't have a pipeline yet
uv run examples/run_mjwp.py \
+override=humanoid_object \
dataset_name=${DATASET_NAME} \
task=${TASK} \
data_id=${DATA_ID} \
robot_type=${ROBOT_TYPE} \
embodiment_type=${HAND_TYPE}
```
## Understanding the Output
After running, you'll find these files in the output directory:
```
example_datasets/processed/oakink/allegro/bimanual/pick_spoon_bowl/0/
├── trajectory_kinematic.npz # IK result
├── trajectory_mjwp.npz # Optimized trajectory
├── visualization_mjwp.mp4 # Video output
└── metrics.json # Success metrics
```
### Trajectory Data Format
The NPZ files contain:
```python
import numpy as np
data = np.load('trajectory_mjwp.npz')
# Available keys:
# - qpos: [T, nq] joint positions
# - qvel: [T, nv] joint velocities
# - ctrl: [T, nu] control commands
# - time: [T] timestamps
```
## Customize Parameters
Override configuration parameters from command line:
```bash
# Increase optimization samples
uv run examples/run_mjwp.py num_samples=2048
# Change planning horizon
uv run examples/run_mjwp.py horizon=2.0
# Use different viewer
uv run examples/run_mjwp.py viewer=rerun
# Combine multiple overrides
uv run examples/run_mjwp.py \
robot_type=inspire \
num_samples=2048 \
horizon=2.0 \
viewer=mujoco-rerun
```
## Remote Development
For headless servers, use Rerun viewer:
```bash
# On remote server, start Rerun server
uv run rerun --serve-web --port 9876
# Run SPIDER with Rerun viewer
uv run examples/run_mjwp.py viewer=rerun
# On local machine, open browser to:
# http://your-server:9876
```
## Common Tasks
### Run Different Robot
```bash
# Allegro hand
uv run examples/run_mjwp.py robot_type=allegro
# Inspire hand
uv run examples/run_mjwp.py robot_type=inspire
# Xhand
uv run examples/run_mjwp.py robot_type=xhand
```
### Run Different Dataset
```bash
# GigaHand
uv run examples/run_mjwp.py +override=gigahand
# Hot3D
uv run examples/run_mjwp.py +override=hot3d
# Humanoid task
uv run examples/run_mjwp.py +override=humanoid
```
### Single Hand Configuration
```bash
# Right hand only
uv run examples/run_mjwp.py embodiment_type=right
# Left hand only
uv run examples/run_mjwp.py embodiment_type=left
```
third_party/spider/docs/index.md
Vendored
+49
View File
@@ -0,0 +1,49 @@
---
layout: home
hero:
name: "SPIDER"
text: "Scalable Physics-Informed DExterous Retargeting"
tagline: A general framework for physics-based retargeting from human to diverse robot embodiments
image:
src: /figs/teaser.png
alt: SPIDER
actions:
- theme: brand
text: Get Started
link: /guide/quick-start
- theme: alt
text: View on GitHub
link: https://github.com/facebookresearch/spider
features:
- icon: 🔬
title: Physics-Based
details: First general physics-based retargeting pipeline for both dexterous hand and humanoid robot manipulation
- icon: ⚡
title: Fast Simulation
details: GPU-accelerated batched simulation with MuJoCo Warp achieving 10-20x speedup over sequential execution
- icon: 📊
title: Rich Datasets, Robots and Simulators
details: Works with 6+ datasets out of the box including GigaHand, Hot3D, OakInk, and more. Supports 9+ robot embodiments including dexterous hands (Allegro, Inspire, Xhand) and humanoid robots (G1, H1, T1). Supports multiple simulators including MuJoCo Warp, Genesis, and HDMI.
- icon: 🔄
title: Sim2Real Ready
details: Optimized trajectories can be directly deployed to real-world robots with minimal adjustments
---
## Quick Example
```bash
# Clone example datasets
git clone https://huggingface.co/datasets/retarget/retarget_example example_datasets
# Install with uv
uv sync --python 3.12
pip install --ignore-requires-python --no-deps -e .
# Run retargeting
uv run examples/run_mjwp.py
```
third_party/spider/docs/package-lock.json
Generated Vendored
+2379
View File
File diff suppressed because it is too large Load Diff
third_party/spider/docs/package.json
Vendored
+18
View File
@@ -0,0 +1,18 @@
{
"name": "spider-docs",
"version": "1.0.0",
"description": "SPIDER Documentation",
"scripts": {
"docs:dev": "vitepress dev",
"docs:build": "vitepress build",
"docs:preview": "vitepress preview"
},
"overrides": {
"esbuild": "0.27.1",
"mdast-util-to-hast": "13.2.1"
},
"devDependencies": {
"vitepress": "^1.6.4",
"vue": "^3.5.24"
}
}
third_party/spider/docs/usage/data-structure.md
Vendored
+389
View File
@@ -0,0 +1,389 @@
# Data Structure
SPIDER uses a structured file organization to handle multiple datasets, robots, and tasks efficiently.
## Directory Layout
```
example_datasets/
├── raw/ # Raw human motion data
│ └── {dataset_name}/
│ ├── {data_id}.pkl
│ └── meshes/
│
└── processed/ # Processed robot trajectories
└── {dataset_name}/
├── dataset_summary.json # High-level metadata
├── assets/ # Shared assets (avoid duplication)
│ ├── objects/
│ └── robots/
├── mano/ # Extracted human hand data
│ └── {embodiment_type}/
│ └── {task}/
│ └── {data_id}/
│ ├── trajectory_keypoint.npz
│ └── info.json
└── {robot_type}/ # Robot-specific data
└── {embodiment_type}/
└── {task}/
└── {data_id}/
├── scene.xml
├── trajectory_kinematic.npz
├── trajectory_mjwp.npz
├── visualization_mjwp.mp4
└── metrics.json
```
## Path Resolution
SPIDER automatically resolves paths based on task parameters:
```python
from spider.io import get_processed_data_dir
# Resolve processed directory
processed_dir = get_processed_data_dir(
dataset_dir="/path/to/datasets",
dataset_name="oakink",
robot_type="allegro",
embodiment_type="bimanual",
task="pick_spoon_bowl",
data_id=0
)
# Returns: /path/to/datasets/processed/oakink/allegro/bimanual/pick_spoon_bowl/0/
```
## File Formats
### Raw Data (`raw/{dataset_name}/{data_id}.pkl`)
Raw human motion data from dataset processing:
```python
{
'qpos_wrist_right': np.ndarray, # [T, 7] position + quaternion
'qpos_finger_right': np.ndarray, # [T, n_joints] joint angles
'qpos_wrist_left': np.ndarray, # [T, 7]
'qpos_finger_left': np.ndarray, # [T, n_joints]
'qpos_obj_right': np.ndarray, # [T, 7] object pose
'qpos_obj_left': np.ndarray, # [T, 7]
'contact': np.ndarray, # [T, ncon] contact indicators
'contact_pos': np.ndarray # [T, ncon, 3] contact positions
}
```
### Trajectory Files (`.npz`)
#### `trajectory_keypoint.npz`
Human hand keypoints (from dataset processor):
```python
{
'qpos_wrist_right': [T, 7],
'qpos_finger_right': [T, 15],
'qpos_obj_right': [T, 7],
# ... left hand equivalents
}
```
#### `trajectory_kinematic.npz`
Kinematic retargeting result (from IK):
```python
{
'qpos': [T, nq], # Full robot configuration
'qvel': [T, nv], # Velocities
'ctrl': [T, nu], # Control targets
'contact': [T, ncon], # Contact indicators
'contact_pos': [T, ncon, 3] # Contact positions
}
```
#### `trajectory_mjwp.npz`
Physics-optimized trajectory (from MJWP):
```python
{
'qpos': [N_steps, ctrl_steps, nq], # Per-step configs
'qvel': [N_steps, ctrl_steps, nv], # Velocities
'ctrl': [N_steps, ctrl_steps, nu], # Control commands
'time': [N_steps, ctrl_steps], # Timestamps
'opt_steps': [N_steps], # Iterations per step
'improvement': [N_steps], # Convergence metric
'rewards': [N_steps, num_samples], # Per-sample rewards
}
```
### Scene Files
#### `scene.xml`
MuJoCo scene definition:
```xml
<mujoco model="scene">
<!-- Solver settings -->
<option timestep="0.01" iterations="20" ...>
<!-- Robot model -->
<include file="../../assets/robots/allegro/bimanual.xml"/>
<!-- Object geometries -->
<body name="object_right" pos="0 0 0">
<freejoint/>
<geom type="mesh" mesh="obj_visual"/>
<geom type="mesh" mesh="obj_collision_0" .../>
...
</body>
<!-- Contact pairs -->
<contact>
<pair geom1="finger_tip" geom2="obj_collision_0"/>
...
</contact>
</mujoco>
```
#### `scene_eq.xml`
Scene with equality constraints (for `num_dyn > 1`):
Includes additional `<equality>` elements for virtual contact constraints used in annealing.
### Metadata Files
#### `task_info.json`
Task-level metadata:
```json
{
"dataset_name": "oakink",
"task": "pick_spoon_bowl",
"embodiment_type": "bimanual",
"ref_dt": 0.02,
"object_names": ["spoon", "bowl"],
"num_frames": 250
}
```
#### `info.json`
Trial-level metadata:
```json
{
"robot_type": "allegro",
"data_id": 0,
"timestamp": "2025-01-15T10:30:00",
"object_asset_paths": {
"spoon": "../../../../assets/objects/spoon/visual.obj",
"bowl": "../../../../assets/objects/bowl/visual.obj"
}
}
```
#### `metrics.json`
Performance metrics:
```json
{
"success_rate": 0.95,
"avg_position_error": 0.012,
"avg_rotation_error": 0.08,
"avg_optimization_time": 0.234,
"total_simulation_steps": 300
}
```
## Asset Organization
### Object Assets
Shared object meshes to avoid duplication:
```
assets/objects/{object_name}/
├── visual.obj # High-res visual mesh
└── convex/ # Convex decomposition
├── 0.obj
├── 1.obj
├── 2.obj
└── ...
```
### Robot Assets
Robot MJCF files:
```
assets/robots/{robot_type}/
├── left.xml # Left hand only
├── right.xml # Right hand only
└── bimanual.xml # Both hands
```
## Loading Data
### In Python
```python
import numpy as np
from spider.io import load_data
from spider.config import Config, process_config
# Create and process config
config = Config(
dataset_name="oakink",
robot_type="allegro",
embodiment_type="bimanual",
task="pick_spoon_bowl",
data_id=0
)
config = process_config(config)
# Load trajectory
qpos, qvel, ctrl, contact, contact_pos = load_data(
config,
config.data_path
)
print(f"Trajectory length: {qpos.shape[0]}")
print(f"DOF: {qpos.shape[1]}")
```
### Manual Loading
```python
import numpy as np
# Load trajectory
data = np.load('trajectory_mjwp.npz')
# Access arrays
qpos = data['qpos'] # [T, nq]
qvel = data['qvel'] # [T, nv]
ctrl = data['ctrl'] # [T, nu]
# Reshape if needed (MJWP saves per-step)
if qpos.ndim == 3:
qpos = qpos.reshape(-1, qpos.shape[-1])
```
## Data Flow Through Pipeline
```mermaid
graph TD
A[Raw Data<br/>raw/dataset/data.pkl] --> B[Dataset Processor]
B --> C[Keypoints<br/>mano/.../trajectory_keypoint.npz]
C --> D[Preprocessing Pipeline]
D --> E[Object Decomposition<br/>assets/objects/*/convex/*.obj]
D --> F[Scene XML<br/>robot/.../scene.xml]
D --> G[Kinematic Trajectory<br/>robot/.../trajectory_kinematic.npz]
G --> H[Physics Optimization]
H --> I[Optimized Trajectory<br/>robot/.../trajectory_mjwp.npz]
H --> J[Video<br/>robot/.../visualization_mjwp.mp4]
style C fill:#e1f5ff
style G fill:#fff4e1
style I fill:#ffe1e1
```
## Configuration Paths
Paths in `Config` are automatically resolved:
```python
@dataclass
class Config:
dataset_dir: str = "example_datasets" # Base directory
dataset_name: str = "oakink"
robot_type: str = "allegro"
embodiment_type: str = "bimanual"
task: str = "pick_spoon_bowl"
data_id: int = 0
# Auto-set by process_config():
model_path: str = "" # Path to scene.xml
data_path: str = "" # Path to trajectory_kinematic.npz
output_dir: str = "" # Where to save results
```
After `process_config()`:
```python
config.model_path = "{dataset_dir}/processed/{dataset_name}/{embodiment_type}/{task}/{data_id}/scene.xml"
config.data_path = "{...}/{robot_type}/{embodiment_type}/{task}/{data_id}/trajectory_kinematic.npz"
config.output_dir = "{...}/{robot_type}/{embodiment_type}/{task}/{data_id}/"
```
## Best Practices
### Organization
1. **Keep raw data separate**: Never modify `raw/` directory
2. **Share assets**: Use `assets/` for common objects/robots
3. **One trial per directory**: Each `{data_id}` is self-contained
4. **Relative paths in JSON**: Use relative paths for portability
### Naming Conventions
- **Tasks**: Use descriptive names (e.g., `pick_cup`, `open_jar`)
- **Objects**: Match names across datasets
- **Robots**: Use official model names (e.g., `allegro`, `inspire`)
### Cleanup
Remove intermediate files after successful optimization:
```bash
# Keep only final results
rm trajectory_kinematic.npz
rm scene.xml # If not needed for replay
```
## Troubleshooting
### Path Not Found
Check that all path components match:
```bash
# Verify structure
ls example_datasets/processed/oakink/allegro/bimanual/pick_spoon_bowl/0/
# Check config values
python -c "from spider.config import Config, process_config; \
c = process_config(Config()); print(c.output_dir)"
```
### Missing Assets
Ensure assets are in shared location:
```bash
# Check object assets
ls example_datasets/processed/oakink/assets/objects/
# Check robot assets
ls example_datasets/processed/oakink/assets/robots/
```
### Corrupted NPZ Files
Validate NPZ integrity:
```python
import numpy as np
try:
data = np.load('trajectory.npz')
print("Keys:", list(data.keys()))
print("Shapes:", {k: v.shape for k, v in data.items()})
except Exception as e:
print(f"Error loading: {e}")
```
third_party/spider/docs/usage/deployment.md
Vendored
+11
View File
@@ -0,0 +1,11 @@
# Deployment to Real Robots
This guide covers deploying SPIDER-optimized trajectories to real-world robots, including coordinate system transformations and practical deployment considerations.
## Overview
SPIDER saves all trajectories in a **minimum coordinate system** defined in the corresponding scene XML file. While joint positions can be used directly, it's important to convert the robot base position and orientation to the world coordinate system for correct execution on real robots.
Please refer to `spider/postprocess/read_to_robot.py` for the conversion code. It is recommended to read from a site from the robot to get wrist orientation that align with your real world setup.
![Deployment Example](/figs/real/pick_cup_real.gif)
third_party/spider/docs/usage/parameter-tuning.md
Vendored
+266
View File
@@ -0,0 +1,266 @@
# Parameter Tuning
This guide helps you tune SPIDER parameters for optimal performance on your tasks.
## Overview
Like reinforcement learning, different motion characteristics require different parameter settings. Start with defaults and iteratively refine based on results.
## Control Parameters
### Simulation Timestep (`sim_dt`)
**Default**: `0.01` (100 Hz)
The fundamental simulation timestep. Smaller values increase accuracy but slow down simulation. Start from a smaller one and gradually increase for faster simulation.
::: warning
All other timing parameters must be divisible by `sim_dt`
:::
### Control Interval (`ctrl_dt`)
**Default**: `0.4s` (2.5 Hz)
How frequently the controller updates. Affects:
- Optimization speed (larger = fewer optimizations)
- Responsiveness (smaller = more reactive)
**Tuning guidelines:**
- **Agile motions**: `0.05-0.2s`
- **Smooth motions**: `0.2-0.4s`
### Planning Horizon (`horizon`)
**Default**: `1.6s`
**Most important parameter**. How far ahead the optimizer looks.
**Impact:**
- ✅ Larger horizon: Better global plan, handles complex motions
- ❌ Larger horizon: Slower optimization, requires more samples
- ✅ Smaller horizon: Faster optimization
- ❌ Smaller horizon: Myopic behavior, may fail on complex tasks
**Tuning strategy:**
1. Start large (2.0s) for complex/unknown tasks
2. Gradually decrease until performance degrades
3. Monitor planned trajectories in Rerun viewer
```bash
# Complex manipulation
uv run examples/run_mjwp.py horizon=2.0
# Simple tracking
uv run examples/run_mjwp.py horizon=1.0
```
::: tip Visual Debugging
Use Rerun viewer to see the planned trajectory (blue/red traces). If the plan looks good but execution fails, the horizon might be too short.
:::
### Knot Spacing (`knot_dt`)
**Default**: `0.4s`
Temporal resolution of control parameterization. Actions are interpolated between knot points.
**Purpose**: Reduces search space by smoothing actions.
**Guidelines:**
- **Agile motions** (jumping, quick grasps): `0.1-0.2s`
- **Smooth motions** (walking, pouring): `0.2-0.4s`
```bash
# Fine control for agile task
uv run examples/run_mjwp.py knot_dt=0.1
# Smooth interpolation
uv run examples/run_mjwp.py knot_dt=0.3
```
## Sampling Parameters
### Number of Samples (`num_samples`)
**Default**: `1024`
Number of parallel trajectories to evaluate.
**Impact:**
- ✅ More samples: Better exploration, more robust
- ❌ More samples: Higher memory usage, slower per-iteration
- Scales linearly with GPU memory and compute
**Tuning:** try it out on your GPU and pick up the largest number that doesn't cause significant slow down.
### Temperature (`temperature`)
**Default**: `0.1`
Softmax temperature for sample weighting.
**Impact:**
- **Lower** (0.01-0.05): Sharper distribution, more optimal but can be unstable
- **Higher** (0.2-0.5): Smoother distribution, more stable but less optimal
**Tuning strategy:**
1. Start at `0.1`
2. If motion is shaky/unstable, increase to `0.15-0.2`
3. If motion is stable, decrease to `0.05-0.08` for better quality
Generally, this parameter is less important.
```bash
# Stable but less optimal
uv run examples/run_mjwp.py temperature=0.2
# Optimal but may be shaky
uv run examples/run_mjwp.py temperature=0.05
```
::: tip Monitoring
Check the `ctrl` panel in rerun viewer. If the action is shaky, increase the temperature.
:::
### Max Iterations (`max_num_iterations`)
**Default**: `32`
Maximum optimization iterations per control step.
**Tuning:**
1. Run with default and monitor `improvement` in Rerun
2. If `improvement` plateaus early (< 10 iters), decrease
3. If `improvement` is still high at end, increase
```bash
# Quick convergence
uv run examples/run_mjwp.py max_num_iterations=8
# Thorough optimization
uv run examples/run_mjwp.py max_num_iterations=64
```
### Early Stopping (`improvement_threshold`)
**Default**: `0.01`
Stop optimization if improvement falls below threshold.
**Usage:**
- Start at `0` (disabled) to see full convergence
- Gradually increase for speed: `0.01` → `0.02` → `0.05`
```bash
# No early stopping
uv run examples/run_mjwp.py improvement_threshold=0.0
# Aggressive early stopping
uv run examples/run_mjwp.py improvement_threshold=0.05
```
## Noise Scheduling
### Control Noise Scales
These control exploration throughout the horizon:
| Parameter | Default | Description |
|-----------|---------|-------------|
| `first_ctrl_noise_scale` | `0.5` | Noise at start of horizon |
| `last_ctrl_noise_scale` | `1.0` | Noise at end of horizon |
| `final_noise_scale` | `0.1` | Global annealing factor |
**Tuning:**
- Motion is stable? Decrease `first_ctrl_noise_scale` to `0.01-0.1`
- Need more exploration? Increase `last_ctrl_noise_scale` to `0.1-1.0`
- High precision task? Decrease `final_noise_scale` to `0.05`
### Component-Specific Scales
For dexterous hands, noise is applied differently to each component:
| Parameter | Default | Purpose |
|-----------|---------|---------|
| `joint_noise_scale` | `0.3` | Robot joint angles |
| `pos_noise_scale` | `0.01` | Hand base position |
| `rot_noise_scale` | `0.03` | Hand base rotation |
**Tuning guidelines:**
- Base is stable? Decrease `pos_noise_scale` and `rot_noise_scale`
- Finger control needs more exploration? Increase `joint_noise_scale`
```bash
# Stable base, explore fingers
uv run examples/run_mjwp.py \
pos_noise_scale=0.005 \
rot_noise_scale=0.01 \
joint_noise_scale=0.5
```
## Reward Scaling
Balance different tracking objectives:
### Position vs Rotation
```yaml
pos_rew_scale: 1.0 # End-effector position
rot_rew_scale: 0.3 # End-effector rotation
```
**Tuning:**
- Task requires precise orientation (e.g., key insertion)? Increase `rot_rew_scale`
- Position more important? Keep `rot_rew_scale` low
### Joint Tracking
```yaml
joint_rew_scale: 0.003
```
Lower weight since joint tracking is less critical than end-effector pose.
**Tuning:**
- Specific joint configuration needed? Increase to `0.01-0.03`
- Only care about end-effector? Keep very low (`0.001-0.003`)
### Velocity Regularization
```yaml
vel_rew_scale: 0.0001
```
Penalizes large velocities to smooth motion.
**Best practices:**
- ✅ Adding velocity tracking helps reduce latency
- ❌ Don't make it too large (causes sluggish motion)
```bash
# Reduce tracking latency
uv run examples/run_mjwp.py vel_rew_scale=0.001
# Disable velocity penalty
uv run examples/run_mjwp.py vel_rew_scale=0.0
```
## Common Issues
### Motion is Unstable/Shaky
**Symptoms**: Robot vibrates, erratic movements, or fails to complete the task
**Solutions:**
1. Iecrease `num_samples` (most effective)
2. Check if `horizon` is too short
3. Increase `knot_dt` for smoother actions or decrease it for more reactive motion
### Optimization Too Slow
**Symptoms**: RTR (realtime rate) < 0.1
**Solutions:**
1. Reduce `horizon` (most effective)
2. Increase `ctrl_dt` (also effective, make sure it doesn't cause instability)
third_party/spider/docs/usage/viewers.md
Vendored
+120
View File
@@ -0,0 +1,120 @@
# Viewers
SPIDER supports multiple viewer backends for visualization during retargeting. This guide covers setting up and using different viewers, including remote development workflows.
## Available Viewers
SPIDER supports the following viewer configurations:
| Viewer | Description | Use Case |
|--------|-------------|----------|
| `mujoco` | Native MuJoCo viewer | Fast local visualization |
| `rerun` | Rerun.io viewer | Remote development, rich logging |
| `viser` | Viser web viewer | Lightweight remote visualization |
To enable both, try `mujoco-rerun` or `mujoco-viser` viewer.
## Viser Viewer
[Viser](https://github.com/nerfstudio-project/viser) provides a lightweight web-based viewer for 3D scenes.
### Local Usage
```bash
uv run examples/run_mjwp.py viewer=viser
```
This will:
1. Start a Viser server
2. Log simulation data in real-time
3. Print a URL to open in your browser
::: warning
If you run headless (no X display), you may need to set a display before launching:
```bash
export DISPLAY=:1
# or
export MUJOCO_GL=egl
```
:::
::: info
If you see an import error, install the dependency:
```bash
pip install viser
```
:::
## Rerun Viewer
[Rerun](https://rerun.io/) provides a powerful logging and visualization system with time-travel capabilities, perfect for remote development.
### Local Usage
```bash
uv run examples/run_mjwp.py viewer=rerun
```
This will:
1. Automatically spawn a Rerun viewer
2. Log simulation data in real-time
3. Display 3D visualization and metrics
### Remote Development
For headless servers, use Rerun's web viewer:
#### Step 1: Start Rerun Server
On your remote server:
```bash
# Start Rerun web server
uv run rerun --serve-web --port 9876
# Or use rerun datapoints for more features
uv run rerun datapoints --serve-web --port 9876
```
#### Step 2: Run SPIDER
In another terminal on the remote server:
```bash
uv run examples/run_mjwp.py viewer=rerun
```
#### Step 3: Connect from Local Machine
Open your local browser to:
```
http://your-server:9876
```
Or connect via the Rerun client:
```bash
# On local machine
rerun --connect rerun+http://your-server:9876/proxy
```
::: warning
Make sure port 9876 is accessible through your firewall. You may need to configure SSH port forwarding:
```bash
ssh -L 9876:localhost:9876 user@remote-server
```
:::
### Logged Data
SPIDER logs the following data to Rerun:
- 3D scene visualization (robot and objects)
- Joint positions and velocities
- Optimization metrics (improvement, reward)
- Contact forces and positions
- Trace trajectories
third_party/spider/docs/workflows/workflow-dexmachina.md
Vendored
+175
View File
@@ -0,0 +1,175 @@
# Workflow: DexMachina (Genesis)
The DexMachina workflow integrates SPIDER with [Genesis](https://genesis.github.io/) simulator and [DexMachina](https://github.com/MandiZhao/dexmachina) framework for dexterous manipulation with downstream RL training.
## Quick Start (Reproducible Setup)
### 1. Download Data
All required data is hosted on HuggingFace. Download it into the spider directory:
```bash
sudo apt install git-lfs
git lfs install
cd /path/to/spider
git clone https://huggingface.co/datasets/retarget/retarget_example example_datasets
```
### 2. Install DexMachina Environment
```bash
# Create conda environment
conda create -n dexmachina python=3.10
conda activate dexmachina
# Install PyTorch
pip install torch==2.5.1
# Clone and install the Genesis fork (use the 'older' branch for API compatibility)
git clone https://github.com/MandiZhao/Genesis.git
cd Genesis
git checkout older
pip install -e .
pip install libigl==2.5.1 # Required: fixes igl.signed_distance API mismatch
cd ..
# Clone and install rl_games fork
git clone https://github.com/MandiZhao/rl_games.git
cd rl_games
pip install -e .
cd ..
# Clone and install dexmachina
git clone https://github.com/MandiZhao/dexmachina.git
cd dexmachina
pip install -e .
cd ..
# Install additional packages for video export and RL training
pip install moviepy==1.0.3 gymnasium ray seaborn
```
### 3. Install SPIDER (Minimal)
Install SPIDER without MuJoCo Warp (only need optimization components):
```bash
cd /path/to/spider
conda activate dexmachina
# Install SPIDER without dependencies (DexMachina has its own)
pip install --ignore-requires-python --no-deps -e .
# Install other dependencies
pip install loguru tyro hydra-core rerun-sdk==0.26.2
```
### 4. Copy Pre-Generated Data to DexMachina Assets
Pre-generated inspire_hand data (7 clips) is stored in the HuggingFace dataset so you don't need to re-run the retargeting pipeline.
```
example_datasets/raw/dexmachina/
contact_retarget/inspire_hand/s01/ # contact mapping (.npy) + videos (.mp4)
retargeted/inspire_hand/s01/ # retargeted kinematics (.pt)
retargeter_results/inspire_hand/s01/ # retargeter results (.npy)
```
Clips: `box_use_01`, `ketchup_use_01`, `ketchup_use_02`, `laptop_use_01`, `mixer_use_01`, `notebook_use_01`, `waffleiron_use_01`
After installing DexMachina, copy the data into its assets directory:
```bash
DEXMACHINA_ASSETS=$(python -c "from dexmachina.asset_utils import get_asset_path; print(get_asset_path(''))")
SPIDER_DATA=/path/to/spider/example_datasets/raw/dexmachina
cp -r $SPIDER_DATA/retargeted/inspire_hand/ $DEXMACHINA_ASSETS/retargeted/inspire_hand/
cp -r $SPIDER_DATA/contact_retarget/inspire_hand/ $DEXMACHINA_ASSETS/contact_retarget/inspire_hand/
cp -r $SPIDER_DATA/retargeter_results/inspire_hand/ $DEXMACHINA_ASSETS/retargeter_results/inspire_hand/
```
If the data is missing at runtime, the error message will print the exact copy commands for your paths.
### 5. Run SPIDER
```bash
conda activate dexmachina
cd /path/to/spider
# Run with default config (inspire hand, box-30-230 task)
python examples/run_dexmachina.py
```
This will:
1. Initialize Genesis environment with DexMachina task
2. Load reference motion from task
3. Optimize trajectory with Sampling
4. Save optimized trajectory and video
Output is saved to `example_datasets/processed/arctic/inspire_hand/bimanual/{task}/{data_id}/`:
- `trajectory_dexmachina.npz` --- trajectory data
- `trajectory_dexmachina.mp4` --- video
## Evaluate Trajectories
After running `run_dexmachina.py`, evaluate object tracking metrics (position, rotation, articulation distances) with:
```bash
conda activate dexmachina
cd /path/to/spider
# Evaluate all tasks under the default data directory
python spider/postprocess/evaluate_dexmachina.py
# Evaluate specific tasks
python spider/postprocess/evaluate_dexmachina.py --tasks box-30-230 laptop-30-230
# Use a different dataset/robot configuration
python spider/postprocess/evaluate_dexmachina.py --dataset_name arctic --robot_type inspire_hand --embodiment_type bimanual
```
The script uses the same dataset path resolution as `config.py` (`dataset_dir/processed/{dataset_name}/{robot_type}/{embodiment_type}/{task}/{data_id}/`). You can also pass `--data_dir` to override with an explicit path.
## Compare SPIDER vs RL (Multi-Seed)
To compare SPIDER (MPC) against RL rollouts across multiple seeds, use `--compare` mode. This loads both `trajectory_dexmachina.npz` (SPIDER) and `rollout_rl.npz` (RL) from each `{task}/{seed}/` directory and prints mean +/- std tables.
```bash
# Compare across seeds 0-4 (default), print Markdown tables
python spider/postprocess/evaluate_dexmachina.py --compare
# Compare with LaTeX tables and per-seed detail
python spider/postprocess/evaluate_dexmachina.py --compare --latex --detail
# Compare specific tasks / seeds
python spider/postprocess/evaluate_dexmachina.py --compare --tasks box-30-230 laptop-30-230 --seeds 0 1 2
```
## Configuration
Key parameters in `examples/config/dexmachina.yaml`:
| Parameter | Default | Description |
|-----------|---------|-------------|
| `task` | `box-30-230` | Object task (format: `object_name-start_frame-end_frame`) |
| `robot_type` | `inspire_hand` | Dexterous hand model |
| `embodiment_type` | `bimanual` | Hand configuration |
| `num_samples` | `1024` | Parallel CEM samples (Genesis envs) |
| `max_num_iterations` | `32` | CEM iterations per MPC step |
| `horizon` | `0.40` | Planning horizon (seconds) |
| `ctrl_dt` | `0.20` | Control timestep |
## Common Issues
1. **Missing DexMachina data**: Copy pre-generated data from `example_datasets/raw/dexmachina/` into the DexMachina assets directory (see "Copy Pre-Generated Data" section above).
2. **Genesis branch compatibility**: You must use the `older` branch of the Genesis fork (`git checkout older`). The `main` and `0122` branches have upstream merges that break API compatibility with DexMachina (e.g., `set_dofs_kp` input shape broadcasting, `inertial_quat` None handling).
3. **`igl.signed_distance` unpacking error** (`ValueError: too many values to unpack`): Install `libigl==2.5.1` explicitly. Newer versions of libigl change the return signature.
4. **`moviepy.editor` not found**: Install `moviepy==1.0.3` (not the newer 2.x which removed `moviepy.editor`). This is needed for video export.
5. **torch weights loading error**: If failed to load trajectory from dexmachina, ensure the data is loaded with `torch.load(data_fname, weights_only=False)`.
6. **numpy version**: Use `numpy==1.26.4` to avoid compatibility issues with Genesis and taichi.
7. **First run is slow**: The first run compiles taichi kernels (JIT) which can appear to hang for several minutes with no output. Subsequent runs use cached kernels and are faster.
third_party/spider/docs/workflows/workflow-hdmi.md
Vendored
+46
View File
@@ -0,0 +1,46 @@
# Workflow: HDMI (Humanoid-Object Interaction)
The HDMI workflow integrates SPIDER with [HDMI (Humanoid Manipulation)](https://github.com/lecar-lab/hdmi) framework for humanoid robot manipulation with downstream sim2real RL training.
## Prerequisites
### Install HDMI Environment
Follow the [official HDMI installation guide](https://github.com/lecar-lab/hdmi):
```bash
# Clone HDMI repository
git clone https://github.com/lecar-lab/hdmi.git
cd hdmi
# Install with uv (recommended)
uv sync
# Install SPIDER into HDMI environment
uv pip install --no-deps -e /path/to/spider
```
## Running HDMI Workflow
### Basic Example
```bash
cd /path/to/spider
# Run with default config (G1 humanoid, suitcase task)
python examples/run_hdmi.py
# OR refer to
./examples/run_hdmi.sh
```
### Configuration Options
You can enable `mjlab` native viewer by setting `viewer` to `hdmi`.
## Train RL
Write the motion as one HDMI reference motion by running the following command:
```bash
python examples/postprocess/read_to_hdmi.py
```
third_party/spider/docs/workflows/workflow-maniptrans.md
Vendored
+227
View File
@@ -0,0 +1,227 @@
# Workflow: ManipTrans (IsaacGym)
SPIDER MPC for dexterous hand-object manipulation using [IsaacGym](https://developer.nvidia.com/isaac-gym) and [HO-Tracker-Baseline](https://github.com/KailinLi/HO-Tracker-Baseline) (ManipTrans).
## Quick Start (Reproducible Setup)
### 1. Download Data
All required data is hosted on HuggingFace. Download it into the spider directory:
```bash
sudo apt install git-lfs
git lfs install
cd /path/to/spider
git clone https://huggingface.co/datasets/retarget/retarget_example example_datasets
```
This downloads `example_datasets/` containing:
- `raw/maniptrans/HO-Tracker/` --- 16 trajectory demos (8 left-hand, 8 right-hand)
- `raw/maniptrans/assets/` --- pre-trained base imitator checkpoints
- `raw/maniptrans/rl_runs/` --- 16 trained RL baseline checkpoints
- `raw/maniptrans/retargeting/` --- optional retargeting initialization
- `processed/maniptrans/` --- pre-computed rollouts, videos, and metrics (160 runs)
### 2. Create Data Symlink
ManipTrans uses relative paths (`data/HO-Tracker/...`). Create a symlink:
```bash
cd /path/to/spider
ln -sf example_datasets/raw/maniptrans data
```
### 3. Install Dependencies
Requires Python 3.8 + PyTorch 1.13.1 + IsaacGym (older than SPIDER's default Python 3.12+):
```bash
# Create conda environment
conda create -n maniptrans python=3.8
conda activate maniptrans
# Install PyTorch 1.13.1 with CUDA 11.7
pip install torch==1.13.1+cu117 torchvision==0.14.1+cu117 --extra-index-url https://download.pytorch.org/whl/cu117
# Install IsaacGym (requires NVIDIA GPU)
cd /path/to/isaacgym/python && pip install -e .
# Install pytorch3d (prebuilt wheel for py38 + cu117 + pytorch 1.13.1)
pip install pytorch3d==0.7.3 -f https://dl.fbaipublicfiles.com/pytorch3d/packaging/wheels/py38_cu117_pyt1131/download.html
# Install additional dependencies
pip install git+https://github.com/otaheri/chamfer_distance.git
pip install git+https://github.com/otaheri/bps_torch.git
pip install git+https://github.com/lixiny/manotorch.git
pip install numpy==1.23.5 gym==0.23.1 opencv-python-headless scikit-learn smplx trimesh
# Clone and install HO-Tracker-Baseline
cd /path/to
git clone https://github.com/KailinLi/HO-Tracker-Baseline.git
cd HO-Tracker-Baseline
touch requirements.txt # Create empty file (prevents pip install error)
pip install -e .
# Install SPIDER (minimal, no-deps to avoid conflicts)
cd /path/to/spider
pip install --ignore-requires-python --no-deps -e .
pip install loguru mujoco hydra-core omegaconf imageio[ffmpeg]
```
### 4. Set Up HO-Tracker-Baseline Data
HO-Tracker-Baseline also needs access to the trajectory data and imitator checkpoints. Point its data paths to spider's `example_datasets/`:
```bash
# Link HO-Tracker data
cd /path/to/HO-Tracker-Baseline
rm -f data/HO-Tracker
ln -sf /path/to/spider/example_datasets/raw/maniptrans/HO-Tracker data/HO-Tracker
# Link retargeting data
rm -rf data/retargeting/HO-Tracker
ln -sf /path/to/spider/example_datasets/raw/maniptrans/retargeting/HO-Tracker data/retargeting/HO-Tracker
# Copy imitator checkpoints
cp /path/to/spider/example_datasets/raw/maniptrans/assets/imitator_*.pth assets/
```
### 5. Patch HO-Tracker-Baseline for Compatibility
The `dexhandimitator` module may fail to import in some environments. Patch `maniptrans_envs/lib/envs/tasks/__init__.py` to make it optional:
```python
# Replace:
from .dexhandimitator import DexHandImitatorRHEnv, DexHandImitatorLHEnv
# With:
try:
from .dexhandimitator import DexHandImitatorRHEnv, DexHandImitatorLHEnv
except ImportError:
DexHandImitatorRHEnv = None
DexHandImitatorLHEnv = None
```
### 6. Run SPIDER
```bash
conda activate maniptrans
cd /path/to/spider
export LD_LIBRARY_PATH="/path/to/miniconda3/envs/maniptrans/lib:${LD_LIBRARY_PATH:-}"
# Default: 0f900@0, inspire hand, left hand
python examples/run_maniptrans.py
# Different trajectory:
python examples/run_maniptrans.py task=81a95@5
# Right hand:
python examples/run_maniptrans.py task=07bb1@1 embodiment_type=right
# Fewer iterations (faster):
python examples/run_maniptrans.py max_num_iterations=16
```
Output is saved to `example_datasets/processed/maniptrans/inspire/{side}/{task}/{data_id}/`:
- `rollout_isaac.npz` --- trajectory data
- `rollout_isaac.mp4` --- video (when `save_video=true`)
- `metrics_isaac.json` --- evaluation metrics
## Available Trajectories
16 trajectories from HO-Tracker dataset (8 per hand):
| Side | Task IDs |
|------|----------|
| Left | `0f900@0`, `81a95@5`, `0f4a7@1`, `70ca2@0`, `39e5e@5`, `380c7@0`, `2ddc8@2`, `0b3a0@6` |
| Right | `07bb1@1`, `235ac@0`, `07301@0`, `49793@0`, `01857@10`, `031b7@0`, `03ac9@0`, `155cb@0` |
## Configuration
Key parameters in `examples/config/maniptrans.yaml`:
| Parameter | Default | Description |
|-----------|---------|-------------|
| `task` | `0f900@0` | HO-Tracker data sequence |
| `embodiment_type` | `left` | `left` or `right` |
| `num_samples` | `1024` | Parallel CEM samples (IsaacGym envs) |
| `max_num_iterations` | `32` | CEM iterations per MPC step |
| `horizon` | `0.6` | Planning horizon (seconds) |
| `ctrl_dt` | `0.2` | Control timestep |
| `seed` | `0` | Random seed |
| `data_id` | `0` | Output subdirectory index |
## RL Baseline Evaluation
Trained RL checkpoints are included for all 16 trajectories. To evaluate:
```bash
cd /path/to/HO-Tracker-Baseline
# Example: evaluate left-hand trajectory
python main/rl/train.py \
task=ResDexHand dexhand=inspire side=lh \
headless=true num_envs=256 test=true \
+forceFullLength=true seed=0 \
rolloutStateInit=false randomStateInit=false \
"dataIndices=[0f900@0]" \
checkpoint=/path/to/spider/example_datasets/raw/maniptrans/rl_runs/baseline_0f900@0_inspire_lh/nn/baseline_0f900@0.pth \
actionsMovingAverage=0.4 \
rh_base_model_checkpoint=assets/imitator_rh_inspire.pth \
lh_base_model_checkpoint=assets/imitator_lh_inspire.pth
```
Side tags: `lh` for left-hand trajectories, `rh` for right-hand trajectories.
## Evaluation Metrics
Four metrics computed against ground-truth demo data:
| Metric | Description |
|--------|-------------|
| `e_t` | Object translation error (cm) |
| `e_r` | Object rotation error (degrees, geodesic) |
| `e_j` | Joint position error (cm) |
| `e_ft` | Fingertip position error (cm) |
Run evaluation across all trajectories:
```bash
# Single-seed evaluation (data_id=0 only)
python spider/postprocess/evaluate_maniptrans.py
# Multi-seed evaluation (all data_ids, mean +/- std)
python spider/postprocess/evaluate_maniptrans.py --multiseed
```
## Common Issues
1. **DexHandFactory registration failure** (`KeyError: 'inspire_rh'`): This is a circular import issue. SPIDER includes a fix in `spider/simulators/maniptrans.py` that lazily triggers hand auto-registration. Ensure you are using the latest SPIDER code.
2. **`dexhandimitator` import error**: Apply the patch in step 5 above to make this import optional.
3. **pytorch3d installation**: Use the prebuilt wheel for your PyTorch/CUDA combination. Building from source requires matching CUDA toolkit version (not just driver) and compatible GCC. The command in step 3 uses prebuilt wheels for py38 + cu117 + pytorch 1.13.1.
4. **`np.float` deprecation** (`AttributeError: module 'numpy' has no attribute 'float'`): Install `numpy==1.23.5` which still supports the deprecated aliases.
5. **Missing `requirements.txt` in HO-Tracker-Baseline**: Run `touch requirements.txt` in the HO-Tracker-Baseline root before `pip install -e .`.
6. **`imageio[ffmpeg]` required for video**: Install with `pip install imageio[ffmpeg]` for video recording support.
7. **`LD_LIBRARY_PATH` must include conda env lib**: Without this, IsaacGym may fail to find shared libraries. Set `export LD_LIBRARY_PATH="/path/to/miniconda3/envs/maniptrans/lib:${LD_LIBRARY_PATH:-}"`.
## Important Notes
### Python 3.8 Compatibility
ManipTrans requires Python 3.8 + PyTorch 1.13.1 + IsaacGym. SPIDER handles this via:
- `from __future__ import annotations` in `spider/config.py` (for `tuple[float, float]` syntax)
- `hasattr(torch, "compile")` guards in `spider/optimizers/sampling.py`
### Action Space
SPIDER optimizes 18-dim residual actions (6 wrist force/torque + 12 DOF positions). The base imitator provides nominal tracking actions. These combine in the env's `pre_physics_step` to produce 36-dim actions for the full base+residual architecture.
### ManipTrans Config Fields
ManipTrans-specific fields (e.g., `maniptrans_episode_length`) are **not** part of the SPIDER `Config` dataclass. In `examples/run_maniptrans.py`, they are extracted from the Hydra config and attached as attributes after `Config` construction.
third_party/spider/docs/workflows/workflow-mjwp.md
Vendored
+317
View File
@@ -0,0 +1,317 @@
# Workflow: MuJoCo Warp (MJWP)
The MuJoCo Warp (MJWP) workflow is SPIDER's default and most versatile pipeline, supporting both dexterous hand and humanoid robot retargeting with high-performance physics simulation.
## Overview
The MJWP workflow uses MuJoCo with Warp backend for GPU-accelerated physics simulation. It provides the complete pipeline from raw human motion data to physics-based optimized robot trajectories.
**Supported Embodiments:**
- Dexterous hands (Allegro, Inspire, Shadow, Xhand, etc.)
- Humanoid robots (G1, H1, T1, etc.)
- Bimanual configurations
## Workflow Steps
```mermaid
graph LR
A[Raw Data] --> B[Process Dataset]
B --> C[Decompose Mesh]
C --> D[Detect Contact]
D --> E[Generate Scene]
E --> F[IK Retargeting]
F --> G[Physics Optimization]
G --> H[Output Trajectory]
```
### Step 1: Setup Environment Variables
Define task parameters:
```bash
export TASK=pick_spoon_bowl
export HAND_TYPE=bimanual
export DATA_ID=0
export ROBOT_TYPE=allegro
export DATASET_NAME=oakink
```
### Step 2: Process Dataset
Convert raw human motion data to standardized format:
```bash
uv run spider/process_datasets/oakink.py \
--task=${TASK} \
--embodiment-type=${HAND_TYPE} \
--data-id=${DATA_ID}
```
**What it does:**
- Reads raw motion capture data (MANO poses, object poses)
- Extracts hand kinematics (joint positions, wrist poses)
- Saves processed data in standardized format
**Output:**
```
example_datasets/processed/oakink/mano/bimanual/pick_spoon_bowl/0/
├── trajectory_keypoint.npz
└── info.json
```
::: tip Dataset-Specific Processors
Replace `oakink.py` with the appropriate processor for your dataset:
- `gigahand.py` - GigaHand dataset
- `hot3d.py` - Hot3D dataset
- `fair_fre.py` - FAIR FRE dataset
:::
### Step 3: Decompose Object Mesh
Create convex decomposition for efficient collision detection:
```bash
uv run spider/preprocess/decompose_fast.py \
--task=${TASK} \
--dataset-name=${DATASET_NAME} \
--data-id=${DATA_ID} \
--embodiment-type=${HAND_TYPE}
```
If you want higher quality convex decomposition, you can use the following command:
```bash
uv run spider/preprocess/decompose.py \
--task=${TASK} \
--dataset-name=${DATASET_NAME} \
--data-id=${DATA_ID} \
--embodiment-type=${HAND_TYPE}
```
**What it does:**
- Loads object mesh from dataset
- Performs convex decomposition using V-HACD or CoACD
- Saves decomposed mesh components
**Output:**
```
example_datasets/processed/oakink/assets/objects/spoon/convex/
├── 0.obj
├── 1.obj
├── 2.obj
└── visual.obj
```
::: warning
Convex decomposition quality significantly affects simulation accuracy. Complex objects may need manual tuning of decomposition parameters.
:::
### Step 4: Detect Contact (Optional)
Identify hand-object contact points for contact-aware retargeting:
```bash
uv run spider/preprocess/detect_contact.py \
--task=${TASK} \
--dataset-name=${DATASET_NAME} \
--data-id=${DATA_ID} \
--embodiment-type=${HAND_TYPE}
```
**What it does:**
- Analyzes hand-object proximity throughout trajectory
- Identifies contact points on hand surface
- Saves contact information for trajectory optimization
**Output:**
- Updates `trajectory_keypoint.npz` with contact data
::: tip
Contact detection helps maintain grasp stability during retargeting, especially for manipulation tasks.
:::
### Step 5: Generate Scene
Create MuJoCo scene XML with robot and objects:
```bash
uv run spider/preprocess/generate_xml.py \
--task=${TASK} \
--dataset-name=${DATASET_NAME} \
--data-id=${DATA_ID} \
--embodiment-type=${HAND_TYPE} \
--robot-type=${ROBOT_TYPE}
```
**What it does:**
- Loads robot URDF/XML configuration
- Creates scene with objects from dataset
- Sets up collision pairs and contact parameters
- Configures camera and lighting
**Output:**
```
example_datasets/processed/oakink/allegro/bimanual/pick_spoon_bowl/0/
└── scene.xml
```
**Scene structure:**
```xml
<mujoco>
<compiler angle="radian" meshdir="assets"/>
<option timestep="0.01"/>
<!-- Robot -->
<worldbody>
<body name="left_hand">...</body>
<body name="right_hand">...</body>
<!-- Objects -->
<body name="spoon">...</body>
<body name="bowl">...</body>
</worldbody>
<!-- Contact pairs -->
<contact>
<pair geom1="left_index_tip" geom2="spoon"/>
...
</contact>
</mujoco>
```
### Step 6: Inverse Kinematics (IK)
A minimum IK scripts based on mink can be used by:
```bash
uv run spider/preprocess/ik_fast.py \
--task=${TASK} \
--dataset-name=${DATASET_NAME} \
--data-id=${DATA_ID} \
--embodiment-type=${HAND_TYPE} \
--robot-type=${ROBOT_TYPE}
```
You can also use the original IK script with mujoco by:
```bash
uv run spider/preprocess/ik.py \
--task=${TASK} \
--dataset-name=${DATASET_NAME} \
--data-id=${DATA_ID} \
--embodiment-type=${HAND_TYPE} \
--robot-type=${ROBOT_TYPE} \
--open-hand
```
**What it does:**
- Loads MANO hand poses
- Solves inverse kinematics for robot joints
- Retargets hand poses to robot configuration
- Initializes with open hand pose (`--open-hand`)
**Output:**
```
example_datasets/processed/oakink/allegro/bimanual/pick_spoon_bowl/0/
└── trajectory_kinematic.npz
```
**Data format:**
```python
# trajectory_kinematic.npz contains:
qpos: [T, nq] # Joint positions
qvel: [T, nv] # Joint velocities (zero for kinematic)
ctrl: [T, nu] # Control targets (same as qpos)
time: [T] # Timestamps
```
::: tip IK Options
- `--open-hand`: Start IK from open hand pose (recommended)
- `--use-contacts`: Use detected contacts in IK optimization
- `--smooth-qvel`: Add velocity smoothness term
:::
### Step 7: Physics-Based Optimization
Optimize trajectory with physics constraints using DIAL-MPC:
```bash
uv run examples/run_mjwp.py \
+override=${DATASET_NAME} \
task=${TASK} \
data_id=${DATA_ID} \
robot_type=${ROBOT_TYPE} \
embodiment_type=${HAND_TYPE}
```
**What it does:**
- Loads kinematic trajectory as reference
- Sets up parallel physics simulation (GPU)
- Runs sampling-based optimization (DIAL-MPC)
- Tracks reference while maintaining physics feasibility
**Real-time visualization** shows:
- Current robot state
- Reference trajectory (transparent)
- Optimization metrics
- Contact forces
**Output:**
```
example_datasets/processed/oakink/allegro/bimanual/pick_spoon_bowl/0/
├── trajectory_mjwp.npz # Optimized trajectory
├── visualization_mjwp.mp4 # Rendered video
└── metrics.json # Success metrics
```
## Augmentation Methods
The original `mjwp_eq` pipeline is no longer supported due to latest mujoco warp release make it unstable.
The newer guidance methods are supported by the `mjwp_act` (e.g. `configs/override/oakink_act.yaml`) configuration instead. You will also need to add the `--act-scene` flag when running the `generate_xml.py` and `ik.py` scripts to ensure the necessary assets and logic are generated properly. This approach will provide the fair comparison you are looking for. Related assets is also available in our huggingface repo.
## Dataset-Specific Examples
### OakInk Dataset
```bash
export TASK=pick_spoon_bowl
export DATASET_NAME=oakink
export DATA_ID=0
# Full workflow
uv run spider/process_datasets/oakink.py --task=${TASK} --embodiment-type=bimanual --data-id=${DATA_ID}
uv run spider/preprocess/decompose_fast.py --task=${TASK} --dataset-name=${DATASET_NAME} --data-id=${DATA_ID} --embodiment-type=bimanual
uv run spider/preprocess/generate_xml.py --task=${TASK} --dataset-name=${DATASET_NAME} --data-id=${DATA_ID} --embodiment-type=bimanual --robot-type=allegro
uv run spider/preprocess/ik.py --task=${TASK} --dataset-name=${DATASET_NAME} --data-id=${DATA_ID} --embodiment-type=bimanual --robot-type=allegro --open-hand
uv run examples/run_mjwp.py +override=oakink task=${TASK} data_id=${DATA_ID} robot_type=allegro embodiment_type=bimanual
```
### GigaHand Dataset
```bash
export TASK=p36-tea
export DATASET_NAME=gigahand
export DATA_ID=10
# Full workflow
uv run spider/process_datasets/gigahand.py --task=${TASK} --embodiment-type=bimanual --data-id=${DATA_ID}
uv run spider/preprocess/decompose_fast.py --task=${TASK} --dataset-name=${DATASET_NAME} --data-id=${DATA_ID} --embodiment-type=bimanual
uv run spider/preprocess/generate_xml.py --task=${TASK} --dataset-name=${DATASET_NAME} --data-id=${DATA_ID} --embodiment-type=bimanual --robot-type=xhand
uv run spider/preprocess/ik.py --task=${TASK} --dataset-name=${DATASET_NAME} --data-id=${DATA_ID} --embodiment-type=bimanual --robot-type=xhand --open-hand
uv run examples/run_mjwp.py +override=gigahand task=${TASK} data_id=${DATA_ID} robot_type=xhand embodiment_type=bimanual
```
### Hot3D Dataset
```bash
export TASK=P0001_4bf4e21a-obj96945373046044
export DATASET_NAME=hot3d
export DATA_ID=0
# Full workflow
uv run spider/process_datasets/hot3d.py --task=${TASK} --embodiment-type=right --data-id=${DATA_ID}
uv run spider/preprocess/decompose_fast.py --task=${TASK} --dataset-name=${DATASET_NAME} --data-id=${DATA_ID} --embodiment-type=right
uv run spider/preprocess/generate_xml.py --task=${TASK} --dataset-name=${DATASET_NAME} --data-id=${DATA_ID} --embodiment-type=right --robot-type=xhand
uv run spider/preprocess/ik.py --task=${TASK} --dataset-name=${DATASET_NAME} --data-id=${DATA_ID} --embodiment-type=right --robot-type=xhand --open-hand
uv run examples/run_mjwp.py +override=hot3d task=${TASK} data_id=${DATA_ID} robot_type=xhand embodiment_type=right
```