Initial snapshot: hand motion pipeline (Dyn-HaMR + dex-retargeting + SPIDER)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
+317
@@ -0,0 +1,317 @@
|
||||
# Workflow: MuJoCo Warp (MJWP)
|
||||
|
||||
The MuJoCo Warp (MJWP) workflow is SPIDER's default and most versatile pipeline, supporting both dexterous hand and humanoid robot retargeting with high-performance physics simulation.
|
||||
|
||||
## Overview
|
||||
|
||||
The MJWP workflow uses MuJoCo with Warp backend for GPU-accelerated physics simulation. It provides the complete pipeline from raw human motion data to physics-based optimized robot trajectories.
|
||||
|
||||
**Supported Embodiments:**
|
||||
- Dexterous hands (Allegro, Inspire, Shadow, Xhand, etc.)
|
||||
- Humanoid robots (G1, H1, T1, etc.)
|
||||
- Bimanual configurations
|
||||
|
||||
|
||||
## Workflow Steps
|
||||
|
||||
```mermaid
|
||||
graph LR
|
||||
A[Raw Data] --> B[Process Dataset]
|
||||
B --> C[Decompose Mesh]
|
||||
C --> D[Detect Contact]
|
||||
D --> E[Generate Scene]
|
||||
E --> F[IK Retargeting]
|
||||
F --> G[Physics Optimization]
|
||||
G --> H[Output Trajectory]
|
||||
```
|
||||
|
||||
### Step 1: Setup Environment Variables
|
||||
|
||||
Define task parameters:
|
||||
|
||||
```bash
|
||||
export TASK=pick_spoon_bowl
|
||||
export HAND_TYPE=bimanual
|
||||
export DATA_ID=0
|
||||
export ROBOT_TYPE=allegro
|
||||
export DATASET_NAME=oakink
|
||||
```
|
||||
|
||||
### Step 2: Process Dataset
|
||||
|
||||
Convert raw human motion data to standardized format:
|
||||
|
||||
```bash
|
||||
uv run spider/process_datasets/oakink.py \
|
||||
--task=${TASK} \
|
||||
--embodiment-type=${HAND_TYPE} \
|
||||
--data-id=${DATA_ID}
|
||||
```
|
||||
|
||||
**What it does:**
|
||||
- Reads raw motion capture data (MANO poses, object poses)
|
||||
- Extracts hand kinematics (joint positions, wrist poses)
|
||||
- Saves processed data in standardized format
|
||||
|
||||
**Output:**
|
||||
```
|
||||
example_datasets/processed/oakink/mano/bimanual/pick_spoon_bowl/0/
|
||||
├── trajectory_keypoint.npz
|
||||
└── info.json
|
||||
```
|
||||
|
||||
::: tip Dataset-Specific Processors
|
||||
Replace `oakink.py` with the appropriate processor for your dataset:
|
||||
- `gigahand.py` - GigaHand dataset
|
||||
- `hot3d.py` - Hot3D dataset
|
||||
- `fair_fre.py` - FAIR FRE dataset
|
||||
:::
|
||||
|
||||
### Step 3: Decompose Object Mesh
|
||||
|
||||
Create convex decomposition for efficient collision detection:
|
||||
|
||||
```bash
|
||||
uv run spider/preprocess/decompose_fast.py \
|
||||
--task=${TASK} \
|
||||
--dataset-name=${DATASET_NAME} \
|
||||
--data-id=${DATA_ID} \
|
||||
--embodiment-type=${HAND_TYPE}
|
||||
```
|
||||
|
||||
If you want higher quality convex decomposition, you can use the following command:
|
||||
```bash
|
||||
uv run spider/preprocess/decompose.py \
|
||||
--task=${TASK} \
|
||||
--dataset-name=${DATASET_NAME} \
|
||||
--data-id=${DATA_ID} \
|
||||
--embodiment-type=${HAND_TYPE}
|
||||
```
|
||||
|
||||
**What it does:**
|
||||
- Loads object mesh from dataset
|
||||
- Performs convex decomposition using V-HACD or CoACD
|
||||
- Saves decomposed mesh components
|
||||
|
||||
**Output:**
|
||||
```
|
||||
example_datasets/processed/oakink/assets/objects/spoon/convex/
|
||||
├── 0.obj
|
||||
├── 1.obj
|
||||
├── 2.obj
|
||||
└── visual.obj
|
||||
```
|
||||
|
||||
::: warning
|
||||
Convex decomposition quality significantly affects simulation accuracy. Complex objects may need manual tuning of decomposition parameters.
|
||||
:::
|
||||
|
||||
### Step 4: Detect Contact (Optional)
|
||||
|
||||
Identify hand-object contact points for contact-aware retargeting:
|
||||
|
||||
```bash
|
||||
uv run spider/preprocess/detect_contact.py \
|
||||
--task=${TASK} \
|
||||
--dataset-name=${DATASET_NAME} \
|
||||
--data-id=${DATA_ID} \
|
||||
--embodiment-type=${HAND_TYPE}
|
||||
```
|
||||
|
||||
**What it does:**
|
||||
- Analyzes hand-object proximity throughout trajectory
|
||||
- Identifies contact points on hand surface
|
||||
- Saves contact information for trajectory optimization
|
||||
|
||||
**Output:**
|
||||
- Updates `trajectory_keypoint.npz` with contact data
|
||||
|
||||
::: tip
|
||||
Contact detection helps maintain grasp stability during retargeting, especially for manipulation tasks.
|
||||
:::
|
||||
|
||||
### Step 5: Generate Scene
|
||||
|
||||
Create MuJoCo scene XML with robot and objects:
|
||||
|
||||
```bash
|
||||
uv run spider/preprocess/generate_xml.py \
|
||||
--task=${TASK} \
|
||||
--dataset-name=${DATASET_NAME} \
|
||||
--data-id=${DATA_ID} \
|
||||
--embodiment-type=${HAND_TYPE} \
|
||||
--robot-type=${ROBOT_TYPE}
|
||||
```
|
||||
|
||||
**What it does:**
|
||||
- Loads robot URDF/XML configuration
|
||||
- Creates scene with objects from dataset
|
||||
- Sets up collision pairs and contact parameters
|
||||
- Configures camera and lighting
|
||||
|
||||
**Output:**
|
||||
```
|
||||
example_datasets/processed/oakink/allegro/bimanual/pick_spoon_bowl/0/
|
||||
└── scene.xml
|
||||
```
|
||||
|
||||
**Scene structure:**
|
||||
```xml
|
||||
<mujoco>
|
||||
<compiler angle="radian" meshdir="assets"/>
|
||||
<option timestep="0.01"/>
|
||||
|
||||
<!-- Robot -->
|
||||
<worldbody>
|
||||
<body name="left_hand">...</body>
|
||||
<body name="right_hand">...</body>
|
||||
|
||||
<!-- Objects -->
|
||||
<body name="spoon">...</body>
|
||||
<body name="bowl">...</body>
|
||||
</worldbody>
|
||||
|
||||
<!-- Contact pairs -->
|
||||
<contact>
|
||||
<pair geom1="left_index_tip" geom2="spoon"/>
|
||||
...
|
||||
</contact>
|
||||
</mujoco>
|
||||
```
|
||||
|
||||
### Step 6: Inverse Kinematics (IK)
|
||||
|
||||
A minimum IK scripts based on mink can be used by:
|
||||
|
||||
```bash
|
||||
uv run spider/preprocess/ik_fast.py \
|
||||
--task=${TASK} \
|
||||
--dataset-name=${DATASET_NAME} \
|
||||
--data-id=${DATA_ID} \
|
||||
--embodiment-type=${HAND_TYPE} \
|
||||
--robot-type=${ROBOT_TYPE}
|
||||
```
|
||||
|
||||
You can also use the original IK script with mujoco by:
|
||||
|
||||
```bash
|
||||
uv run spider/preprocess/ik.py \
|
||||
--task=${TASK} \
|
||||
--dataset-name=${DATASET_NAME} \
|
||||
--data-id=${DATA_ID} \
|
||||
--embodiment-type=${HAND_TYPE} \
|
||||
--robot-type=${ROBOT_TYPE} \
|
||||
--open-hand
|
||||
```
|
||||
|
||||
**What it does:**
|
||||
- Loads MANO hand poses
|
||||
- Solves inverse kinematics for robot joints
|
||||
- Retargets hand poses to robot configuration
|
||||
- Initializes with open hand pose (`--open-hand`)
|
||||
|
||||
**Output:**
|
||||
```
|
||||
example_datasets/processed/oakink/allegro/bimanual/pick_spoon_bowl/0/
|
||||
└── trajectory_kinematic.npz
|
||||
```
|
||||
|
||||
**Data format:**
|
||||
```python
|
||||
# trajectory_kinematic.npz contains:
|
||||
qpos: [T, nq] # Joint positions
|
||||
qvel: [T, nv] # Joint velocities (zero for kinematic)
|
||||
ctrl: [T, nu] # Control targets (same as qpos)
|
||||
time: [T] # Timestamps
|
||||
```
|
||||
|
||||
::: tip IK Options
|
||||
- `--open-hand`: Start IK from open hand pose (recommended)
|
||||
- `--use-contacts`: Use detected contacts in IK optimization
|
||||
- `--smooth-qvel`: Add velocity smoothness term
|
||||
:::
|
||||
|
||||
### Step 7: Physics-Based Optimization
|
||||
|
||||
Optimize trajectory with physics constraints using DIAL-MPC:
|
||||
|
||||
```bash
|
||||
uv run examples/run_mjwp.py \
|
||||
+override=${DATASET_NAME} \
|
||||
task=${TASK} \
|
||||
data_id=${DATA_ID} \
|
||||
robot_type=${ROBOT_TYPE} \
|
||||
embodiment_type=${HAND_TYPE}
|
||||
```
|
||||
|
||||
**What it does:**
|
||||
- Loads kinematic trajectory as reference
|
||||
- Sets up parallel physics simulation (GPU)
|
||||
- Runs sampling-based optimization (DIAL-MPC)
|
||||
- Tracks reference while maintaining physics feasibility
|
||||
|
||||
**Real-time visualization** shows:
|
||||
- Current robot state
|
||||
- Reference trajectory (transparent)
|
||||
- Optimization metrics
|
||||
- Contact forces
|
||||
|
||||
**Output:**
|
||||
```
|
||||
example_datasets/processed/oakink/allegro/bimanual/pick_spoon_bowl/0/
|
||||
├── trajectory_mjwp.npz # Optimized trajectory
|
||||
├── visualization_mjwp.mp4 # Rendered video
|
||||
└── metrics.json # Success metrics
|
||||
```
|
||||
|
||||
## Augmentation Methods
|
||||
|
||||
The original `mjwp_eq` pipeline is no longer supported due to latest mujoco warp release make it unstable.
|
||||
The newer guidance methods are supported by the `mjwp_act` (e.g. `configs/override/oakink_act.yaml`) configuration instead. You will also need to add the `--act-scene` flag when running the `generate_xml.py` and `ik.py` scripts to ensure the necessary assets and logic are generated properly. This approach will provide the fair comparison you are looking for. Related assets is also available in our huggingface repo.
|
||||
|
||||
## Dataset-Specific Examples
|
||||
|
||||
### OakInk Dataset
|
||||
|
||||
```bash
|
||||
export TASK=pick_spoon_bowl
|
||||
export DATASET_NAME=oakink
|
||||
export DATA_ID=0
|
||||
|
||||
# Full workflow
|
||||
uv run spider/process_datasets/oakink.py --task=${TASK} --embodiment-type=bimanual --data-id=${DATA_ID}
|
||||
uv run spider/preprocess/decompose_fast.py --task=${TASK} --dataset-name=${DATASET_NAME} --data-id=${DATA_ID} --embodiment-type=bimanual
|
||||
uv run spider/preprocess/generate_xml.py --task=${TASK} --dataset-name=${DATASET_NAME} --data-id=${DATA_ID} --embodiment-type=bimanual --robot-type=allegro
|
||||
uv run spider/preprocess/ik.py --task=${TASK} --dataset-name=${DATASET_NAME} --data-id=${DATA_ID} --embodiment-type=bimanual --robot-type=allegro --open-hand
|
||||
uv run examples/run_mjwp.py +override=oakink task=${TASK} data_id=${DATA_ID} robot_type=allegro embodiment_type=bimanual
|
||||
```
|
||||
|
||||
### GigaHand Dataset
|
||||
|
||||
```bash
|
||||
export TASK=p36-tea
|
||||
export DATASET_NAME=gigahand
|
||||
export DATA_ID=10
|
||||
|
||||
# Full workflow
|
||||
uv run spider/process_datasets/gigahand.py --task=${TASK} --embodiment-type=bimanual --data-id=${DATA_ID}
|
||||
uv run spider/preprocess/decompose_fast.py --task=${TASK} --dataset-name=${DATASET_NAME} --data-id=${DATA_ID} --embodiment-type=bimanual
|
||||
uv run spider/preprocess/generate_xml.py --task=${TASK} --dataset-name=${DATASET_NAME} --data-id=${DATA_ID} --embodiment-type=bimanual --robot-type=xhand
|
||||
uv run spider/preprocess/ik.py --task=${TASK} --dataset-name=${DATASET_NAME} --data-id=${DATA_ID} --embodiment-type=bimanual --robot-type=xhand --open-hand
|
||||
uv run examples/run_mjwp.py +override=gigahand task=${TASK} data_id=${DATA_ID} robot_type=xhand embodiment_type=bimanual
|
||||
```
|
||||
|
||||
### Hot3D Dataset
|
||||
|
||||
```bash
|
||||
export TASK=P0001_4bf4e21a-obj96945373046044
|
||||
export DATASET_NAME=hot3d
|
||||
export DATA_ID=0
|
||||
|
||||
# Full workflow
|
||||
uv run spider/process_datasets/hot3d.py --task=${TASK} --embodiment-type=right --data-id=${DATA_ID}
|
||||
uv run spider/preprocess/decompose_fast.py --task=${TASK} --dataset-name=${DATASET_NAME} --data-id=${DATA_ID} --embodiment-type=right
|
||||
uv run spider/preprocess/generate_xml.py --task=${TASK} --dataset-name=${DATASET_NAME} --data-id=${DATA_ID} --embodiment-type=right --robot-type=xhand
|
||||
uv run spider/preprocess/ik.py --task=${TASK} --dataset-name=${DATASET_NAME} --data-id=${DATA_ID} --embodiment-type=right --robot-type=xhand --open-hand
|
||||
uv run examples/run_mjwp.py +override=hot3d task=${TASK} data_id=${DATA_ID} robot_type=xhand embodiment_type=right
|
||||
```
|
||||
Reference in New Issue
Block a user