Initial snapshot: hand motion pipeline (Dyn-HaMR + dex-retargeting + SPIDER)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
+114
@@ -0,0 +1,114 @@
|
||||
# Installation
|
||||
|
||||
This guide walks you through installing SPIDER and its dependencies.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- **Python**: 3.12 or higher
|
||||
- **CUDA**: 11.8 or higher (for GPU acceleration)
|
||||
- **Git**: For cloning repositories
|
||||
- **Git LFS**: For large file storage (datasets)
|
||||
|
||||
## Option 1: Using uv (Recommended)
|
||||
|
||||
[uv](https://github.com/astral-sh/uv) is a fast Python package installer that we recommend for development.
|
||||
|
||||
### Install uv
|
||||
|
||||
```bash
|
||||
# Linux/macOS
|
||||
curl -LsSf https://astral.sh/uv/install.sh | sh
|
||||
|
||||
# Windows
|
||||
powershell -c "irm https://astral.sh/uv/install.ps1 | iex"
|
||||
```
|
||||
|
||||
### Install SPIDER
|
||||
|
||||
```bash
|
||||
# Clone the repository
|
||||
git clone https://github.com/facebookresearch/spider.git
|
||||
cd spider
|
||||
|
||||
uv sync
|
||||
```
|
||||
|
||||
### Clone Example Datasets
|
||||
|
||||
```bash
|
||||
# Install git-lfs
|
||||
sudo apt install git-lfs # Ubuntu/Debian
|
||||
# brew install git-lfs # macOS
|
||||
|
||||
git lfs install
|
||||
git clone https://huggingface.co/datasets/retarget/retarget_example example_datasets
|
||||
```
|
||||
|
||||
## Option 2: Using Conda
|
||||
|
||||
If you prefer conda for environment management:
|
||||
|
||||
```bash
|
||||
# Clone the repository
|
||||
git clone https://github.com/facebookresearch/spider.git
|
||||
cd spider
|
||||
|
||||
# Create conda environment
|
||||
conda create -n spider python=3.12
|
||||
conda activate spider
|
||||
|
||||
# Install dependencies
|
||||
python -m pip install --upgrade pip
|
||||
python -m pip install -r requirements.txt
|
||||
python -m pip install --no-deps -e .
|
||||
```
|
||||
|
||||
## Verify Installation
|
||||
|
||||
Test your installation by running a simple example:
|
||||
|
||||
```bash
|
||||
# With uv
|
||||
uv run examples/run_mjwp.py
|
||||
|
||||
# With conda
|
||||
python examples/run_mjwp.py
|
||||
```
|
||||
|
||||
If everything is installed correctly, you should see the MuJoCo viewer open with a retargeting simulation.
|
||||
|
||||
## Optional: Install Additional Backends
|
||||
|
||||
### DexMachina (Genesis Simulator)
|
||||
|
||||
For RL training with Genesis simulator:
|
||||
|
||||
```bash
|
||||
# Follow DexMachina installation instructions
|
||||
# https://mandizhao.github.io/dexmachina-docs/0_install.html
|
||||
|
||||
conda activate dexmachina
|
||||
pip install --ignore-requires-python --no-deps -e .
|
||||
```
|
||||
|
||||
### HDMI (Humanoid RL)
|
||||
|
||||
For humanoid robot RL training:
|
||||
|
||||
```bash
|
||||
# Clone and install HDMI
|
||||
git clone https://github.com/lecar-lab/hdmi.git
|
||||
cd hdmi
|
||||
|
||||
# Install SPIDER in HDMI environment
|
||||
uv pip install --no-deps -e ../spider
|
||||
```
|
||||
|
||||
### IDE Setup
|
||||
|
||||
For VSCode, recommended extensions:
|
||||
- Python
|
||||
- Pylance
|
||||
- Ruff
|
||||
|
||||
You can load the `.vscode/settings.json` file to your VSCode to get the recommended settings.
|
||||
+281
@@ -0,0 +1,281 @@
|
||||
# Quick Start
|
||||
|
||||
This guide will help you run your first SPIDER retargeting example in minutes.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
Make sure you have completed the [installation](/guide/installation) and cloned the example datasets.
|
||||
|
||||
## Run Preprocessed Example
|
||||
|
||||
The quickest way to see SPIDER in action is to run a preprocessed example:
|
||||
|
||||
```bash
|
||||
# Run with default config (Xhand bimanual, OakInk dataset)
|
||||
uv run examples/run_mjwp.py
|
||||
```
|
||||
|
||||
This will:
|
||||
1. Load preprocessed kinematic trajectory
|
||||
2. Optimize trajectory with physics constraints
|
||||
3. Display real-time visualization in MuJoCo viewer
|
||||
4. Save optimized trajectory and video
|
||||
|
||||
## Full Workflow Example For Dexterous Hand
|
||||
|
||||
To process a task from scratch, follow these steps:
|
||||
|
||||
### 1. Set Task Parameters
|
||||
|
||||
```bash
|
||||
export TASK=p36-tea
|
||||
export HAND_TYPE=bimanual
|
||||
export DATA_ID=10
|
||||
export ROBOT_TYPE=xhand
|
||||
export DATASET_NAME=gigahand
|
||||
```
|
||||
|
||||
### 2. Process Dataset
|
||||
|
||||
Convert raw human motion data to standardized format:
|
||||
|
||||
```bash
|
||||
uv run spider/process_datasets/${DATASET_NAME}.py \
|
||||
--task=${TASK} \
|
||||
--embodiment-type=${HAND_TYPE} \
|
||||
--data-id=${DATA_ID}
|
||||
```
|
||||
|
||||
### 3. Decompose Object Mesh
|
||||
|
||||
Create convex decomposition for physics simulation:
|
||||
|
||||
```bash
|
||||
uv run spider/preprocess/decompose_fast.py \
|
||||
--task=${TASK} \
|
||||
--dataset-name=${DATASET_NAME} \
|
||||
--data-id=${DATA_ID} \
|
||||
--embodiment-type=${HAND_TYPE}
|
||||
```
|
||||
|
||||
### 4. Detect Contacts (Optional)
|
||||
|
||||
Identify hand-object contact points:
|
||||
|
||||
```bash
|
||||
uv run spider/preprocess/detect_contact.py \
|
||||
--task=${TASK} \
|
||||
--dataset-name=${DATASET_NAME} \
|
||||
--data-id=${DATA_ID} \
|
||||
--embodiment-type=${HAND_TYPE}
|
||||
```
|
||||
|
||||
### 5. Generate Scene
|
||||
|
||||
Create MuJoCo scene XML:
|
||||
|
||||
```bash
|
||||
uv run spider/preprocess/generate_xml.py \
|
||||
--task=${TASK} \
|
||||
--dataset-name=${DATASET_NAME} \
|
||||
--data-id=${DATA_ID} \
|
||||
--embodiment-type=${HAND_TYPE} \
|
||||
--robot-type=${ROBOT_TYPE}
|
||||
```
|
||||
|
||||
### 6. Run Inverse Kinematics
|
||||
|
||||
Convert human poses to robot joint angles:
|
||||
|
||||
```bash
|
||||
uv run spider/preprocess/ik.py \
|
||||
--task=${TASK} \
|
||||
--dataset-name=${DATASET_NAME} \
|
||||
--data-id=${DATA_ID} \
|
||||
--embodiment-type=${HAND_TYPE} \
|
||||
--robot-type=${ROBOT_TYPE} \
|
||||
--open-hand
|
||||
```
|
||||
|
||||
### 7. Physics-Based Retargeting
|
||||
|
||||
Optimize trajectory with physics constraints:
|
||||
|
||||
```bash
|
||||
uv run examples/run_mjwp.py \
|
||||
task=${TASK} \
|
||||
dataset_name=${DATASET_NAME} \
|
||||
data_id=${DATA_ID} \
|
||||
robot_type=${ROBOT_TYPE} \
|
||||
embodiment_type=${HAND_TYPE}
|
||||
```
|
||||
|
||||
## Full Workflow Example For Humanoid Robot
|
||||
|
||||
Set environment variables:
|
||||
|
||||
```bash
|
||||
# humanoid only
|
||||
export TASK=dance
|
||||
export HAND_TYPE=humanoid
|
||||
export DATA_ID=0
|
||||
export ROBOT_TYPE=unitree_g1
|
||||
export DATASET_NAME=lafan
|
||||
|
||||
# humanoid + object
|
||||
export TASK=move_largebox
|
||||
export HAND_TYPE=humanoid_object
|
||||
export DATA_ID=0
|
||||
export ROBOT_TYPE=unitree_g1
|
||||
export DATASET_NAME=omomo
|
||||
```
|
||||
|
||||
### Run IK
|
||||
|
||||
```bash
|
||||
# with GMR (remember to generate GMR data trajectoru_gmr.pkl first with their official code. )
|
||||
uv run spider/process_datasets/gmr.py \
|
||||
--task=${TASK} \
|
||||
--dataset-name=${DATASET_NAME} \
|
||||
--data-id=${DATA_ID} \
|
||||
--robot-type=${ROBOT_TYPE} \
|
||||
--embodiment-type=${HAND_TYPE} \
|
||||
--contact-detection-mode=one
|
||||
|
||||
# with locomujoco
|
||||
uv run spider/process_datasets/locomujoco.py \
|
||||
--task=${TASK} \
|
||||
--dataset-name=${DATASET_NAME} \
|
||||
--data-id=${DATA_ID} \
|
||||
--robot-type=${ROBOT_TYPE} \
|
||||
--embodiment-type=${HAND_TYPE}
|
||||
|
||||
# for humanoid + object, we don't have a pipeline yet
|
||||
# you can directly download data from omniretarget
|
||||
# https://huggingface.co/datasets/omniretarget/OmniRetarget_Dataset
|
||||
```
|
||||
|
||||
### Run Physics-Based Retargeting
|
||||
|
||||
```bash
|
||||
# for humanoid only
|
||||
uv run examples/run_mjwp.py \
|
||||
+override=humanoid \
|
||||
dataset_name=${DATASET_NAME} \
|
||||
task=${TASK} \
|
||||
data_id=${DATA_ID} \
|
||||
robot_type=${ROBOT_TYPE} \
|
||||
embodiment_type=${HAND_TYPE}
|
||||
|
||||
# for humanoid + object, we don't have a pipeline yet
|
||||
uv run examples/run_mjwp.py \
|
||||
+override=humanoid_object \
|
||||
dataset_name=${DATASET_NAME} \
|
||||
task=${TASK} \
|
||||
data_id=${DATA_ID} \
|
||||
robot_type=${ROBOT_TYPE} \
|
||||
embodiment_type=${HAND_TYPE}
|
||||
```
|
||||
|
||||
## Understanding the Output
|
||||
|
||||
After running, you'll find these files in the output directory:
|
||||
|
||||
```
|
||||
example_datasets/processed/oakink/allegro/bimanual/pick_spoon_bowl/0/
|
||||
├── trajectory_kinematic.npz # IK result
|
||||
├── trajectory_mjwp.npz # Optimized trajectory
|
||||
├── visualization_mjwp.mp4 # Video output
|
||||
└── metrics.json # Success metrics
|
||||
```
|
||||
|
||||
### Trajectory Data Format
|
||||
|
||||
The NPZ files contain:
|
||||
|
||||
```python
|
||||
import numpy as np
|
||||
|
||||
data = np.load('trajectory_mjwp.npz')
|
||||
|
||||
# Available keys:
|
||||
# - qpos: [T, nq] joint positions
|
||||
# - qvel: [T, nv] joint velocities
|
||||
# - ctrl: [T, nu] control commands
|
||||
# - time: [T] timestamps
|
||||
```
|
||||
|
||||
## Customize Parameters
|
||||
|
||||
Override configuration parameters from command line:
|
||||
|
||||
```bash
|
||||
# Increase optimization samples
|
||||
uv run examples/run_mjwp.py num_samples=2048
|
||||
|
||||
# Change planning horizon
|
||||
uv run examples/run_mjwp.py horizon=2.0
|
||||
|
||||
# Use different viewer
|
||||
uv run examples/run_mjwp.py viewer=rerun
|
||||
|
||||
# Combine multiple overrides
|
||||
uv run examples/run_mjwp.py \
|
||||
robot_type=inspire \
|
||||
num_samples=2048 \
|
||||
horizon=2.0 \
|
||||
viewer=mujoco-rerun
|
||||
```
|
||||
|
||||
## Remote Development
|
||||
|
||||
For headless servers, use Rerun viewer:
|
||||
|
||||
```bash
|
||||
# On remote server, start Rerun server
|
||||
uv run rerun --serve-web --port 9876
|
||||
|
||||
# Run SPIDER with Rerun viewer
|
||||
uv run examples/run_mjwp.py viewer=rerun
|
||||
|
||||
# On local machine, open browser to:
|
||||
# http://your-server:9876
|
||||
```
|
||||
|
||||
## Common Tasks
|
||||
|
||||
### Run Different Robot
|
||||
|
||||
```bash
|
||||
# Allegro hand
|
||||
uv run examples/run_mjwp.py robot_type=allegro
|
||||
|
||||
# Inspire hand
|
||||
uv run examples/run_mjwp.py robot_type=inspire
|
||||
|
||||
# Xhand
|
||||
uv run examples/run_mjwp.py robot_type=xhand
|
||||
```
|
||||
|
||||
### Run Different Dataset
|
||||
|
||||
```bash
|
||||
# GigaHand
|
||||
uv run examples/run_mjwp.py +override=gigahand
|
||||
|
||||
# Hot3D
|
||||
uv run examples/run_mjwp.py +override=hot3d
|
||||
|
||||
# Humanoid task
|
||||
uv run examples/run_mjwp.py +override=humanoid
|
||||
```
|
||||
|
||||
### Single Hand Configuration
|
||||
|
||||
```bash
|
||||
# Right hand only
|
||||
uv run examples/run_mjwp.py embodiment_type=right
|
||||
|
||||
# Left hand only
|
||||
uv run examples/run_mjwp.py embodiment_type=left
|
||||
```
|
||||
Reference in New Issue
Block a user