174 lines
5.0 KiB
Markdown
174 lines
5.0 KiB
Markdown
# Add New Dataset
|
|
|
|
This guide explains how to add a new dataset to SPIDER for retargeting human motion data to robots.
|
|
|
|
## Overview
|
|
|
|
Adding a new dataset involves:
|
|
1. Preparing raw data in standardized format
|
|
2. Creating a dataset processor script
|
|
3. Extracting hand kinematics and object poses
|
|
4. Converting meshes to MuJoCo-compatible format
|
|
5. Testing the integration
|
|
|
|
## Dataset Requirements
|
|
|
|
Your dataset should include:
|
|
- **Hand motion data**: Either MANO parameters or joint angles
|
|
- **Object information**: 3D meshes and 6D poses (position + orientation)
|
|
- **Temporal alignment**: Synchronized hand and object trajectories
|
|
|
|
Supported formats:
|
|
- MANO parameters (shape, pose)
|
|
- Direct joint angles
|
|
- Wrist poses (position + rotation)
|
|
- Object 6D poses
|
|
|
|
## Data File Structure
|
|
|
|
SPIDER uses a standardized directory structure:
|
|
|
|
```
|
|
example_datasets/
|
|
├── raw/ # Raw data from original dataset
|
|
│ └── my_dataset/
|
|
│ ├── task_name_01.pkl # Raw motion capture data
|
|
│ ├── task_name_02.pkl
|
|
│ └── meshes/ # Object meshes
|
|
│ ├── cup.obj
|
|
│ └── spoon.obj
|
|
│
|
|
└── processed/ # Processed data for SPIDER
|
|
└── my_dataset/
|
|
├── dataset_summary.json # Dataset metadata
|
|
├── assets/ # Shared assets
|
|
│ ├── objects/ # Object meshes
|
|
│ │ └── cup/
|
|
│ │ ├── convex/ # Convex decomposition
|
|
│ │ │ ├── 0.obj
|
|
│ │ │ ├── 1.obj
|
|
│ │ │ └── ...
|
|
│ │ └── visual.obj
|
|
│ └── robots/ # Robot models
|
|
│ └── allegro/
|
|
│ ├── left.xml
|
|
│ └── right.xml
|
|
└── mano/ # Processed MANO data
|
|
└── bimanual/
|
|
└── task_name/
|
|
└── 0/
|
|
├── trajectory_keypoint.npz
|
|
└── info.json
|
|
```
|
|
|
|
## Step 1: Prepare Raw Data
|
|
|
|
Place your raw data in the appropriate directory:
|
|
|
|
```bash
|
|
mkdir -p example_datasets/raw/my_dataset
|
|
# Copy your raw data files here
|
|
```
|
|
|
|
### Raw Data Format
|
|
|
|
Your raw data file (`.pkl` or `.npz`) should contain:
|
|
|
|
```python
|
|
{
|
|
# Hand data (one of the following):
|
|
'mano_pose': [...], # [T, 48] MANO pose parameters
|
|
'mano_shape': [...], # [10] or [T, 10] MANO shape parameters
|
|
# OR
|
|
'qpos_finger_left': [...], # [T, n_joints] Left finger joints
|
|
'qpos_finger_right': [...], # [T, n_joints] Right finger joints
|
|
'qpos_wrist_left': [...], # [T, 7] Left wrist pose (xyz + quat)
|
|
'qpos_wrist_right': [...], # [T, 7] Right wrist pose (xyz + quat)
|
|
|
|
# Object data:
|
|
'object_pose_left': [...], # [T, 7] Object pose (xyz + quat) for left hand
|
|
'object_pose_right': [...], # [T, 7] Object pose (xyz + quat) for right hand
|
|
'object_name_left': 'cup', # Object identifier
|
|
'object_name_right': 'spoon',
|
|
|
|
# Metadata:
|
|
'fps': 30.0, # Frame rate
|
|
'task_name': 'pick_cup', # Task identifier
|
|
}
|
|
```
|
|
|
|
## Step 2: Create Dataset Processor
|
|
|
|
Create a processor script at `spider/process_datasets/my_dataset.py`:
|
|
|
|
## Step 3: Handle Object Meshes
|
|
|
|
Convert Meshes to OBJ Format
|
|
|
|
Ensure meshes are in MuJoCo-compatible OBJ format:
|
|
|
|
```python
|
|
def convert_mesh_to_obj(input_mesh: str, output_mesh: str):
|
|
"""
|
|
Convert mesh to MuJoCo-compatible OBJ format.
|
|
|
|
Supports various input formats (STL, PLY, glb, etc.)
|
|
"""
|
|
import trimesh
|
|
|
|
# Load mesh
|
|
mesh = trimesh.load(input_mesh)
|
|
|
|
# Ensure single mesh (merge if needed)
|
|
if isinstance(mesh, trimesh.Scene):
|
|
mesh = mesh.dump(concatenate=True)
|
|
|
|
# Export as OBJ
|
|
mesh.export(output_mesh)
|
|
|
|
print(f"Converted {input_mesh} to {output_mesh}")
|
|
```
|
|
|
|
Place Meshes in Assets
|
|
|
|
Place meshes in assets folder, for example, for cup:
|
|
|
|
```bash
|
|
mkdir -p example_datasets/processed/my_dataset/assets/objects/cup
|
|
cp example_datasets/raw/my_dataset/meshes/cup.obj example_datasets/processed/my_dataset/assets/objects/cup/visual.obj
|
|
```
|
|
|
|
## Step 4: Test Your Dataset Processor
|
|
|
|
Run Processor
|
|
|
|
```bash
|
|
# Process a single sample
|
|
uv run spider/process_datasets/my_dataset.py \
|
|
--task=pick_cup \
|
|
--embodiment-type=bimanual \
|
|
--data-id=0
|
|
```
|
|
|
|
### Verify Output
|
|
|
|
Check that the output files exist and have correct format.
|
|
|
|
```python
|
|
import numpy as np
|
|
|
|
# Load processed data
|
|
data = np.load('example_datasets/processed/my_dataset/mano/bimanual/pick_cup/0/trajectory_keypoint.npz')
|
|
|
|
print("Keys:", list(data.keys()))
|
|
print("qpos_finger shape:", data['qpos_finger'].shape)
|
|
print("qpos_wrist shape:", data['qpos_wrist'].shape)
|
|
```
|
|
|
|
## Resources
|
|
|
|
- [OakInk Dataset](https://oakink.net/)
|
|
- [GigaHand Dataset](https://gigahands.github.io/)
|
|
- [Hot3D Dataset](https://github.com/facebookresearch/hot3d)
|
|
- [MANO Hand Model](https://mano.is.tue.mpg.de/)
|