Initial snapshot: hand motion pipeline (Dyn-HaMR + dex-retargeting + SPIDER)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
+389
@@ -0,0 +1,389 @@
|
||||
# Data Structure
|
||||
|
||||
SPIDER uses a structured file organization to handle multiple datasets, robots, and tasks efficiently.
|
||||
|
||||
## Directory Layout
|
||||
|
||||
```
|
||||
example_datasets/
|
||||
├── raw/ # Raw human motion data
|
||||
│ └── {dataset_name}/
|
||||
│ ├── {data_id}.pkl
|
||||
│ └── meshes/
|
||||
│
|
||||
└── processed/ # Processed robot trajectories
|
||||
└── {dataset_name}/
|
||||
├── dataset_summary.json # High-level metadata
|
||||
├── assets/ # Shared assets (avoid duplication)
|
||||
│ ├── objects/
|
||||
│ └── robots/
|
||||
├── mano/ # Extracted human hand data
|
||||
│ └── {embodiment_type}/
|
||||
│ └── {task}/
|
||||
│ └── {data_id}/
|
||||
│ ├── trajectory_keypoint.npz
|
||||
│ └── info.json
|
||||
└── {robot_type}/ # Robot-specific data
|
||||
└── {embodiment_type}/
|
||||
└── {task}/
|
||||
└── {data_id}/
|
||||
├── scene.xml
|
||||
├── trajectory_kinematic.npz
|
||||
├── trajectory_mjwp.npz
|
||||
├── visualization_mjwp.mp4
|
||||
└── metrics.json
|
||||
```
|
||||
|
||||
## Path Resolution
|
||||
|
||||
SPIDER automatically resolves paths based on task parameters:
|
||||
|
||||
```python
|
||||
from spider.io import get_processed_data_dir
|
||||
|
||||
# Resolve processed directory
|
||||
processed_dir = get_processed_data_dir(
|
||||
dataset_dir="/path/to/datasets",
|
||||
dataset_name="oakink",
|
||||
robot_type="allegro",
|
||||
embodiment_type="bimanual",
|
||||
task="pick_spoon_bowl",
|
||||
data_id=0
|
||||
)
|
||||
# Returns: /path/to/datasets/processed/oakink/allegro/bimanual/pick_spoon_bowl/0/
|
||||
```
|
||||
|
||||
## File Formats
|
||||
|
||||
### Raw Data (`raw/{dataset_name}/{data_id}.pkl`)
|
||||
|
||||
Raw human motion data from dataset processing:
|
||||
|
||||
```python
|
||||
{
|
||||
'qpos_wrist_right': np.ndarray, # [T, 7] position + quaternion
|
||||
'qpos_finger_right': np.ndarray, # [T, n_joints] joint angles
|
||||
'qpos_wrist_left': np.ndarray, # [T, 7]
|
||||
'qpos_finger_left': np.ndarray, # [T, n_joints]
|
||||
'qpos_obj_right': np.ndarray, # [T, 7] object pose
|
||||
'qpos_obj_left': np.ndarray, # [T, 7]
|
||||
'contact': np.ndarray, # [T, ncon] contact indicators
|
||||
'contact_pos': np.ndarray # [T, ncon, 3] contact positions
|
||||
}
|
||||
```
|
||||
|
||||
### Trajectory Files (`.npz`)
|
||||
|
||||
#### `trajectory_keypoint.npz`
|
||||
|
||||
Human hand keypoints (from dataset processor):
|
||||
|
||||
```python
|
||||
{
|
||||
'qpos_wrist_right': [T, 7],
|
||||
'qpos_finger_right': [T, 15],
|
||||
'qpos_obj_right': [T, 7],
|
||||
# ... left hand equivalents
|
||||
}
|
||||
```
|
||||
|
||||
#### `trajectory_kinematic.npz`
|
||||
|
||||
Kinematic retargeting result (from IK):
|
||||
|
||||
```python
|
||||
{
|
||||
'qpos': [T, nq], # Full robot configuration
|
||||
'qvel': [T, nv], # Velocities
|
||||
'ctrl': [T, nu], # Control targets
|
||||
'contact': [T, ncon], # Contact indicators
|
||||
'contact_pos': [T, ncon, 3] # Contact positions
|
||||
}
|
||||
```
|
||||
|
||||
#### `trajectory_mjwp.npz`
|
||||
|
||||
Physics-optimized trajectory (from MJWP):
|
||||
|
||||
```python
|
||||
{
|
||||
'qpos': [N_steps, ctrl_steps, nq], # Per-step configs
|
||||
'qvel': [N_steps, ctrl_steps, nv], # Velocities
|
||||
'ctrl': [N_steps, ctrl_steps, nu], # Control commands
|
||||
'time': [N_steps, ctrl_steps], # Timestamps
|
||||
'opt_steps': [N_steps], # Iterations per step
|
||||
'improvement': [N_steps], # Convergence metric
|
||||
'rewards': [N_steps, num_samples], # Per-sample rewards
|
||||
}
|
||||
```
|
||||
|
||||
### Scene Files
|
||||
|
||||
#### `scene.xml`
|
||||
|
||||
MuJoCo scene definition:
|
||||
|
||||
```xml
|
||||
<mujoco model="scene">
|
||||
<!-- Solver settings -->
|
||||
<option timestep="0.01" iterations="20" ...>
|
||||
|
||||
<!-- Robot model -->
|
||||
<include file="../../assets/robots/allegro/bimanual.xml"/>
|
||||
|
||||
<!-- Object geometries -->
|
||||
<body name="object_right" pos="0 0 0">
|
||||
<freejoint/>
|
||||
<geom type="mesh" mesh="obj_visual"/>
|
||||
<geom type="mesh" mesh="obj_collision_0" .../>
|
||||
...
|
||||
</body>
|
||||
|
||||
<!-- Contact pairs -->
|
||||
<contact>
|
||||
<pair geom1="finger_tip" geom2="obj_collision_0"/>
|
||||
...
|
||||
</contact>
|
||||
</mujoco>
|
||||
```
|
||||
|
||||
#### `scene_eq.xml`
|
||||
|
||||
Scene with equality constraints (for `num_dyn > 1`):
|
||||
|
||||
Includes additional `<equality>` elements for virtual contact constraints used in annealing.
|
||||
|
||||
### Metadata Files
|
||||
|
||||
#### `task_info.json`
|
||||
|
||||
Task-level metadata:
|
||||
|
||||
```json
|
||||
{
|
||||
"dataset_name": "oakink",
|
||||
"task": "pick_spoon_bowl",
|
||||
"embodiment_type": "bimanual",
|
||||
"ref_dt": 0.02,
|
||||
"object_names": ["spoon", "bowl"],
|
||||
"num_frames": 250
|
||||
}
|
||||
```
|
||||
|
||||
#### `info.json`
|
||||
|
||||
Trial-level metadata:
|
||||
|
||||
```json
|
||||
{
|
||||
"robot_type": "allegro",
|
||||
"data_id": 0,
|
||||
"timestamp": "2025-01-15T10:30:00",
|
||||
"object_asset_paths": {
|
||||
"spoon": "../../../../assets/objects/spoon/visual.obj",
|
||||
"bowl": "../../../../assets/objects/bowl/visual.obj"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
#### `metrics.json`
|
||||
|
||||
Performance metrics:
|
||||
|
||||
```json
|
||||
{
|
||||
"success_rate": 0.95,
|
||||
"avg_position_error": 0.012,
|
||||
"avg_rotation_error": 0.08,
|
||||
"avg_optimization_time": 0.234,
|
||||
"total_simulation_steps": 300
|
||||
}
|
||||
```
|
||||
|
||||
## Asset Organization
|
||||
|
||||
### Object Assets
|
||||
|
||||
Shared object meshes to avoid duplication:
|
||||
|
||||
```
|
||||
assets/objects/{object_name}/
|
||||
├── visual.obj # High-res visual mesh
|
||||
└── convex/ # Convex decomposition
|
||||
├── 0.obj
|
||||
├── 1.obj
|
||||
├── 2.obj
|
||||
└── ...
|
||||
```
|
||||
|
||||
### Robot Assets
|
||||
|
||||
Robot MJCF files:
|
||||
|
||||
```
|
||||
assets/robots/{robot_type}/
|
||||
├── left.xml # Left hand only
|
||||
├── right.xml # Right hand only
|
||||
└── bimanual.xml # Both hands
|
||||
```
|
||||
|
||||
## Loading Data
|
||||
|
||||
### In Python
|
||||
|
||||
```python
|
||||
import numpy as np
|
||||
from spider.io import load_data
|
||||
from spider.config import Config, process_config
|
||||
|
||||
# Create and process config
|
||||
config = Config(
|
||||
dataset_name="oakink",
|
||||
robot_type="allegro",
|
||||
embodiment_type="bimanual",
|
||||
task="pick_spoon_bowl",
|
||||
data_id=0
|
||||
)
|
||||
config = process_config(config)
|
||||
|
||||
# Load trajectory
|
||||
qpos, qvel, ctrl, contact, contact_pos = load_data(
|
||||
config,
|
||||
config.data_path
|
||||
)
|
||||
|
||||
print(f"Trajectory length: {qpos.shape[0]}")
|
||||
print(f"DOF: {qpos.shape[1]}")
|
||||
```
|
||||
|
||||
### Manual Loading
|
||||
|
||||
```python
|
||||
import numpy as np
|
||||
|
||||
# Load trajectory
|
||||
data = np.load('trajectory_mjwp.npz')
|
||||
|
||||
# Access arrays
|
||||
qpos = data['qpos'] # [T, nq]
|
||||
qvel = data['qvel'] # [T, nv]
|
||||
ctrl = data['ctrl'] # [T, nu]
|
||||
|
||||
# Reshape if needed (MJWP saves per-step)
|
||||
if qpos.ndim == 3:
|
||||
qpos = qpos.reshape(-1, qpos.shape[-1])
|
||||
```
|
||||
|
||||
## Data Flow Through Pipeline
|
||||
|
||||
```mermaid
|
||||
graph TD
|
||||
A[Raw Data<br/>raw/dataset/data.pkl] --> B[Dataset Processor]
|
||||
B --> C[Keypoints<br/>mano/.../trajectory_keypoint.npz]
|
||||
C --> D[Preprocessing Pipeline]
|
||||
D --> E[Object Decomposition<br/>assets/objects/*/convex/*.obj]
|
||||
D --> F[Scene XML<br/>robot/.../scene.xml]
|
||||
D --> G[Kinematic Trajectory<br/>robot/.../trajectory_kinematic.npz]
|
||||
G --> H[Physics Optimization]
|
||||
H --> I[Optimized Trajectory<br/>robot/.../trajectory_mjwp.npz]
|
||||
H --> J[Video<br/>robot/.../visualization_mjwp.mp4]
|
||||
|
||||
style C fill:#e1f5ff
|
||||
style G fill:#fff4e1
|
||||
style I fill:#ffe1e1
|
||||
```
|
||||
|
||||
## Configuration Paths
|
||||
|
||||
Paths in `Config` are automatically resolved:
|
||||
|
||||
```python
|
||||
@dataclass
|
||||
class Config:
|
||||
dataset_dir: str = "example_datasets" # Base directory
|
||||
dataset_name: str = "oakink"
|
||||
robot_type: str = "allegro"
|
||||
embodiment_type: str = "bimanual"
|
||||
task: str = "pick_spoon_bowl"
|
||||
data_id: int = 0
|
||||
|
||||
# Auto-set by process_config():
|
||||
model_path: str = "" # Path to scene.xml
|
||||
data_path: str = "" # Path to trajectory_kinematic.npz
|
||||
output_dir: str = "" # Where to save results
|
||||
```
|
||||
|
||||
After `process_config()`:
|
||||
|
||||
```python
|
||||
config.model_path = "{dataset_dir}/processed/{dataset_name}/{embodiment_type}/{task}/{data_id}/scene.xml"
|
||||
config.data_path = "{...}/{robot_type}/{embodiment_type}/{task}/{data_id}/trajectory_kinematic.npz"
|
||||
config.output_dir = "{...}/{robot_type}/{embodiment_type}/{task}/{data_id}/"
|
||||
```
|
||||
|
||||
## Best Practices
|
||||
|
||||
### Organization
|
||||
|
||||
1. **Keep raw data separate**: Never modify `raw/` directory
|
||||
2. **Share assets**: Use `assets/` for common objects/robots
|
||||
3. **One trial per directory**: Each `{data_id}` is self-contained
|
||||
4. **Relative paths in JSON**: Use relative paths for portability
|
||||
|
||||
### Naming Conventions
|
||||
|
||||
- **Tasks**: Use descriptive names (e.g., `pick_cup`, `open_jar`)
|
||||
- **Objects**: Match names across datasets
|
||||
- **Robots**: Use official model names (e.g., `allegro`, `inspire`)
|
||||
|
||||
### Cleanup
|
||||
|
||||
Remove intermediate files after successful optimization:
|
||||
|
||||
```bash
|
||||
# Keep only final results
|
||||
rm trajectory_kinematic.npz
|
||||
rm scene.xml # If not needed for replay
|
||||
```
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Path Not Found
|
||||
|
||||
Check that all path components match:
|
||||
|
||||
```bash
|
||||
# Verify structure
|
||||
ls example_datasets/processed/oakink/allegro/bimanual/pick_spoon_bowl/0/
|
||||
|
||||
# Check config values
|
||||
python -c "from spider.config import Config, process_config; \
|
||||
c = process_config(Config()); print(c.output_dir)"
|
||||
```
|
||||
|
||||
### Missing Assets
|
||||
|
||||
Ensure assets are in shared location:
|
||||
|
||||
```bash
|
||||
# Check object assets
|
||||
ls example_datasets/processed/oakink/assets/objects/
|
||||
|
||||
# Check robot assets
|
||||
ls example_datasets/processed/oakink/assets/robots/
|
||||
```
|
||||
|
||||
### Corrupted NPZ Files
|
||||
|
||||
Validate NPZ integrity:
|
||||
|
||||
```python
|
||||
import numpy as np
|
||||
|
||||
try:
|
||||
data = np.load('trajectory.npz')
|
||||
print("Keys:", list(data.keys()))
|
||||
print("Shapes:", {k: v.shape for k, v in data.items()})
|
||||
except Exception as e:
|
||||
print(f"Error loading: {e}")
|
||||
```
|
||||
+11
@@ -0,0 +1,11 @@
|
||||
# Deployment to Real Robots
|
||||
|
||||
This guide covers deploying SPIDER-optimized trajectories to real-world robots, including coordinate system transformations and practical deployment considerations.
|
||||
|
||||
## Overview
|
||||
|
||||
SPIDER saves all trajectories in a **minimum coordinate system** defined in the corresponding scene XML file. While joint positions can be used directly, it's important to convert the robot base position and orientation to the world coordinate system for correct execution on real robots.
|
||||
|
||||
Please refer to `spider/postprocess/read_to_robot.py` for the conversion code. It is recommended to read from a site from the robot to get wrist orientation that align with your real world setup.
|
||||
|
||||
|
||||
+266
@@ -0,0 +1,266 @@
|
||||
# Parameter Tuning
|
||||
|
||||
This guide helps you tune SPIDER parameters for optimal performance on your tasks.
|
||||
|
||||
## Overview
|
||||
|
||||
Like reinforcement learning, different motion characteristics require different parameter settings. Start with defaults and iteratively refine based on results.
|
||||
|
||||
## Control Parameters
|
||||
|
||||
### Simulation Timestep (`sim_dt`)
|
||||
|
||||
**Default**: `0.01` (100 Hz)
|
||||
|
||||
The fundamental simulation timestep. Smaller values increase accuracy but slow down simulation. Start from a smaller one and gradually increase for faster simulation.
|
||||
|
||||
::: warning
|
||||
All other timing parameters must be divisible by `sim_dt`
|
||||
:::
|
||||
|
||||
### Control Interval (`ctrl_dt`)
|
||||
|
||||
**Default**: `0.4s` (2.5 Hz)
|
||||
|
||||
How frequently the controller updates. Affects:
|
||||
- Optimization speed (larger = fewer optimizations)
|
||||
- Responsiveness (smaller = more reactive)
|
||||
|
||||
**Tuning guidelines:**
|
||||
- **Agile motions**: `0.05-0.2s`
|
||||
- **Smooth motions**: `0.2-0.4s`
|
||||
|
||||
### Planning Horizon (`horizon`)
|
||||
|
||||
**Default**: `1.6s`
|
||||
|
||||
**Most important parameter**. How far ahead the optimizer looks.
|
||||
|
||||
**Impact:**
|
||||
- ✅ Larger horizon: Better global plan, handles complex motions
|
||||
- ❌ Larger horizon: Slower optimization, requires more samples
|
||||
- ✅ Smaller horizon: Faster optimization
|
||||
- ❌ Smaller horizon: Myopic behavior, may fail on complex tasks
|
||||
|
||||
**Tuning strategy:**
|
||||
1. Start large (2.0s) for complex/unknown tasks
|
||||
2. Gradually decrease until performance degrades
|
||||
3. Monitor planned trajectories in Rerun viewer
|
||||
|
||||
```bash
|
||||
# Complex manipulation
|
||||
uv run examples/run_mjwp.py horizon=2.0
|
||||
|
||||
# Simple tracking
|
||||
uv run examples/run_mjwp.py horizon=1.0
|
||||
```
|
||||
|
||||
::: tip Visual Debugging
|
||||
Use Rerun viewer to see the planned trajectory (blue/red traces). If the plan looks good but execution fails, the horizon might be too short.
|
||||
:::
|
||||
|
||||
### Knot Spacing (`knot_dt`)
|
||||
|
||||
**Default**: `0.4s`
|
||||
|
||||
Temporal resolution of control parameterization. Actions are interpolated between knot points.
|
||||
|
||||
**Purpose**: Reduces search space by smoothing actions.
|
||||
|
||||
**Guidelines:**
|
||||
- **Agile motions** (jumping, quick grasps): `0.1-0.2s`
|
||||
- **Smooth motions** (walking, pouring): `0.2-0.4s`
|
||||
|
||||
```bash
|
||||
# Fine control for agile task
|
||||
uv run examples/run_mjwp.py knot_dt=0.1
|
||||
|
||||
# Smooth interpolation
|
||||
uv run examples/run_mjwp.py knot_dt=0.3
|
||||
```
|
||||
|
||||
## Sampling Parameters
|
||||
|
||||
### Number of Samples (`num_samples`)
|
||||
|
||||
**Default**: `1024`
|
||||
|
||||
Number of parallel trajectories to evaluate.
|
||||
|
||||
**Impact:**
|
||||
- ✅ More samples: Better exploration, more robust
|
||||
- ❌ More samples: Higher memory usage, slower per-iteration
|
||||
- Scales linearly with GPU memory and compute
|
||||
|
||||
**Tuning:** try it out on your GPU and pick up the largest number that doesn't cause significant slow down.
|
||||
|
||||
### Temperature (`temperature`)
|
||||
|
||||
**Default**: `0.1`
|
||||
|
||||
Softmax temperature for sample weighting.
|
||||
|
||||
**Impact:**
|
||||
- **Lower** (0.01-0.05): Sharper distribution, more optimal but can be unstable
|
||||
- **Higher** (0.2-0.5): Smoother distribution, more stable but less optimal
|
||||
|
||||
**Tuning strategy:**
|
||||
1. Start at `0.1`
|
||||
2. If motion is shaky/unstable, increase to `0.15-0.2`
|
||||
3. If motion is stable, decrease to `0.05-0.08` for better quality
|
||||
|
||||
Generally, this parameter is less important.
|
||||
|
||||
```bash
|
||||
# Stable but less optimal
|
||||
uv run examples/run_mjwp.py temperature=0.2
|
||||
|
||||
# Optimal but may be shaky
|
||||
uv run examples/run_mjwp.py temperature=0.05
|
||||
```
|
||||
|
||||
::: tip Monitoring
|
||||
Check the `ctrl` panel in rerun viewer. If the action is shaky, increase the temperature.
|
||||
:::
|
||||
|
||||
### Max Iterations (`max_num_iterations`)
|
||||
|
||||
**Default**: `32`
|
||||
|
||||
Maximum optimization iterations per control step.
|
||||
|
||||
**Tuning:**
|
||||
1. Run with default and monitor `improvement` in Rerun
|
||||
2. If `improvement` plateaus early (< 10 iters), decrease
|
||||
3. If `improvement` is still high at end, increase
|
||||
|
||||
```bash
|
||||
# Quick convergence
|
||||
uv run examples/run_mjwp.py max_num_iterations=8
|
||||
|
||||
# Thorough optimization
|
||||
uv run examples/run_mjwp.py max_num_iterations=64
|
||||
```
|
||||
|
||||
### Early Stopping (`improvement_threshold`)
|
||||
|
||||
**Default**: `0.01`
|
||||
|
||||
Stop optimization if improvement falls below threshold.
|
||||
|
||||
**Usage:**
|
||||
- Start at `0` (disabled) to see full convergence
|
||||
- Gradually increase for speed: `0.01` → `0.02` → `0.05`
|
||||
|
||||
```bash
|
||||
# No early stopping
|
||||
uv run examples/run_mjwp.py improvement_threshold=0.0
|
||||
|
||||
# Aggressive early stopping
|
||||
uv run examples/run_mjwp.py improvement_threshold=0.05
|
||||
```
|
||||
|
||||
## Noise Scheduling
|
||||
|
||||
### Control Noise Scales
|
||||
|
||||
These control exploration throughout the horizon:
|
||||
|
||||
| Parameter | Default | Description |
|
||||
|-----------|---------|-------------|
|
||||
| `first_ctrl_noise_scale` | `0.5` | Noise at start of horizon |
|
||||
| `last_ctrl_noise_scale` | `1.0` | Noise at end of horizon |
|
||||
| `final_noise_scale` | `0.1` | Global annealing factor |
|
||||
|
||||
**Tuning:**
|
||||
- Motion is stable? Decrease `first_ctrl_noise_scale` to `0.01-0.1`
|
||||
- Need more exploration? Increase `last_ctrl_noise_scale` to `0.1-1.0`
|
||||
- High precision task? Decrease `final_noise_scale` to `0.05`
|
||||
|
||||
### Component-Specific Scales
|
||||
|
||||
For dexterous hands, noise is applied differently to each component:
|
||||
|
||||
| Parameter | Default | Purpose |
|
||||
|-----------|---------|---------|
|
||||
| `joint_noise_scale` | `0.3` | Robot joint angles |
|
||||
| `pos_noise_scale` | `0.01` | Hand base position |
|
||||
| `rot_noise_scale` | `0.03` | Hand base rotation |
|
||||
|
||||
**Tuning guidelines:**
|
||||
- Base is stable? Decrease `pos_noise_scale` and `rot_noise_scale`
|
||||
- Finger control needs more exploration? Increase `joint_noise_scale`
|
||||
|
||||
```bash
|
||||
# Stable base, explore fingers
|
||||
uv run examples/run_mjwp.py \
|
||||
pos_noise_scale=0.005 \
|
||||
rot_noise_scale=0.01 \
|
||||
joint_noise_scale=0.5
|
||||
```
|
||||
|
||||
## Reward Scaling
|
||||
|
||||
Balance different tracking objectives:
|
||||
|
||||
### Position vs Rotation
|
||||
|
||||
```yaml
|
||||
pos_rew_scale: 1.0 # End-effector position
|
||||
rot_rew_scale: 0.3 # End-effector rotation
|
||||
```
|
||||
|
||||
**Tuning:**
|
||||
- Task requires precise orientation (e.g., key insertion)? Increase `rot_rew_scale`
|
||||
- Position more important? Keep `rot_rew_scale` low
|
||||
|
||||
### Joint Tracking
|
||||
|
||||
```yaml
|
||||
joint_rew_scale: 0.003
|
||||
```
|
||||
|
||||
Lower weight since joint tracking is less critical than end-effector pose.
|
||||
|
||||
**Tuning:**
|
||||
- Specific joint configuration needed? Increase to `0.01-0.03`
|
||||
- Only care about end-effector? Keep very low (`0.001-0.003`)
|
||||
|
||||
### Velocity Regularization
|
||||
|
||||
```yaml
|
||||
vel_rew_scale: 0.0001
|
||||
```
|
||||
|
||||
Penalizes large velocities to smooth motion.
|
||||
|
||||
**Best practices:**
|
||||
- ✅ Adding velocity tracking helps reduce latency
|
||||
- ❌ Don't make it too large (causes sluggish motion)
|
||||
|
||||
```bash
|
||||
# Reduce tracking latency
|
||||
uv run examples/run_mjwp.py vel_rew_scale=0.001
|
||||
|
||||
# Disable velocity penalty
|
||||
uv run examples/run_mjwp.py vel_rew_scale=0.0
|
||||
```
|
||||
|
||||
## Common Issues
|
||||
|
||||
### Motion is Unstable/Shaky
|
||||
|
||||
**Symptoms**: Robot vibrates, erratic movements, or fails to complete the task
|
||||
|
||||
**Solutions:**
|
||||
1. Iecrease `num_samples` (most effective)
|
||||
2. Check if `horizon` is too short
|
||||
3. Increase `knot_dt` for smoother actions or decrease it for more reactive motion
|
||||
|
||||
### Optimization Too Slow
|
||||
|
||||
**Symptoms**: RTR (realtime rate) < 0.1
|
||||
|
||||
**Solutions:**
|
||||
1. Reduce `horizon` (most effective)
|
||||
2. Increase `ctrl_dt` (also effective, make sure it doesn't cause instability)
|
||||
+120
@@ -0,0 +1,120 @@
|
||||
# Viewers
|
||||
|
||||
SPIDER supports multiple viewer backends for visualization during retargeting. This guide covers setting up and using different viewers, including remote development workflows.
|
||||
|
||||
## Available Viewers
|
||||
|
||||
SPIDER supports the following viewer configurations:
|
||||
|
||||
| Viewer | Description | Use Case |
|
||||
|--------|-------------|----------|
|
||||
| `mujoco` | Native MuJoCo viewer | Fast local visualization |
|
||||
| `rerun` | Rerun.io viewer | Remote development, rich logging |
|
||||
| `viser` | Viser web viewer | Lightweight remote visualization |
|
||||
|
||||
To enable both, try `mujoco-rerun` or `mujoco-viser` viewer.
|
||||
|
||||
## Viser Viewer
|
||||
|
||||
[Viser](https://github.com/nerfstudio-project/viser) provides a lightweight web-based viewer for 3D scenes.
|
||||
|
||||
### Local Usage
|
||||
|
||||
```bash
|
||||
uv run examples/run_mjwp.py viewer=viser
|
||||
```
|
||||
|
||||
This will:
|
||||
1. Start a Viser server
|
||||
2. Log simulation data in real-time
|
||||
3. Print a URL to open in your browser
|
||||
|
||||
::: warning
|
||||
If you run headless (no X display), you may need to set a display before launching:
|
||||
|
||||
```bash
|
||||
export DISPLAY=:1
|
||||
# or
|
||||
export MUJOCO_GL=egl
|
||||
```
|
||||
:::
|
||||
|
||||
::: info
|
||||
If you see an import error, install the dependency:
|
||||
|
||||
```bash
|
||||
pip install viser
|
||||
```
|
||||
:::
|
||||
|
||||
## Rerun Viewer
|
||||
|
||||
[Rerun](https://rerun.io/) provides a powerful logging and visualization system with time-travel capabilities, perfect for remote development.
|
||||
|
||||
### Local Usage
|
||||
|
||||
```bash
|
||||
uv run examples/run_mjwp.py viewer=rerun
|
||||
```
|
||||
|
||||
This will:
|
||||
1. Automatically spawn a Rerun viewer
|
||||
2. Log simulation data in real-time
|
||||
3. Display 3D visualization and metrics
|
||||
|
||||
### Remote Development
|
||||
|
||||
For headless servers, use Rerun's web viewer:
|
||||
|
||||
#### Step 1: Start Rerun Server
|
||||
|
||||
On your remote server:
|
||||
|
||||
```bash
|
||||
# Start Rerun web server
|
||||
uv run rerun --serve-web --port 9876
|
||||
|
||||
# Or use rerun datapoints for more features
|
||||
uv run rerun datapoints --serve-web --port 9876
|
||||
```
|
||||
|
||||
#### Step 2: Run SPIDER
|
||||
|
||||
In another terminal on the remote server:
|
||||
|
||||
```bash
|
||||
uv run examples/run_mjwp.py viewer=rerun
|
||||
```
|
||||
|
||||
#### Step 3: Connect from Local Machine
|
||||
|
||||
Open your local browser to:
|
||||
|
||||
```
|
||||
http://your-server:9876
|
||||
```
|
||||
|
||||
Or connect via the Rerun client:
|
||||
|
||||
```bash
|
||||
# On local machine
|
||||
rerun --connect rerun+http://your-server:9876/proxy
|
||||
```
|
||||
|
||||
::: warning
|
||||
Make sure port 9876 is accessible through your firewall. You may need to configure SSH port forwarding:
|
||||
|
||||
```bash
|
||||
ssh -L 9876:localhost:9876 user@remote-server
|
||||
```
|
||||
:::
|
||||
|
||||
### Logged Data
|
||||
|
||||
SPIDER logs the following data to Rerun:
|
||||
|
||||
- 3D scene visualization (robot and objects)
|
||||
- Joint positions and velocities
|
||||
- Optimization metrics (improvement, reward)
|
||||
- Contact forces and positions
|
||||
- Trace trajectories
|
||||
Reference in New Issue
Block a user