Add protocols for authoring simulation environments. This is for preview only, we discourage users from using this in production code.
PiperOrigin-RevId: 897857927 Change-Id: I82ce0a23465429e7025334e83ba00dfbe5085649
This commit is contained in:
Tom Erez
+876
@@ -0,0 +1,876 @@
|
||||
---
|
||||
name: mujoco-python
|
||||
description: >
|
||||
Build, manipulate, and simulate MuJoCo physics models using the Python
|
||||
bindings (MjSpec, MjModel, MjData). Covers choosing between compute
|
||||
backends (C++ for full features and noslip, MJWarp for GPU batch RL). Use when constructing scenes
|
||||
programmatically via the spec API, compiling and stepping simulations,
|
||||
reading sensor/body/geom data, attaching sub-models, composing specs with
|
||||
prefixed names, using contact sensors for fixed-size observation spaces,
|
||||
configuring collision filtering, offscreen rendering (context management,
|
||||
cameras, depth/segmentation), or performing spatial math (quaternion, pose,
|
||||
rotation conversions via mju_). Covers gotchas around compilation
|
||||
lifecycle, named indexing vs bind, geom size semantics, camera conventions,
|
||||
and orientation representations.
|
||||
---
|
||||
|
||||
# MuJoCo Python Bindings
|
||||
|
||||
## Compilation Lifecycle
|
||||
|
||||
```
|
||||
MjSpec ──spec.compile()──▶ MjModel ──MjData(model)──▶ MjData
|
||||
│ │ │
|
||||
│ (mutable blueprint) │ (compiled, mostly frozen) │ (simulation state)
|
||||
│ │ │
|
||||
└── spec.recompile(m, d) ─────┴────────────────────────────┘
|
||||
```
|
||||
|
||||
1. **MjSpec** — mutable data structure you edit to define the simulation.
|
||||
2. **`spec.compile()`** — produces `MjModel` + you create `MjData(model)`.
|
||||
After this, changing the spec has **no effect** until you recompile.
|
||||
3. **Most `MjModel` fields are unsafe to mutate.** Changing them requires
|
||||
`spec.recompile(model, data)`, which returns **new** model and data objects
|
||||
(preserving physics state for existing elements).
|
||||
|
||||
```python
|
||||
import mujoco
|
||||
|
||||
spec = mujoco.MjSpec()
|
||||
body = spec.worldbody.add_body(pos=[0, 0, 1])
|
||||
geom = body.add_geom(type=mujoco.mjtGeom.mjGEOM_SPHERE, size=[0.1])
|
||||
body.add_freejoint()
|
||||
|
||||
model = spec.compile()
|
||||
data = mujoco.MjData(model)
|
||||
mujoco.mj_forward(model, data)
|
||||
|
||||
# Later: add another body, recompile keeping state
|
||||
body2 = spec.worldbody.add_body(pos=[1, 0, 1])
|
||||
body2.add_geom(size=[0.1])
|
||||
body2.add_freejoint()
|
||||
model, data = spec.recompile(model, data) # state preserved
|
||||
```
|
||||
|
||||
> [!CAUTION]
|
||||
> `recompile` returns **new** objects. Always reassign: `model, data = spec.recompile(model, data)`.
|
||||
|
||||
### Loading and Serializing
|
||||
|
||||
```python
|
||||
spec = mujoco.MjSpec() # empty
|
||||
spec = mujoco.MjSpec.from_string(xml_string) # from XML string
|
||||
spec = mujoco.MjSpec.from_file('/path/to.xml') # from file
|
||||
model = mujoco.MjModel.from_xml_string(xml) # direct to model (no spec)
|
||||
|
||||
xml_out = spec.to_xml() # serialize back
|
||||
```
|
||||
|
||||
### Compile Error Debugging
|
||||
|
||||
Use the `.info` field on spec elements for traceability:
|
||||
|
||||
```python
|
||||
geom = spec.worldbody.add_geom()
|
||||
geom.info = 'created at my_file.py:42'
|
||||
spec.compile() # Error: "size 0 must be positive in geom\nElement name '', id 0, created at my_file.py:42"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Compute Backends
|
||||
|
||||
MuJoCo has two compute backends. **Choose early** — the backend determines
|
||||
which features, solvers, and APIs are available.
|
||||
|
||||
| | C++ (default) | MJWarp (NVIDIA GPU) |
|
||||
|---|---|---|
|
||||
| **Import** | `import mujoco` | `import mujoco_warp as mjw` |
|
||||
| **Optimized for** | Latency (single scene) | Throughput (big batches) |
|
||||
| **Hardware** | CPU | NVIDIA GPU |
|
||||
| **Solvers** | All (Newton, CG, PGS, **noslip**) | All except PGS, **noslip**, islands |
|
||||
| **Plugins** | ✅ All | SDF only |
|
||||
| **Precision** | float64 | float32 |
|
||||
| **Named access / bind** | ✅ | Via wrapper libraries or MJX `bind()` |
|
||||
| **Contact sensors** | ✅ | ✅ |
|
||||
| **Sparse Jacobians** | ✅ | ❌ (dense only) |
|
||||
| **Batch rendering** | ❌ | ✅ (BVH ray tracing) |
|
||||
|
||||
### When to use which
|
||||
|
||||
- **C++ (default)**: Real-time control, model predictive control, interactive
|
||||
visualization, any workflow needing full feature support (noslip solver,
|
||||
PGS, islands, plugins, ellipsoidal fluid model, sparse Jacobians). Also the
|
||||
only backend with native `bind()`. Use this unless you need massive
|
||||
parallelism. (For MJWarp, named access is available via wrapper
|
||||
libraries or MJX `bind()`.)
|
||||
|
||||
- **MJWarp**: Reinforcement learning with large batch sizes on NVIDIA GPUs.
|
||||
Scales better for contact-rich scenes and large meshes than the legacy
|
||||
MJX-JAX backend. Not differentiable. May degrade for scenes beyond ~60 DoFs.
|
||||
|
||||
> [!IMPORTANT]
|
||||
> The `noslip` solver (post-constraint velocity correction for exact zero
|
||||
> slip at contacts) is **only available in the C++ backend**. If your task
|
||||
> requires accurate friction modeling without any tangential sliding at
|
||||
> contacts, you must use C++.
|
||||
|
||||
> [!WARNING]
|
||||
> MJWarp uses **float32**, which can cause numerical differences vs C++
|
||||
> (float64). Solver convergence, small friction values, and long rollouts
|
||||
> may be sensitive to this. If you see NaNs or instability on GPU, try
|
||||
> increasing solver iterations or simplifying the model.
|
||||
|
||||
---
|
||||
|
||||
## Building Models with MjSpec
|
||||
|
||||
### Adding elements
|
||||
|
||||
Most `add_*` methods accept keyword arguments matching MJCF XML attributes:
|
||||
|
||||
```python
|
||||
spec = mujoco.MjSpec()
|
||||
|
||||
body = spec.worldbody.add_body(name='arm', pos=[0, 0, 1], quat=[1, 0, 0, 0])
|
||||
geom = body.add_geom(
|
||||
name='arm_geom',
|
||||
type=mujoco.mjtGeom.mjGEOM_CAPSULE,
|
||||
size=[0.05, 0.3],
|
||||
rgba=[1, 0, 0, 1],
|
||||
)
|
||||
joint = body.add_joint(
|
||||
name='hinge1',
|
||||
type=mujoco.mjtJoint.mjJNT_HINGE,
|
||||
axis=[0, 1, 0],
|
||||
range=[-1.57, 1.57],
|
||||
)
|
||||
site = body.add_site(name='sensor_site', pos=[0, 0, 0.3])
|
||||
cam = body.add_camera(name='arm_cam', pos=[0, -2, 0], xyaxes=[1,0,0, 0,0,1])
|
||||
```
|
||||
|
||||
### Orientation alternatives
|
||||
|
||||
In addition to `quat`, you can specify orientation with `euler`, `axisangle`,
|
||||
`xyaxes`, or `zaxis`. Only one can be set at a time:
|
||||
|
||||
```python
|
||||
body.add_geom(euler=[0, 90, 0]) # Euler angles (degrees by default)
|
||||
body.add_geom(axisangle=[0, 1, 0, 1.57]) # axis + angle
|
||||
body.add_geom(zaxis=[0, 1, 0]) # minimal rotation to align Z
|
||||
body.add_geom(xyaxes=[1,0,0, 0,0,1]) # explicit X and Y axes
|
||||
```
|
||||
|
||||
### Top-level elements
|
||||
|
||||
Sensors, actuators, tendons, materials, textures, and meshes are added directly
|
||||
to the spec (not to bodies):
|
||||
|
||||
```python
|
||||
spec.add_material(name='red', rgba=[1, 0, 0, 1])
|
||||
spec.add_actuator(name='motor', joint=joint.name, gear=[1, 0, 0, 0, 0, 0])
|
||||
spec.add_sensor(
|
||||
name='joint_pos',
|
||||
type=mujoco.mjtSensor.mjSENS_JOINTPOS,
|
||||
objtype=mujoco.mjtObj.mjOBJ_JOINT,
|
||||
objname=joint.name,
|
||||
)
|
||||
```
|
||||
|
||||
> [!IMPORTANT]
|
||||
> Always use `element.name` (e.g., `joint.name`, `geom.name`, `site.name`)
|
||||
> instead of hardcoded strings when referencing spec elements. This keeps
|
||||
> references correct if the element is renamed or attached with a prefix.
|
||||
|
||||
### Geom size semantics
|
||||
|
||||
| Type | Size params |
|
||||
| --------- | ----------------------------------------------- |
|
||||
| sphere | `[radius]` |
|
||||
| capsule | `[radius, half_length]` or `[radius]` + fromto |
|
||||
| cylinder | `[radius, half_length]` or `[radius]` + fromto |
|
||||
| box | `[half_x, half_y, half_z]` |
|
||||
| ellipsoid | `[radius_x, radius_y, radius_z]` |
|
||||
| plane | `[half_x, half_y, grid_spacing]` |
|
||||
|
||||
> [!WARNING]
|
||||
> Capsule/cylinder `size` changes meaning with `fromto`. Without `fromto`,
|
||||
> `size=[radius, half_length]`. With `fromto`, `size=[radius]` only — the
|
||||
> length is computed from the two endpoints.
|
||||
|
||||
---
|
||||
|
||||
## Accessing Compiled Data: Named Access vs Bind
|
||||
|
||||
There are **two** recommended ways to read/write compiled model and data fields.
|
||||
**Prefer `bind`** when working with spec elements; use **named access** otherwise.
|
||||
|
||||
### 1. Named Access (on MjModel / MjData)
|
||||
|
||||
```python
|
||||
model.geom('my_geom').size # → numpy view of geom_size for 'my_geom'
|
||||
data.body('torso').xpos # → numpy view of body_xpos
|
||||
data.joint('knee').qpos # → shape depends on joint type
|
||||
data.actuator('motor').ctrl = 1.0 # writable view
|
||||
```
|
||||
|
||||
Aliases: `joint` / `jnt`, `camera` / `cam`, `tendon` / `ten`, `material` / `mat`,
|
||||
`texture` / `tex`, `equality` / `eq`, `keyframe` / `key`.
|
||||
|
||||
> [!WARNING]
|
||||
> Named access returns **views, not copies.** After `mj_step`, old references
|
||||
> reflect new values. Use `.copy()` when logging:
|
||||
> `positions.append(data.body('torso').xpos.copy())`
|
||||
|
||||
### 2. Bind (bridges MjSpec elements → MjModel / MjData)
|
||||
|
||||
`bind()` connects spec elements (or lists of them) to their compiled
|
||||
counterparts. **Use `.set()` to write through bind:**
|
||||
|
||||
```python
|
||||
geom = spec.worldbody.add_geom(name='ball', size=[0.1], type=mujoco.mjtGeom.mjGEOM_SPHERE)
|
||||
joint = body.add_joint(name='j1', type=mujoco.mjtJoint.mjJNT_HINGE)
|
||||
model = spec.compile()
|
||||
data = mujoco.MjData(model)
|
||||
mujoco.mj_forward(model, data)
|
||||
|
||||
# Reading via bind
|
||||
model.bind(geom).size # → array([0.1, 0., 0.])
|
||||
data.bind(geom).xpos # → array([0., 0., 0.])
|
||||
|
||||
# Writing via bind — always use .set()
|
||||
data.bind(joint).set('qpos', 1.5) # sets the joint's qpos
|
||||
|
||||
# Bind a list of spec elements
|
||||
joints = [spec.joint('j1'), spec.joint('j2')]
|
||||
data.bind(joints).qpos # → concatenated array
|
||||
data.bind(joints).set('qpos', np.array([0.5, 1.0])) # write to both
|
||||
```
|
||||
|
||||
> [!CAUTION]
|
||||
> The spec must match the compiled model. If you modify the spec after
|
||||
> `compile()`, you must recompile before calling `bind()`, or you get:
|
||||
> `ValueError: 'The mjSpec does not match mjModel. Please recompile the mjSpec.'`
|
||||
|
||||
---
|
||||
|
||||
## Attachments: Composing Specs
|
||||
|
||||
Attach child specs/bodies to parent specs via frames or sites:
|
||||
|
||||
```python
|
||||
parent = mujoco.MjSpec()
|
||||
child = mujoco.MjSpec()
|
||||
child_body = child.worldbody.add_body(name='arm')
|
||||
child_body.add_geom(name='arm_geom', size=[0.05, 0.3], type=mujoco.mjtGeom.mjGEOM_CAPSULE)
|
||||
child_body.add_joint(name='arm_joint', type=mujoco.mjtJoint.mjJNT_HINGE)
|
||||
|
||||
frame = parent.worldbody.add_frame(pos=[0, 0, 1])
|
||||
frame.attach_body(child_body, prefix='left_')
|
||||
# 'arm' → 'left_arm', 'arm_geom' → 'left_arm_geom', 'arm_joint' → 'left_arm_joint'
|
||||
|
||||
# Or attach entire spec to a site
|
||||
site = parent.worldbody.add_site(name='attach_point', pos=[0, 0, 2])
|
||||
parent.attach(child, site=site, prefix='right_', suffix='_v2')
|
||||
```
|
||||
|
||||
> [!IMPORTANT]
|
||||
> **Cross-spec references require a shared parent.**
|
||||
> If you need to create an element (e.g., an equality constraint) that
|
||||
> references elements from *two different child specs*, you must first
|
||||
> attach both children to the same parent, then add the cross-referencing
|
||||
> element to the **parent** spec using the final prefixed/suffixed names:
|
||||
|
||||
```python
|
||||
# Two robot arms, each defined as a separate spec
|
||||
arm_spec = mujoco.MjSpec()
|
||||
arm_body = arm_spec.worldbody.add_body(name='hand')
|
||||
arm_body.add_geom(name='hand_geom', size=[0.05])
|
||||
wrist_joint = arm_body.add_joint(name='wrist', type=mujoco.mjtJoint.mjJNT_HINGE)
|
||||
|
||||
# Attach both to the parent with different prefixes
|
||||
parent = mujoco.MjSpec()
|
||||
left_prefix, right_prefix = 'left_', 'right_'
|
||||
|
||||
frame_l = parent.worldbody.add_frame(pos=[-0.5, 0, 1])
|
||||
frame_l.attach_body(arm_body, prefix=left_prefix) # left_wrist, left_hand, ...
|
||||
|
||||
frame_r = parent.worldbody.add_frame(pos=[0.5, 0, 1])
|
||||
frame_r.attach_body(arm_body, prefix=right_prefix) # right_wrist, right_hand, ...
|
||||
|
||||
# NOW add a constraint linking both arms — look up the prefixed joints
|
||||
# from the parent spec, don't hardcode the names
|
||||
left_wrist = parent.joint(f'{left_prefix}{wrist_joint.name}')
|
||||
right_wrist = parent.joint(f'{right_prefix}{wrist_joint.name}')
|
||||
parent.add_equality(type=mujoco.mjtEq.mjEQ_JOINT,
|
||||
name1=left_wrist.name, name2=right_wrist.name)
|
||||
model = parent.compile()
|
||||
```
|
||||
|
||||
### Attachment Transforms
|
||||
|
||||
When attaching to a site or frame, the child body's position is transformed
|
||||
relative to the parent's attachment point. Attachment also handles unit
|
||||
conversion (degrees vs radians) between parent and child specs automatically.
|
||||
|
||||
### Assets Get Renamed Too
|
||||
|
||||
Prefix/suffix changes apply to asset filenames:
|
||||
```python
|
||||
child.assets = {'mesh.obj': data}
|
||||
parent.attach(child, prefix='robot_')
|
||||
# Asset key becomes 'robot_mesh.obj' in parent
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Cameras
|
||||
|
||||
### Orientation
|
||||
|
||||
MuJoCo cameras look down the **negative Z axis**. The camera frame is:
|
||||
- **-Z** → forward (viewing direction)
|
||||
- **+X** → right
|
||||
- **+Y** → up
|
||||
|
||||
To point a camera downward (looking at the ground), set its Z axis to `[0, 0, 1]`:
|
||||
|
||||
```python
|
||||
body.add_camera(
|
||||
name='overhead',
|
||||
xyaxes=[1, 0, 0, 0, 1, 0], # x=[1,0,0], y=[0,1,0] → z=[0,0,1] → looks DOWN (-z)
|
||||
pos=[0, 0, 5],
|
||||
)
|
||||
```
|
||||
|
||||
### Geom Group Visibility
|
||||
|
||||
Each camera/viewer has 6 geom groups (0–5). Default visibility:
|
||||
|
||||
| Group | Default Visible | Typical Use |
|
||||
|-------|----------------|-------------|
|
||||
| 0 | ✅ Yes | Standard geoms (default group for new geoms) |
|
||||
| 1 | ✅ Yes | Secondary visual geoms |
|
||||
| 2 | ✅ Yes | Tertiary visual geoms |
|
||||
| 3 | ❌ No | Collision-only or debug geoms |
|
||||
| 4 | ❌ No | Hidden geoms |
|
||||
| 5 | ❌ No | Hidden geoms |
|
||||
|
||||
A newly created geom is in **group 0** by default. Toggle visibility at runtime
|
||||
via `mjvOption.geomgroup[i]`. The same 3-on/3-off default applies to sites,
|
||||
joints, tendons, actuators, flexes, and skins.
|
||||
|
||||
---
|
||||
|
||||
## Contacts: Use Sensors, Not the Contact Array
|
||||
|
||||
### The problem with `data.contact`
|
||||
|
||||
`data.contact` is a **variable-length** array that changes size every timestep
|
||||
depending on what's colliding. Iterating over it directly is fragile and
|
||||
**incompatible with learning-based agents** and fixed-size observation spaces.
|
||||
|
||||
```python
|
||||
# ❌ WRONG — don't iterate data.contact for reward/observation logic
|
||||
for c in data.contact:
|
||||
if c.geom1 == target_geom_id:
|
||||
force = ... # fragile, variable-length, non-deterministic order
|
||||
```
|
||||
|
||||
> [!CAUTION]
|
||||
> Never iterate `data.contact` to build observations or compute rewards.
|
||||
> The array's length and ordering can change between timesteps and even
|
||||
> between MuJoCo versions. Use **contact sensors** instead.
|
||||
|
||||
### Contact sensors: fixed-size, declarative contact queries
|
||||
|
||||
A `<contact>` sensor selects contacts via declarative matching criteria, reduces
|
||||
them to a fixed number of slots, and extracts requested data fields into
|
||||
`data.sensordata` — always the same size, every timestep.
|
||||
|
||||
The pipeline has three stages:
|
||||
1. **Matching** — filter contacts by geom, body, subtree, or site volume
|
||||
2. **Reduction** — keep the top `num` contacts (by order, min distance, max force, or net force)
|
||||
3. **Extraction** — copy requested fields (`found`, `force`, `torque`, `dist`, `pos`, `normal`, `tangent`)
|
||||
|
||||
### Example: detect contact force between a gripper and an object
|
||||
|
||||
```python
|
||||
import mujoco
|
||||
import numpy as np
|
||||
|
||||
spec = mujoco.MjSpec()
|
||||
|
||||
# Build a simple scene: floor + falling object
|
||||
floor = spec.worldbody.add_geom(
|
||||
name='floor', type=mujoco.mjtGeom.mjGEOM_PLANE, size=[1, 1, 0.01]
|
||||
)
|
||||
obj_body = spec.worldbody.add_body(name='obj', pos=[0, 0, 0.5])
|
||||
obj_body.add_freejoint()
|
||||
obj_geom = obj_body.add_geom(
|
||||
name='obj_geom', type=mujoco.mjtGeom.mjGEOM_SPHERE,
|
||||
size=[0.05], mass=0.1,
|
||||
)
|
||||
|
||||
# Add a contact sensor: report force for contacts involving obj_geom
|
||||
contact_sensor = spec.add_sensor(
|
||||
name='obj_contact',
|
||||
type=mujoco.mjtSensor.mjSENS_CONTACT,
|
||||
# Match any contact involving this geom — use .name, not a literal string:
|
||||
objname=obj_geom.name, objtype=mujoco.mjtObj.mjOBJ_GEOM,
|
||||
)
|
||||
|
||||
model = spec.compile()
|
||||
data = mujoco.MjData(model)
|
||||
|
||||
# Step the simulation until the object lands
|
||||
mujoco.mj_step(model, data, nstep=500)
|
||||
mujoco.mj_forward(model, data)
|
||||
|
||||
# Read the contact sensor via bind — always fixed-size in data.sensordata
|
||||
contact_data = data.bind(contact_sensor).sensordata
|
||||
print(f'Contact sensor output: {contact_data}')
|
||||
```
|
||||
|
||||
### XML-based contact sensor (common pattern)
|
||||
|
||||
When loading from XML, contact sensors are even cleaner:
|
||||
|
||||
```xml
|
||||
<sensor>
|
||||
<!-- Is the gripper touching the object? Report force and normal for up to 3 contacts -->
|
||||
<contact name="grip_contact"
|
||||
body1="gripper" body2="object"
|
||||
num="3" data="found force normal"
|
||||
reduce="maxforce"/>
|
||||
|
||||
<!-- Total wrench from all contacts on a body -->
|
||||
<contact name="object_net"
|
||||
body1="object"
|
||||
data="force torque"
|
||||
reduce="netforce"/>
|
||||
</sensor>
|
||||
```
|
||||
|
||||
The output size is deterministic: `num × size(data fields)`. For `"found force
|
||||
normal"` with `num=3`, you get 3 × (1+3+3) = 21 numbers every timestep, padded
|
||||
with zeros if fewer contacts match.
|
||||
|
||||
### Touch sensor: simpler alternative for scalar normal force
|
||||
|
||||
If you only need a scalar "how hard is something pressing on this site", use a
|
||||
`touch` sensor instead:
|
||||
|
||||
```python
|
||||
site = body.add_site(name='fingertip', pos=[0, 0, 0.05], size=[0.02])
|
||||
spec.add_sensor(
|
||||
name='fingertip_touch',
|
||||
type=mujoco.mjtSensor.mjSENS_TOUCH,
|
||||
objname=site.name, objtype=mujoco.mjtObj.mjOBJ_SITE,
|
||||
)
|
||||
```
|
||||
|
||||
The touch sensor sums normal contact forces within the site volume — one scalar
|
||||
output, always present in `sensordata`.
|
||||
|
||||
---
|
||||
|
||||
## Spatial Math Utilities (mju_)
|
||||
|
||||
MuJoCo ships a library of spatial computation functions under the `mju_`
|
||||
namespace — quaternion algebra, rotation conversions, pose composition, and
|
||||
coordinate transforms. **Always check for an existing `mju_` function before
|
||||
implementing spatial math from scratch.** For basic vector arithmetic (add,
|
||||
subtract, dot product, norm), just use NumPy/JAX/Torch directly.
|
||||
|
||||
### Quaternion Operations
|
||||
|
||||
```python
|
||||
res = np.zeros(3)
|
||||
mujoco.mju_rotVecQuat(res, vec, quat) # rotate vector by quaternion
|
||||
|
||||
quat = np.zeros(4)
|
||||
mujoco.mju_mat2Quat(quat, mat3x3) # 3x3 rotation matrix → quaternion
|
||||
mujoco.mju_quat2Mat(mat, quat) # quaternion → 3x3 matrix
|
||||
mujoco.mju_axisAngle2Quat(quat, axis, angle) # axis-angle → quaternion
|
||||
mujoco.mju_euler2Quat(quat, euler, 'xyz') # Euler angles → quaternion
|
||||
mujoco.mju_mulQuat(res, q1, q2) # multiply quaternions
|
||||
mujoco.mju_negQuat(res, quat) # conjugate
|
||||
mujoco.mju_quatZ2Vec(quat, vec) # quat that rotates z-axis to vec
|
||||
mujoco.mju_quatIntegrate(quat, vel, scale) # integrate quat with angular velocity
|
||||
```
|
||||
|
||||
> [!TIP]
|
||||
> `mju_quatZ2Vec` is particularly useful: given a target direction vector, it
|
||||
> returns the quaternion that rotates the Z-axis to point in that direction.
|
||||
|
||||
### Pose Operations
|
||||
|
||||
```python
|
||||
mujoco.mju_mulPose(pos_res, quat_res, pos1, quat1, pos2, quat2) # compose poses
|
||||
mujoco.mju_negPose(pos_res, quat_res, pos, quat) # invert pose
|
||||
mujoco.mju_trnVecPose(res, pos, quat, vec) # transform vector by pose
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Common Gotchas
|
||||
|
||||
### 1. Computed fields are read-only
|
||||
|
||||
`data.xpos`, `data.xmat`, `data.xquat`, `data.geom_xpos` are **output** fields
|
||||
computed by `mj_forward()`. You cannot assign to them directly. Instead, modify
|
||||
input fields (`data.qpos`, `data.qvel`, `data.ctrl`) and call `mj_forward()` or
|
||||
`mj_step()`.
|
||||
|
||||
### 2. Duplicate names are forbidden
|
||||
|
||||
```python
|
||||
spec.add_material(name='yellow')
|
||||
spec.add_material(name='yellow') # ValueError: "repeated name 'yellow' in material"
|
||||
```
|
||||
|
||||
Names must be unique within each element type.
|
||||
|
||||
### 3. Orientation keywords are mutually exclusive
|
||||
|
||||
```python
|
||||
body.add_geom(axisangle=[1, 0, 0, 1.57], euler=[0, 0, 0])
|
||||
# ValueError: 'Only one of: axisangle, xyaxes, zaxis, or euler can be set.'
|
||||
```
|
||||
|
||||
Pick one orientation representation. Quaternion (`quat`) is the native format.
|
||||
|
||||
### 4. `size` must be positive for geoms
|
||||
|
||||
A geom with `size[0] == 0` will fail compilation. Always set at least
|
||||
`size=[radius]` for spheres/capsules, or `size=[hx, hy, hz]` for boxes.
|
||||
|
||||
### 5. `mj_step` with `nstep` repeats the same control
|
||||
|
||||
```python
|
||||
mujoco.mj_step(model, data, nstep=100) # 100 steps, same ctrl each step
|
||||
```
|
||||
|
||||
This is much faster than a Python loop and is fine for passive simulation or
|
||||
constant-control scenarios. But if you need to update `data.ctrl` between steps,
|
||||
you must step one at a time.
|
||||
|
||||
### 6. Euler sequence matters
|
||||
|
||||
`mju_euler2Quat` takes a 3-character sequence string. Lowercase = intrinsic
|
||||
rotations, uppercase = extrinsic:
|
||||
|
||||
```python
|
||||
mujoco.mju_euler2Quat(quat, [roll, pitch, yaw], 'xyz') # intrinsic x-y-z
|
||||
mujoco.mju_euler2Quat(quat, [roll, pitch, yaw], 'XYZ') # extrinsic X-Y-Z
|
||||
```
|
||||
|
||||
The sequence must be exactly 3 characters from `xyzXYZ`.
|
||||
|
||||
### 7. `copy()` vs view semantics
|
||||
|
||||
NumPy arrays from MjModel/MjData are **views** into C memory. `mj_step` changes
|
||||
them in-place. Always `.copy()` when storing values for later comparison.
|
||||
|
||||
### 8. Default class handling
|
||||
|
||||
```python
|
||||
main = spec.default # global default class (always named 'main')
|
||||
child_class = spec.add_default('high_friction', main)
|
||||
child_class.geom.friction = [1.5, 0.005, 0.0001]
|
||||
|
||||
geom = body.add_geom(child_class) # use specific default class
|
||||
geom = body.add_geom() # uses 'main' class implicitly
|
||||
```
|
||||
|
||||
### 9. Gravity is -Z by default
|
||||
|
||||
MuJoCo convention: **+Z is up**, gravity is `[0, 0, -9.81]`. The viewer and
|
||||
all built-in models assume this. Don't fight it — orient your scene accordingly.
|
||||
|
||||
### 10. Capsule/cylinder size with and without fromto
|
||||
|
||||
```python
|
||||
# With explicit pos/quat: size = [radius, half_length]
|
||||
body.add_geom(type=mujoco.mjtGeom.mjGEOM_CAPSULE, size=[0.05, 0.3])
|
||||
|
||||
# With fromto: size = [radius] only — length is inferred from endpoints
|
||||
body.add_geom(
|
||||
type=mujoco.mjtGeom.mjGEOM_CAPSULE,
|
||||
size=[0.05],
|
||||
fromto=[0, 0, 0, 0, 0, 0.6],
|
||||
)
|
||||
```
|
||||
|
||||
### 11. Collision filtering with contype/conaffinity
|
||||
|
||||
Two geoms collide only if `(g1.contype & g2.conaffinity) || (g2.contype & g1.conaffinity)`.
|
||||
By default both are `1`, so everything collides with everything.
|
||||
|
||||
```python
|
||||
# Visual-only geom: set contype=0, conaffinity=0 to disable collisions
|
||||
body.add_geom(size=[0.1], contype=0, conaffinity=0, group=1)
|
||||
|
||||
# Separate collision groups using bitmasks:
|
||||
robot_geom = body.add_geom(size=[0.05], contype=1, conaffinity=2)
|
||||
tool_geom = body.add_geom(size=[0.03], contype=2, conaffinity=1)
|
||||
# Robot and tool collide (1&1=0, but 2&2=0… wait):
|
||||
# contype=1 & conaffinity=1 → collide; contype=2 & conaffinity=2 → collide
|
||||
```
|
||||
|
||||
> [!TIP]
|
||||
> **`condim` and `friction` interact.** Each geom has `friction=[tangential, torsional, rolling]`
|
||||
> (default `[1, 0.005, 0.0001]`). The `condim` value controls which friction coefficients are
|
||||
> *active* in a contact:
|
||||
>
|
||||
> | condim | Active friction | Geom `friction` indices used |
|
||||
> |--------|----------------|------------------------------|
|
||||
> | 1 | None (frictionless, normal force only) | — |
|
||||
> | 3 | Tangential (opposes sliding) | `friction[0]` |
|
||||
> | 4 | Tangential + torsional (opposes sliding and twisting around contact normal) | `friction[0:2]` |
|
||||
> | 6 | Tangential + torsional + rolling (also opposes rolling around tangent axes) | `friction[0:3]` |
|
||||
>
|
||||
> Torsional friction models a surface contact patch resisting twist — useful for soft fingers.
|
||||
> Rolling friction dissipates energy from local deformations — useful for stopping balls from rolling
|
||||
> forever. Both torsional and rolling coefficients have **units of length** (roughly the contact
|
||||
> patch diameter or deformation depth).
|
||||
>
|
||||
> ```python
|
||||
> # A soft finger pad: enable torsional friction for stable grasping
|
||||
> finger_geom = body.add_geom(
|
||||
> type=mujoco.mjtGeom.mjGEOM_CAPSULE,
|
||||
> size=[0.01, 0.02],
|
||||
> condim=4,
|
||||
> friction=[1.0, 0.01, 0.0001], # tangential=1.0, torsional=0.01
|
||||
> )
|
||||
>
|
||||
> # A ball that should stop rolling on a surface
|
||||
> ball_geom = body.add_geom(
|
||||
> type=mujoco.mjtGeom.mjGEOM_SPHERE,
|
||||
> size=[0.05],
|
||||
> condim=6,
|
||||
> friction=[0.8, 0.005, 0.002], # tangential=0.8, torsional=0.005, rolling=0.002
|
||||
> )
|
||||
> ```
|
||||
|
||||
---
|
||||
|
||||
## Offscreen Rendering
|
||||
|
||||
Offscreen rendering produces images (RGB, depth, segmentation) without a
|
||||
display. It requires an OpenGL context — MuJoCo auto-detects the best
|
||||
available backend (EGL on headless Linux, GLFW on desktop, OSMesa as
|
||||
fallback).
|
||||
|
||||
### The `Renderer` class
|
||||
|
||||
`mujoco.Renderer` wraps GL context creation, scene management, and buffer
|
||||
readback. **Always use it as a context manager** to ensure GPU resources are
|
||||
freed:
|
||||
|
||||
```python
|
||||
import mujoco
|
||||
import numpy as np
|
||||
|
||||
# Define a camera in the spec and keep a reference
|
||||
overhead_cam = spec.worldbody.add_camera(
|
||||
name='overhead',
|
||||
pos=[0, 0, 3],
|
||||
quat=[0.707, 0.707, 0, 0], # looking down
|
||||
fovy=60,
|
||||
)
|
||||
|
||||
model = spec.compile()
|
||||
data = mujoco.MjData(model)
|
||||
|
||||
# Create renderer — width/height must not exceed offscreen buffer (see below)
|
||||
with mujoco.Renderer(model, height=480, width=640) as renderer:
|
||||
mujoco.mj_forward(model, data)
|
||||
|
||||
# Use the spec element's .name — never a literal string
|
||||
renderer.update_scene(data, camera=overhead_cam.name)
|
||||
rgb = renderer.render() # → np.ndarray (H, W, 3), dtype=uint8
|
||||
|
||||
# Depth rendering
|
||||
renderer.enable_depth_rendering()
|
||||
renderer.update_scene(data, camera=overhead_cam.name)
|
||||
depth = renderer.render() # → np.ndarray (H, W), dtype=float32 (meters)
|
||||
renderer.disable_depth_rendering()
|
||||
|
||||
# Segmentation rendering
|
||||
renderer.enable_segmentation_rendering()
|
||||
renderer.update_scene(data, camera=overhead_cam.name)
|
||||
seg = renderer.render() # → np.ndarray (H, W, 2), dtype=int32
|
||||
# seg[:,:,0] = object ID, seg[:,:,1] = object type; background = (-1, -1)
|
||||
renderer.disable_segmentation_rendering()
|
||||
```
|
||||
|
||||
> [!WARNING]
|
||||
> Forgetting to close the renderer (or not using `with`) leaks GPU memory and
|
||||
> GL contexts. In loops, create the renderer **once** outside the loop.
|
||||
|
||||
### What the `Renderer` holds internally
|
||||
|
||||
When you create `mujoco.Renderer(model, height, width)`, it allocates three
|
||||
internal objects that must be freed together:
|
||||
|
||||
1. **`GLContext`** — an offscreen OpenGL context (EGL, GLFW, or OSMesa,
|
||||
auto-detected). Created with the requested `width × height`.
|
||||
2. **`MjrContext`** — MuJoCo's GPU rendering resources (shaders, textures,
|
||||
framebuffers), bound to the GLContext. Set to the offscreen framebuffer.
|
||||
3. **`MjvScene`** — geometry buffer holding the scene snapshot passed to the
|
||||
GPU each frame.
|
||||
|
||||
The context manager (`with Renderer(...) as r:`) calls `r.close()` on exit,
|
||||
which frees the MjrContext first and then the GLContext — **order matters**.
|
||||
If you use the renderer without `with`, call `renderer.close()` manually.
|
||||
|
||||
> [!CAUTION]
|
||||
> Internally, `MjrContext.free()` must be called **before** `GLContext.free()`.
|
||||
> Reversing the order leaks GPU resources or segfaults. The `Renderer` class
|
||||
> handles this automatically — prefer it over manual context management.
|
||||
|
||||
### Offscreen framebuffer size
|
||||
|
||||
The renderer cannot exceed the offscreen buffer dimensions. The defaults are
|
||||
640×480. Set larger buffers **before compilation** via `spec.visual`:
|
||||
|
||||
```python
|
||||
spec.visual.global_.offwidth = 1920
|
||||
spec.visual.global_.offheight = 1080
|
||||
model = spec.compile()
|
||||
|
||||
# Now you can render up to 1920×1080
|
||||
with mujoco.Renderer(model, height=1080, width=1920) as renderer:
|
||||
...
|
||||
```
|
||||
|
||||
> [!IMPORTANT]
|
||||
> Increasing offscreen buffer size consumes GPU memory. For batch rendering
|
||||
> of many cameras, keep the per-frame resolution modest.
|
||||
|
||||
### Cameras
|
||||
|
||||
MuJoCo has two camera systems: **fixed cameras** defined in the model, and
|
||||
the **free camera** for interactive viewing.
|
||||
|
||||
#### Defining cameras in MjSpec
|
||||
|
||||
Always store the return value of `add_camera` and use its `.name` or `.id`
|
||||
to reference the camera later — never hardcode literal strings:
|
||||
|
||||
```python
|
||||
# Fixed camera on worldbody — good for evaluation/recording
|
||||
overhead_cam = spec.worldbody.add_camera(
|
||||
name='overhead',
|
||||
pos=[0, 0, 3],
|
||||
quat=[0.707, 0.707, 0, 0],
|
||||
fovy=60,
|
||||
)
|
||||
|
||||
# Camera attached to a body — moves with the body
|
||||
wrist_cam = wrist_body.add_camera(
|
||||
name='wrist_cam',
|
||||
pos=[0.05, 0, 0],
|
||||
xyaxes=[0, -1, 0, 0, 0, -1],
|
||||
fovy=90,
|
||||
)
|
||||
```
|
||||
|
||||
#### Selecting a camera for rendering
|
||||
|
||||
`update_scene` accepts a camera **name** (str), **id** (int), or an
|
||||
`MjvCamera` object. Always derive from the spec element:
|
||||
|
||||
```python
|
||||
# By name via spec element (recommended — survives recompilation)
|
||||
renderer.update_scene(data, camera=overhead_cam.name)
|
||||
|
||||
# By id via spec element (after compile; matches model.cam_* arrays)
|
||||
renderer.update_scene(data, camera=overhead_cam.id)
|
||||
|
||||
# Free camera (default) — no camera argument needed
|
||||
renderer.update_scene(data)
|
||||
|
||||
# Custom free camera with explicit lookat/distance/angles
|
||||
cam = mujoco.MjvCamera()
|
||||
cam.type = mujoco.mjtCamera.mjCAMERA_FREE
|
||||
cam.lookat[:] = [0, 0, 0.5]
|
||||
cam.distance = 3.0
|
||||
cam.azimuth = 135
|
||||
cam.elevation = -25
|
||||
renderer.update_scene(data, camera=cam)
|
||||
```
|
||||
|
||||
#### Camera properties reference
|
||||
|
||||
| Property | Type | Description |
|
||||
|----------|------|-------------|
|
||||
| `pos` | `real(3)` | Position in parent body frame |
|
||||
| `quat` | `real(4)` | Orientation quaternion (w, x, y, z) |
|
||||
| `xyaxes` | `real(6)` | Alternative orientation: `[x_axis(3), y_axis(3)]` |
|
||||
| `fovy` | `real` | Vertical field of view (degrees, default 45) |
|
||||
| `resolution` | `int(2)` | Sensor resolution — only for camera-based sensors |
|
||||
| `targetbody` | `str` | Track this body (camera always looks at it) |
|
||||
| `mode` | `str` | `"fixed"`, `"track"`, `"trackcom"`, `"targetbody"`, `"targetbodycom"` |
|
||||
|
||||
### Scene options
|
||||
|
||||
Control what is visualized via `MjvOption`:
|
||||
|
||||
```python
|
||||
scene_option = mujoco.MjvOption()
|
||||
# geomgroup is a bool array indexed by group number (0–5).
|
||||
# Each geom's `group` attribute (default 0) assigns it to a group.
|
||||
# Toggle visibility of each group:
|
||||
scene_option.geomgroup[:] = False # hide all groups
|
||||
scene_option.geomgroup[0] = True # show group 0 (e.g. ground plane)
|
||||
scene_option.geomgroup[3] = True # show group 3 (e.g. visualization geoms)
|
||||
|
||||
# Toggle rendering flags
|
||||
scene_option.flags[mujoco.mjtVisFlag.mjVIS_CONTACTFORCE] = True
|
||||
scene_option.flags[mujoco.mjtVisFlag.mjVIS_JOINT] = True
|
||||
|
||||
renderer.update_scene(data, camera=overhead_cam.name, scene_option=scene_option)
|
||||
```
|
||||
|
||||
### Filament backend (experimental)
|
||||
|
||||
MuJoCo's default renderer uses OpenGL. An alternative **Filament** backend
|
||||
(Vulkan-based) is available experimentally and provides higher-quality
|
||||
rendering. Filament does **not** require vertical flip (`np.flipud` is a
|
||||
no-op). It is selected via build flags — see the MuJoCo Filament
|
||||
[source](../src/experimental/filament) for details.
|
||||
|
||||
---
|
||||
|
||||
## Key References
|
||||
|
||||
### Documentation
|
||||
|
||||
| Document | Description |
|
||||
|----------|-------------|
|
||||
| [XMLreference.rst](XMLreference.rst) | Complete MJCF XML element and attribute reference |
|
||||
| [python.rst](python.rst) | Python bindings API: named access, bind, enums, callbacks |
|
||||
| [modeling.rst](modeling.rst) | MJCF modeling guide: coordinate frames, defaults, attachments |
|
||||
| [simulation.rst](programming/simulation.rst) | Simulation loop, state, forward/inverse dynamics |
|
||||
| [modeledit.rst](programming/modeledit.rst) | Procedural model editing with MjSpec |
|
||||
| [visualization.rst](programming/visualization.rst) | Rendering, cameras, scene management |
|
||||
| [APIfunctions.rst](APIreference/APIfunctions.rst) | C API function reference (mj_, mju_, mjv_, mjr_) |
|
||||
| [APItypes.rst](APIreference/APItypes.rst) | All MuJoCo structs and enums |
|
||||
|
||||
### Test Files (Executable Examples)
|
||||
|
||||
| Test file | Key patterns demonstrated |
|
||||
|-----------|--------------------------|
|
||||
| [specs_test.py](../../py/mujoco/specs_test.py) | MjSpec API: compile, recompile, attach, bind, defaults, delete, actuator shortcuts |
|
||||
| [bindings_test.py](../../py/mujoco/bindings_test.py) | Named indexing, mju_ functions, copy/pickle, contacts, mj_step |
|
||||
| [support_test.py](../../py/mujoco/mjx/_src/support_test.py) | MJX bind `.set()` pattern, JAX functional updates |
|
||||
|
||||
### Source Code
|
||||
|
||||
| File | Description |
|
||||
|------|-------------|
|
||||
| [mujoco.h](../include/mujoco.h) | Main C API header with all mju_ function signatures |
|
||||
| [XMLschema.rst](XMLschema.rst) | Schema-level XML structure documentation |
|
||||
Reference in New Issue
Block a user