Merge pull request #3111 from StafaH:main
PiperOrigin-RevId: 871825088 Change-Id: I1e9c4a559a19427753cfdbacc7c7254ad0cb9c6c
This commit is contained in:
Copybara-Service
+106
-8
@@ -21,15 +21,16 @@ MJWarp is developed and maintained as a joint effort by `NVIDIA <https://nvidia.
|
||||
Tutorial notebook
|
||||
=================
|
||||
|
||||
The MJWarp basics are covered in a
|
||||
`tutorial
|
||||
notebook <https://colab.research.google.com/github/google-deepmind/mujoco_warp/blob/main/notebooks/tutorial.ipynb>`__.
|
||||
The MJWarp basics are covered in a tutorial |notebook| |open in colab|.
|
||||
|
||||
.. |notebook| replace:: `[notebook] <https://github.com/google-deepmind/mujoco_warp/blob/main/notebooks/tutorial.ipynb>`__
|
||||
|
||||
.. |open in colab| replace:: `[open in colab] <https://colab.research.google.com/github/google-deepmind/mujoco_warp/blob/main/notebooks/tutorial.ipynb>`__
|
||||
|
||||
|
||||
When To Use MJWarp?
|
||||
===================
|
||||
|
||||
.. TODO(robotics-simulation): batch renderer
|
||||
|
||||
High throughput
|
||||
---------------
|
||||
|
||||
@@ -365,9 +366,6 @@ To enable this routine set ``Model.opt.ls_parallel=True`` or add a custom numeri
|
||||
|
||||
The parallel linesearch is currently an experimental feature.
|
||||
|
||||
.. _mjwBatch:
|
||||
|
||||
|
||||
Memory
|
||||
------
|
||||
|
||||
@@ -409,6 +407,8 @@ total contacts with per-collider contacts: mesh-mesh: 30 (CCD), ellipsoid-ellips
|
||||
(primitive) should set `nconmax`_ / `naconmax`_ to at least 8 / 80 (may require more for broadphase) and ``nccdmax`` /
|
||||
``naccdmax`` to 3 / 30.
|
||||
|
||||
.. _mjwBatch:
|
||||
|
||||
Batched :class:`Model <mujoco_warp.Model>` Fields
|
||||
=================================================
|
||||
|
||||
@@ -464,6 +464,104 @@ subset of fields.
|
||||
|
||||
Heterogeneous worlds, for example: per-world meshes or number of degrees of freedom, are not currently available.
|
||||
|
||||
Batch Rendering
|
||||
===============
|
||||
|
||||
MJWarp provides a high-throughput ray-tracing batch renderer built on
|
||||
`Warp's accelerated BVHs <https://nvidia.github.io/warp/api_reference/_generated/warp.Bvh.html#warp.Bvh>`__ for
|
||||
rendering worlds with multiple cameras in parallel on device.
|
||||
|
||||
Key features:
|
||||
|
||||
- **Mesh rendering with textures**: BVH-accelerated mesh rendering with full texture support.
|
||||
- **Heightfield rendering**: Optimized rendering for heightfields.
|
||||
- **Flex rendering**: Render 2D and 3D :ref:`flex<deformable-flex>` objects.
|
||||
- **Lighting and shadows**: Dynamic lighting with configurable shadows; domain randomizable: `light_active`,
|
||||
`light_type`, `light_castshadow`, `light_xpos`, `light_xdir`.
|
||||
- **Heterogeneous multi-camera**: Multiple cameras per world and each camera can have a different resolution
|
||||
(`cam_resolution`), field of view (`cam_fovy`, `cam_sensorsize`, `cam_intrinsic`), and output mode (`cam_output`).
|
||||
- **Domain Randomization**: Per-world :class:`mjw.Model <mujoco_warp.Model>` fields (see
|
||||
:ref:`Batched Model Fields <mjwBatch>` above): `geom_matid`, `geom_size`, `geom_rgba`, `mat_texid`, `mat_texrepeat`,
|
||||
`mat_rgba`.
|
||||
- **BVH-accelerated ray/rays API**: Ray casting: Accelerated :func:`mjw.ray <mujoco_warp.ray>`,
|
||||
:func:`mjw.rays <mujoco_warp.rays>`, and :ref:`rangefinder sensors <sensor-rangefinder>` via
|
||||
`Warp's BVHs <https://nvidia.github.io/warp/api_reference/_generated/warp.Bvh.html#warp.Bvh>`__.
|
||||
|
||||
Basic Usage
|
||||
----------
|
||||
|
||||
Rendering or raycasting requires a :class:`mjw.RenderContext <mujoco_warp.RenderContext>` which contains BVH structures
|
||||
and rendering and output buffers.
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
rc = mjw.create_render_context(
|
||||
mjm,
|
||||
nworld=1,
|
||||
cam_res=(256, 256), # Override camera resolution (or per-camera list)
|
||||
render_rgb=True, # Enable RGB output (or per-camera list)
|
||||
render_depth=True, # Enable depth output (or per-camera list)
|
||||
use_textures=True, # Apply material textures
|
||||
use_shadows=False, # Enable shadow casting (slower)
|
||||
enabled_geom_groups=[0, 1], # Only render geoms in groups 0 and 1
|
||||
cam_active=[True, False], # Selectively enable/disable cameras
|
||||
flex_render_smooth=True, # Smooth shading for soft bodies
|
||||
)
|
||||
|
||||
Each :class:`mjw.RenderContext <mujoco_warp.RenderContext>` parameter can be applied globally or per camera.
|
||||
Additionally, values for :class:`mjw.RenderContext <mujoco_warp.RenderContext>` parameters can be parsed from XML:
|
||||
|
||||
.. code-block:: xml
|
||||
|
||||
<camera name="front_camera" pos="3 0 2" xyaxes="0 1 0 -0.6 0 0.8" resolution="64 64" output="rgb depth"/>
|
||||
|
||||
or set via :ref:`mjSpec <mjspec>` for camera customization.
|
||||
|
||||
To render, first call :func:`mjw.refit_bvh <mujoco_warp.refit_bvh>` to update the BVH trees,
|
||||
followed by :func:`mjw.render <mujoco_warp.render>` to write to output buffers.
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
mjw.refit_bvh(m, d, rc)
|
||||
mjw.render(m, d, rc)
|
||||
|
||||
The output buffers contain stacked pixels for all cameras with shape `(nworld, npixel)` and RGB data is
|
||||
packed into one `unit32` variable. `RenderContext.rgb_adr` and `RenderContext.depth_adr` provide per-camera indexing.
|
||||
For convenience, :func:`mjw.get_rgb <mujoco_warp.get_rgb>` and :func:`mjw.get_depth <mujoco_warp.get_depth>`
|
||||
provide per-camera batched post-processing.
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
nworld = 1
|
||||
cam_index = 0
|
||||
resolution = rc.cam_res.numpy()[cam_index]
|
||||
rgb_data = wp.zeros((nworld, resolution[1], resolution[0]), dtype=wp.vec3)
|
||||
mjw.get_rgb(rc, rgb_data=rgb_data, cam_id=cam_index)
|
||||
|
||||
A complete example can be found in the MJWarp tutorial |notebook| |open in colab|.
|
||||
|
||||
Benchmarks
|
||||
----------
|
||||
|
||||
Rendering can be benchmarked using `testspeed`_:
|
||||
|
||||
.. code-block:: shell
|
||||
|
||||
mjwarp-testspeed benchmarks/primitives.xml --function=render
|
||||
|
||||
For benchmark results across a variety of scenes, see the
|
||||
`released benchmarks <https://github.com/google-deepmind/mujoco_warp/pull/1113>`__.
|
||||
|
||||
Notes
|
||||
-----
|
||||
|
||||
- **Meshes**: Rendering computation scales with mesh complexity. A primitive is expected to have better
|
||||
performance (i.e., higher throughput) compared to a similar sized :ref:`mesh<body-geom-mesh>` or
|
||||
:ref:`heightfield <body-geom-hfield>`.
|
||||
- **Flex**: Currently limited to 2D and 3D :ref:`flex<deformable-flex>` objects. Performance is expected to improved as
|
||||
this feature is further developed.
|
||||
- **Scaling**: Rendering scales linearly with resolution (total number of pixels) and number of cameras.
|
||||
|
||||
.. _mjwFAQ:
|
||||
|
||||
Frequently Asked Questions
|
||||
|
||||
Reference in New Issue
Block a user