diff --git a/doc/mjwarp/index.rst b/doc/mjwarp/index.rst index 3e6db4d6..4fe9dfe3 100644 --- a/doc/mjwarp/index.rst +++ b/doc/mjwarp/index.rst @@ -21,15 +21,16 @@ MJWarp is developed and maintained as a joint effort by `NVIDIA `__. +The MJWarp basics are covered in a tutorial |notebook| |open in colab|. + +.. |notebook| replace:: `[notebook] `__ + +.. |open in colab| replace:: `[open in colab] `__ + When To Use MJWarp? =================== -.. TODO(robotics-simulation): batch renderer - High throughput --------------- @@ -365,9 +366,6 @@ To enable this routine set ``Model.opt.ls_parallel=True`` or add a custom numeri The parallel linesearch is currently an experimental feature. -.. _mjwBatch: - - Memory ------ @@ -409,6 +407,8 @@ total contacts with per-collider contacts: mesh-mesh: 30 (CCD), ellipsoid-ellips (primitive) should set `nconmax`_ / `naconmax`_ to at least 8 / 80 (may require more for broadphase) and ``nccdmax`` / ``naccdmax`` to 3 / 30. +.. _mjwBatch: + Batched :class:`Model ` Fields ================================================= @@ -464,6 +464,104 @@ subset of fields. Heterogeneous worlds, for example: per-world meshes or number of degrees of freedom, are not currently available. +Batch Rendering +=============== + +MJWarp provides a high-throughput ray-tracing batch renderer built on +`Warp's accelerated BVHs `__ for +rendering worlds with multiple cameras in parallel on device. + +Key features: + +- **Mesh rendering with textures**: BVH-accelerated mesh rendering with full texture support. +- **Heightfield rendering**: Optimized rendering for heightfields. +- **Flex rendering**: Render 2D and 3D :ref:`flex` objects. +- **Lighting and shadows**: Dynamic lighting with configurable shadows; domain randomizable: `light_active`, + `light_type`, `light_castshadow`, `light_xpos`, `light_xdir`. +- **Heterogeneous multi-camera**: Multiple cameras per world and each camera can have a different resolution + (`cam_resolution`), field of view (`cam_fovy`, `cam_sensorsize`, `cam_intrinsic`), and output mode (`cam_output`). +- **Domain Randomization**: Per-world :class:`mjw.Model ` fields (see + :ref:`Batched Model Fields ` above): `geom_matid`, `geom_size`, `geom_rgba`, `mat_texid`, `mat_texrepeat`, + `mat_rgba`. +- **BVH-accelerated ray/rays API**: Ray casting: Accelerated :func:`mjw.ray `, + :func:`mjw.rays `, and :ref:`rangefinder sensors ` via + `Warp's BVHs `__. + +Basic Usage +---------- + +Rendering or raycasting requires a :class:`mjw.RenderContext ` which contains BVH structures +and rendering and output buffers. + +.. code-block:: python + + rc = mjw.create_render_context( + mjm, + nworld=1, + cam_res=(256, 256), # Override camera resolution (or per-camera list) + render_rgb=True, # Enable RGB output (or per-camera list) + render_depth=True, # Enable depth output (or per-camera list) + use_textures=True, # Apply material textures + use_shadows=False, # Enable shadow casting (slower) + enabled_geom_groups=[0, 1], # Only render geoms in groups 0 and 1 + cam_active=[True, False], # Selectively enable/disable cameras + flex_render_smooth=True, # Smooth shading for soft bodies + ) + +Each :class:`mjw.RenderContext ` parameter can be applied globally or per camera. +Additionally, values for :class:`mjw.RenderContext ` parameters can be parsed from XML: + +.. code-block:: xml + + + +or set via :ref:`mjSpec ` for camera customization. + +To render, first call :func:`mjw.refit_bvh ` to update the BVH trees, +followed by :func:`mjw.render ` to write to output buffers. + +.. code-block:: python + + mjw.refit_bvh(m, d, rc) + mjw.render(m, d, rc) + +The output buffers contain stacked pixels for all cameras with shape `(nworld, npixel)` and RGB data is +packed into one `unit32` variable. `RenderContext.rgb_adr` and `RenderContext.depth_adr` provide per-camera indexing. +For convenience, :func:`mjw.get_rgb ` and :func:`mjw.get_depth ` +provide per-camera batched post-processing. + +.. code-block:: python + + nworld = 1 + cam_index = 0 + resolution = rc.cam_res.numpy()[cam_index] + rgb_data = wp.zeros((nworld, resolution[1], resolution[0]), dtype=wp.vec3) + mjw.get_rgb(rc, rgb_data=rgb_data, cam_id=cam_index) + +A complete example can be found in the MJWarp tutorial |notebook| |open in colab|. + +Benchmarks +---------- + +Rendering can be benchmarked using `testspeed`_: + +.. code-block:: shell + + mjwarp-testspeed benchmarks/primitives.xml --function=render + +For benchmark results across a variety of scenes, see the +`released benchmarks `__. + +Notes +----- + +- **Meshes**: Rendering computation scales with mesh complexity. A primitive is expected to have better + performance (i.e., higher throughput) compared to a similar sized :ref:`mesh` or + :ref:`heightfield `. +- **Flex**: Currently limited to 2D and 3D :ref:`flex` objects. Performance is expected to improved as + this feature is further developed. +- **Scaling**: Rendering scales linearly with resolution (total number of pixels) and number of cameras. + .. _mjwFAQ: Frequently Asked Questions