diff --git a/doc/mjwarp/index.rst b/doc/mjwarp/index.rst index 93e6e611..44843d35 100644 --- a/doc/mjwarp/index.rst +++ b/doc/mjwarp/index.rst @@ -28,8 +28,6 @@ notebook + +To render all cameras, users must first call ``refit_bvh`` to update the BVH trees, and then call ``render`` which will render all cameras +that are meant to be rendered into the output buffers. + +.. code-block:: python + + mjw.refit_bvh(m, d, rc) + mjw.render(m, d, rc) + +The result can be accessed through output buffers. The output buffers are linear with a shape of ``(nworld, pixels)``. +We provide ``rgb_adr`` and ``depth_adr`` for users to correctly access camera data. RGB data is packed into a ``uint32`` +and needs to be unpacked for downstream use. To facilitate all of this, we provide two helper functions in the +public API: ``get_rgb`` and ``get_depth``. + +.. code-block:: python + + nworld = 1 + cam_index = 0 + resolution = rc.cam_res.numpy()[cam_index] + rgb_data = wp.zeros((nworld, resolution[1], resolution[0]), dtype=wp.vec3) + mjw.get_rgb(rc, rgb_data=rgb_data, cam_id=cam_index) + +A complete example can be found in the +`tutorial +notebook `__. + +Benchmarks +---------- + +Rendering can be benchmarked from the CLI using ``testspeed``: + +.. code-block:: shell + + mjwarp-testspeed benchmarks/primitives.xml --function=render + +For benchmark results across a variety of scenes, see the +`released benchmarks `__. + +Limitations +----------- + +- **Visual meshes vs primitives**: Where possible, users should use primitives over visual meshes. + Mesh rendering is costly and scales with mesh complexity. For vision-based learning, especially for + non-sim2real usage, it is recommended to use the primitives of roughly the same shape as the original + mesh. +- **Flex rendering**: Flex rendering is currently in a prototype state, limited to 2D and 3D flex objects. Performance and features + will continue to improve over time. +- **Higher resolution / more cameras**: The renderer currently scales linearly with resolution and number of cameras, + as it scales with the total number of rays that need to be raycast. Higher resolutions or more cameras will lead to + lower throughput. Improving high resolution rendering performance is on the roadmap. + .. _mjwFAQ: Frequently Asked Questions