Minor updates to documentation across multiple sections.
PiperOrigin-RevId: 898652783 Change-Id: Idb671910c18ed3406722b8a1307f97ae9a194139
This commit is contained in:
Yuval Tassa
committed by
Copybara-Service
Copybara-Service
parent
c08d181d52
commit
239aa1f856
@@ -3,7 +3,7 @@
|
||||
Extensions
|
||||
----------
|
||||
|
||||
This section describes MuJoCo's mechanisms for user-authored extensions. At present, extensibility is provided by
|
||||
This section describes MuJoCo's mechanisms for user-authored extensions. At present, extensibility is provided
|
||||
via :ref:`engine plugins<exPlugin>` and :ref:`resource providers<exProvider>`.
|
||||
|
||||
.. _exPlugin:
|
||||
@@ -11,7 +11,7 @@ via :ref:`engine plugins<exPlugin>` and :ref:`resource providers<exProvider>`.
|
||||
Engine plugins
|
||||
~~~~~~~~~~~~~~
|
||||
|
||||
Engine plugins, introduced in MuJoCo 2.3.0, allow user-defined logic to be inserted into various parts of MuJoCo's
|
||||
Engine plugins allow user-defined logic to be inserted into various parts of MuJoCo's
|
||||
computational pipeline. For example, custom sensor and actuator types can be implemented as plugins. Plugin features are
|
||||
referenced in the XML content of an MJCF model, allowing MJCF to remain an abstract physical description of
|
||||
a system even if the simulation requirements extend beyond MuJoCo's built-in capabilities.
|
||||
@@ -462,6 +462,6 @@ Now we can write assets as strings in our MJCF files:
|
||||
|
||||
<asset>
|
||||
<texture name="grid" file="grid.png" type="2d"/>
|
||||
<mesh content-type="model/obj" file="data:model/obj;base65,I215IG9iamVjdA0KdiAxIDAgMA0KdiAwIDEgMA0KdiAwIDAgMQ=="/>
|
||||
<mesh content-type="model/obj" file="data:model/obj;base64,I215IG9iamVjdA0KdiAxIDAgMA0KdiAwIDEgMA0KdiAwIDAgMQ=="/>
|
||||
...
|
||||
</asset>
|
||||
|
||||
@@ -31,9 +31,9 @@ OpenGL renderer
|
||||
state-of-the-art rendering engines (and can be replaced with such an engine if desired) but nevertheless it provides
|
||||
efficient and informative 3D rendering.
|
||||
Thread
|
||||
The Threading framework (new in MuJoCo 3.0) is written in C++ and exposed in C. It provides a ThreadPool interface
|
||||
to process Tasks asynchronously. To enable use in MuJoCo, create a ThreadPool and assign it to the thread_pool field
|
||||
in mjData.
|
||||
The threading framework is written in C++ and exposed in C. It provides a :ref:`mjThreadPool<mjThreadPool>` interface
|
||||
to process tasks asynchronously. To enable use in MuJoCo, create a thread pool and assign it to the
|
||||
``mjData.threadpool`` field.
|
||||
UI framework
|
||||
The UI framework is written in C. UI elements are rendered in OpenGL. It has its own event
|
||||
mechanism and abstract hooks for keyboard and mouse input. The code samples use it with GLFW, but it can also be used
|
||||
@@ -80,7 +80,7 @@ working development environment. We provide a cross-platform `CMake
|
||||
applications independently of the MuJoCo library itself.
|
||||
|
||||
On macOS, the DMG disk image contains ``MuJoCo.app``, which you can double-click to launch the ``simulate`` GUI. You can
|
||||
also drag ``MuJoCo.app`` into the ``/Application`` on your system, as you would to install any other app. As well as the
|
||||
also drag ``MuJoCo.app`` into the ``/Applications`` on your system, as you would to install any other app. As well as the
|
||||
``MuJoCo.app`` `Application Bundle <https://developer.apple.com/go/?id=bundle-
|
||||
structure>`__, the DMG includes the ``mujoco.framework`` subdirectory containing the MuJoCo dynamic library and all of
|
||||
its public headers. If you are using Xcode, you can import it as a framework dependency on your project. (This also
|
||||
@@ -94,7 +94,7 @@ Building from source
|
||||
|
||||
To build MuJoCo from source, you will need CMake and a working C++17 compiler installed. The steps are:
|
||||
|
||||
#. Clone the ``mujoco`` repository: ``git clone https://github.com/deepmind/mujoco.git``
|
||||
#. Clone the ``mujoco`` repository: ``git clone https://github.com/google-deepmind/mujoco.git``
|
||||
#. Create a new build directory and ``cd`` into it.
|
||||
#. Run :shell:`cmake $PATH_TO_CLONED_REPO` to configure the build.
|
||||
#. Run ``cmake --build .`` to build.
|
||||
@@ -109,7 +109,7 @@ section of the documentation.
|
||||
Additionally, the CMake setup also implements an installation phase which will copy and organize the output files to a
|
||||
target directory.
|
||||
|
||||
5. Select the directory: :shell:`cmake $PATH_TO_CLONED_REPO -DCMAKE_INSTALL_PREFIX=<my_install_dir>`
|
||||
#. Select the directory: :shell:`cmake $PATH_TO_CLONED_REPO -DCMAKE_INSTALL_PREFIX=<my_install_dir>`
|
||||
#. After building, install with ``cmake --install .``
|
||||
#. If desired, proceed to building the Python bindings - see
|
||||
:ref:`PyBuild`.
|
||||
@@ -132,7 +132,7 @@ Building the docs
|
||||
|
||||
If you wish to build the documentation locally, for example to test pull-requests that improve it, do:
|
||||
|
||||
1. Clone the ``mujoco`` repository: ``git clone https://github.com/deepmind/mujoco.git``
|
||||
1. Clone the ``mujoco`` repository: ``git clone https://github.com/google-deepmind/mujoco.git``
|
||||
2. Go to the ``doc/`` directory: ``cd mujoco/doc``
|
||||
3. Install the dependencies: ``pip install -r requirements.txt``
|
||||
|br| Note that the MuJoCo Warp API documentation is autogenerated and requires additional dependencies.
|
||||
@@ -238,7 +238,7 @@ to which the symbol belongs. First we list the prefixes corresponding to type de
|
||||
``mjui``
|
||||
Data structure related to UI framework, for example :ref:`mjuiSection`.
|
||||
``mjs``
|
||||
Data structure related :doc:`procedural model editing <modeledit>`, for example :ref:`mjsJoint`.
|
||||
Data structure related to :doc:`procedural model editing <modeledit>`, for example :ref:`mjsJoint`.
|
||||
|
||||
Next we list the prefixes corresponding to function definitions. Note that function prefixes always end with underscore.
|
||||
|
||||
@@ -280,11 +280,6 @@ thread. On Windows and macOS, there is a canonical OpenGL library provided by th
|
||||
currently supports GLX for rendering to an X11 window, OSMesa for headless software rendering, and EGL for hardware
|
||||
accelerated headless rendering.
|
||||
|
||||
Before version 2.1.4, MuJoCo used GLEW rather than GLAD to manage OpenGL symbols, which required linking against
|
||||
different GLEW libraries at build time depending on the GL implementation used. In order to avoid having manage OpenGL
|
||||
dependency when no rendering was required, "nogl" builds of the library was made available. Since OpenGL symbols are
|
||||
now lazily resolved at runtime after the switch to GLAD, the "nogl" libraries are no longer provided.
|
||||
|
||||
.. toctree::
|
||||
:hidden:
|
||||
|
||||
|
||||
@@ -1,14 +1,8 @@
|
||||
Model Editing
|
||||
-------------
|
||||
|
||||
.. admonition:: New API
|
||||
:class: note
|
||||
|
||||
The API described below is new but feature complete. It is recommended for general use, but latent bugs are still
|
||||
possible. Please report any issues on GitHub.
|
||||
|
||||
As of MuJoCo 3.2.0, it is possible to create and modify models using the :ref:`mjSpec` struct and related API.
|
||||
This datastructure is in one-to-one correspondence with MJCF and indeed, MuJoCo's own XML parsers (both MJCF and URDF)
|
||||
It is possible to create and modify models using the :ref:`mjSpec` struct and related API.
|
||||
This data structure is in one-to-one correspondence with MJCF and indeed, MuJoCo's own XML parsers (both MJCF and URDF)
|
||||
use this API when loading a model.
|
||||
|
||||
|
||||
@@ -17,16 +11,16 @@ use this API when loading a model.
|
||||
Overview
|
||||
~~~~~~~~
|
||||
|
||||
The new API augments the traditional workflow of creating and editing models using XML files, breaking up the *parse* and
|
||||
The API augments the traditional workflow of creating and editing models using XML files, breaking up the *parse* and
|
||||
*compile* steps. As summarized in the :ref:`Overview chapter<Instance>`, the traditional workflow is:
|
||||
|
||||
1. Create an XML model description file (MJCF or URDF) and associated assets. |br|
|
||||
2. Call :ref:`mj_loadXML`, obtain an :ref:`mjModel` instance.
|
||||
|
||||
The new workflow using :ref:`mjSpec` is:
|
||||
The workflow using :ref:`mjSpec` is:
|
||||
|
||||
1. Create an empty :ref:`mjSpec` using :ref:`mj_makeSpec` or parse an existing XML file using :ref:`mj_parseXML`.
|
||||
2. Programmatically edit the :ref:`mjSpec` datastructure by adding, modifying and removing elements.
|
||||
2. Programmatically edit the :ref:`mjSpec` data structure by adding, modifying, and removing elements.
|
||||
3. Compile the :ref:`mjSpec` to an :ref:`mjModel` instance using :ref:`mj_compile`.
|
||||
|
||||
After compilation, the :ref:`mjSpec` remains editable, so steps 2 and 3 are interchangeable.
|
||||
@@ -40,9 +34,9 @@ Usage
|
||||
Here we describe the C API for procedural model editing, but it is also exposed in the :ref:`Python
|
||||
bindings<PyModelEdit>`. Advanced users can refer to `user_api_test.cc
|
||||
<https://github.com/google-deepmind/mujoco/blob/main/test/user/user_api_test.cc>`__ and the MJCF parser in
|
||||
`xml_native_reader.cc <https://github.com/google-deepmind/mujoco/blob/main/src/xml/xml_native_reader.cc>`__ for more
|
||||
usage examples. After creating a new :ref:`mjSpec` or parsing an existing XML file to an :ref:`mjSpec`, procedural
|
||||
editing corresponds to setting attributes. For example, in order to change the timestep, one can do:
|
||||
`xml_native_reader.cc <https://github.com/google-deepmind/mujoco/blob/main/src/xml/xml_native_reader.cc>`__ for
|
||||
more usage examples. After creating a new :ref:`mjSpec` or parsing an existing XML file to an :ref:`mjSpec`,
|
||||
procedural editing corresponds to setting attributes. For example, in order to change the timestep, one can do:
|
||||
|
||||
.. code-block:: C
|
||||
|
||||
@@ -56,7 +50,7 @@ In C one uses the provided :ref:`getters<AttributeGetters>` and :ref:`setters<At
|
||||
|
||||
.. code-block:: C
|
||||
|
||||
mjs_setString(model->modelname, "my_model");
|
||||
mjs_setString(spec->modelname, "my_model");
|
||||
|
||||
In C++, one can use vectors and strings directly:
|
||||
|
||||
@@ -76,7 +70,7 @@ Loading a spec from XML can be done as follows:
|
||||
|
||||
Model elements
|
||||
^^^^^^^^^^^^^^
|
||||
Model elements corresponding to MJCF are exposed to the user as C structs with the ``mjs`` prefix, the definitions are
|
||||
Model elements corresponding to MJCF are exposed to the user as C structs with the ``mjs`` prefix. The definitions are
|
||||
listed under the :ref:`Model Editing<tySpecStructure>` section of the struct reference. For example, an MJCF
|
||||
:ref:`geom<body-geom>` corresponds to an :ref:`mjsGeom`.
|
||||
|
||||
@@ -120,7 +114,7 @@ Attachment
|
||||
^^^^^^^^^^
|
||||
|
||||
This framework introduces a powerful new feature: attaching and deleting model subtrees. This feature is already used to
|
||||
power the :ref:`attach<body-attach>` an :ref:`replicate<replicate>` meta-elements in MJCF. Attachment allows the user to
|
||||
power the :ref:`attach<body-attach>` and :ref:`replicate<replicate>` meta-elements in MJCF. Attachment allows the user to
|
||||
move or copy a subtree from one model into another, while also copying or moving related referenced assets and
|
||||
referencing elements from outside the kinematic tree (e.g., actuators and sensors). Similarly, deleting a subtree will
|
||||
remove all associated elements from the model. The default behavior ("shallow copy") is to move the child into the
|
||||
@@ -199,7 +193,7 @@ already initialized elements.
|
||||
.. admonition:: Possible future change
|
||||
:class: note
|
||||
|
||||
The behaviour described above, where defaults are only applied at initialization, is a remnant of the old, XML-only
|
||||
The behavior described above, where defaults are only applied at initialization, is a remnant of the old, XML-only
|
||||
loading pipeline. A future API change could allow defaults to be changed and applied after initialization. If you
|
||||
think this feature is important to you, please let us know on GitHub.
|
||||
|
||||
|
||||
@@ -13,7 +13,7 @@ initialized by the corresponding API functions. These are very elaborate data st
|
||||
structures, preallocated data arrays for all intermediate results, as well as an :ref:`internal stack <siStack>`. Our
|
||||
strategy is to allocate all necessary heap memory at the beginning of the simulation, and free it after the simulation
|
||||
is done, so that we never have to call the C memory allocation and deallocation functions during the simulation. This is
|
||||
done for speed, avoidance of memory fragmentation, future GPU portability, and ease of managing the state of the entire
|
||||
done for speed, avoidance of memory fragmentation, GPU portability, and ease of managing the state of the entire
|
||||
simulator during a reset. It also means however that the maximal variable-memory allocation given by the :at:`memory`
|
||||
attribute in the :ref:`size <size>` MJCF element, which affects the allocation of :ref:`mjData`, must be set to a
|
||||
sufficiently large value. If this maximal size is exceeded during simulation, it is not increased dynamically, but
|
||||
@@ -331,6 +331,8 @@ Auxiliary Controls: ``qfrc_applied`` and ``xfrc_applied``
|
||||
| Note that the effects of ``qfrc_applied`` and ``xfrc_applied`` can be recreated by appropriate actuator
|
||||
definitions.
|
||||
|
||||
.. _siMocap:
|
||||
|
||||
MoCap poses: ``mocap_pos`` and ``mocap_quat``
|
||||
``mjData.mocap_pos`` and ``mjData.mocap_quat`` are special optional kinematic states :ref:`described here<CMocap>`,
|
||||
which allow the user to set the positions and orientations of static bodies in real-time, for example when streaming
|
||||
@@ -560,14 +562,12 @@ external force computed by inverse dynamics.
|
||||
Multi-threading
|
||||
~~~~~~~~~~~~~~~
|
||||
|
||||
When MuJoCo is used for simulation as explained in the :ref:`simulation loop <siSimulation>` section, it runs in a
|
||||
single thread. We have experimented with multi-threading parts of the simulation pipeline that are computationally
|
||||
expensive and amenable to parallel processing, and have concluded that the speedup is not worth using up the extra
|
||||
processor cores. This is because MuJoCo is already fast compared to the overhead of launching and synchronizing
|
||||
multiple threads within the same time step. If users start working with large simulations involving many floating
|
||||
bodies, we may eventually implement within-step multi-threading, but for now this use case is not common.
|
||||
MuJoCo has experimental support for within-step multi-threading. When a :ref:`mjThreadPool` is assigned to
|
||||
``mjData.threadpool``, parts of the simulation pipeline — such as collision detection and constraint solving across
|
||||
:ref:`islands<siSleep>` — can be distributed across worker threads. Note that within-step threading currently has
|
||||
significant memory overhead and is still a work in progress.
|
||||
|
||||
Rather than speed up a single simulation, we prefer to use multi-threading to speed up sampling operations that are
|
||||
The more common and well-supported use of multi-threading is to speed up sampling operations that are
|
||||
common in more advanced applications. Simulation is inherently serial over time (the output of one mj_step is the
|
||||
input to the next), while in sampling many calls to either forward or inverse dynamics can be executed in parallel
|
||||
since there are no dependencies among them, except perhaps for a common initial state.
|
||||
@@ -784,7 +784,7 @@ difference between row-major and column-major formats.
|
||||
When possible, MuJoCo exploits sparsity. This can make all the difference between O(N) and O(N^3) scaling. The inertia
|
||||
matrix ``mjData.qM`` and its LTDL factorization ``mjData.qLD`` are always represented as sparse. ``qM`` uses a custom
|
||||
indexing format designed for matrices that correspond to tree topology, while ``qLD`` uses the standard CSR format.
|
||||
``qM`` will be migrated to CSR in and upcoming change. The functions :ref:`mj_factorM`, :ref:`mj_solveM`,
|
||||
``qM`` will be migrated to CSR in an upcoming change. The functions :ref:`mj_factorM`, :ref:`mj_solveM`,
|
||||
:ref:`mj_solveM2` and :ref:`mj_mulM` are used for sparse factorization, substitution and matrix-vector multiplication.
|
||||
The user can also convert these matrices to dense format with the function :ref:`mj_fullM` although MuJoCo never does
|
||||
that internally.
|
||||
@@ -994,7 +994,7 @@ in MJCF which are sufficient for most models, and allow the user to adjust them
|
||||
the simulator runs out of dynamic memory at runtime it will trigger an error. When such errors are triggered, the user
|
||||
should increase :at:`memory`. The field ``mjData.maxuse_arena`` is designed to help with this adjustment. It keeps track
|
||||
of the maximum arena use since the last reset. So one strategy is to make very large allocation, then monitor
|
||||
``mjData.maxuse_memory`` statistics during typical simulations, and use it to reduce the allocation.
|
||||
``mjData.maxuse_arena`` statistics during typical simulations, and use it to reduce the allocation.
|
||||
|
||||
The kinetic and potential energy are computed and stored in ``mjData.energy`` when the corresponding flag in
|
||||
``mjModel.opt.enableflags`` is set. This can be used as another diagnostic. In general, simulation instability is
|
||||
@@ -1061,7 +1061,7 @@ non-convex mesh collisions, or to replace some of the convex collision functions
|
||||
beyond the ones provided by MuJoCo. The global 2D array :ref:`mjCOLLISIONFUNC` contains the collision function pointer
|
||||
for each pair of geom types (in the upper-left triangle). To replace them, simply set these pointers to your
|
||||
functions. The collision function type is :ref:`mjfCollision`. When user collision functions detect contacts, they
|
||||
should construct an mjvContact structure for each contact and then call the function :ref:`mj_addContact` to add that
|
||||
should construct an :ref:`mjContact` structure for each contact and then call the function :ref:`mj_addContact` to add that
|
||||
contact to ``mjData.contact``. The reference documentation of mj_addContact explains which fields of mjContact must be
|
||||
filled in by custom collision functions. Note that the functions we are talking about here correspond to near-phase
|
||||
collisions, and are called only after the list of candidate geom pairs has been constructed by the internal
|
||||
@@ -1176,7 +1176,7 @@ which are initialized asleep. These can be placed in mid-air or in deep collisio
|
||||
Notes
|
||||
^^^^^
|
||||
|
||||
.. admonition:: New feature
|
||||
.. admonition:: Subject to change
|
||||
:class: warning
|
||||
|
||||
Sleeping is a new feature (Nov 2025) that is subject to change and may have latent bugs.
|
||||
@@ -1270,7 +1270,7 @@ Notes
|
||||
The RK4 integrator is not currently supported, due to the subtleties of waking inside the sub-steps.
|
||||
|
||||
**Latent bugs**
|
||||
Sleeping is a new feature (Nov 2025) and may have latent bugs. These bugs may generally come in two varieties:
|
||||
Sleeping may have latent bugs. These bugs may generally come in two varieties:
|
||||
|
||||
- Quantities which could be skipped are instead recomputed. The only observable effect of such a bug would be that
|
||||
the simulation is slower than it could be. This type of bug can only be diagnosed with detailed profiling.
|
||||
|
||||
@@ -7,23 +7,23 @@ MuJoCo has a native 3D visualizer. Its use is illustrated in the :ref:`simulate.
|
||||
the simpler :ref:`basic.cc <saBasic>` code sample. While it is not a full-featured rendering engine, it is a
|
||||
convenient, efficient and reasonably good-looking visualizer that facilitates research and development. It renders not
|
||||
only the simulation state but also decorative elements such as contact points and forces, equivalent inertia boxes,
|
||||
convex hulls, kinematic trees, constraint violations, spatial frames and text labels; these can provide insight into
|
||||
convex hulls, kinematic trees, constraint violations, spatial frames, and text labels; these can provide insight into
|
||||
the physics simulation and help fine-tune the model.
|
||||
|
||||
The visualizer is tightly integrated with the simulator and supports both onscreen and offscreen rendering, as
|
||||
illustrated in the :ref:`record.cc <saRecord>` code sample. This makes it suitable for synthetic computer vision and
|
||||
machine learning applications, especially in cloud environments. VR integration is also available as of MuJoCo version
|
||||
1.40, facilitating applications that utilize new head-mounted displays such as Oculus Rift and HTC Vive.
|
||||
machine learning applications, especially in cloud environments. VR integration is also available, facilitating
|
||||
applications that utilize head-mounted displays.
|
||||
|
||||
Visualization in MuJoCo is a two-stage process:
|
||||
|
||||
Abstract visualization and interaction
|
||||
This stage populates the :ref:`mjvScene` data structure with a list of geometric objects, lights, cameras and
|
||||
This stage populates the :ref:`mjvScene` data structure with a list of geometric objects, lights, cameras, and
|
||||
everything else needed to produce a 3D rendering. It also provides abstract keyboard and mouse hooks for user
|
||||
interaction. The relevant data structure and function names have the prefix ``mjv``.
|
||||
OpenGL rendering
|
||||
This stage takes the mjvScene data structure populated in the abstract visualization stage, and renders it. It also
|
||||
provides basic 2d drawing and framebuffer access, so that most applications would not need to call OpenGL directly.
|
||||
provides basic 2D drawing and framebuffer access, so that most applications would not need to call OpenGL directly.
|
||||
The relevant data structure and function names have the prefix ``mjr``.
|
||||
|
||||
There are several reasons for this separation. First, the two stages are conceptually different and separating them is
|
||||
@@ -105,8 +105,8 @@ Abstract visualization and interaction
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
This stage populates the :ref:`mjvScene` data structure with a list of geometric objects,
|
||||
lights, cameras and everything else needed to produce a 3D rendering. It also provides abstract keyboard and mouse hooks
|
||||
for user interaction.
|
||||
lights, cameras, and everything else needed to produce a 3D rendering. It also provides abstract keyboard and mouse
|
||||
hooks for user interaction.
|
||||
|
||||
.. _viCamera:
|
||||
|
||||
@@ -125,14 +125,14 @@ are defined by the enum mjtCamera:
|
||||
|
||||
mjCAMERA_FREE
|
||||
This is the most commonly used abstract camera. It can be freely moved with the mouse. It has a lookat point,
|
||||
distance to the lookat point, azimuth and elevation; twist around the line of sight is not allowed. The function
|
||||
:ref:`mjv_moveCamera` is a mouse hook for controlling all these camera properties interactively with the mouse. When
|
||||
:ref:`simulate.cc <saSimulate>` first starts, it uses the free camera.
|
||||
distance to the lookat point, azimuth, and elevation; twist around the line of sight is not allowed. The function
|
||||
:ref:`mjv_moveCamera` is a mouse hook for controlling all these camera properties interactively with the mouse.
|
||||
When :ref:`simulate.cc <saSimulate>` first starts, it uses the free camera.
|
||||
mjCAMERA_TRACKING
|
||||
This is similar to the free camera, except the lookat point is no longer a free parameter but instead is coupled to
|
||||
the MuJoCo body whose id is given by mjvCamera.trackbodyid. At each update, the lookat point is set to the center of
|
||||
mass of the kinematic subtree rooted at the specified body. There is also some filtering which produces smooth camera
|
||||
motion. The distance, azimuth and elevation are controlled by the user and are not modified automatically. This is
|
||||
motion. The distance, azimuth, and elevation are controlled by the user and are not modified automatically. This is
|
||||
useful for tracking a body as it moves around, without turning the camera. To switch from the free to the tracking
|
||||
camera in :ref:`simulate.cc <saSimulate>`, hold Ctrl and right-double-click on the body of interest. Press Esc to go
|
||||
back to the free camera.
|
||||
@@ -171,7 +171,7 @@ because it needs information about the camera and viewport.
|
||||
The function mjv_select returns the index of the geom at the specified window coordinates, or -1 if there is no geom
|
||||
at those coordinates. The 3D position is also returned. See the code sample :ref:`simulate.cc <saSimulate>` for an
|
||||
example of how to use this function. Internally, mjv_select calls the engine-level function :ref:`mj_ray` which in turn
|
||||
calls the per-geom functions :ref:`mj_rayMesh`, :ref:`mj_rayHfield` and :ref:`mju_rayGeom`. The user can implement
|
||||
calls the per-geom functions :ref:`mj_rayMesh`, :ref:`mj_rayHfield`, and :ref:`mju_rayGeom`. The user can implement
|
||||
custom selection mechanisms by calling these functions directly. In a VR application for example, it would make sense to
|
||||
use the hand-held controller as a "laser pointer" that can select objects.
|
||||
|
||||
@@ -184,9 +184,8 @@ Interactive perturbations have proven very useful in exploring the model dynamic
|
||||
systems. The user is free to implement any perturbation mechanism of their choice by setting ``mjData.qfrc_applied`` or
|
||||
``mjData.xfrc_applied`` to suitable forces (in generalized and Cartesian coordinates respectively).
|
||||
|
||||
Prior to MuJoCo version 1.40, user code had to maintain a collection of objects in order to implement perturbations.
|
||||
All these objects are now grouped into the data structure :ref:`mjvPerturb`. Its use is illustrated in
|
||||
:ref:`simulate.cc <saSimulate>`.
|
||||
All objects needed to implement interactive perturbations are grouped into the data structure :ref:`mjvPerturb`.
|
||||
Its use is illustrated in :ref:`simulate.cc <saSimulate>`.
|
||||
The idea is to select a MuJoCo body of interest, and provide a reference pose (i.e., a 3D position and quaternion
|
||||
orientation) for that body. These are stored in mjPerturb.refpos/quat. The function :ref:`mjv_movePerturb` is a mouse
|
||||
hook for controlling the reference pose with the mouse. The function :ref:`mjv_initPerturb` is used to set the
|
||||
@@ -275,40 +274,26 @@ Since we have introduced two spaces, namely model space and room space, we need
|
||||
which spatial quantities are defined with respect to which spatial frame. Everything accessible by the simulator lives
|
||||
in the model space. The room space is only accessible by the visualizer. The only quantities defined in room space are
|
||||
the mjvGLCamera parameters. The functions :ref:`mjv_room2model`, :ref:`mjv_model2room`, :ref:`mjv_cameraInModel`,
|
||||
:ref:`mjv_cameraInRoom` perform the necessary transformations, and are needed for VR applications.
|
||||
and :ref:`mjv_cameraInRoom` perform the necessary transformations, and are needed for VR applications.
|
||||
|
||||
We now outline the procedure for hooking up head tracking to MuJoCo's visualizer in a VR application. A code sample
|
||||
illustrating this will soon be posted. We assume that a tracking device provides in real-time the positions of the two
|
||||
eyes (usually generated by tracking the position and orientation of the head and assuming a user-specific ipd), as
|
||||
well as the forward and up camera directions. We copy these data directly into the two mjvGLCameras, which are in
|
||||
mjvScene.camera[n] where n=0 is the left eye and n=1 is the right eye. Note that the forward direction is normal to
|
||||
the projection surface, and not necessarily aligned with the gaze direction; indeed the gaze direction is unknown
|
||||
(unless we also have an eye-tracking device) and does not affect the rendering.
|
||||
While MuJoCo does not provide a built-in VR application, it provides data structures and functions to support VR
|
||||
integration in user code.
|
||||
|
||||
We must also set the mjvGLCamera frustum. How this is done depends on the nature of the VR system. For head-mounted
|
||||
displays such as the Oculus Rift and HTC Vive, the projection surface moves with the head, and so the frustum is fixed
|
||||
and provided by the SDK. In this case we simply copy it into mjvGLCamera, averaging the left and right edges to
|
||||
compute the frustum_center parameter. Alternatively, the projection surface can be a monitor which is stationary in
|
||||
the room (which is the case in the zSpace system). For such systems we must compute the frustum at each frame, by
|
||||
taking into account the spatial relations between the monitor and the eyes/cameras. This assumes that the monitor is
|
||||
also tracked. The natural approach here is to define the monitor as the center of the room coordinate frame, and track
|
||||
the head relative to it. In the zSpace system this is done by embedding the motion capture cameras in the monitor
|
||||
itself.
|
||||
**Head tracking and cameras**
|
||||
In a typical VR application, a tracking device provides the positions and orientations of the user's eyes in
|
||||
real-time. These data can be copied directly into the two ``mjvGLCamera`` structures in ``mjvScene.camera[n]``
|
||||
(where ``n=0`` is the left eye and ``n=1`` is the right eye). The ``mjvGLCamera`` frustum parameters must also be
|
||||
set according to the physical characteristics of the tracked display.
|
||||
|
||||
Apart from tracking the head and using the correct perspective projection, VR applications typically involve hand-held
|
||||
spatial controllers that must be mapped to the motion of simulated objects or otherwise interact with the simulation.
|
||||
The pose of these controllers is recorded by the motion capture system in room space. The transformation functions we
|
||||
provide (mjv_room2model in particular) can be used to map to model space. Once we have the pose of the controller in
|
||||
model space, we can use a MuJoCo mocap body (defined in the model) to insert the controller in the simulation. This is
|
||||
precisely why mocap bodies were introduced in MuJoCo. Such bodies are treated as fixed from the viewpoint of physics,
|
||||
yet the user is expected to move them programmatically at each simulation step. They can interact with the simulation
|
||||
through contacts, or better yet, through soft equality constraints to regular bodies which in turn make contacts. The
|
||||
latter approach is illustrated in the MPL models available on the Forum. It provides effective dynamic filtering and
|
||||
avoids contacts involving bodies that behave as if they are infinitely heavy (which is what a fixed body is). Note
|
||||
that the time-varying positions and orientations of the mocap bodies are stored in ``mjData.mocap_pos/quat``, as opposed
|
||||
to storing them in mjModel. This is because mjModel is supposed to remain constant. The fixed mocap body pose stored
|
||||
in mjModel is only used at initialization and reset, when user code has not yet had a chance to update
|
||||
mjData.mocap_pos/quat.
|
||||
**Controllers and mocap bodies**
|
||||
Hand-held spatial controllers are also tracked in room space. The function :ref:`mjv_room2model` can map these
|
||||
poses to model space. Once in model space, the controller poses can be used to update the position of MuJoCo
|
||||
*mocap bodies*. Mocap bodies are treated as fixed from the viewpoint of physics, yet the user is expected to move
|
||||
them programmatically at each simulation step. They can interact with the simulation through contacts, or better
|
||||
yet, through soft equality constraints to regular bodies which in turn make contacts. This provides effective
|
||||
dynamic filtering and avoids contacts involving bodies that behave as if they are infinitely heavy. The
|
||||
time-varying positions and orientations of the mocap bodies are stored in ``mjData.mocap_pos`` and
|
||||
``mjData.mocap_quat``.
|
||||
|
||||
.. _Rendering:
|
||||
|
||||
@@ -316,14 +301,14 @@ OpenGL Rendering
|
||||
~~~~~~~~~~~~~~~~
|
||||
|
||||
This stage takes the mjvScene data structure populated in the abstract visualization stage, and renders it. It also
|
||||
provides basic 2d drawing and framebuffer access, so that most applications would not need to call OpenGL directly.
|
||||
provides basic 2D drawing and framebuffer access, so that most applications would not need to call OpenGL directly.
|
||||
|
||||
.. _reContext:
|
||||
|
||||
Context and GPU resources
|
||||
'''''''''''''''''''''''''
|
||||
|
||||
The first step in the rendering process is create the model-specific GPU context :ref:`mjrContext`. This is done by
|
||||
The first step in the rendering process is to create the model-specific GPU context :ref:`mjrContext`. This is done by
|
||||
first clearing the data structure with the function :ref:`mjr_defaultContext`, and then calling the function
|
||||
:ref:`mjr_makeContext`. This was already illustrated earlier; the relevant code is:
|
||||
|
||||
@@ -371,7 +356,7 @@ mjrContext.currentBuffer which changes whenever the active buffer changes. Some
|
||||
because the user can upload modified resources with the functions :ref:`mjr_uploadTexture`, :ref:`mjr_uploadMesh`,
|
||||
:ref:`mjr_uploadHField`. This can be used to achieve dynamic effects such as inserting a video feed into the
|
||||
rendering, or modulating a terrain map. Such modifications affect the resources residing on the GPU, but their OpenGL
|
||||
names are reused, thus the change is not actually visible in mjrContext.
|
||||
names are reused; thus, the change is not actually visible in mjrContext.
|
||||
|
||||
The user should **never** make changes to mjrContext directly. MuJoCo's renderer assumes that only it can manage
|
||||
mjrContext. In fact this kind of object would normally be opaque and its internal structure would not be exposed to
|
||||
@@ -434,7 +419,7 @@ be obtained with the function :ref:`mjr_maxViewport`. Note that while the offscr
|
||||
window buffer size changes whenever the user resizes or maximizes the window. Therefore user code should not assume
|
||||
fixed viewport size. In the code sample :ref:`simulate.cc <saSimulate>` we use a callback which is triggered whenever
|
||||
the window size changes, while in :ref:`basic.cc <saBasic>` we simply check the window size every time we render. On
|
||||
certain scaled displays (only on OSX it seems) the window size and framebuffer size can be different. So if you are
|
||||
certain scaled displays (notably on MacOS) the window size and framebuffer size can be different. So if you are
|
||||
getting the size with GLFW functions, use glfwGetFramebufferSize rather than glfwGetWindowSize. On the other hand,
|
||||
mouse coordinates are returned by the operating system in window rather than framebuffer units; thus the mouse
|
||||
interaction functions discussed earlier should use glfwGetWindowSize to obtain the window height needed to normalize
|
||||
@@ -468,10 +453,10 @@ mjSTEREO_SIDEBYSIDE
|
||||
side. In principle users can cross their eyes and see stereo on a regular monitor, but the goal here is to show it in
|
||||
a stereoscopic device. Most head-mounted displays support this stereo mode.
|
||||
|
||||
In addition to the main mjr_render function, we provide several functions for "decorating" the image. These are 2d
|
||||
rendering functions and include :ref:`mjr_overlay`, :ref:`mjr_text`, :ref:`mjr_rectangle`, :ref:`mjr_figure`. The user
|
||||
can draw additional decorations with their own OpenGL code. This should be done after mjr_render, because mjr_render
|
||||
clears the viewport.
|
||||
In addition to the main mjr_render function, we provide several functions for "decorating" the image. These are 2D
|
||||
rendering functions and include :ref:`mjr_overlay`, :ref:`mjr_text`, :ref:`mjr_rectangle`, and :ref:`mjr_figure`. The
|
||||
user can draw additional decorations with their own OpenGL code. This should be done after mjr_render, because
|
||||
mjr_render clears the viewport.
|
||||
|
||||
We also provide the functions :ref:`mjr_finish` and :ref:`mjr_getError` for explicit synchronization with the GPU and
|
||||
for OpenGL error checking. They simply call glFinish and glGetError internally. This together with the basic 2d
|
||||
|
||||
Reference in New Issue
Block a user