chenlin/Mujoco_WASM
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Expose a handle for the Python viewer.

Browse Source 
This change also requires user scripts to explicitly synchronize changes to physics state to the viewer. The Simulate class was reconfigured so that certain UI events are handled during this sync operation, outside of the render loop on the main thread. These correspond to operations that require access to the full mjModel/mjData.

To support other, more interactive operations (e.g. camera movements), a new mjvSceneState struct is introduced which captures only the portion of the physics state required for scene re-rendering. The mjvSceneState is updated from mjModel/mjData during the viewer sync operation, and is significantly cheaper than a full mj_copyModel and mj_copyData.

Fixes https://github.com/deepmind/mujoco/issues/796

PiperOrigin-RevId: 525723636
Change-Id: Id08d0210a2c067d5afe85e2bf104f276aeddd75e
This commit is contained in:
Saran Tunyasuvunakool
2023-04-20 05:58:32 -07:00
committed by Copybara-Service
parent 4f5da9c554
commit b362cb4972
34 changed files with 4644 additions and 1088 deletions
Download Patch File Download Diff File Expand all files Collapse all files
doc/APIreference/APItypes.rst
+11
View File
@@ -814,6 +814,17 @@ This structure contains everything needed to render the 3D scene in OpenGL.
.. mujoco-include:: mjvScene
.. _mjvSceneState:
mjvSceneState
~~~~~~~~~~~~~
This structure contains the portions of :ref:`mjModel` and :ref:`mjData` that are required for
various ``mjv_*`` functions.
.. mujoco-include:: mjvScene
.. _mjvFigure:
mjvFigure
doc/APIreference/functions.rst
+72
View File
@@ -1332,6 +1332,15 @@ mjv_moveCamera
Move camera with mouse; action is mjtMouse.
.. _mjv_moveCameraFromState:
mjv_moveCameraFromState
~~~~~~~~~~~~~~~~~~~~~~~
.. mujoco-include:: mjv_moveCameraFromState
Move camera with mouse given a scene state; action is mjtMouse.
.. _mjv_movePerturb:
mjv_movePerturb
@@ -1341,6 +1350,15 @@ mjv_movePerturb
Move perturb object with mouse; action is mjtMouse.
.. _mjv_movePerturbFromState:
mjv_movePerturbFromState
~~~~~~~~~~~~~~~~~~~~~~~~
.. mujoco-include:: mjv_movePerturbFromState
Move perturb object with mouse given a scene state; action is mjtMouse.
.. _mjv_moveModel:
mjv_moveModel
@@ -1483,6 +1501,51 @@ mjv_updateScene
Update entire scene given model state.
.. _mjv_updateSceneFromState:
mjv_updateSceneFromState
~~~~~~~~~~~~~~~~~~~~~~~~
.. mujoco-include:: mjv_updateSceneFromState
Update entire scene from a scene state, return the number of new mjWARN_VGEOMFULL warnings.
.. _mjv_defaultSceneState:
mjv_defaultSceneState
~~~~~~~~~~~~~~~~~~~~~
.. mujoco-include:: mjv_defaultSceneState
Set default scene state.
.. _mjv_makeSceneState:
mjv_makeSceneState
~~~~~~~~~~~~~~~~~~
.. mujoco-include:: mjv_makeSceneState
Allocate resources and initialize a scene state object.
.. _mjv_freeSceneState:
mjv_freeSceneState
~~~~~~~~~~~~~~~~~~
.. mujoco-include:: mjv_freeSceneState
Free scene state.
.. _mjv_updateSceneState:
mjv_updateSceneState
~~~~~~~~~~~~~~~~~~~~
.. mujoco-include:: mjv_updateSceneState
Update a scene state from model and data.
.. _mjv_addGeoms:
mjv_addGeoms
@@ -1572,6 +1635,15 @@ mjr_freeContext
Free resources in custom OpenGL context, set to default.
.. _mjr_resizeOffscreen:
mjr_resizeOffscreen
~~~~~~~~~~~~~~~~~~~
.. mujoco-include:: mjr_resizeOffscreen
Resize offscreen buffers.
.. _mjr_uploadTexture:
mjr_uploadTexture
doc/changelog.rst
+5 -1
View File
@@ -35,6 +35,10 @@ Python bindings
state concurrently with the internal ``mj_forward``, resulting in e.g.
`MuJoCo stack overflow error <https://github.com/deepmind/mujoco/issues/783>`_
or `segmentation fault <https://github.com/deepmind/mujoco/issues/790>`_.
- The ``viewer.launch_passive`` function now returns a handle which can be used to interact with the viewer. The passive
viewer now also requires an explicit call to ``sync`` on its handle to pick up any update to the physics state. This
is to avoid race conditions that can result in visual artifacts. See :ref:`documentation<PyViewer>` for details.
- The ``viewer.launch_repl`` function has been removed since its functionality is superceded by ``launch_passive``.
- Added a small number of missing struct fields discovered through the new ``introspect`` metadata.
Bug fixes
@@ -102,7 +106,7 @@ Python bindings
#. Added ``viewer.launch_passive`` which launches the interactive viewer in a passive, non-blocking mode. Calls to
``launch_passive`` return immediately, allowing user code to continue execution, with the viewer automatically
reflecting any changes to the physics state. (Note that this functionality is currently in experimental/beta stage,
and is not yet described in our :ref:`viewer documentation<PyViewer>`.)
and is not yet described in our :ref:`viewer documentation<PyViewer>`.)
#. Added the ``mjpython`` launcher for macOS, which is required for ``viewer.launch_passive`` to function there.
#. Removed ``efc_`` fields from joint indexers. Since the introduction of arena memory, these fields now have dynamic
sizes that change between time steps depending on the number of active constraints, breaking strict correspondence
doc/includes/references.h
+229 -1
View File
@@ -673,7 +673,6 @@ struct mjVisual_ { // visualization options
float realtime; // initial real-time factor (1: real time)
int offwidth; // width of offscreen buffer
int offheight; // height of offscreen buffer
int treedepth; // depth of the bounding volume hierarchy
int ellipsoidinertia; // geom for inertia visualization (0: box, 1: ellipsoid)
} global;
@@ -1779,6 +1778,7 @@ struct mjvOption_ { // abstract visualization options
mjtByte actuatorgroup[mjNGROUP]; // actuator visualization by group
mjtByte skingroup[mjNGROUP]; // skin visualization by group
mjtByte flags[mjNVISFLAG]; // visualization flags (indexed by mjtVisFlag)
int bvh_depth; // depth of the bounding volume hierarchy to be visualized
};
typedef struct mjvOption_ mjvOption;
struct mjvScene_ { // abstract scene passed to OpenGL renderer
@@ -1865,6 +1865,219 @@ struct mjvFigure_ { // abstract 2D figure passed to OpenGL rendere
float yaxisdata[2]; // range of y-axis in data units
};
typedef struct mjvFigure_ mjvFigure;
struct mjvSceneState_ {
int nbuffer; // size of the buffer in bytes
void* buffer; // heap-allocated memory for all arrays in this struct
int maxgeom; // maximum number of mjvGeom supported by this state object
mjvScene plugincache; // scratch space for vis geoms inserted by plugins
// fields in mjModel that are necessary to re-render a scene
struct {
int nu;
int na;
int nbody;
int nbvh;
int njnt;
int ngeom;
int nsite;
int ncam;
int nlight;
int nmesh;
int nskin;
int nskinvert;
int nskinface;
int nskinbone;
int nskinbonevert;
int nmat;
int neq;
int ntendon;
int nwrap;
int nsensor;
int nnames;
int nsensordata;
mjOption opt;
mjVisual vis;
mjStatistic stat;
int* body_parentid;
int* body_rootid;
int* body_weldid;
int* body_mocapid;
int* body_jntnum;
int* body_jntadr;
int* body_geomnum;
int* body_geomadr;
mjtNum* body_iquat;
mjtNum* body_mass;
mjtNum* body_inertia;
int* body_bvhadr;
int* body_bvhnum;
int* bvh_depth;
int* bvh_child;
int* bvh_geomid;
mjtNum* bvh_aabb;
int* jnt_type;
int* jnt_bodyid;
int* jnt_group;
int* geom_type;
int* geom_bodyid;
int* geom_dataid;
int* geom_matid;
int* geom_group;
mjtNum* geom_size;
mjtNum* geom_aabb;
mjtNum* geom_rbound;
float* geom_rgba;
int* site_type;
int* site_bodyid;
int* site_matid;
int* site_group;
mjtNum* site_size;
float* site_rgba;
mjtNum* cam_fovy;
mjtNum* cam_ipd;
mjtByte* light_directional;
mjtByte* light_castshadow;
mjtByte* light_active;
float* light_attenuation;
float* light_cutoff;
float* light_exponent;
float* light_ambient;
float* light_diffuse;
float* light_specular;
int* mesh_texcoordadr;
int* mesh_graphadr;
int* skin_matid;
int* skin_group;
float* skin_rgba;
float* skin_inflate;
int* skin_vertadr;
int* skin_vertnum;
int* skin_texcoordadr;
int* skin_faceadr;
int* skin_facenum;
int* skin_boneadr;
int* skin_bonenum;
float* skin_vert;
int* skin_face;
int* skin_bonevertadr;
int* skin_bonevertnum;
float* skin_bonebindpos;
float* skin_bonebindquat;
int* skin_bonebodyid;
int* skin_bonevertid;
float* skin_bonevertweight;
int* mat_texid;
mjtByte* mat_texuniform;
float* mat_texrepeat;
float* mat_emission;
float* mat_specular;
float* mat_shininess;
float* mat_reflectance;
float* mat_rgba;
int* eq_type;
int* eq_obj1id;
int* eq_obj2id;
mjtByte* eq_active;
mjtNum* eq_data;
int* tendon_num;
int* tendon_matid;
int* tendon_group;
mjtByte* tendon_limited;
mjtNum* tendon_width;
mjtNum* tendon_range;
mjtNum* tendon_stiffness;
mjtNum* tendon_damping;
mjtNum* tendon_frictionloss;
mjtNum* tendon_lengthspring;
float* tendon_rgba;
int* actuator_trntype;
int* actuator_dyntype;
int* actuator_trnid;
int* actuator_actadr;
int* actuator_actnum;
int* actuator_group;
mjtByte* actuator_ctrllimited;
mjtByte* actuator_actlimited;
mjtNum* actuator_ctrlrange;
mjtNum* actuator_actrange;
mjtNum* actuator_cranklength;
int* sensor_type;
int* sensor_objid;
int* sensor_adr;
int* name_bodyadr;
int* name_jntadr;
int* name_geomadr;
int* name_siteadr;
int* name_camadr;
int* name_lightadr;
int* name_eqadr;
int* name_tendonadr;
int* name_actuatoradr;
char* names;
} model;
// fields in mjData that are necessary to re-render a scene
struct {
mjWarningStat warning[mjNWARNING];
int nefc;
int ncon;
mjtNum time;
mjtNum* act;
mjtNum* ctrl;
mjtNum* xfrc_applied;
mjtNum* sensordata;
mjtNum* xpos;
mjtNum* xquat;
mjtNum* xmat;
mjtNum* xipos;
mjtNum* ximat;
mjtNum* xanchor;
mjtNum* xaxis;
mjtNum* geom_xpos;
mjtNum* geom_xmat;
mjtNum* site_xpos;
mjtNum* site_xmat;
mjtNum* cam_xpos;
mjtNum* cam_xmat;
mjtNum* light_xpos;
mjtNum* light_xdir;
mjtNum* subtree_com;
int* ten_wrapadr;
int* ten_wrapnum;
int* wrap_obj;
mjtNum* wrap_xpos;
mjtByte* bvh_active;
mjContact* contact;
mjtNum* efc_force;
} data;
};
typedef struct mjvSceneState_ mjvSceneState;
//----------------------------- MJAPI FUNCTIONS --------------------------------
void mj_defaultVFS(mjVFS* vfs);
@@ -2020,8 +2233,14 @@ mjtNum mjv_frustumHeight(const mjvScene* scn);
void mjv_alignToCamera(mjtNum res[3], const mjtNum vec[3], const mjtNum forward[3]);
void mjv_moveCamera(const mjModel* m, int action, mjtNum reldx, mjtNum reldy,
const mjvScene* scn, mjvCamera* cam);
void mjv_moveCameraFromState(const mjvSceneState* scnstate, int action,
mjtNum reldx, mjtNum reldy,
const mjvScene* scn, mjvCamera* cam);
void mjv_movePerturb(const mjModel* m, const mjData* d, int action, mjtNum reldx,
mjtNum reldy, const mjvScene* scn, mjvPerturb* pert);
void mjv_movePerturbFromState(const mjvSceneState* scnstate, int action,
mjtNum reldx, mjtNum reldy,
const mjvScene* scn, mjvPerturb* pert);
void mjv_moveModel(const mjModel* m, int action, mjtNum reldx, mjtNum reldy,
const mjtNum roomup[3], mjvScene* scn);
void mjv_initPerturb(const mjModel* m, mjData* d, const mjvScene* scn, mjvPerturb* pert);
@@ -2044,6 +2263,14 @@ void mjv_makeScene(const mjModel* m, mjvScene* scn, int maxgeom);
void mjv_freeScene(mjvScene* scn);
void mjv_updateScene(const mjModel* m, mjData* d, const mjvOption* opt,
const mjvPerturb* pert, mjvCamera* cam, int catmask, mjvScene* scn);
int mjv_updateSceneFromState(const mjvSceneState* scnstate, const mjvOption* opt,
const mjvPerturb* pert, mjvCamera* cam, int catmask,
mjvScene* scn);
void mjv_defaultSceneState(mjvSceneState* scnstate);
void mjv_makeSceneState(const mjModel* m, const mjData* d,
mjvSceneState* scnstate, int maxgeom);
void mjv_freeSceneState(mjvSceneState* scnstate);
void mjv_updateSceneState(const mjModel* m, mjData* d, mjvSceneState* scnstate);
void mjv_addGeoms(const mjModel* m, mjData* d, const mjvOption* opt,
const mjvPerturb* pert, int catmask, mjvScene* scn);
void mjv_makeLights(const mjModel* m, mjData* d, mjvScene* scn);
@@ -2054,6 +2281,7 @@ void mjr_makeContext(const mjModel* m, mjrContext* con, int fontscale);
void mjr_changeFont(int fontscale, mjrContext* con);
void mjr_addAux(int index, int width, int height, int samples, mjrContext* con);
void mjr_freeContext(mjrContext* con);
void mjr_resizeOffscreen(int width, int height, mjrContext* con);
void mjr_uploadTexture(const mjModel* m, const mjrContext* con, int texid);
void mjr_uploadMesh(const mjModel* m, const mjrContext* con, int meshid);
void mjr_uploadHField(const mjModel* m, const mjrContext* con, int hfieldid);
doc/python.rst
+78 -12
View File
@@ -117,19 +117,19 @@ As a reference, a working build configuration can be found in MuJoCo's
Interactive viewer
==================
An interactive GUI viewer is available as part of the Python package. (This is the same viewer as the ``simulate``
application that ships with the MuJoCo binary releases.)
An interactive GUI viewer is provided as part of the Python package in the ``mujoco.viewer`` module. This is the same
viewer as the ``simulate`` application that ships with the MuJoCo binary releases.
Three distinct use cases are supported:
#. Launching as a standalone application:
#. As a **standalone application**:
- ``python -m mujoco.viewer`` launches an empty visualization session, where a model can be loaded by drag-and-drop.
- ``python -m mujoco.viewer --mjcf=/path/to/some/mjcf.xml`` launches a visualization session for the specified
model file.
#. Launching from a Python program/script -- import the module via ``from mujoco import viewer`` and launch the GUI
using one of the following invocations:
#. As a **fully managed viewer** in a Python program/script, through the function ``viewer.launch``. This function
**blocks the user's script completely** to take care of running and timing a physics loop.
- ``viewer.launch()`` launches an empty visualization session, where a model can be loaded by drag-and-drop.
- ``viewer.launch(model)`` launches a visualization session for the given ``mjModel`` where the visualizer
@@ -137,13 +137,79 @@ Three distinct use cases are supported:
- ``viewer.launch(model, data)`` is the same as above, except that the visualizer operates directly on the given
``mjData`` instance -- upon exit the ``data`` object will have been modified.
#. Launching from an interactive Python session (aka REPL): when working interactively either in a ``python`` or
``ipython`` shell, the visualizer can be launched in a "passive" mode via ``viewer.launch_repl(model, data)``, where
the user remains in full control of modifying or stepping the physics. In this mode, the user can interact with the
visualizer using the mouse and keyboard as usual, however the physics will be frozen unless the user explicitly calls
``mj_step`` (or perform any other modification of the ``mjData`` or ``mjModel``) in the REPL terminal. Note that since
the visualizer does not modify ``mjData`` in this mode, mouse-drag perturbations will not work unless the user
explicitly handles incoming GUI perturbation events in the REPL session.
#. As a **passive viewer**, by calling ``viewer.launch_passive(model, data)``. This function **does not block**,
allowing the user script to continue execution. In this mode, the user's script is responsible for timing and
advancing the physics state, and mouse-drag perturbations will not work unless the user explicitly handles incoming
events.
.. warning::
On macOS, ``launch_passive`` requires that the user script is executed via a special ``mjpython`` launcher.
The ``mjpython`` command is installed as part of the ``mujoco`` package, and can be used as a drop-in replacement
for the usual ``python`` command and supports an identical set of command line flags and arguments. For example,
a script can be executed via ``mjpython my_script.py``, and an IPython shell can be launched via
``mjpython -m IPython``.
The ``launch_passive`` function returns a handle which can be used to interact with the viewer. It has the following
attributes:
- ``scn``, ``cam``, ``opt``, and ``pert`` properties: correspond to :ref:`mjvScene`, :ref:`mjvCamera`,
:ref:`mjvOption`, and :ref:`mjvPerturb` structs, respectively.
- ``lock()``: provides a mutex lock for the viewer as a context manager. Since the viewer operates its own
thread, user code must ensure that it is holding the viewer lock before modifying any physics or visualization
state. These include the ``mjModel`` and ``mjData`` instance passed to ``launch_passive``, and also the ``scn``,
``cam``, ``opt``, and ``pert`` properties of the viewer handle.
- ``sync()``: synchronizes state between ``mjModel``, ``mjData``, and GUI user inputs since the previous call to
``sync``. In order to allow user scripts to make arbitrary modifications to ``mjModel`` and ``mjData`` without
needing to hold the viewer lock, the passive viewer does not access or modify these structs outside of ``sync``
calls.
User scripts must call ``sync`` in order for the viewer to reflect physics state changes. The ``sync`` function
also transfers user inputs from the GUI back into ``mjOption`` (inside ``mjModel``) and ``mjData``, including
enable/disable flags, control inputs, and mouse perturbations.
- ``close()``: programmatically closes the viewer window. This method can be safely called without locking.
- ``is_running()``: returns ``True`` if the viewer window is running and ``False`` if it is closed.
This method can be safely called without locking.
The viewer handle can also be used as a context manager which calls ``close()`` automatically upon exit. A minimal
example of a user script that uses ``launch_passive`` might look like the following. (Note that example is a simple
illustrative example that does **not** necessarily keep the physics ticking at the correct wallclock rate.)
.. code-block:: python
import time
import mujoco
import mujoco.viewer
m = mujoco.MjModel.from_xml_path('/path/to/mjcf.xml')
d = mujoco.MjData(m)
with mujoco.viewer.launch_passive(m, d) as viewer:
# Close the viewer automatically after 30 seconds.
start = time.time()
while viewer.is_running() and time.time() - start < 30:
step_start = time.time()
# The mj_step call can be replaced with a user-defined function that evaluates
# a policy, applies a control signal, and steps an environment.
mujoco.mj_step(m, d)
# Example of modifying a viewer option: toggle contact points every second.
with viewer.lock():
viewer.opt.flags[mujoco.mjtVisFlag.mjVIS_CONTACTPOINT] = int(d.time % 2)
# Synchronize so that the viewer picks up changes to the physics state.
viewer.sync()
# Rudimentary time keeping, doesn't attempt to catch up if physics stepping
# takes too long.
time_until_next_step = m.opt.timestep - (time.time() - step_start)
if time_until_next_step > 0:
time.sleep(time_until_next_step)
.. _PyUsage: