Expose a handle for the Python viewer.

This change also requires user scripts to explicitly synchronize changes to physics state to the viewer. The Simulate class was reconfigured so that certain UI events are handled during this sync operation, outside of the render loop on the main thread. These correspond to operations that require access to the full mjModel/mjData.

To support other, more interactive operations (e.g. camera movements), a new mjvSceneState struct is introduced which captures only the portion of the physics state required for scene re-rendering. The mjvSceneState is updated from mjModel/mjData during the viewer sync operation, and is significantly cheaper than a full mj_copyModel and mj_copyData.

Fixes https://github.com/deepmind/mujoco/issues/796

PiperOrigin-RevId: 525723636
Change-Id: Id08d0210a2c067d5afe85e2bf104f276aeddd75e

doc/APIreference/APItypes.rst

@@ -814,6 +814,17 @@ This structure contains everything needed to render the 3D scene in OpenGL.
 .. mujoco-include:: mjvScene
 
 
+.. _mjvSceneState:
+
+mjvSceneState
+~~~~~~~~~~~~~
+
+This structure contains the portions of :ref:`mjModel` and :ref:`mjData` that are required for
+various ``mjv_*`` functions.
+
+.. mujoco-include:: mjvScene
+
+
 .. _mjvFigure:
 
 mjvFigure

doc/APIreference/functions.rst

@@ -1332,6 +1332,15 @@ mjv_moveCamera
 
 Move camera with mouse; action is mjtMouse.
 
+.. _mjv_moveCameraFromState:
+
+mjv_moveCameraFromState
+~~~~~~~~~~~~~~~~~~~~~~~
+
+.. mujoco-include:: mjv_moveCameraFromState
+
+Move camera with mouse given a scene state; action is mjtMouse.
+
 .. _mjv_movePerturb:
 
 mjv_movePerturb
@@ -1341,6 +1350,15 @@ mjv_movePerturb
 
 Move perturb object with mouse; action is mjtMouse.
 
+.. _mjv_movePerturbFromState:
+
+mjv_movePerturbFromState
+~~~~~~~~~~~~~~~~~~~~~~~~
+
+.. mujoco-include:: mjv_movePerturbFromState
+
+Move perturb object with mouse given a scene state; action is mjtMouse.
+
 .. _mjv_moveModel:
 
 mjv_moveModel
@@ -1483,6 +1501,51 @@ mjv_updateScene
 
 Update entire scene given model state.
 
+.. _mjv_updateSceneFromState:
+
+mjv_updateSceneFromState
+~~~~~~~~~~~~~~~~~~~~~~~~
+
+.. mujoco-include:: mjv_updateSceneFromState
+
+Update entire scene from a scene state, return the number of new mjWARN_VGEOMFULL warnings.
+
+.. _mjv_defaultSceneState:
+
+mjv_defaultSceneState
+~~~~~~~~~~~~~~~~~~~~~
+
+.. mujoco-include:: mjv_defaultSceneState
+
+Set default scene state.
+
+.. _mjv_makeSceneState:
+
+mjv_makeSceneState
+~~~~~~~~~~~~~~~~~~
+
+.. mujoco-include:: mjv_makeSceneState
+
+Allocate resources and initialize a scene state object.
+
+.. _mjv_freeSceneState:
+
+mjv_freeSceneState
+~~~~~~~~~~~~~~~~~~
+
+.. mujoco-include:: mjv_freeSceneState
+
+Free scene state.
+
+.. _mjv_updateSceneState:
+
+mjv_updateSceneState
+~~~~~~~~~~~~~~~~~~~~
+
+.. mujoco-include:: mjv_updateSceneState
+
+Update a scene state from model and data.
+
 .. _mjv_addGeoms:
 
 mjv_addGeoms
@@ -1572,6 +1635,15 @@ mjr_freeContext
 
 Free resources in custom OpenGL context, set to default.
 
+.. _mjr_resizeOffscreen:
+
+mjr_resizeOffscreen
+~~~~~~~~~~~~~~~~~~~
+
+.. mujoco-include:: mjr_resizeOffscreen
+
+Resize offscreen buffers.
+
 .. _mjr_uploadTexture:
 
 mjr_uploadTexture