Add User Interface documentation.
PiperOrigin-RevId: 575800982 Change-Id: I9b2e03932d0d3b1844b90e5da7c32a640d999e69
This commit is contained in:
Yuval Tassa
committed by
Copybara-Service
Copybara-Service
parent
c6ef4f71c3
commit
2bb8652bf6
@@ -34,6 +34,8 @@ MuJoCo defines a large number of types:
|
||||
- Structs used by :ref:`engine plugins<tyPluginStructure>`.
|
||||
|
||||
- Several :ref:`tyFunction` for user-defined callbacks.
|
||||
- :ref:`tyNotes` regarding specific data structures that require detailed description.
|
||||
|
||||
|
||||
|
||||
.. _tyPrimitive:
|
||||
@@ -909,6 +911,7 @@ This structure contains the custom OpenGL rendering context, with the ids of all
|
||||
User Interface
|
||||
^^^^^^^^^^^^^^
|
||||
|
||||
For a high-level description of the UI framework, see :ref:`UI`.
|
||||
The names of these struct types are prefixed with ``mjui``, except for the main :ref:`mjUI` struct itself.
|
||||
|
||||
|
||||
@@ -917,7 +920,9 @@ The names of these struct types are prefixed with ``mjui``, except for the main
|
||||
mjuiState
|
||||
~~~~~~~~~
|
||||
|
||||
This structure contains the keyboard and mouse state used by the UI framework.
|
||||
This C struct represents the global state of the window, keyboard and mouse, input event descriptors, and all window
|
||||
rectangles (including the visible UI rectangles). There is only one ``mjuiState`` per application, even if there are
|
||||
multiple UIs. This struct would normally be defined as a global variable.
|
||||
|
||||
.. mujoco-include:: mjuiState
|
||||
|
||||
@@ -967,7 +972,9 @@ This structure defines one section of the UI.
|
||||
mjuiDef
|
||||
~~~~~~~
|
||||
|
||||
This structure defines one entry in the definition table used for simplified UI construction.
|
||||
This structure defines one entry in the definition table used for simplified UI construction. It contains everything
|
||||
needed to define one UI item. Some translation is performed by the helper functions, so that multiple mjuiDefs can be
|
||||
defined as a static table.
|
||||
|
||||
.. mujoco-include:: mjuiDef
|
||||
|
||||
@@ -977,7 +984,12 @@ This structure defines one entry in the definition table used for simplified UI
|
||||
mjUI
|
||||
~~~~
|
||||
|
||||
This structure defines the entire UI.
|
||||
This C struct represents an entire UI. The same application could have multiple UIs, for example on the left and the
|
||||
right of the window. This would normally be defined as a global variable. As explained earlier, it contains static
|
||||
allocation for a maximum number of supported UI sections (:ref:`mjuiSection<mjuiSection>`) each with a maximum number
|
||||
of supported items (:ref:`mjuiItem<mjuiItem>`). It also contains the color and spacing themes, enable/disable
|
||||
callback, virtual window descriptor, text edit state, mouse focus. Some of these fields are set only once when the UI
|
||||
is initialized, others change at runtime.
|
||||
|
||||
.. mujoco-include:: mjUI
|
||||
|
||||
|
||||
@@ -1945,6 +1945,8 @@ Find first rectangle containing mouse, -1: not found.
|
||||
UI framework
|
||||
^^^^^^^^^^^^
|
||||
|
||||
For a high-level description of the UI framework, see :ref:`UI`.
|
||||
|
||||
.. _mjui_themeSpacing:
|
||||
|
||||
mjui_themeSpacing
|
||||
@@ -1970,7 +1972,12 @@ mjui_add
|
||||
|
||||
.. mujoco-include:: mjui_add
|
||||
|
||||
Add definitions to UI.
|
||||
This is the helper function used to construct a UI. The second argument points to an array of :ref:`mjuiDef` structs,
|
||||
each corresponding to one item. The last (unused) item has its type set to -1, to mark termination. The items are added
|
||||
after the end of the last used section. There is also another version of this function
|
||||
(:ref:`mjui_addToSection<mjui_addToSection>`) which adds items to a specified section instead of adding them at the end
|
||||
of the UI. Keep in mind that there is a maximum preallocated number of sections and items per section, given by
|
||||
:ref:`mjMAXUISECT<glNumeric>` and :ref:`mjMAXUIITEM<glNumeric>`. Exceeding these maxima results in low-level errors.
|
||||
|
||||
.. _mjui_addToSection:
|
||||
|
||||
@@ -1997,7 +2004,12 @@ mjui_update
|
||||
|
||||
.. mujoco-include:: mjui_update
|
||||
|
||||
Update specific section/item; -1: update all.
|
||||
This is the main UI update function. It needs to be called whenever the user data (pointed to by the item data pointers)
|
||||
changes, or when the UI state itself changes. It is normally called by a higher-level function implemented by the user
|
||||
(``UiModify`` in :ref:`simulate.cc <saSimulate>`) which also recomputes the layout of all rectangles and associated
|
||||
auxiliary buffers. The function updates the pixels in the offscreen OpenGL buffer. To perform minimal updates, the user
|
||||
specifies the section and the item that was modified. A value of -1 means all items and/or sections need to be updated
|
||||
(which is needed following major changes.)
|
||||
|
||||
.. _mjui_event:
|
||||
|
||||
@@ -2006,7 +2018,10 @@ mjui_event
|
||||
|
||||
.. mujoco-include:: mjui_event
|
||||
|
||||
Handle UI event, return pointer to changed item, NULL if no change.
|
||||
This function is the low-level event handler. It makes the necessary changes in the UI and returns a pointer to the item
|
||||
that received the event (or ``NULL`` if no valid event was recorded). This is normally called within the event handler
|
||||
implemented by the user (``UiEvent`` in :ref:`simulate.cc <saSimulate>`), and then some action is taken by user code
|
||||
depending on which UI item was modified and what the state of that item is after the event is handled.
|
||||
|
||||
.. _mjui_render:
|
||||
|
||||
@@ -2015,7 +2030,9 @@ mjui_render
|
||||
|
||||
.. mujoco-include:: mjui_render
|
||||
|
||||
Copy UI image to current buffer.
|
||||
This function is called in the screen refresh loop. It copies the offscreen OpenGL buffer to the window framebuffer. If
|
||||
there are multiple UIs in the application, it should be called once for each UI. Thus ``mjui_render`` is called all the
|
||||
time, while :ref:`mjui_update` is called only when changes in the UI take place.
|
||||
|
||||
.. _Errorandmemory:
|
||||
|
||||
|
||||
@@ -218,6 +218,43 @@ of how to use these functions.
|
||||
|
||||
.. _UIframework:
|
||||
|
||||
For a high-level description of the UI framework, see :ref:`UI`.
|
||||
|
||||
.. _mjui_add:
|
||||
|
||||
This is the helper function used to construct a UI. The second argument points to an array of :ref:`mjuiDef` structs,
|
||||
each corresponding to one item. The last (unused) item has its type set to -1, to mark termination. The items are added
|
||||
after the end of the last used section. There is also another version of this function
|
||||
(:ref:`mjui_addToSection<mjui_addToSection>`) which adds items to a specified section instead of adding them at the end
|
||||
of the UI. Keep in mind that there is a maximum preallocated number of sections and items per section, given by
|
||||
:ref:`mjMAXUISECT<glNumeric>` and :ref:`mjMAXUIITEM<glNumeric>`. Exceeding these maxima results in low-level errors.
|
||||
|
||||
.. _mjui_update:
|
||||
|
||||
This is the main UI update function. It needs to be called whenever the user data (pointed to by the item data pointers)
|
||||
changes, or when the UI state itself changes. It is normally called by a higher-level function implemented by the user
|
||||
(``UiModify`` in :ref:`simulate.cc <saSimulate>`) which also recomputes the layout of all rectangles and associated
|
||||
auxiliary buffers. The function updates the pixels in the offscreen OpenGL buffer. To perform minimal updates, the user
|
||||
specifies the section and the item that was modified. A value of -1 means all items and/or sections need to be updated
|
||||
(which is needed following major changes.)
|
||||
|
||||
.. _mjui_event:
|
||||
|
||||
This function is the low-level event handler. It makes the necessary changes in the UI and returns a pointer to the item
|
||||
that received the event (or ``NULL`` if no valid event was recorded). This is normally called within the event handler
|
||||
implemented by the user (``UiEvent`` in :ref:`simulate.cc <saSimulate>`), and then some action is taken by user code
|
||||
depending on which UI item was modified and what the state of that item is after the event is handled.
|
||||
|
||||
|
||||
.. _mjui_render:
|
||||
|
||||
This function is called in the screen refresh loop. It copies the offscreen OpenGL buffer to the window framebuffer. If
|
||||
there are multiple UIs in the application, it should be called once for each UI. Thus ``mjui_render`` is called all the
|
||||
time, while :ref:`mjui_update` is called only when changes in the UI take place.
|
||||
|
||||
|
||||
|
||||
|
||||
.. _Errorandmemory:
|
||||
|
||||
.. _Standardmath:
|
||||
|
||||
@@ -13,6 +13,11 @@ Bug fixes
|
||||
3. Fixed bug where mixed ``jnt_limited`` joints were not being constrained correctly.
|
||||
4. Made ``device_put`` type validation more verbose (fixes :github:issue:`1113`).
|
||||
|
||||
Documentation
|
||||
^^^^^^^^^^^^^
|
||||
|
||||
5. Added documentation for the :ref:`UI` framework.
|
||||
|
||||
Version 3.0.0 (October 18, 2023)
|
||||
--------------------------------
|
||||
|
||||
|
||||
@@ -275,5 +275,6 @@ now lazily resolved at runtime after the switch to GLAD, the "nogl" libraries ar
|
||||
|
||||
simulation
|
||||
visualization
|
||||
ui
|
||||
samples
|
||||
extension
|
||||
|
||||
@@ -0,0 +1,105 @@
|
||||
.. _UI:
|
||||
|
||||
User Interface
|
||||
--------------
|
||||
|
||||
MuJoCo has a native UI framework. Its use is illustrated in the :ref:`simulate.cc <saSimulate>` viewer. It is
|
||||
designed to be fast in terms of updating and rendering, easy to use both for the developer and for the user,
|
||||
cross-platform, and integrated with the native MuJoCo renderer. In order to achieve these design goals, we have omitted
|
||||
many features and customization options that are available in other UI frameworks, and instead focused on efficiency and
|
||||
automation.
|
||||
|
||||
|
||||
|
||||
.. _uiDesign:
|
||||
|
||||
Design overview
|
||||
~~~~~~~~~~~~~~~
|
||||
|
||||
|
||||
Native OpenGL rendering
|
||||
We do not use any helper tools or libraries. Instead we provide C code for rendering all UI elements directly in
|
||||
OpenGL. We support multiple UIs, each of which is a virtual rectangle that can be taller than the visible window. The
|
||||
elements of each UI are rendered offscreen in auxiliary OpenGL buffers, via minimal updates, only when changes are
|
||||
necessary. At each screen refresh we then copy the pixels from these auxiliary buffers to the window framebuffer, and
|
||||
also implement vertical scroll bars when the window is smaller than the UI. This copy operation is done on the GPU and
|
||||
is very fast.
|
||||
|
||||
|
||||
Platform abstraction
|
||||
The software design has 3 layers: OpenGL rendering of UI elements working in conjunction with the MuJoCo renderer
|
||||
(which is fully cross-platform); abstract functions for access to windows, keyboard and mouse, defined as pure virtual
|
||||
functions in the ``PlatformUIAdapter`` class; and an implementation of those functions in the derived class
|
||||
``GlfwAdapter``. `GLFW <https://www.glfw.org/>`__ itself is cross-platform. Nevertheless, we have opted for this
|
||||
layered design in order to separate generic from platform-specific functionality. If GLFW needs to be replaced with
|
||||
another similar framework for some reason, only ``GlfwAdapter`` will need to be rewritten.
|
||||
|
||||
|
||||
Themes and appearance
|
||||
Individual UI elements do not allow customization in terms of appearance or layout. Instead, we use themes for colors
|
||||
and spacing, and arrange all UI elements automatically. Several built-in themes are provided and the user can design
|
||||
custom themes, however the entire UI uses a single theme for all elements. Appearance is minimalist: mostly colored
|
||||
rectangles with text. Bitmaps and other custom decorations are not supported. The UI element types are check boxes,
|
||||
radio button groups, selection lists, sliders, text edit boxes, static text, buttons, separators. These elements are
|
||||
grouped into sections which can be expanded and collapsed.
|
||||
|
||||
|
||||
Layout and rectangles
|
||||
Each UI is one virtual rectangle, whose width is determined by the theme and whose height is determined by the
|
||||
sections, items within each section, and also the expand/collapse state of each section. The sizes and auxiliary
|
||||
buffers for these virtual rectangles are handled automatically when the UI is updated. Each UI has a visible rectangle
|
||||
on the screen, and in addition there are other rectangles -- for 3D rendering, 2D figures, and possibly custom OpenGL
|
||||
rendering. All these visible rectangles are saved (in :ref:`mjuiState`) and are used to determine where mouse events
|
||||
should be directed. The rectangle layout is updated by a callback provided by the user.
|
||||
|
||||
|
||||
Static allocation and creation
|
||||
Rather than allocating and deallocating a large number of objects corresponding to UI elements and linking them
|
||||
together, we create a single C struct (type :ref:`mjUI`) with static allocation supporting some maximum number of
|
||||
sections and elements; and then keep a record of how many are in use. UI creation is simplified by helper functions
|
||||
whose input is a C struct (type :ref:`mjuiDef`) that is essentially a table where each row describes one UI element
|
||||
(see below). This makes it possible to construct elaborate user interfaces with surprisingly little C code.
|
||||
Programmatic UI creation is also possible, for example when populating a UI with sliders corresponding to MuJoCo model
|
||||
joints.
|
||||
|
||||
|
||||
Minimal state
|
||||
The UI is designed to be as stateless as possible, so as to simplify development. This has two aspects. First, instead
|
||||
of replicating user data within the UI elements, we store pointers to user data. For example, we might create a UI
|
||||
slider and set its data pointer to ``mjData* d->qpos+7``. This slider will visualize as well as control the 7th scalar
|
||||
component of the qpos vector of a MuJoCo model. Thus, when the simulation is updated, we have to remember to update
|
||||
the UI as well. And furthermore we have to disable UI editing when the simulation is being updated. But the advantage
|
||||
is that the UI becomes easier to construct, and there is no danger of discrepancies between user data and the UI.
|
||||
Second, the UI elements themselves are mostly stateless. Instead we keep track of a minimal set of global states, in
|
||||
particular mouse and keyboard state, section expand/collapse, contents of the text box being edited if any.
|
||||
|
||||
|
||||
Automated enable and disable
|
||||
While each UI item can be set in enabled or disabled state directly, we also provide automation as follows. Each UI
|
||||
item can be assigned an integer category. Then a :ref:`mjfItemEnable` callback determines whether each category should
|
||||
be enabled or disabled, based on some program-specific conditions. For example, sliders that can change the values of
|
||||
MuJoCo model joints should be disabled when the simulation state is being updated.
|
||||
|
||||
|
||||
|
||||
.. _uiAPI:
|
||||
|
||||
Main API
|
||||
~~~~~~~~
|
||||
|
||||
Click on the links below for detailed API reference of the main UI data structures and functions.
|
||||
|
||||
|
||||
**Main data structures:**
|
||||
|
||||
- :ref:`mjUI<mjUI>`: An entire UI.
|
||||
- :ref:`mjuiState<mjuiState>`: Global UI state.
|
||||
- :ref:`mjuiDef<mjuiDef>`: One entry in the definition table used for UI construction.
|
||||
|
||||
|
||||
**Main functions:**
|
||||
|
||||
- :ref:`mjui_update<mjui_update>`: Main UI update function.
|
||||
- :ref:`mjui_render<mjui_render>`: Renders the UI.
|
||||
- :ref:`mjui_event<mjui_event>`: Low-level event handler.
|
||||
- :ref:`mjui_add<mjui_add>`: Helper function used to construct a UI.
|
||||
Reference in New Issue
Block a user