diff --git a/doc/APIreference/APItypes.rst b/doc/APIreference/APItypes.rst index d2a812d6..684e810b 100644 --- a/doc/APIreference/APItypes.rst +++ b/doc/APIreference/APItypes.rst @@ -34,6 +34,8 @@ MuJoCo defines a large number of types: - Structs used by :ref:`engine plugins`. - Several :ref:`tyFunction` for user-defined callbacks. +- :ref:`tyNotes` regarding specific data structures that require detailed description. + .. _tyPrimitive: @@ -909,6 +911,7 @@ This structure contains the custom OpenGL rendering context, with the ids of all User Interface ^^^^^^^^^^^^^^ +For a high-level description of the UI framework, see :ref:`UI`. The names of these struct types are prefixed with ``mjui``, except for the main :ref:`mjUI` struct itself. @@ -917,7 +920,9 @@ The names of these struct types are prefixed with ``mjui``, except for the main mjuiState ~~~~~~~~~ -This structure contains the keyboard and mouse state used by the UI framework. +This C struct represents the global state of the window, keyboard and mouse, input event descriptors, and all window +rectangles (including the visible UI rectangles). There is only one ``mjuiState`` per application, even if there are +multiple UIs. This struct would normally be defined as a global variable. .. mujoco-include:: mjuiState @@ -967,7 +972,9 @@ This structure defines one section of the UI. mjuiDef ~~~~~~~ -This structure defines one entry in the definition table used for simplified UI construction. +This structure defines one entry in the definition table used for simplified UI construction. It contains everything +needed to define one UI item. Some translation is performed by the helper functions, so that multiple mjuiDefs can be +defined as a static table. .. mujoco-include:: mjuiDef @@ -977,7 +984,12 @@ This structure defines one entry in the definition table used for simplified UI mjUI ~~~~ -This structure defines the entire UI. +This C struct represents an entire UI. The same application could have multiple UIs, for example on the left and the +right of the window. This would normally be defined as a global variable. As explained earlier, it contains static +allocation for a maximum number of supported UI sections (:ref:`mjuiSection`) each with a maximum number +of supported items (:ref:`mjuiItem`). It also contains the color and spacing themes, enable/disable +callback, virtual window descriptor, text edit state, mouse focus. Some of these fields are set only once when the UI +is initialized, others change at runtime. .. mujoco-include:: mjUI diff --git a/doc/APIreference/functions.rst b/doc/APIreference/functions.rst index 09622ed0..91c35b1b 100644 --- a/doc/APIreference/functions.rst +++ b/doc/APIreference/functions.rst @@ -1945,6 +1945,8 @@ Find first rectangle containing mouse, -1: not found. UI framework ^^^^^^^^^^^^ +For a high-level description of the UI framework, see :ref:`UI`. + .. _mjui_themeSpacing: mjui_themeSpacing @@ -1970,7 +1972,12 @@ mjui_add .. mujoco-include:: mjui_add -Add definitions to UI. +This is the helper function used to construct a UI. The second argument points to an array of :ref:`mjuiDef` structs, +each corresponding to one item. The last (unused) item has its type set to -1, to mark termination. The items are added +after the end of the last used section. There is also another version of this function +(:ref:`mjui_addToSection`) which adds items to a specified section instead of adding them at the end +of the UI. Keep in mind that there is a maximum preallocated number of sections and items per section, given by +:ref:`mjMAXUISECT` and :ref:`mjMAXUIITEM`. Exceeding these maxima results in low-level errors. .. _mjui_addToSection: @@ -1997,7 +2004,12 @@ mjui_update .. mujoco-include:: mjui_update -Update specific section/item; -1: update all. +This is the main UI update function. It needs to be called whenever the user data (pointed to by the item data pointers) +changes, or when the UI state itself changes. It is normally called by a higher-level function implemented by the user +(``UiModify`` in :ref:`simulate.cc `) which also recomputes the layout of all rectangles and associated +auxiliary buffers. The function updates the pixels in the offscreen OpenGL buffer. To perform minimal updates, the user +specifies the section and the item that was modified. A value of -1 means all items and/or sections need to be updated +(which is needed following major changes.) .. _mjui_event: @@ -2006,7 +2018,10 @@ mjui_event .. mujoco-include:: mjui_event -Handle UI event, return pointer to changed item, NULL if no change. +This function is the low-level event handler. It makes the necessary changes in the UI and returns a pointer to the item +that received the event (or ``NULL`` if no valid event was recorded). This is normally called within the event handler +implemented by the user (``UiEvent`` in :ref:`simulate.cc `), and then some action is taken by user code +depending on which UI item was modified and what the state of that item is after the event is handled. .. _mjui_render: @@ -2015,7 +2030,9 @@ mjui_render .. mujoco-include:: mjui_render -Copy UI image to current buffer. +This function is called in the screen refresh loop. It copies the offscreen OpenGL buffer to the window framebuffer. If +there are multiple UIs in the application, it should be called once for each UI. Thus ``mjui_render`` is called all the +time, while :ref:`mjui_update` is called only when changes in the UI take place. .. _Errorandmemory: diff --git a/doc/APIreference/functions_override.rst b/doc/APIreference/functions_override.rst index 98241ae5..675cb0e2 100644 --- a/doc/APIreference/functions_override.rst +++ b/doc/APIreference/functions_override.rst @@ -218,6 +218,43 @@ of how to use these functions. .. _UIframework: +For a high-level description of the UI framework, see :ref:`UI`. + +.. _mjui_add: + +This is the helper function used to construct a UI. The second argument points to an array of :ref:`mjuiDef` structs, +each corresponding to one item. The last (unused) item has its type set to -1, to mark termination. The items are added +after the end of the last used section. There is also another version of this function +(:ref:`mjui_addToSection`) which adds items to a specified section instead of adding them at the end +of the UI. Keep in mind that there is a maximum preallocated number of sections and items per section, given by +:ref:`mjMAXUISECT` and :ref:`mjMAXUIITEM`. Exceeding these maxima results in low-level errors. + +.. _mjui_update: + +This is the main UI update function. It needs to be called whenever the user data (pointed to by the item data pointers) +changes, or when the UI state itself changes. It is normally called by a higher-level function implemented by the user +(``UiModify`` in :ref:`simulate.cc `) which also recomputes the layout of all rectangles and associated +auxiliary buffers. The function updates the pixels in the offscreen OpenGL buffer. To perform minimal updates, the user +specifies the section and the item that was modified. A value of -1 means all items and/or sections need to be updated +(which is needed following major changes.) + +.. _mjui_event: + +This function is the low-level event handler. It makes the necessary changes in the UI and returns a pointer to the item +that received the event (or ``NULL`` if no valid event was recorded). This is normally called within the event handler +implemented by the user (``UiEvent`` in :ref:`simulate.cc `), and then some action is taken by user code +depending on which UI item was modified and what the state of that item is after the event is handled. + + +.. _mjui_render: + +This function is called in the screen refresh loop. It copies the offscreen OpenGL buffer to the window framebuffer. If +there are multiple UIs in the application, it should be called once for each UI. Thus ``mjui_render`` is called all the +time, while :ref:`mjui_update` is called only when changes in the UI take place. + + + + .. _Errorandmemory: .. _Standardmath: diff --git a/doc/changelog.rst b/doc/changelog.rst index 859d1558..6daa2ae9 100644 --- a/doc/changelog.rst +++ b/doc/changelog.rst @@ -13,6 +13,11 @@ Bug fixes 3. Fixed bug where mixed ``jnt_limited`` joints were not being constrained correctly. 4. Made ``device_put`` type validation more verbose (fixes :github:issue:`1113`). +Documentation +^^^^^^^^^^^^^ + +5. Added documentation for the :ref:`UI` framework. + Version 3.0.0 (October 18, 2023) -------------------------------- diff --git a/doc/programming/index.rst b/doc/programming/index.rst index b397b80c..381d8754 100644 --- a/doc/programming/index.rst +++ b/doc/programming/index.rst @@ -275,5 +275,6 @@ now lazily resolved at runtime after the switch to GLAD, the "nogl" libraries ar simulation visualization + ui samples extension diff --git a/doc/programming/ui.rst b/doc/programming/ui.rst new file mode 100644 index 00000000..df96ccc3 --- /dev/null +++ b/doc/programming/ui.rst @@ -0,0 +1,105 @@ +.. _UI: + +User Interface +-------------- + +MuJoCo has a native UI framework. Its use is illustrated in the :ref:`simulate.cc ` viewer. It is +designed to be fast in terms of updating and rendering, easy to use both for the developer and for the user, +cross-platform, and integrated with the native MuJoCo renderer. In order to achieve these design goals, we have omitted +many features and customization options that are available in other UI frameworks, and instead focused on efficiency and +automation. + + + +.. _uiDesign: + +Design overview +~~~~~~~~~~~~~~~ + + +Native OpenGL rendering + We do not use any helper tools or libraries. Instead we provide C code for rendering all UI elements directly in + OpenGL. We support multiple UIs, each of which is a virtual rectangle that can be taller than the visible window. The + elements of each UI are rendered offscreen in auxiliary OpenGL buffers, via minimal updates, only when changes are + necessary. At each screen refresh we then copy the pixels from these auxiliary buffers to the window framebuffer, and + also implement vertical scroll bars when the window is smaller than the UI. This copy operation is done on the GPU and + is very fast. + + +Platform abstraction + The software design has 3 layers: OpenGL rendering of UI elements working in conjunction with the MuJoCo renderer + (which is fully cross-platform); abstract functions for access to windows, keyboard and mouse, defined as pure virtual + functions in the ``PlatformUIAdapter`` class; and an implementation of those functions in the derived class + ``GlfwAdapter``. `GLFW `__ itself is cross-platform. Nevertheless, we have opted for this + layered design in order to separate generic from platform-specific functionality. If GLFW needs to be replaced with + another similar framework for some reason, only ``GlfwAdapter`` will need to be rewritten. + + +Themes and appearance + Individual UI elements do not allow customization in terms of appearance or layout. Instead, we use themes for colors + and spacing, and arrange all UI elements automatically. Several built-in themes are provided and the user can design + custom themes, however the entire UI uses a single theme for all elements. Appearance is minimalist: mostly colored + rectangles with text. Bitmaps and other custom decorations are not supported. The UI element types are check boxes, + radio button groups, selection lists, sliders, text edit boxes, static text, buttons, separators. These elements are + grouped into sections which can be expanded and collapsed. + + +Layout and rectangles + Each UI is one virtual rectangle, whose width is determined by the theme and whose height is determined by the + sections, items within each section, and also the expand/collapse state of each section. The sizes and auxiliary + buffers for these virtual rectangles are handled automatically when the UI is updated. Each UI has a visible rectangle + on the screen, and in addition there are other rectangles -- for 3D rendering, 2D figures, and possibly custom OpenGL + rendering. All these visible rectangles are saved (in :ref:`mjuiState`) and are used to determine where mouse events + should be directed. The rectangle layout is updated by a callback provided by the user. + + +Static allocation and creation + Rather than allocating and deallocating a large number of objects corresponding to UI elements and linking them + together, we create a single C struct (type :ref:`mjUI`) with static allocation supporting some maximum number of + sections and elements; and then keep a record of how many are in use. UI creation is simplified by helper functions + whose input is a C struct (type :ref:`mjuiDef`) that is essentially a table where each row describes one UI element + (see below). This makes it possible to construct elaborate user interfaces with surprisingly little C code. + Programmatic UI creation is also possible, for example when populating a UI with sliders corresponding to MuJoCo model + joints. + + +Minimal state + The UI is designed to be as stateless as possible, so as to simplify development. This has two aspects. First, instead + of replicating user data within the UI elements, we store pointers to user data. For example, we might create a UI + slider and set its data pointer to ``mjData* d->qpos+7``. This slider will visualize as well as control the 7th scalar + component of the qpos vector of a MuJoCo model. Thus, when the simulation is updated, we have to remember to update + the UI as well. And furthermore we have to disable UI editing when the simulation is being updated. But the advantage + is that the UI becomes easier to construct, and there is no danger of discrepancies between user data and the UI. + Second, the UI elements themselves are mostly stateless. Instead we keep track of a minimal set of global states, in + particular mouse and keyboard state, section expand/collapse, contents of the text box being edited if any. + + +Automated enable and disable + While each UI item can be set in enabled or disabled state directly, we also provide automation as follows. Each UI + item can be assigned an integer category. Then a :ref:`mjfItemEnable` callback determines whether each category should + be enabled or disabled, based on some program-specific conditions. For example, sliders that can change the values of + MuJoCo model joints should be disabled when the simulation state is being updated. + + + +.. _uiAPI: + +Main API +~~~~~~~~ + +Click on the links below for detailed API reference of the main UI data structures and functions. + + +**Main data structures:** + +- :ref:`mjUI`: An entire UI. +- :ref:`mjuiState`: Global UI state. +- :ref:`mjuiDef`: One entry in the definition table used for UI construction. + + +**Main functions:** + +- :ref:`mjui_update`: Main UI update function. +- :ref:`mjui_render`: Renders the UI. +- :ref:`mjui_event`: Low-level event handler. +- :ref:`mjui_add`: Helper function used to construct a UI.