diff --git a/doc/APIreference/APIfunctions.rst b/doc/APIreference/APIfunctions.rst index 9686131d..a1ec8da6 100644 --- a/doc/APIreference/APIfunctions.rst +++ b/doc/APIreference/APIfunctions.rst @@ -38,6 +38,7 @@ API function can be classified as: - :ref:`Abstract interaction`: mouse control of cameras and perturbations. - :ref:`Abstract Visualization`. - :ref:`OpenGL rendering`. + - :ref:`Filament rendering`. - :ref:`UI framework`. - **Threads, Plugins, Derivatives** diff --git a/doc/APIreference/APItypes.rst b/doc/APIreference/APItypes.rst index 8b7e8416..77b9fc70 100644 --- a/doc/APIreference/APItypes.rst +++ b/doc/APIreference/APItypes.rst @@ -4,13 +4,16 @@ Types MuJoCo defines a large number of types: -- Two :ref:`primitive types`. +- Four :ref:`primitive types`: :ref:`mjtNum`, :ref:`mjtByte`, :ref:`mjtBool`, and + :ref:`mjtSize`. + - :ref:`C enum types` used to define categorical values. These can be classified as: - Enums used in :ref:`mjModel`. - Enums used in :ref:`mjData`. - Enums for abstract :ref:`visualization`. - - Enums used by the :ref:`renderer`. + - Enums used by the :ref:`classic renderer`. + - Enums used by the :ref:`filament renderer`. - Enums used by the :ref:`mjUI` user interface package. - Enums used by :ref:`engine plugins`. - Enums used for :ref:`procedural model manipulation`. @@ -31,7 +34,8 @@ MuJoCo defines a large number of types: - :ref:`Auxiliary struct types`, also used by the engine. - Structs for collecting :ref:`simulation statistics`. - Structs for :ref:`abstract visualization`. - - Structs used by the :ref:`renderer`. + - Structs used by the :ref:`classic renderer`. + - Structs used by the :ref:`filament renderer`. - Structs used by the :ref:`UI framework`. - Structs used for :ref:`procedural model manipulation`. - Structs used by :ref:`engine plugins`. @@ -737,7 +741,7 @@ These are the possible font types. mjrPixelFormat ~~~~~~~~~~~~~~ -There are the possible values: +These are the possible values: .. mujoco-include:: mjrPixelFormat @@ -746,7 +750,7 @@ There are the possible values: mjrVertexAttributeUsage ~~~~~~~~~~~~~~~~~~~~~~~ -There are the possible values: +These are the possible values: .. mujoco-include:: mjrVertexAttributeUsage @@ -755,7 +759,7 @@ There are the possible values: mjrVertexAttributeType ~~~~~~~~~~~~~~~~~~~~~~ -There are the possible values: +These are the possible values: .. mujoco-include:: mjrVertexAttributeType @@ -764,7 +768,7 @@ There are the possible values: mjrIndexType ~~~~~~~~~~~~ -There are the possible values: +These are the possible values: .. mujoco-include:: mjrIndexType @@ -778,6 +782,34 @@ There are the possible values: .. mujoco-include:: mjrMeshPrimitiveType +.. _tyFilamentRenderEnums: + +Filament Rendering +^^^^^^^^^^^^^^^^^^ + +The enums below are defined in `mjrfilament.h `__. + + +.. _mjrGraphicsApi: + +mjrGraphicsApi +~~~~~~~~~~~~~~ + +The underlying graphics API to use for Filament rendering. + +.. mujoco-include:: mjrGraphicsApi + + +.. _mjrDrawMode: + +mjrDrawMode +~~~~~~~~~~~ + +High-level control for how to draw objects in the scene for Filament rendering. + +.. mujoco-include:: mjrDrawMode + + .. _tyUIEnums: User Interface @@ -1306,6 +1338,263 @@ This structure contains the custom OpenGL rendering context, with the ids of all .. mujoco-include:: mjrContext +.. _tyFilamentRenderStructure: + +Filament Rendering +^^^^^^^^^^^^^^^^^^ + +The names of these struct types are prefixed with ``mjrf``. They are defined in +`mjrfilament.h `__. + +There are seven key types defined by this API: :ref:`mjrfContext`, :ref:`mjrfTexture`, +:ref:`mjrfMesh`, :ref:`mjrfLight`, :ref:`mjrfRenderable`, +:ref:`mjrfScene`, and :ref:`mjrfRenderTarget`. + +Each object is created using a `create` function and destroyed using a `destroy` function, e.g. `mjrf_createTexture` and +`mjrf_destroyTexture`. All objects require a :ref:`Context` in order to be created (with the exception of +the :ref:`Context` object itself). Additionally, the `create` functions accept a pointer to a configuration +struct (e.g. `mjrTextureConfig`) which describes the parameters for the object to be created. Each of these structs has +a corresponding `default` function (e.g. `mjrf_defaultTextureConfig`) which can be used to initialize the struct to +default values. The default values are assumed to be `0` or `NULL` unless otherwise specified. + + +.. _mjrfContext: + +mjrfContext +~~~~~~~~~~~ + +The Context is the main entry point for the filament rendering library. It manages all the core filament objects that +are responsible for the rendering of an image. All other objects (e.g. Textures, Meshes, Scenes, etc.) need a Context in +order to be created. + +Otherwise, the main function to use with the Context is :ref:`mjrf_render()` which performs the actual +rendering of an image. + +Filament uses a separate thread for rendering. However, despite that, this API is not thread-safe; calls are expected to +be made from a single thread. Due to the asynchronous nature of filament, some APIs provide handles or callbacks to +signal when an operation is complete. (Note: for WASM builds, filament does not use a separate thread.) + +There are two key differences between the :ref:`mjrfContext` and the classic :ref:`mjrContext`. +Firstly, the filament context will manage the underlying graphics context itself. This means users do not need to +initialize EGL or similar libraries beforehand. Secondly, the filament context is independent of a MuJoCo model. That +means you can use a single :ref:`mjrfContext` instance to render images for multiple models. + +.. _mjrfContextConfig: + +mjrfContextConfig +~~~~~~~~~~~~~~~~~ + +Parameters for creating :ref:`filament graphics context`. + +.. mujoco-include:: mjrfContextConfig + + +.. _mjrfTexture: + +mjrfTexture +~~~~~~~~~~~ + +A texture is a 2D or 3D (cubemap) image that adds visual detail to a rendered model, such as color or bumpiness, without +increasing geometric complexity. A texture is simply a memory buffer holds pixel data, as well as metadata such as the +dimensions of the image or the format of the pixels (e.g. 8-bit RGB). + +.. _mjrfTextureConfig: + +mjrfTextureConfig +~~~~~~~~~~~~~~~~~ + +Parameters for creating a :ref:`texture`. + +.. mujoco-include:: mjrfTextureConfig + + +.. _mjrfTextureData: + +mjrfTextureData +~~~~~~~~~~~~~~~ + +Binary data payload for a :ref:`texture`. + +.. mujoco-include:: mjrfTextureData + + +.. _mjrfMesh: + +mjrfMesh +~~~~~~~~ + +A mesh describes the surface geometry of an object to be rendered. It is defined as a collection of vertices (i.e. a +VertexBuffer), a set of indices (i.e. an IndexBuffer) that describes the order in which the vertices should be +processed, and a primitive type that defined how the vertices are to be interpreted (e.g. triangles, lines, etc.) when +rendering the surface. + +Filament does not directly support normals. Instead, it encodes the normal, tangent, and bitangent into a 4-component +quaternion describing the "orientation" of the vertex. Ideally, you should preprocess your assets to generate this data +offline, but we will compute it on the fly if needed (at a performance cost). + +Vertex data may or may not be interleaved. Interleaved data assumes that the attributes are packed in the order +specified in the attributes array, with no padding in-between. Additionally, the `data` pointer for each attribute is +assumed to point to the first element of that type. For non-interleaved data, each attribute is assumed to be stored in +a separate array. + +Additionally, the bounds of the mesh should be computed in order to allow the filament renderer to perform frustum-based +culling. Alternatively, the bounds can be computed at runtime (though there is a small performance cost). If no bounds +are provided (or calculated), then frustum culling will not be performed. + + +.. _mjrfMeshData: + +mjrfMeshData +~~~~~~~~~~~~ + +Binary data used for creating a :ref:`mesh`. + +.. mujoco-include:: mjrfMeshData + + +.. _mjrfScene: + +mjrfScene +~~~~~~~~~ + +A Scene is a collection of :ref:`Lights` and :ref:`Renderables` that describes what is to be +rendered. + + +.. _mjrfSceneParams: + +mjrfSceneParams +~~~~~~~~~~~~~~~ + +Parameters for creating a :ref:`scene`. + +.. mujoco-include: mjrfSceneParams + + +.. _mjrfLight: + +mjrfLight +~~~~~~~~~ + +A light is a source of illumination. (Without lights, a rendered image will be completely black.) There are several +different types of lights such as directional, spot, point, and image lights. + +The primary light in a scene is the image light (also sometimes known as the environment light). This is a light that +"surrounds" the entire scene and is defined as a 3D texture. Each "pixel" of the cubemap is interpreted as the color of +projected into the scene from a particular direction. + +The texture used for image-based lighting can be generated using filament's `cmgen` tool. The tool should be configured +to output a KTX file from your source image. This tool calculates additional data (i.e. the spherical harmonics) and +encodes that information into the KTX file. + +Directional lights are the next most common type of light and is usually used to simulate the sun; a uniformly colored +light that is emitted in a single direction. + +Filament only supports a single image and directional light. You can define as many point or spot lights as you want. +Each light source (except image based lights) may or may not cast shadows. Each shadow-casting light incurs a +performance cost. + + +.. _mjrfLightParams: + +mjrfLightParams +~~~~~~~~~~~~~~~ + +Parameters for creating a :ref:`light`. + +.. mujoco-include:: mjrfLightParams + + +.. _mjrfRenderable: + +mjrfRenderable +~~~~~~~~~~~~~~ + +A renderable is a single object that is to be drawn. It is defined as a combination of a :ref:`mjrfMesh` +(i.e. the shape or surface geometry, as described above) and a :ref:`mjrfMaterial` (i.e. a description of +how the surface interacts with lights to product the final visual appearance). + + +.. _mjrfRenderableParams: + +mjrfRenderableParams +~~~~~~~~~~~~~~~~~~~~ + +Parameters for creating a :ref:`renderable`. + +.. mujoco-include:: mjrfRenderableParams + + +.. _mjrfMaterial: + +mjrfMaterial +~~~~~~~~~~~~ + +Materials describe the properties of the surface of a renderable, effectively dictating how the surface interacts with +lights to produce a final pixel color in the output image. Different lighting models will be applied to the surface +depending on the values of the material properties. There there are three lighting models currently supported: + +1. Metallic-roughness (PBR): this is the preferred model for rendering models based standard +metallic-roughness workflows. + +2. Specular-glossiness (non-PBR): this is a legacy model designed to be compatible with classic +:ref:`mjr` renderer, though it is not 100% identical. + +3. Unlit: this model ignores lighting and is used for rendering UX or decorative elements like +contact forces and labels. + +.. mujoco-include:: mjrfMaterial + + +.. _mjrfRenderTarget: + +mjrfRenderTarget +~~~~~~~~~~~~~~~~ + +A RenderTarget is a memory buffer that holds the results of a rendering operation. (This is an alternative to rendering +directly to the screen.) + + +.. _mjrfRenderTargetConfig: + +mjrfRenderTargetConfig +~~~~~~~~~~~~~~~~~~~~~~ + +Parameters for creating a :ref:`render target`. + +.. mujoco-include:: mjrfRenderTargetConfig + + +.. _mjrfRenderRequest: + +mjrfRenderRequest +~~~~~~~~~~~~~~~~~ + +A single rendering operation. + +.. mujoco-include:: mjrfRenderRequest + + +.. _mjrfReadPixelsRequest: + +mjrfReadPixelsRequest +~~~~~~~~~~~~~~~~~~~~~ + +A single pixel read operation. + +.. mujoco-include:: mjrfReadPixelsRequest + + +.. _mjrfFrameStats: + +mjrfFrameStats +~~~~~~~~~~~~~~ + +Information about a single frame of rendering. + +.. mujoco-include:: mjrfFrameStats + + .. _tyUIStructure: User Interface diff --git a/doc/APIreference/functions.rst b/doc/APIreference/functions.rst index c8a3d087..2f30b473 100644 --- a/doc/APIreference/functions.rst +++ b/doc/APIreference/functions.rst @@ -3048,6 +3048,506 @@ Call glGetError and return result. Find first rectangle containing mouse, -1: not found. +.. _FilamentRenderingApi: + +Filament rendering +^^^^^^^^^^^^^^^^^^ + +Rendering functions using the Filament rendering engine. These functions are prefixed with ``mjrf``. See +:ref:`Filament Rendering` for an overview of the core types and their uses. + +.. _mjrf_defaultContextConfig: + +`mjrf_defaultContextConfig <#mjrf_defaultContextConfig>`__ +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +.. mujoco-include:: mjrf_defaultContextConfig + +Initializes the mjrfContextConfig to default values. + +.. _mjrf_createContext: + +`mjrf_createContext <#mjrf_createContext>`__ +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +.. mujoco-include:: mjrf_createContext + +Creates a filament rendering context. + +.. _mjrf_destroyContext: + +`mjrf_destroyContext <#mjrf_destroyContext>`__ +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +.. mujoco-include:: mjrf_destroyContext + +Destroys the filament rendering context. + +.. _mjrf_getRendererInfo: + +`mjrf_getRendererInfo <#mjrf_getRendererInfo>`__ +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +.. mujoco-include:: mjrf_getRendererInfo + +Gets active renderer information for the given filament context. + +.. _mjrf_defaultRenderRequest: + +`mjrf_defaultRenderRequest <#mjrf_defaultRenderRequest>`__ +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +.. mujoco-include:: mjrf_defaultRenderRequest + +Initializes the mjrfRenderRequest to default values. + +.. _mjrf_defaultReadPixelsRequest: + +`mjrf_defaultReadPixelsRequest <#mjrf_defaultReadPixelsRequest>`__ +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +.. mujoco-include:: mjrf_defaultReadPixelsRequest + +Initializes the mjrfReadPixelsRequest to default values. + +.. _mjrf_render: + +`mjrf_render <#mjrf_render>`__ +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +.. mujoco-include:: mjrf_render + +Submits the given requests for rendering. Because rendering happens asynchronously, callers have +to submit both the render and read requests in the same call. Multiple requests and reads can be +submitted in a single call. These requests will be processed in order, so some care must be +taken. Firstly, requests should be grouped by target. Next, the combined area of the viewports +for all requests for a given target must be contained within the dimensions of the target itself. + +Callbacks will be invoked from within this function, though there is no guarantee on which +invocation of this function it will be done. + +.. _mjrf_waitForFrame: + +`mjrf_waitForFrame <#mjrf_waitForFrame>`__ +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +.. mujoco-include:: mjrf_waitForFrame + +Waits for all rendering operations to complete for the given frame handle, triggering any +callbacks as needed. + +.. _mjrf_setClearColor: + +`mjrf_setClearColor <#mjrf_setClearColor>`__ +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +.. mujoco-include:: mjrf_setClearColor + +Sets the clear color for the renderer. + +.. _mjrf_defaultFrameStats: + +`mjrf_defaultFrameStats <#mjrf_defaultFrameStats>`__ +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +.. mujoco-include:: mjrf_defaultFrameStats + +Initializes the mjrFrameStats to default values. + +.. _mjrf_getFrameStats: + +`mjrf_getFrameStats <#mjrf_getFrameStats>`__ +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +.. mujoco-include:: mjrf_getFrameStats + +Returns the stats for the given frame but updating the given `stats_out`. + +.. _mjrf_defaultTextureConfig: + +`mjrf_defaultTextureConfig <#mjrf_defaultTextureConfig>`__ +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +.. mujoco-include:: mjrf_defaultTextureConfig + +Initializes the mjrfTextureConfig to default values. + +.. _mjrf_createTexture: + +`mjrf_createTexture <#mjrf_createTexture>`__ +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +.. mujoco-include:: mjrf_createTexture + +Creates a filament texture. Note that the texture will not be created on the GPU until +`mjrf_setTextureData()` is called. + +.. _mjrf_destroyTexture: + +`mjrf_destroyTexture <#mjrf_destroyTexture>`__ +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +.. mujoco-include:: mjrf_destroyTexture + +Destroys the texture. + +.. _mjrf_defaultTextureData: + +`mjrf_defaultTextureData <#mjrf_defaultTextureData>`__ +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +.. mujoco-include:: mjrf_defaultTextureData + +Initializes the mjrfTextureData to default values. + +.. _mjrf_setTextureData: + +`mjrf_setTextureData <#mjrf_setTextureData>`__ +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +.. mujoco-include:: mjrf_setTextureData + +Uploads the given texture data to the texture. + +.. _mjrf_getTextureWidth: + +`mjrf_getTextureWidth <#mjrf_getTextureWidth>`__ +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +.. mujoco-include:: mjrf_getTextureWidth + +Returns the width of the texture. + +.. _mjrf_getTextureHeight: + +`mjrf_getTextureHeight <#mjrf_getTextureHeight>`__ +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +.. mujoco-include:: mjrf_getTextureHeight + +Returns the height of the texture. + +.. _mjrf_getTextureSamplerType: + +`mjrf_getTextureSamplerType <#mjrf_getTextureSamplerType>`__ +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +.. mujoco-include:: mjrf_getTextureSamplerType + +Returns the sampler type (mjrSamplerType) used by the texture. +[returns: mjrSamplerType] + +.. _mjrf_defaultMeshData: + +`mjrf_defaultMeshData <#mjrf_defaultMeshData>`__ +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +.. mujoco-include:: mjrf_defaultMeshData + +Initializes the mjrfMeshData to default values. + +.. _mjrf_createMesh: + +`mjrf_createMesh <#mjrf_createMesh>`__ +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +.. mujoco-include:: mjrf_createMesh + +Creates a mesh with the given data. + +.. _mjrf_destroyMesh: + +`mjrf_destroyMesh <#mjrf_destroyMesh>`__ +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +.. mujoco-include:: mjrf_destroyMesh + +Destroys the mesh. + +.. _mjrf_defaultSceneParams: + +`mjrf_defaultSceneParams <#mjrf_defaultSceneParams>`__ +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +.. mujoco-include:: mjrf_defaultSceneParams + +Initializes the mjrfSceneParams to default values. + +.. _mjrf_createScene: + +`mjrf_createScene <#mjrf_createScene>`__ +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +.. mujoco-include:: mjrf_createScene + +Creates a scene with the given parameters. + +.. _mjrf_destroyScene: + +`mjrf_destroyScene <#mjrf_destroyScene>`__ +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +.. mujoco-include:: mjrf_destroyScene + +Destroys the scene. + +.. _mjrf_addLightToScene: + +`mjrf_addLightToScene <#mjrf_addLightToScene>`__ +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +.. mujoco-include:: mjrf_addLightToScene + +Adds a light to the scene. + +.. _mjrf_removeLightFromScene: + +`mjrf_removeLightFromScene <#mjrf_removeLightFromScene>`__ +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +.. mujoco-include:: mjrf_removeLightFromScene + +Removes the light from the scene. + +.. _mjrf_addRenderableToScene: + +`mjrf_addRenderableToScene <#mjrf_addRenderableToScene>`__ +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +.. mujoco-include:: mjrf_addRenderableToScene + +Adds a renderable to the scene. + +.. _mjrf_removeRenderableFromScene: + +`mjrf_removeRenderableFromScene <#mjrf_removeRenderableFromScene>`__ +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +.. mujoco-include:: mjrf_removeRenderableFromScene + +Removes the renderable from the scene. + +.. _mjrf_setSceneSkybox: + +`mjrf_setSceneSkybox <#mjrf_setSceneSkybox>`__ +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +.. mujoco-include:: mjrf_setSceneSkybox + +Sets the skybox (cube texture) for the scene. + +.. _mjrf_configureSceneFromModel: + +`mjrf_configureSceneFromModel <#mjrf_configureSceneFromModel>`__ +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +.. mujoco-include:: mjrf_configureSceneFromModel + +Configures the scene based on the parameters in an mjModel. + +.. _mjrf_defaultLightParams: + +`mjrf_defaultLightParams <#mjrf_defaultLightParams>`__ +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +.. mujoco-include:: mjrf_defaultLightParams + +Initializes the mjrfLightParams to default values. + +.. _mjrf_createLight: + +`mjrf_createLight <#mjrf_createLight>`__ +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +.. mujoco-include:: mjrf_createLight + +Creates a light for the filament renderer. + +.. _mjrf_destroyLight: + +`mjrf_destroyLight <#mjrf_destroyLight>`__ +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +.. mujoco-include:: mjrf_destroyLight + +Destroys the light. + +.. _mjrf_setLightEnabled: + +`mjrf_setLightEnabled <#mjrf_setLightEnabled>`__ +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +.. mujoco-include:: mjrf_setLightEnabled + +Enables or disables the light. + +.. _mjrf_setLightIntensity: + +`mjrf_setLightIntensity <#mjrf_setLightIntensity>`__ +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +.. mujoco-include:: mjrf_setLightIntensity + +Sets the intensity of the light, in candela. + +.. _mjrf_setLightColor: + +`mjrf_setLightColor <#mjrf_setLightColor>`__ +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +.. mujoco-include:: mjrf_setLightColor + +Sets the RGB color of the light. + +.. _mjrf_setLightTransform: + +`mjrf_setLightTransform <#mjrf_setLightTransform>`__ +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +.. mujoco-include:: mjrf_setLightTransform + +Sets the position and direction of the light. + +.. _mjrf_getLightType: + +`mjrf_getLightType <#mjrf_getLightType>`__ +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +.. mujoco-include:: mjrf_getLightType + +Returns the type of the light (mjrLightType). + +.. _mjrf_defaultMaterial: + +`mjrf_defaultMaterial <#mjrf_defaultMaterial>`__ +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +.. mujoco-include:: mjrf_defaultMaterial + +Initializes the mjrfMaterial to default values. + +.. _mjrf_defaultRenderableParams: + +`mjrf_defaultRenderableParams <#mjrf_defaultRenderableParams>`__ +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +.. mujoco-include:: mjrf_defaultRenderableParams + +Initializes the mjrfRenderableParams to default values. + +.. _mjrf_createRenderable: + +`mjrf_createRenderable <#mjrf_createRenderable>`__ +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +.. mujoco-include:: mjrf_createRenderable + +Creates a renderable with the given parameters. + +.. _mjrf_destroyRenderable: + +`mjrf_destroyRenderable <#mjrf_destroyRenderable>`__ +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +.. mujoco-include:: mjrf_destroyRenderable + +Destroys the renderable. + +.. _mjrf_setRenderableMesh: + +`mjrf_setRenderableMesh <#mjrf_setRenderableMesh>`__ +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +.. mujoco-include:: mjrf_setRenderableMesh + +Sets the mesh of the renderable. + +.. _mjrf_setRenderableGeomMesh: + +`mjrf_setRenderableGeomMesh <#mjrf_setRenderableGeomMesh>`__ +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +.. mujoco-include:: mjrf_setRenderableGeomMesh + +Sets the mesh of the renderable to a built-in mesh based on the geom type. Note: using the same +parameters (nstack, nslice, nquad) will have better performance as the internal mesh data can be +shared across renderables. +[type: mjtGeom] + +.. _mjrf_setRenderableMaterial: + +`mjrf_setRenderableMaterial <#mjrf_setRenderableMaterial>`__ +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +.. mujoco-include:: mjrf_setRenderableMaterial + +Sets the material properties and textures of the renderable. + +.. _mjrf_getRenderableMaterial: + +`mjrf_getRenderableMaterial <#mjrf_getRenderableMaterial>`__ +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +.. mujoco-include:: mjrf_getRenderableMaterial + +Copies the material properties of the renderable into the given mjrfMaterial. + +.. _mjrf_setRenderableTransform: + +`mjrf_setRenderableTransform <#mjrf_setRenderableTransform>`__ +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +.. mujoco-include:: mjrf_setRenderableTransform + +Sets the transform position and rotation of the renderable. + +.. _mjrf_setRenderableSize: + +`mjrf_setRenderableSize <#mjrf_setRenderableSize>`__ +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +.. mujoco-include:: mjrf_setRenderableSize + +Sets the size of the renderable. Note that, for most renderables, this is equivalent to setting +the scale. However, for some geom-based renderables, the size scale is not applied uniformly +(e.g. the spherical ends of a capsule are scaled such that they always remain spherical). + +.. _mjrf_defaultRenderTargetConfig: + +`mjrf_defaultRenderTargetConfig <#mjrf_defaultRenderTargetConfig>`__ +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +.. mujoco-include:: mjrf_defaultRenderTargetConfig + +Initializes the RenderTargetConfig to default values. + +.. _mjrf_createRenderTarget: + +`mjrf_createRenderTarget <#mjrf_createRenderTarget>`__ +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +.. mujoco-include:: mjrf_createRenderTarget + +Creates a render target for the filament renderer. + +.. _mjrf_destroyRenderTarget: + +`mjrf_destroyRenderTarget <#mjrf_destroyRenderTarget>`__ +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +.. mujoco-include:: mjrf_destroyRenderTarget + +Destroys the render target. + +.. _mjrf_resizeRenderTarget: + +`mjrf_resizeRenderTarget <#mjrf_resizeRenderTarget>`__ +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +.. mujoco-include:: mjrf_resizeRenderTarget + +Resizes the render target to the given width and height. + .. _UIframework: UI framework diff --git a/doc/APIreference/functions_override.rst b/doc/APIreference/functions_override.rst index bd673a6f..cf61a14d 100644 --- a/doc/APIreference/functions_override.rst +++ b/doc/APIreference/functions_override.rst @@ -434,6 +434,11 @@ The functions in this section implement abstract visualization. The results are also be used by users wishing to implement their own renderer, or hook up MuJoCo to advanced rendering tools such as Unity or Unreal Engine. See :ref:`simulate` for illustration of how to use these functions. +.. _FilamentRenderingApi: + +Rendering functions using the Filament rendering engine. These functions are prefixed with ``mjrf``. See +:ref:`Filament Rendering` for an overview of the core types and their uses. + .. _OpenGLrendering: These functions expose the OpenGL renderer. See :ref:`simulate` for an illustration diff --git a/doc/ext/header_reader.py b/doc/ext/header_reader.py index 3d21207d..2afb56e3 100644 --- a/doc/ext/header_reader.py +++ b/doc/ext/header_reader.py @@ -115,7 +115,9 @@ def read(lines: List[str]) -> Dict[str, ApiDefinition]: if stripped_functions: s.code = f'{s.code}{line}' else: - s.code = f'{s.code}{line[6:]}' + decl = line.replace('MJAPI ', '') + decl = decl[6:] if decl.startswith(' ' * 6) else decl + s.code = f'{s.code}{decl}' s.token = token s.start('FUNCTION') if _is_function_end(line): @@ -142,7 +144,9 @@ def read(lines: List[str]) -> Dict[str, ApiDefinition]: if stripped_functions: s.code = f'{s.code}{line}' else: - s.code = f'{s.code}{line[6:]}' + decl = line.replace('MJAPI ', '') + decl = decl[6:] if decl.startswith(' ' * 6) else decl + s.code = f'{s.code}{decl}' if _is_function_end(line): api[s.token] = s.export_definition() s.end() @@ -176,6 +180,12 @@ def read(lines: List[str]) -> Dict[str, ApiDefinition]: s.start('STRUCT') s.code = f'{s.code}{line}' + match = _STRUCT_END_REGEX_1.search(line) + if match is not None: + s.token = match.group('token') + api[s.token] = s.export_definition() + s.end() + if line.startswith('//'): s.doc = f'{s.doc}{line[3:]}' s.start('DOC') @@ -185,7 +195,9 @@ def read(lines: List[str]) -> Dict[str, ApiDefinition]: if stripped_functions: s.code = f'{s.code}{line}' else: - s.code = f'{s.code}{line[6:]}' + decl = line.replace('MJAPI ', '') + decl = decl[6:] if decl.startswith(' ' * 6) else decl + s.code = f'{s.code}{decl}' s.token = token s.start('FUNCTION') if _is_function_end(line): @@ -203,7 +215,13 @@ def _find_section(line) -> Optional[str]: def _find_function_start(line, stripped) -> Optional[str]: - if (line.startswith('MJAPI') and 'extern' not in line) or stripped: + if ( + 'extern' not in line + and 'typedef' not in line + and '#define' not in line + and '_DEBUG_' not in line + and not line.startswith('//') + ) or stripped: match = _FUNCTION_REGEX.search(line) if match is not None: return match.group('token') diff --git a/doc/includes/references.h b/doc/includes/references.h index 45352ae7..e813ba53 100644 --- a/doc/includes/references.h +++ b/doc/includes/references.h @@ -1479,6 +1479,131 @@ typedef struct mjrContext_ { // custom OpenGL context // depth output format int readDepthMap; // depth mapping: mjDEPTH_ZERONEAR or mjDEPTH_ZEROFAR } mjrContext; +typedef struct mjrfContext_ mjrfContext; +typedef struct mjrfTexture_ mjrfTexture; +typedef struct mjrfMesh_ mjrfMesh; +typedef struct mjrfScene_ mjrfScene; +typedef struct mjrfLight_ mjrfLight; +typedef struct mjrfRenderable_ mjrfRenderable; +typedef struct mjrfRenderTarget_ mjrfRenderTarget; +typedef enum mjrGraphicsApi_ { + mjGRAPHICS_API_DEFAULT = 0, // default (platform-dependent) + mjGRAPHICS_API_OPENGL, // desktop, mobile (GLES), web (WebGL) + mjGRAPHICS_API_VULKAN, // vulkan +} mjrGraphicsApi; +typedef struct mjrfContextConfig_ { + int graphics_api; // rendering graphics API [mjrGraphicsApi] + mjtBool force_software_rendering; // force backend to use software rendering + void* native_window; // platform-dependent window handle (or nullptr for windowless) +} mjrfContextConfig; +typedef enum mjrDrawMode_ { + mjDRAW_MODE_DEFAULT, // default colors and lighting + mjDRAW_MODE_DEFAULT_NO_TEXTURES, // default, but without textures + mjDRAW_MODE_WIREFRAME, // wireframe rendering + mjDRAW_MODE_DEPTH, // grayscale depth map + mjDRAW_MODE_ISLANDS, // color objects based on island and sleep state + mjDRAW_MODE_SEGMENTATION_BY_ID, // color objects based on segmentation id + mjDRAW_MODE_SEGMENTATION_BY_COLOR, // generate visually distinct colors using segmentation id +} mjrDrawMode; +typedef struct mjrfRenderRequest_ { + mjrfScene* scene; // scene to render + mjrCamera camera; // camera (viewpoint) from which to render scene + mjrRect viewport; // viewport (rect area) into which to render + mjrfRenderTarget* target; // target used for rendering (or nullptr for window rendering) + int draw_mode; // method to use for drawing objects [mjrDrawMode] + mjtBool enable_post_processing; // enable post processing, enabled by default + mjtBool enable_reflections; // enable reflections, enabled by default + mjtBool enable_shadows; // enable shadows, enabled by default +} mjrfRenderRequest; +typedef struct mjrfReadPixelsRequest_ { + mjrfRenderTarget* target; // render target from which to read the image pixels + void* output; // buffer into which the pixels will be stored + mjtSize num_bytes; // size of output buffer + mjrfCallback read_completed; // callback when read is complete; can use to free output + void* user_data; // user data for read_completed_callback +} mjrfReadPixelsRequest; +typedef struct mjrfFrameStats_ { + double frame_rate; // frame rate, in frames per second +} mjrfFrameStats; +typedef struct mjrfTextureConfig_ { + int width; // texture width, or number of bytes for compressed data (e.g. KTX) + int height; // texture height, or 0 for compressed data (e.g. KTX) + int format; // pixel format (e.g. RGB8, RGBA8, KTX, etc.) [mjrPixelFormat] + int color_space; // color space (e.g. LINEAR, sRGB, etc.) [mjrColorSpace] + int sampler_type; // texture sampler (e.g. 2D, cube, etc.) [mjrSamplerType] +} mjrfTextureConfig; +typedef struct mjrfTextureData_ { + const void* bytes; // pointer to image data, or nullptr for empty texture + mjtSize num_bytes; // number of bytes in the image data + mjrfCallback release; // callback when data has finished uploading + void* user_data; // user data for release callback +} mjrfTextureData; +typedef struct mjrfMeshData_ { + mjtSize num_vertices; // number of vertices; all vertex attributes share this size + int num_attributes; // number of attributes defined + mjrVertexAttribute attributes[mjMAX_VERTEX_ATTRIBUTES]; // per-vertex attribute information + mjtBool interleaved; // true if vertex attributes are interleaved + mjtSize num_indices; // number of indices + const void* indices; // indices data array + int index_type; // index data format (e.g. UINT16 or UINT32) [mjrIndexType] + int primitive_type; // index interpretation (e.g. TRIANGLES, etc.) [mjrMeshPrimitiveType] + mjtBool compute_bounds; // if true, compute bounds from vertex positions + float bounds_min[3]; // min/max bounds; assume unset if bounds_min == bounds_max + float bounds_max[3]; + mjrfCallback release; // callback when data has finished uploading + void* user_data; // user data for release callback +} mjrfMeshData; +typedef struct mjrfSceneParams_ { +} mjrfSceneParams; +typedef struct mjrfLightParams_ { + int type; // type of light (e.g. spot, point, image, etc.) [mjrLightType] + const mjrfTexture* texture; // texture; only for image lights + float color[3]; // RGB color + float intensity; // light intensity, in candela + mjtBool cast_shadows; // if true, cast shadows + float range; // effective range of light, in meters + float spot_cone_angle; // spot light cone angle, in degrees + int shadow_map_size; // size of shadow map texture, 0 to use default size + float bulb_radius; // bulb radius, used for soft shadows + float vsm_blur_width; // variance shadow map blur width +} mjrfLightParams; +typedef struct mjrfMaterial_ { + float color[4]; // object color; defaults to white + int32_t segmentation_id; // ID for segmentation rendering; maps to RGB8 color (i.e. 24 bits) + int32_t island_id; // ID to which the renderable belongs + int sleep_state; // sleep state of the renderable [mjtSleepState] + float uv_scale[3]; // scale applied to UV coordinates; defaults to (1,1,1) + float uv_offset[3]; // offset applied to UV coordinates; defaults to (0,0,0) + float scissor[4]; // if non-zero, applies scissor testing when rendering + float metallic; // metallic factory [0, 1]; disabled if < 0 + float roughness; // roughness factor [0, 1]; disabled if < 0 + float specular; // specular factor [0, 1]; disabled if < 0 + float glossiness; // glossiness factor [0, 1]; disabled if < 0 + float emissive; // emissive/glow factor [0, 1]; disabled if < 0 + float reflectance; // blend factor for reflective surfaces [0, 1]; applies only to planes + mjtBool decor_ux; // for ux elements, does not apply any lighting + mjtBool selected; // for "selected" ux elements, adds additional styling + const mjrfTexture* color_texture; // color/albedo texture (RGB8) + const mjrfTexture* opacity_texture; // opacity texture (A8) + const mjrfTexture* normal_texture; // normal map texture (RGB8) + const mjrfTexture* metallic_texture; // metallic map texture (R8) + const mjrfTexture* roughness_texture; // roughness map texture (R8) + const mjrfTexture* occlusion_texture; // ambient occlusion texture (R8) + const mjrfTexture* orm_texture; // occlusion/roughness/metallic texture (RGB8) + const mjrfTexture* emissive_texture; // emissive texture (RGB8) + const mjrfTexture* reflection_texture; // reflection texture, for internal use only +} mjrfMaterial; +typedef struct mjrfRenderableParams_ { + mjtBool cast_shadows; // if true, casts shadows + mjtBool receive_shadows; // if true, receives shadows + uint16_t blend_order; // controls draw order for transparent objects [0, 8] +} mjrfRenderableParams; +typedef struct mjrfRenderTargetConfig_ { + int width; // texture width + int height; // texture height + int color_format; // pixel format for color buffer [mjrPixelFormat] + int depth_format; // pixel format for depth buffer [mjrPixelFormat] +} mjrfRenderTargetConfig; typedef enum mjtGeomInertia { // type of inertia inference mjINERTIA_VOLUME = 0, // mass distributed in the volume mjINERTIA_SHELL, // mass distributed on the surface @@ -3219,6 +3344,63 @@ typedef struct mjvFigure_ { // abstract 2D figure passed to OpenGL rendere } mjvFigure; //----------------------------- MJAPI FUNCTIONS -------------------------------- +void mjrf_defaultContextConfig(mjrfContextConfig* config); +mjrfContext* mjrf_createContext(const mjrfContextConfig* config); +void mjrf_destroyContext(mjrfContext* ctx); +void mjrf_getRendererInfo(mjrfContext* ctx, mjrRendererInfo* info); +void mjrf_defaultRenderRequest(mjrfRenderRequest* request); +void mjrf_defaultReadPixelsRequest(mjrfReadPixelsRequest* request); +mjrfFrameHandle mjrf_render(mjrfContext* ctx, const mjrfRenderRequest* req, int nreq, + const mjrfReadPixelsRequest* read_req, int nread_req); +void mjrf_waitForFrame(mjrfContext* ctx, mjrfFrameHandle frame); +void mjrf_setClearColor(mjrfContext* ctx, const float color[3]); +void mjrf_defaultFrameStats(mjrfFrameStats* stats); +void mjrf_getFrameStats(mjrfContext* ctx, mjrfFrameHandle frame, mjrfFrameStats* stats_out); +void mjrf_defaultTextureConfig(mjrfTextureConfig* config); +mjrfTexture* mjrf_createTexture(mjrfContext* ctx, const mjrfTextureConfig* config); +void mjrf_destroyTexture(mjrfTexture* texture); +void mjrf_defaultTextureData(mjrfTextureData* data); +void mjrf_setTextureData(mjrfTexture* texture, const mjrfTextureData* data); +int mjrf_getTextureWidth(const mjrfTexture* texture); +int mjrf_getTextureHeight(const mjrfTexture* texture); +int mjrf_getTextureSamplerType(const mjrfTexture* texture); +void mjrf_defaultMeshData(mjrfMeshData* data); +mjrfMesh* mjrf_createMesh(mjrfContext* ctx, const mjrfMeshData* data); +void mjrf_destroyMesh(mjrfMesh* mesh); +void mjrf_defaultSceneParams(mjrfSceneParams* params); +mjrfScene* mjrf_createScene(mjrfContext* ctx, const mjrfSceneParams* params); +void mjrf_destroyScene(mjrfScene* scene); +void mjrf_addLightToScene(mjrfScene* scene, mjrfLight* light); +void mjrf_removeLightFromScene(mjrfScene* scene, mjrfLight* light); +void mjrf_addRenderableToScene(mjrfScene* scene, mjrfRenderable* renderable); +void mjrf_removeRenderableFromScene(mjrfScene* scene, mjrfRenderable* renderable); +void mjrf_setSceneSkybox(mjrfScene* scene, const mjrfTexture* texture); +void mjrf_configureSceneFromModel(mjrfScene* scene, const mjModel* model); +void mjrf_defaultLightParams(mjrfLightParams* params); +mjrfLight* mjrf_createLight(mjrfContext* ctx, const mjrfLightParams* params); +void mjrf_destroyLight(mjrfLight* light); +void mjrf_setLightEnabled(mjrfLight* light, mjtBool enabled); +void mjrf_setLightIntensity(mjrfLight* light, float intensity); +void mjrf_setLightColor(mjrfLight* light, const float color[3]); +void mjrf_setLightTransform(mjrfLight* light, const float position[3], const float direction[3]); +int mjrf_getLightType(const mjrfLight* light); +void mjrf_defaultMaterial(mjrfMaterial* material); +void mjrf_defaultRenderableParams(mjrfRenderableParams* params); +mjrfRenderable* mjrf_createRenderable(mjrfContext* ctx, const mjrfRenderableParams* params); +void mjrf_destroyRenderable(mjrfRenderable* renderable); +void mjrf_setRenderableMesh(mjrfRenderable* renderable, const mjrfMesh* mesh, int elem_offset, + int elem_count); +void mjrf_setRenderableGeomMesh(mjrfRenderable* renderable, int type, int nstack, int nslice, + int nquad); +void mjrf_setRenderableMaterial(mjrfRenderable* renderable, const mjrfMaterial* material); +void mjrf_getRenderableMaterial(mjrfRenderable* renderable, mjrfMaterial* material); +void mjrf_setRenderableTransform(mjrfRenderable* renderable, const float position[3], + const float rotation[9]); +void mjrf_setRenderableSize(mjrfRenderable* renderable, const float size[3]); +void mjrf_defaultRenderTargetConfig(mjrfRenderTargetConfig* config); +mjrfRenderTarget* mjrf_createRenderTarget(mjrfContext* ctx, const mjrfRenderTargetConfig* config); +void mjrf_destroyRenderTarget(mjrfRenderTarget* render_target); +void mjrf_resizeRenderTarget(mjrfRenderTarget* render_target, int width, int height); void mj_defaultVFS(mjVFS* vfs); int mj_mountVFS(mjVFS* vfs, const char* filepath, const mjpResourceProvider* provider); int mj_unmountVFS(mjVFS* vfs, const char* filename); diff --git a/doc/js/linenumbers.js b/doc/js/linenumbers.js index cb46b871..aa107936 100644 --- a/doc/js/linenumbers.js +++ b/doc/js/linenumbers.js @@ -38,6 +38,7 @@ const SRCS = [ 'render/classic/render_gl2.c', 'render/classic/render_gl3.c', 'render/classic/render_util.c', + 'render/filament/mjrfilament.cc', 'render/noop/render_noop.c', 'ui/ui_main.c', 'user/user_api.cc', diff --git a/doc/programming/index.rst b/doc/programming/index.rst index bbb640f1..9110ae8d 100644 --- a/doc/programming/index.rst +++ b/doc/programming/index.rst @@ -22,21 +22,22 @@ Parser Compiler The compiler is written in C++. It takes an mjCModel C++ object constructed by the parser, and converts it into an mjModel C structure used at runtime. -Abstract visualizer - The abstract visualizer is written in C. It generates a list of abstract geometric entities representing the - simulation state, with all information needed for actual rendering. It also provides abstract mouse hooks for camera - and perturbation control. -OpenGL renderer - The renderer is written in C and is based on fixed-function OpenGL. It does not have all the features of - state-of-the-art rendering engines (and can be replaced with such an engine if desired) but nevertheless it provides - efficient and informative 3D rendering. Thread The threading framework is written in C++ and exposed in C. It provides a thread pool interface to process tasks asynchronously. To enable use in MuJoCo, call ``mju_threadpool``. +Rendering + There are two rendering libraries provided by MuJoCo. The :ref:`classic rendering` library is + written in C and uses OpenGL 1.5. It provides a simple and efficient way to visualize MuJoCo models. The + :ref:`filament rendering` library is written in C++ and uses the externally-devloped Filament + rendering engine. It provides more modern and feature-rich real-time rendering capabilities. +Abstract visualizer + The abstract visualizer is written in C. It generates a list of abstract geometric entities representing the + simulation state, with all information needed for rendering with the Classic renderer. It also provides abstract + mouse hooks for camera and perturbation control. UI framework - The UI framework is written in C. UI elements are rendered in OpenGL. It has its own event - mechanism and abstract hooks for keyboard and mouse input. The code samples use it with GLFW, but it can also be used - with other window libraries. + The UI framework is written in C and is designed to work with the :ref:`classic OpenGL renderer`. + UI elements are rendered in OpenGL. It has its own event mechanism and abstract hooks for keyboard and mouse input. + The code samples use it with GLFW, but it can also be used with other window libraries. .. _inStart: @@ -236,14 +237,16 @@ to which the symbol belongs. First we list the prefixes corresponding to type de Primitive type, for example :ref:`mjtNum` and :ref:`mjtGeom`. Most types in this family are enums. ``mjf`` Callback function type, for example :ref:`mjfGeneric`. +``mjs`` + Data structure related to :doc:`procedural model editing `, for example :ref:`mjsJoint`. ``mjv`` Data structure related to abstract visualization, for example :ref:`mjvCamera`. +``mjrf`` + Data structure related to filament rendering, for example :ref:`mjrfContext`. ``mjr`` Data structure related to OpenGL rendering, for example :ref:`mjrContext`. ``mjui`` Data structure related to UI framework, for example :ref:`mjuiSection`. -``mjs`` - Data structure related to :doc:`procedural model editing `, for example :ref:`mjsJoint`. Next we list the prefixes corresponding to function definitions. Note that function prefixes always end with underscore. @@ -256,6 +259,8 @@ Next we list the prefixes corresponding to function definitions. Note that funct in the sense that they do not have mjModel and mjData pointers as their arguments. ``mjv_`` Function related to abstract visualization, for example :ref:`mjv_updateScene`. +``mjrf_`` + Function related to filament rendering, for example :ref:`mjrf_render`. ``mjr_`` Function related to OpenGL rendering, for example :ref:`mjr_render`. ``mjui_`` @@ -268,23 +273,6 @@ Next we list the prefixes corresponding to function definitions. Note that funct ``mjs_`` Functions for :doc:`procedural model editing `, for example :ref:`mjs_addJoint`. -.. _inOpenGL: - -Using OpenGL -~~~~~~~~~~~~ - -The use of MuJoCo's native OpenGL renderer will be explained in :ref:`Rendering`. For rendering, MuJoCo uses OpenGL 1.5 -in the compatibility profile with the ``ARB_framebuffer_object`` and ``ARB_vertex_buffer_object`` extensions. OpenGL -symbols are loaded via `GLAD `_ the first time the :ref:`mjr_makeContext` function -is called. This means that the MuJoCo library itself does not have an explicit dependency on OpenGL and can be used -on systems without OpenGL support, as long as ``mjr_`` functions are not called. - -Applications that use MuJoCo's built-in rendering functionalities are responsible for linking against an appropriate -OpenGL context creation library and for ensuring that there is an OpenGL context that is made current on the running -thread. On Windows and macOS, there is a canonical OpenGL library provided by the operating system. On Linux, MuJoCo -currently supports GLX for rendering to an X11 window, OSMesa for headless software rendering, and EGL for hardware -accelerated headless rendering. - .. toctree:: :hidden: diff --git a/doc/programming/visualization.rst b/doc/programming/visualization.rst index 39800a4e..9b9ef516 100644 --- a/doc/programming/visualization.rst +++ b/doc/programming/visualization.rst @@ -3,6 +3,13 @@ Visualization ------------- +.. admonition:: MuJoCo Studio + :class: note + + We are actively developing a new visualizer platform called + `MuJoCo Studio `__. We will update this + section once it becomes more established. + MuJoCo has a native 3D visualizer. Its use is illustrated in the :ref:`simulate.cc ` code sample and in the simpler :ref:`basic.cc ` code sample. While it is not a full-featured rendering engine, it is a convenient, efficient and reasonably good-looking visualizer that facilitates research and development. It renders not @@ -303,6 +310,23 @@ OpenGL Rendering This stage takes the mjvScene data structure populated in the abstract visualization stage, and renders it. It also provides basic 2D drawing and framebuffer access, so that most applications would not need to call OpenGL directly. +.. _reOpenGL: + +Using OpenGL +''''''''''''' + +MuJoCo uses OpenGL 1.5 in the compatibility profile with the ``ARB_framebuffer_object`` and ``ARB_vertex_buffer_object`` +extensions. OpenGL symbols are loaded via `GLAD `_ the first time the +:ref:`mjr_makeContext` function is called. This means that the MuJoCo library itself does not have an explicit +dependency on OpenGL and can be used on systems without OpenGL support, as long as ``mjr_`` functions are not called. + +Applications that use MuJoCo's built-in rendering functionalities are responsible for linking against an appropriate +OpenGL context creation library and for ensuring that there is an OpenGL context that is made current on the running +thread. On Windows and macOS, there is a canonical OpenGL library provided by the operating system. On Linux, MuJoCo +currently supports GLX for rendering to an X11 window, OSMesa for headless software rendering, and EGL for hardware +accelerated headless rendering. + + .. _reContext: Context and GPU resources @@ -462,3 +486,21 @@ We also provide the functions :ref:`mjr_finish` and :ref:`mjr_getError` for expl for OpenGL error checking. They simply call glFinish and glGetError internally. This together with the basic 2d drawing functions above is meant to provide enough functionality so that most users will not need to write OpenGL code. Of course we cannot achieve this in all cases, short of providing wrappers for all of OpenGL. + + +.. _FilamentRendering: + +Filament Rendering +~~~~~~~~~~~~~~~~~~ + +MuJoCo also provides a `Filament `_ based renderer for 3D visualization of its +simulations. + +Filament is a real-time physically based rendering (PBR) engine developed by Google. It is designed to be as small as +possible and as efficient as possible, while still providing high-quality results. It works across all major platforms +(Linux, Windows, macOS, Android, iOS, Web) and supports OpenGL, Vulkan, and Metal. + +MuJoCo's current integration with the Filament renderer is done by setting `MUJOCO_USE_FILAMENT` to 1 in the CMake Build +configuration. This effectively replaces the OpenGL-based `mjr` function implementations with Filament-based ones. It +also makes the underlying Filament `mjrf` :ref:`types ` and +:ref:`functions ` available for use. diff --git a/include/mujoco/mjrfilament.h b/include/mujoco/mjrfilament.h index 79d88fda..04448652 100644 --- a/include/mujoco/mjrfilament.h +++ b/include/mujoco/mjrfilament.h @@ -25,33 +25,9 @@ extern "C" { #endif -// IMPORTANT: This API should still be considered experimental and is likely change frequently. +//---------------------------------- Filament rendering -------------------------------------------- -// This library provides a C API for the filament rendering library (github.com/google/filament) -// that is designed to work with the MuJoCo library for visualizing simulations. -// -// The filament renderer is a real-time physically based rendering (PBR) engine developed by Google. -// It is designed to be as small as possible and as efficient as possible, while still providing -// high-quality results. It works across all major platforms (Linux, Windows, macOS, Android, iOS, -// Web) and supports OpenGL, Vulkan, and Metal. -// -// For the purposes of this API, we assume the reader has a basic understanding of rendering -// concepts (e.g. textures, vertices, cameras, framebuffers, etc.). We will also highlight some of -// the key differences between this renderer and the legacy/classic MuJoCo (mjr) renderer. -// -// ## API Overview -// -// There are seven key components: Context, Texture, Mesh, Scene, Light, Renderable, and -// RenderTarget. We'll describe these in detail further below. -// -// Each object is created using a `create` function and destroyed using a `destroy` function, e.g. -// `mjrf_createTexture` and `mjrf_destroyTexture`. The `create` functions accept a pointer to a -// configuration struct (e.g. `mjrTextureConfig`) which describes the parameters for the object to -// be created. Each of these structs has a corresponding `default` function (e.g. -// `mjrf_defaultTextureConfig`) which can be used to initialize the struct to default values. Default -// values are assumed to be 0/NULL unless otherwise specified. -// -// For now, we'll just define opaque handles for each of our components. +// Opaque handles for each of our components. typedef struct mjrfContext_ mjrfContext; typedef struct mjrfTexture_ mjrfTexture; typedef struct mjrfMesh_ mjrfMesh; @@ -60,36 +36,18 @@ typedef struct mjrfLight_ mjrfLight; typedef struct mjrfRenderable_ mjrfRenderable; typedef struct mjrfRenderTarget_ mjrfRenderTarget; -// ## Rendering Context (mjrfContext) -// -// The Context is the main entry point for the library. It manages all the core filament objects -// that are responsible for the rendering of an image. -// -// All other objects (e.g. Textures, Meshes, Scenes, etc.) need a Context in order to be created. -// Otherwise, the main function to use with the Context is `mjrf_render()` which does the actual -// rendering. -// -// Filament uses a separate thread for doing the actual rendering. However, despite that, this API -// is not thread-safe; calls are expected to be made from a single thread. Also, due to the -// asynchronous nature of filament, some APIs provide handles or callbacks to signal when an -// operation is complete. (Note: for WASM builds, filament does not use a separate thread.) -// -// There are two key differences between the mjrfContext and the classic mjrContext. Firstly, the -// filament context will manage the underlying graphics context itself. This means users do not need -// to initialize EGL or similar libraries beforehand. Secondly, the filament context is independent -// of a MuJoCo model. That means you can use a single mjrfContext to render images for multiple -// models. - // Callback function type for rendering operations. typedef void (*mjrfCallback)(void* user_data); -typedef enum mjrGraphicsApi_ { // underlying graphics API to use for rendering +// Underlying graphics API to use for rendering. +typedef enum mjrGraphicsApi_ { mjGRAPHICS_API_DEFAULT = 0, // default (platform-dependent) mjGRAPHICS_API_OPENGL, // desktop, mobile (GLES), web (WebGL) mjGRAPHICS_API_VULKAN, // vulkan } mjrGraphicsApi; -typedef struct mjrfContextConfig_ { // parameters for creating filament context (mjrfContext) +// Parameters for creating filament graphics context (mjrfContext). +typedef struct mjrfContextConfig_ { int graphics_api; // rendering graphics API [mjrGraphicsApi] mjtBool force_software_rendering; // force backend to use software rendering void* native_window; // platform-dependent window handle (or nullptr for windowless) @@ -107,7 +65,8 @@ void mjrf_destroyContext(mjrfContext* ctx); // Gets active renderer information for the given filament context. void mjrf_getRendererInfo(mjrfContext* ctx, mjrRendererInfo* info); -typedef enum mjrDrawMode_ { // how to draw objects in the scene +// High-level control for how to draw objects in the scene. +typedef enum mjrDrawMode_ { mjDRAW_MODE_DEFAULT, // default colors and lighting mjDRAW_MODE_DEFAULT_NO_TEXTURES, // default, but without textures mjDRAW_MODE_WIREFRAME, // wireframe rendering @@ -117,7 +76,8 @@ typedef enum mjrDrawMode_ { // how to draw objects in the scene mjDRAW_MODE_SEGMENTATION_BY_COLOR, // generate visually distinct colors using segmentation id } mjrDrawMode; -typedef struct mjrfRenderRequest_ { // a single rendering operation +// A single rendering operation. +typedef struct mjrfRenderRequest_ { mjrfScene* scene; // scene to render mjrCamera camera; // camera (viewpoint) from which to render scene mjrRect viewport; // viewport (rect area) into which to render @@ -131,7 +91,8 @@ typedef struct mjrfRenderRequest_ { // a single rendering operation // Initializes the mjrfRenderRequest to default values. void mjrf_defaultRenderRequest(mjrfRenderRequest* request); -typedef struct mjrfReadPixelsRequest_ { // a single read operation +// A single pixel read operation. +typedef struct mjrfReadPixelsRequest_ { mjrfRenderTarget* target; // render target from which to read the image pixels void* output; // buffer into which the pixels will be stored mjtSize num_bytes; // size of output buffer @@ -152,19 +113,20 @@ typedef uint64_t mjrfFrameHandle; // taken. Firstly, requests should be grouped by target. Next, the combined area of the viewports // for all requests for a given target must be contained within the dimensions of the target itself. // -// Callbacks will be invoked from within this function, though there is no guarantee on when exactly -// that will be done. +// Callbacks will be invoked from within this function, though there is no guarantee on which +// invocation of this function it will be done. mjrfFrameHandle mjrf_render(mjrfContext* ctx, const mjrfRenderRequest* req, int nreq, const mjrfReadPixelsRequest* read_req, int nread_req); -// Waits for all rendering operations to complete for the given frame handle, -// triggering any callbacks as needed. +// Waits for all rendering operations to complete for the given frame handle, triggering any +// callbacks as needed. void mjrf_waitForFrame(mjrfContext* ctx, mjrfFrameHandle frame); // Sets the clear color for the renderer. void mjrf_setClearColor(mjrfContext* ctx, const float color[3]); -typedef struct mjrfFrameStats_ { // stats for a single frame of rendering +// Information for a single frame of rendering. +typedef struct mjrfFrameStats_ { double frame_rate; // frame rate, in frames per second } mjrfFrameStats; @@ -174,16 +136,8 @@ void mjrf_defaultFrameStats(mjrfFrameStats* stats); // Returns the stats for the given frame but updating the given `stats_out`. void mjrf_getFrameStats(mjrfContext* ctx, mjrfFrameHandle frame, mjrfFrameStats* stats_out); -// ## Textures (mjrfTexture) -// -// A texture is a 2D or 3D (cubemap) image that adds visual detail to a rendered model, such as -// color or bumpiness, without increasing geometric complexity. -// -// For textures intended to be used for image-based lights (see `mjrfLight` below), you should use -// filament's `cmgen` tool to generate a KTX image from your source image. This tool will calculate -// additional data (i.e. the spherical harmonics) and encode that information into the KTX file. - -typedef struct mjrfTextureConfig_ { // parameters for creating a texture (mjrfTexture) +// Parameters for creating a texture (mjrfTexture). +typedef struct mjrfTextureConfig_ { int width; // texture width, or number of bytes for compressed data (e.g. KTX) int height; // texture height, or 0 for compressed data (e.g. KTX) int format; // pixel format (e.g. RGB8, RGBA8, KTX, etc.) [mjrPixelFormat] @@ -194,15 +148,15 @@ typedef struct mjrfTextureConfig_ { // parameters for creating a texture (mjrfT // Initializes the mjrfTextureConfig to default values. void mjrf_defaultTextureConfig(mjrfTextureConfig* config); -// Creates a filament texture. Note that the texture will not be created on the -// GPU until `mjrf_setTextureData()` is called. -mjrfTexture* mjrf_createTexture(mjrfContext* ctx, - const mjrfTextureConfig* config); +// Creates a filament texture. Note that the texture will not be created on the GPU until +// `mjrf_setTextureData()` is called. +mjrfTexture* mjrf_createTexture(mjrfContext* ctx, const mjrfTextureConfig* config); // Destroys the texture. void mjrf_destroyTexture(mjrfTexture* texture); -typedef struct mjrfTextureData_ { // binary data for a texture (mjrfTexture) +// Binary data payload for an mjrfTexture. +typedef struct mjrfTextureData_ { const void* bytes; // pointer to image data, or nullptr for empty texture mjtSize num_bytes; // number of bytes in the image data mjrfCallback release; // callback when data has finished uploading @@ -225,32 +179,11 @@ int mjrf_getTextureHeight(const mjrfTexture* texture); // [returns: mjrSamplerType] int mjrf_getTextureSamplerType(const mjrfTexture* texture); -// ## Meshes (mjrfMesh) -// -// A mesh describes the surface geometry of an object to be rendered. It is defined as a collection -// of vertices (i.e. a VertexBuffer), a set of indices (i.e. an IndexBuffer) that describes the -// order in which the vertices should be processed, and a primitive type that defined how the -// vertices are to be interpreted (e.g. triangles, lines, etc.) when rendering the surface. -// -// Filament does not directly support normals. Instead, it encodes the normal, tangent, and -// bitangent into a 4-component quaternion describing the "orientation" of the vertex. Ideally, you -// should preprocess your assets to generate this data offline, but we will compute it on the fly if -// needed (at a performance cost). -// -// Vertex data may or may not be interleaved. Interleaved data assumes that the attributes are -// packed in the order specified in the attributes array, with no padding in-between. Additionally, -// the `data` pointer for each attribute is assumed to point to the first element of that type. For -// non-interleaved data, each attribute is assumed to be stored in a separate array. -// -// Additionally, the bounds of the mesh should be computed in order to allow the filament renderer -// to perform frustum-based culling. Alternatively, the bounds can be computed at runtime (though -// there is a small performance cost). If no bounds are provided (or calculated), then frustum -// culling will not be performed. - // Maximum number of vertex attributes in a mesh. enum { mjMAX_VERTEX_ATTRIBUTES = 16 }; -typedef struct mjrfMeshData_ { // binary data for a mesh (mjrfMesh) +// Binary data used for creating a mesh (mjrfMesh). +typedef struct mjrfMeshData_ { mjtSize num_vertices; // number of vertices; all vertex attributes share this size int num_attributes; // number of attributes defined mjrVertexAttribute attributes[mjMAX_VERTEX_ATTRIBUTES]; // per-vertex attribute information @@ -275,12 +208,8 @@ mjrfMesh* mjrf_createMesh(mjrfContext* ctx, const mjrfMeshData* data); // Destroys the mesh. void mjrf_destroyMesh(mjrfMesh* mesh); -// ## Scenes (mjrfScene) -// -// A scene is a collection of entities (Lights and Renderables) that describes what is to be -// rendered. - -typedef struct mjrfSceneParams_ { // parameters for creating a scene (mjrfScene) +// Parameters for creating a scene (mjrfScene). +typedef struct mjrfSceneParams_ { } mjrfSceneParams; // Initializes the mjrfSceneParams to default values. @@ -310,24 +239,8 @@ void mjrf_setSceneSkybox(mjrfScene* scene, const mjrfTexture* texture); // Configures the scene based on the parameters in an mjModel. void mjrf_configureSceneFromModel(mjrfScene* scene, const mjModel* model); -// ## Lights (mjrfLight) -// -// A light is a source of illumination in the scene. (Without lights, a scene will be completely -// black.) There are several different types of lights such as directional, spot, point, and image -// lights. -// -// The primary light in a scene is the image light (also sometimes known as the environment light). -// This is a light that "surrounds" the entire scene and is defined as a 3D texture. Each "pixel" of -// the cubemap is interpreted as the color of projected into the scene from a particular direction. -// -// Directional lights are the next most common type of light and is usually used to simulate the -// sun; a uniformly colored light that is emitted in a single direction. -// -// Filament only supports a single image and directional light. You can define as many point or spot -// lights as you want. Each light source (except image based lights) may or may not cast shadows. -// Each shadow-casting light incurs a performance cost. - -typedef struct mjrfLightParams_ { // parameters for creating a light (mjrfLight) +// Parameters for creating a light (mjrfLight). +typedef struct mjrfLightParams_ { int type; // type of light (e.g. spot, point, image, etc.) [mjrLightType] const mjrfTexture* texture; // texture; only for image lights float color[3]; // RGB color @@ -364,23 +277,8 @@ void mjrf_setLightTransform(mjrfLight* light, const float position[3], const flo // Returns the type of the light (mjrLightType). int mjrf_getLightType(const mjrfLight* light); -// ## Renderables (mjrfRenderable) -// -// A renderable is a single drawable object in the scene. It is defined as a combination of a mesh -// (i.e. surface geometry) and a material (i.e. surface appearance and properties). -// -// In terms of materials, there are three lighting models currently supported: -// -// 1. Metallic-roughness (PBR): this is the preferred model for rendering models based standard -// metallic-roughness workflows. -// 2. Specular-glossiness (non-PBR): this is a legacy model designed to be compatible with classic -// mjr renderer, though it is not 100% identical. -// 3. Unlit: this model ignores lighting and used for rendering UX or decorative elements like -// contact forces and labels. -// -// Which lighting model is used is determined by the mjrfMaterial properties. - -typedef struct mjrfMaterial_ { // material properties for a renderable (mjrfMaterial) +// Material properties for a renderable (mjrfRenderable). +typedef struct mjrfMaterial_ { float color[4]; // object color; defaults to white int32_t segmentation_id; // ID for segmentation rendering; maps to RGB8 color (i.e. 24 bits) int32_t island_id; // ID to which the renderable belongs @@ -410,7 +308,8 @@ typedef struct mjrfMaterial_ { // material properties for a renderable (mjrfMat // Initializes the mjrfMaterial to default values. void mjrf_defaultMaterial(mjrfMaterial* material); -typedef struct mjrfRenderableParams_ { // parameters for creating a renderable (mjrfRenderable) +// Parameters for creating a renderable (mjrfRenderable). +typedef struct mjrfRenderableParams_ { mjtBool cast_shadows; // if true, casts shadows mjtBool receive_shadows; // if true, receives shadows uint16_t blend_order; // controls draw order for transparent objects [0, 8] @@ -451,12 +350,8 @@ void mjrf_setRenderableTransform(mjrfRenderable* renderable, const float positio // (e.g. the spherical ends of a capsule are scaled such that they always remain spherical). void mjrf_setRenderableSize(mjrfRenderable* renderable, const float size[3]); -// ## Render Targets (mjrfRenderTarget) -// -// A render target is a memory buffer that holds the results of a rendering operation. (This is an -// alternative to rendering directly to the screen.) See mjrf_render for more details. - -typedef struct mjrfRenderTargetConfig_ { // parameters for creating a render target (mjrfRenderTarget) +// Parameters for creating a render target (mjrfRenderTarget). +typedef struct mjrfRenderTargetConfig_ { int width; // texture width int height; // texture height int color_format; // pixel format for color buffer [mjrPixelFormat] @@ -475,8 +370,6 @@ void mjrf_destroyRenderTarget(mjrfRenderTarget* render_target); // Resizes the render target to the given width and height. void mjrf_resizeRenderTarget(mjrfRenderTarget* render_target, int width, int height); -// ## Debug-only functions. - // Draws an ImGui editor for the given scene, exposing filament-specific settings. void mjrf_DEBUG_drawImguiEditor(mjrfScene* scene);