From 700da7c5bd0e963a7f572507dc1bd4a0b2962745 Mon Sep 17 00:00:00 2001
From: Sam Haves <shaves@google.com>
Date: Thu, 31 Jul 2025 09:29:38 -0700
Subject: [PATCH] Add initial documentation for experimental OpenUSD support in
 MuJoCo.

PiperOrigin-RevId: 789365870
Change-Id: I19cce03ec2ffc28cdf4252bfd02c8b5163bb5ba2
---
 doc/OpenUSD/building.rst                | 57 +++++++++++++++
 doc/OpenUSD/exporting.rst               | 15 ++++
 doc/OpenUSD/importing.rst               | 29 ++++++++
 doc/OpenUSD/index.rst                   | 63 ++++++++++++++++
 doc/OpenUSD/mjcPhysics.rst              | 97 +++++++++++++++++++++++++
 doc/OpenUSD/mjcf_file_format_plugin.rst | 87 ++++++++++++++++++++++
 doc/index.rst                           |  1 +
 7 files changed, 349 insertions(+)
 create mode 100644 doc/OpenUSD/building.rst
 create mode 100644 doc/OpenUSD/exporting.rst
 create mode 100644 doc/OpenUSD/importing.rst
 create mode 100644 doc/OpenUSD/index.rst
 create mode 100644 doc/OpenUSD/mjcPhysics.rst
 create mode 100644 doc/OpenUSD/mjcf_file_format_plugin.rst

diff --git a/doc/OpenUSD/building.rst b/doc/OpenUSD/building.rst
new file mode 100644
index 00000000..f6e1c2ce
--- /dev/null
+++ b/doc/OpenUSD/building.rst
@@ -0,0 +1,57 @@
+Building
+========
+
+.. WARNING:: OpenUSD support is currently experimental and subject to frequent change.
+
+Advanced users can start testing out USD support by building against their own USD libraries or USD built from source.
+
+This assumes that you have built MuJoCo in ``~/mujoco`` and have a build directory at ``~/mujoco/build``
+
+Building USD
+------------
+
+USD has a pretty streamlined installation via their ``build_usd.py`` script. It's recommended to use a separate
+installation directory that exists outside of the cloned repository directory.
+
+.. code-block:: bash
+
+   git clone https://github.com/PixarAnimationStudios/OpenUSD
+   python OpenUSD/build_scripts/build_usd.py /path/to/my_usd_install_dir
+
+Enabling USD
+------------
+
+USD is comprised of many plugins. When USD enabled application starts up it looks for an environment variable called
+``PXR_PLUGINPATH_NAME``. Below is an example where we build MuJoCo with USD enabled and set this variable.
+
+.. code-block:: bash
+
+   cd ~/mujoco/build
+   cmake .. -DCMAKE_BUILD_TYPE=Release -DUSD_DIR=/path/to/my_usd_install_dir
+   cmake --build . -j 30; sudo cmake --install .
+   export PXR_PLUGINPATH_NAME=/usr/local/lib/mujocoUsd/resources/*/plugInfo.json
+
+If we now run :ref:`simulate.cc <saSimulate>`, we will be able to drag and drop USD files.
+
+.. code-block:: bash
+
+   simulate
+
+Enabling plugins in Houdini
+---------------------------
+
+Houdini is a procedural content authoring tool with extensive support for USD workflows via their Solaris context. It's
+highly popular in the VFX industry, and it's easy to imagine procedural generation tools for simulation ready assets and
+scenes.
+
+To allow support for loading MJCF files in Solaris, and usage of the mjhcPhysics schemas you can build against Houdini's
+USD libraries. To do so, simply run `source ./houdini_setup` as descrived in the `SideFX documentation
+<https://www.sidefx.com/faq/question/how-do-i-set-up-the-houdini-environment-for-command-line-tools>`__.
+
+.. code-block:: bash
+
+   cd ~/mujoco/build
+   cmake .. -DCMAKE_BUILD_TYPE=Release -DHOUDINI_HFS_DIR=$HFS
+   cmake --build . -j 30; sudo cmake --install .
+   export PXR_PLUGINPATH_NAME=/usr/local/lib/mujocoUsd/resources/*/plugInfo.json
+   houdini
diff --git a/doc/OpenUSD/exporting.rst b/doc/OpenUSD/exporting.rst
new file mode 100644
index 00000000..212abd5c
--- /dev/null
+++ b/doc/OpenUSD/exporting.rst
@@ -0,0 +1,15 @@
+Exporting
+=========
+
+.. WARNING:: OpenUSD support is currently experimental and subject to frequent change.
+
+Currently, exporting MuJoCo scenes to OpenUSD format is an area of active development. The primary method for exporting
+is expected to be through the Python API.
+
+USDExporter
+-----------
+
+At the moment the only way to export USD from MuJoCo is to use the existing USDExporter in :doc:`../python`.
+
+We are working on native support for writing simulations as animations to existing USD scenes, please check back here
+for updates.
diff --git a/doc/OpenUSD/importing.rst b/doc/OpenUSD/importing.rst
new file mode 100644
index 00000000..a0111088
--- /dev/null
+++ b/doc/OpenUSD/importing.rst
@@ -0,0 +1,29 @@
+Importing
+=========
+
+.. WARNING:: OpenUSD support is currently experimental and subject to frequent change.
+
+MuJoCo can load assets from OpenUSD files (``.usd``, ``.usda``, ``.usdc``, ``.usdz``). This allows you to incorporate
+assets and scenes defined in USD into your MuJoCo simulations.
+
+USD in MJCF
+-----------------------------
+
+If you have built mujoco with USD enabled, you can reference USD assets from MJCF via the ``<model`` tag with content
+type ``text/usd``.
+
+.. code-block:: xml
+    :caption: example.xml
+
+    <mujoco>
+      <asset>
+        <model file="chair.usdz" name="chair" content_type="text/usd"/>
+      </asset>
+
+      <worldbody>
+        ...
+      </worldbody>
+    </mujoco>
+
+In this example the ``<model file="chair.usdz"/>`` line in ``<asset>`` tells MuJoCo to load and process the USD file.
+
diff --git a/doc/OpenUSD/index.rst b/doc/OpenUSD/index.rst
new file mode 100644
index 00000000..675bbf6b
--- /dev/null
+++ b/doc/OpenUSD/index.rst
@@ -0,0 +1,63 @@
+OpenUSD
+===========
+
+.. toctree::
+    :hidden:
+
+    building
+    mjcPhysics
+    mjcf_file_format_plugin
+    importing
+    exporting
+
+.. WARNING:: OpenUSD support is currently experimental and subject to frequent change.
+
+Introduction
+------------
+
+This chapter describes MuJoCo's support for `OpenUSD <https://openusd.org/release/intro.html>`__. USD (Universal Scene
+Description) is an open-source framework developed by Pixar for describing 3D scenes. MuJoCo's integration allows users
+to leverage USD's rich ecosystem and tooling.
+
+What is OpenUSD?
+----------------
+
+USD is a high-performance, extensible system for describing, composing, simulating, and collaborating on 3D data.
+Originally developed by Pixar Animation Studios, USD is now used across various industries, including visual effects,
+animation, gaming, and robotics, to streamline complex 3D workflows. It provides a common language for different
+software applications to exchange 3D scene information.
+
+Why do we care about OpenUSD?
+-----------------------------
+
+Integrating USD with MuJoCo offers several advantages:
+
+*   **Interoperability:** USD is supported by a wide range of 3D content creation tools (e.g., Houdini, Maya, Blender).
+    This allows MuJoCo users to easily import scenes and assets created in these tools.
+*   **Rich Scene Description:** USD provides a powerful and flexible way to represent complex scenes, including
+    geometry, materials, lighting, and hierarchies.
+*   **Collaboration:** USD's layering and composition features enable powerful and efficient
+    non-destructive authoring pipelines.
+
+USD support overview
+------------------------------------------------------
+
+*   **Import:** You can load USD assets (specifically ``.usd``, ``.usda``, ``.usdc``, ``.usdz`` files) into MuJoCo via
+    MJCF or dragging and dropping into :ref:`simulate.cc <saSimulate>`.
+*   **Schemas:** MuJoCo primarily uses the standard `UsdPhysics
+    <https://openusd.org/dev/api/usd_physics_page_front.html>`__ schemas for representing physics properties.
+*   **Extensions:** Custom :doc:`mjcPhysics` schemas are provided to cover MuJoCo-specific features not available in
+    ``UsdPhysics``.
+*   **MJCF File Format Plugin:** A :doc:`mjcf_file_format_plugin` allows treating MJCF files as USD layers in any native USD
+    application.
+*   **Export:** MuJoCo scenes can be exported to USD.
+
+Where do I learn more about USD?
+------------------------------------------
+
+*   `Remedy's Book of USD <https://remedy-entertainment.github.io/USDBook>`__: Friendly introduction to USD.
+*   `Official OpenUSD Documentation <https://openusd.org/release/intro.html>`__: Official documentation for API and
+    implementation details.
+*   `Pixar's USD Introduction <https://graphics.pixar.com/usd/release/index.html>`__: Simple example usage of USD.
+*   `NVIDIA's USD Resources <https://developer.nvidia.com/usd>`__: Set of USD resources primarily concerned with asset
+    structure.
diff --git a/doc/OpenUSD/mjcPhysics.rst b/doc/OpenUSD/mjcPhysics.rst
new file mode 100644
index 00000000..b0f6fb96
--- /dev/null
+++ b/doc/OpenUSD/mjcPhysics.rst
@@ -0,0 +1,97 @@
+mjcPhysics
+==========
+
+.. WARNING:: OpenUSD support is currently experimental and subject to frequent change.
+
+The ``mjcPhysics`` `schema <https://openusd.org/release/api/_usd__page__generating_schemas.html>`__ allows for detailed
+specification of a MuJoCo simulation environment directly within a USD file. The aim is not to replace `UsdPhysics
+<https://openusd.org/release/api/usd_physics_page_front.html>`__, but to extend existing concepts and create new types
+only where is necessary.
+
+The schema can be use `codeless <https://openusd.org/dev/api/_usd__page__generating_schemas.html#Codeless_Schemas>`__,
+or can be built with its C++ bindings. We've pre-generated `the code
+<https://github.com/google-deepmind/mujoco/tree/main/src/experimental/usd/mjcPhysics>`__ via `usdGenSchema
+<https://openusd.org/dev/api/_usd__page__generating_schemas.html>`_ for internal MuJoCo usage, but it should also work
+outside of MuJoCo.
+
+API Schemas
+-----------
+
+MjcSceneAPI
+^^^^^^^^^^^
+
+This API schema provides global options for the MuJoCo simulation. It is an
+amalgamation of the ``<option>``, ``<option/flag>`` and ``<compiler>`` elements in
+MJCF. Users should apply this to an existing
+`UsdPhysicsScene <https://openusd.org/dev/api/class_usd_physics_scene.html>`__
+prim.
+
+Key attributes include:
+
+-   **mjc:option**: Attributes in this namespace map to the ``<option>`` element.
+-   **mjc:flag**: Attributes in this namespace map to the ``<option/flag>`` element.
+-   **mjc:compiler**: Attributes in this namespace map to the ``<compiler>`` element.
+
+MjcSiteAPI
+^^^^^^^^^^
+
+This API class is used to define a MuJoCo site, it can be applied to
+`UsdGeomSphere <https://openusd.org/dev/api/class_usd_geom_sphere.html>`__,
+`UsdGeomCapsule <https://openusd.org/dev/api/class_usd_geom_capsule.html>`__,
+`UsdGeomCylinder <https://openusd.org/dev/api/class_usd_geom_cylinder.html>`__, and
+`UsdGeomCube <https://openusd.org/dev/api/class_usd_geom_cube.html>`__.
+
+MjcImageableAPI
+^^^^^^^^^^^^^^^
+
+This API class provides attributes for strictly visual entities in MuJoCo, in
+MuJoCo terms we would quantify these has having ``contype = conaffinity = 0``.
+
+MjcCollisionAPI
+^^^^^^^^^^^^^^^
+
+This API class is applied to prims that represent collision geometry and should
+be applied alongside
+`UsdPhysicsCollisionAPI <https://openusd.org/dev/api/class_usd_physics_collision_a_p_i.html>`__.
+
+MjcMeshCollisionAPI
+^^^^^^^^^^^^^^^^^^^
+
+This API class is applied to prims that represent mesh collision geometry and
+should be applied alongside
+`UsdPhysicsMeshCollisionAPI <https://openusd.org/dev/api/class_usd_physics_mesh_collision_a_p_i.html>`__.
+
+MjcJointAPI
+^^^^^^^^^^^
+
+This API class is applied to `UsdPhysicsJoint <https://openusd.org/dev/api/class_usd_physics_joint.html>`__ prims,
+adding extra attributes to fully describe MuJoCo joints.
+
+MjcMaterialAPI
+^^^^^^^^^^^^^^
+
+This API class provides attributes for physical materials and is an extension of `UsdPhysicsMaterialAPI
+<https://openusd.org/dev/api/class_usd_physics_material_a_p_i.html>`__
+
+Type Schemas
+------------
+
+MjcActuator
+^^^^^^^^^^^
+
+This class represents a MuJoCo actuator, which is responsible for applying force to a transmission target joint, body,
+or site specific via a `relationship <https://openusd.org/dev/api/class_usd_relationship.html>`__.
+
+We do not use the existing `UsdPhysicsDriveAPI <https://openusd.org/dev/api/class_usd_physics_drive_a_p_i.html>`__ as it
+is closer to a runtime construct and the concepts do not map very closely.
+
+MjcKeyframe
+^^^^^^^^^^^
+
+This type holds tensor values representing simulator state at specific time values.
+
+In MJCF this is the ``<keyframe>`` element and has a ``time`` attribute. In USD we map the time attribute to
+`timeSamples <https://openusd.org/release/tut_xforms.html>`__ instead.
+
+The order of the values in the keyframes should map to the depth first ordered traversal of rigidbodies in the composed
+stage.
diff --git a/doc/OpenUSD/mjcf_file_format_plugin.rst b/doc/OpenUSD/mjcf_file_format_plugin.rst
new file mode 100644
index 00000000..1676a33d
--- /dev/null
+++ b/doc/OpenUSD/mjcf_file_format_plugin.rst
@@ -0,0 +1,87 @@
+File Format Plugin
+=========================
+
+What is an SdfFileFormat plugin?
+--------------------------------
+
+In the OpenUSD framework, ``Sdf`` stands for Scene Description Foundations. It's the underlying layer that handles the
+serialization and composition of scene data. A ``SdfFileFormat`` plugin is a component that teaches USD how to read and
+write a specific file format.
+
+By default, USD comes with plugins for its own formats (``.usda``, ``.usdc``, ``.usdz``) and the community has created
+several plugin extension such as the `Adobe File Format Plugins
+<https://github.com/adobe/USD-Fileformat-plugins/tree/main>`__.
+
+The MJCF ``SdfFileFormat`` plugin allows USD-aware applications to directly understand and interact with MuJoCo's native
+``.xml`` (MJCF) files as if they were native USD files.
+
+What does it enable?
+------------------------
+
+This plugin enables:
+
+1.  **Referencing MJCF files in USD:** Using standard USD composition arcs (like references, payloads) to include an
+    MJCF file directly within a larger USD scene. For example, you can place a MuJoCo robot defined in an ``.xml`` file
+    into a room scene modeled in USD.
+2.  **Load MJCF files in USD tools:** Tools like ``usdview`` or other USD-based
+    applications can open, inspect, and render MJCF files, translating the MJCF elements into USD prims and attributes
+    on the fly.
+3.  **Convert MJCF to USD:** The plugin can be used as a basis for converting MJCF files to persistent
+    USD files (e.g., ``.usda`` or ``.usdc``).
+
+Essentially, it makes MJCF a first-class citizen in the USD ecosystem.
+
+Usage
+------------------
+
+1.  **Installation:** refer to :doc:`building`.
+
+2.  **Referencing in a USD file (e.g., ``.usda``):**
+
+    .. code-block:: usd
+        :caption: example.usda
+
+        #usda 1.0
+        (
+            upAxis = "Z"
+        )
+
+        def Xform "world"
+        {
+            def "robot" (
+                prepend references = @./my_robot.xml@
+            )
+            {
+            }
+        }
+
+    In this example, ``my_robot.xml`` is an MJCF file in the same directory. USD will use the plugin to load and
+    interpret its contents.
+
+3.  **Opening in usdview:**
+
+    .. code-block:: bash
+
+       usdview my_robot.xml
+
+    If the plugin is correctly set up, ``usdview`` will render the robot defined in the MJCF file.
+
+4.  **Using in Python (with USD API):**
+
+    .. code-block:: python
+
+        from pxr import Usd
+
+        # Load an MJCF file as a USD stage
+        stage = Usd.Stage.Open('my_robot.xml')
+
+        if stage:
+            print(f"Successfully opened {stage.GetRootLayer().identifier}")
+            # You can now inspect the stage as any other USD stage
+            for prim in stage.TraverseAll():
+                print(prim.GetPath())
+        else:
+            print("Failed to open MJCF file")
+
+    This plugin significantly enhances the interoperability between MuJoCo and USD-based workflows, allowing
+    seamless integration of physics assets defined in MJCF into broader 3D environments.
diff --git a/doc/index.rst b/doc/index.rst
index 0aba3e4c..485fa9cb 100644
--- a/doc/index.rst
+++ b/doc/index.rst
@@ -16,6 +16,7 @@
    python
    MJX <mjx>
    unity
+   OpenUSD/index.rst
    models
    changelog