Add JavaScript bindings and WASM support
Co-authored-by: Matija Kecman <matijak@google.com> Co-authored-by: Sebastian Noreña Rendón <sebas.norena@creativa77.com.ar> Co-authored-by: Kyle Bayes <kylebayes@google.com> PiperOrigin-RevId: 826479070 Change-Id: I25acf36e6c20b091d8492e10798d028ae66f6733
This commit is contained in:
Matias Manevi
committed by
Copybara-Service
Copybara-Service
parent
57f7145806
commit
4086261714
+219
@@ -0,0 +1,219 @@
|
||||
# MuJoCo JavaScript Bindings
|
||||
|
||||
> [!CAUTION]
|
||||
> **These bindings are not yet ready for general use. They have been added
|
||||
> without announcement while we develop the CI we need to accept pull
|
||||
> requests.**
|
||||
|
||||
These are the canonical JavaScript and TypeScript bindings for the MuJoCo
|
||||
physics engine.
|
||||
|
||||
This package provides a high-level API that allows you to interact with the core
|
||||
MuJoCo engine compiled into a high-performance WebAssembly (WASM) module. These
|
||||
bindings are developed and maintained by Google DeepMind and are always up to
|
||||
date with the latest developments in MuJoCo. For brevity, the documentation
|
||||
below will often refer to “JavaScript” but the concepts apply equally to
|
||||
TypeScript.
|
||||
|
||||
> [!IMPORTANT]
|
||||
> _These bindings are still a WIP. For details, see the [Future Work](#future-work)
|
||||
> section. Also note that development has primarily taken place on Linux using
|
||||
> Google Chrome. If you're working on a different OS or browser, you may
|
||||
> encounter some rough edges._
|
||||
|
||||
## Prerequisites
|
||||
|
||||
> [!NOTE]
|
||||
> Run all the commands in this README from the top-level directory.
|
||||
|
||||
- To compile the [`bindings.cc`](codegen/generated/bindings.cc) file, which
|
||||
generates the `.wasm` WebAssembly file, `.js` JavaScript import, and `.d.ts`
|
||||
TypeScript declaration file, you will need Emscripten SDK version `4.0.10`.
|
||||
Later versions may work but are untested. To set up the SDK, do the
|
||||
following, you can run this anywhere but the rest of the commands in this
|
||||
README only work in the shell where you source the `emsdk_env.sh` script.
|
||||
|
||||
```sh
|
||||
git clone https://github.com/emscripten-core/emsdk.git
|
||||
./emsdk/emsdk install 4.0.10
|
||||
./emsdk/emsdk activate 4.0.10
|
||||
source ./emsdk/emsdk_env.sh
|
||||
```
|
||||
|
||||
- To easily run the JavaScript tests and the demo application, `node` and `npm`
|
||||
are required. We recommend managing these using
|
||||
[nvm](https://github.com/nvm-sh/nvm). There are also various JavaScript
|
||||
dependencies needed for the tests, demo, and bindings build process. These
|
||||
dependencies are expected to be located in the `wasm` folder. To install
|
||||
them and ensure they can be found by later commands, run the following:
|
||||
|
||||
```sh
|
||||
npm install --prefix ./wasm
|
||||
export PATH="$(pwd)/wasm/node_modules/.bin:$PATH"
|
||||
```
|
||||
|
||||
- To modify the bindings `python3` is required because the [`bindings.cc`](codegen/generated/bindings.cc)
|
||||
file is generated by a Python script. To run the bindings generator tests,
|
||||
`absl` is required and `pytest` will be helpful. Set up a Python environment
|
||||
with these dependencies as follows:
|
||||
|
||||
```sh
|
||||
python3 -m venv .venv
|
||||
source .venv/bin/activate
|
||||
pip install -r python/build_requirements.txt
|
||||
```
|
||||
|
||||
> [!TIP]
|
||||
> _Emscripten is well-documented. We recommend reading the sections covering the
|
||||
> [Emscripten Compiler Settings](https://emscripten.org/docs/tools_reference/settings_reference.html),
|
||||
> the [Emscripten SDK](https://emscripten.org/docs/tools_reference/emsdk.html),
|
||||
> and the [Embind](https://emscripten.org/docs/porting/connecting_cpp_and_javascript/embind.html)
|
||||
> library. To understand the limitations and caveats related to using the
|
||||
> browser as a platform, see the
|
||||
> [Porting](https://emscripten.org/docs/porting/index.html#porting) section._
|
||||
|
||||
## User Guide
|
||||
|
||||
### Bindings Generation
|
||||
|
||||
The [`bindings.cc`](codegen/generated/bindings.cc) file is compiled to generate
|
||||
to `.wasm` WebAssembly file, `.js` JavaScript import, and `.d.ts` TypeScript
|
||||
declaration file. These are the files you'll use to call MuJoCo from JavaScript.
|
||||
To generate them ensure the npm and Emscripten SDK prerequisites are set up and
|
||||
then run the following:
|
||||
|
||||
```sh
|
||||
emcmake cmake -B build && cmake --build build
|
||||
```
|
||||
|
||||
This command will generate the following folders under the project root:
|
||||
|
||||
- `build`: contains MuJoCo compiled using Emscripten.
|
||||
- `wasm/dist`: contains the WebAssembly module, `.js` and `.d.ts` files.
|
||||
|
||||
### Example Application
|
||||
|
||||
After generating the bindings you will be ready to write web applications using
|
||||
MuJoCo. We have provided a basic web application that uses Three.js to render a
|
||||
simple simulation, to try it run this command:
|
||||
|
||||
```sh
|
||||
npm run dev:demo --prefix ./wasm
|
||||
```
|
||||
|
||||
You may prefer to write your entire app in C++ and compile it using Emscripten.
|
||||
If you do this, you won’t need to use these bindings, since you’ll be writing
|
||||
minimal JavaScript, and the granularity of these bindings may be inappropriate
|
||||
(e.g., you might want to call multiple MuJoCo functions in the C++ callback
|
||||
invoked by `requestAnimationFrame`).
|
||||
|
||||
We have also found that a hybrid approach can be helpful, as it is often more
|
||||
convenient to work with browser APIs directly in JavaScript. If you choose to
|
||||
write your application in C++ and compile it using Emscripten, you may want to
|
||||
copy a subset of the `EMSCRIPTEN_BINDINGS` from `bindings.cc` into your
|
||||
application’s source file.
|
||||
|
||||
## Development
|
||||
|
||||
In order to change the bindings you will need to change the [`bindings.cc`](codegen/generated/bindings.cc)
|
||||
file but this should not be done manually. The file is generated using the
|
||||
Python scripts and template files in the [`codegen`](codegen) folder, to edit
|
||||
the bindings you will need to change those files and re-generate [`bindings.cc`](codegen/generated/bindings.cc)
|
||||
using this command:
|
||||
|
||||
```sh
|
||||
PYTHONPATH=python/mujoco python3 -m wasm.codegen.update
|
||||
```
|
||||
|
||||
The codegen scripts use MuJoCo’s Python introspect library to generate the
|
||||
Embind `EMSCRIPTEN_BINDINGS` block that binds C++ functions and classes to
|
||||
JavaScript. The functions and classes that are bound are wrappers around
|
||||
MuJoCo's C API. These wrappers provide a convenient place to add features like
|
||||
bounds checking and nice error reporting.
|
||||
|
||||
### Testing
|
||||
|
||||
1. **JavaScript API tests.**
|
||||
These verify that a wide variety of MuJoCo functions and classes work
|
||||
correctly when called from JavaScript. There are also preliminary benchmarks
|
||||
for JavaScript/C++ shared memory buffers. The enums test is special because
|
||||
it is generated by a Python script. Run the tests as follows:
|
||||
|
||||
```sh
|
||||
npm run test --prefix ./wasm
|
||||
```
|
||||
|
||||
2. **Bindings generator tests.**
|
||||
These are relevant when developing or extending the bindings. The following
|
||||
command finds and runs all `test_*.py` or `*_test.py` files in the `wasm`
|
||||
folder:
|
||||
|
||||
```sh
|
||||
PYTHONPATH=python/mujoco python3 -m pytest ./wasm
|
||||
```
|
||||
|
||||
> [!NOTE]
|
||||
> If you add/edit an enum in MuJoCo you will need to run the following command
|
||||
> to re-creates the [`enums_tests.ts`](tests/enums_tests.ts) file which checks
|
||||
> that all the enums in the API are bound:
|
||||
>
|
||||
> ```sh
|
||||
> PYTHONPATH=python/mujoco python3 -m wasm.codegen.enums_test_generator
|
||||
> ```
|
||||
|
||||
### Debugging
|
||||
|
||||
We provide a “sandbox” app where you can quickly write code to run in your
|
||||
browser. Write your code in the [`main.ts`](tests/sandbox/main.ts) file and use
|
||||
the following command to execute it in your browser:
|
||||
|
||||
```sh
|
||||
npm run dev:sandbox --prefix ./wasm
|
||||
```
|
||||
|
||||
The page will be blank since the script only logs to the console output. You
|
||||
can add your code at the indicated placeholder and use Chrome DevTools for
|
||||
debugging. It is possible to set up a debug workflow where stack traces and
|
||||
stepping through code across language boundaries work correctly. Our current
|
||||
method to do this only works internally at Google, but it should be possible to
|
||||
replicate the experience with open-source tooling — community suggestions are
|
||||
welcome!
|
||||
|
||||
## Future Work
|
||||
|
||||
1. **Bind all useful APIs.**
|
||||
These bindings are not yet complete. While the main MuJoCo APIs (`mj_step`,
|
||||
`mj_loadXML`, etc.) are well tested, other APIs (e.g., functions from
|
||||
`mjspec.h`) remain untested in real web applications (though test code for
|
||||
the `mjspec` bindings does exist). One notable feature not yet supported in
|
||||
the WASM bindings, which has proved very useful in the Python bindings, is
|
||||
named access methods — where data distributed across multiple arrays in C can
|
||||
be conveniently accessed by name, e.g., `model.geom('mygeom')` or
|
||||
`data.joint('myjoint')`. Currently, this data must be accessed via the
|
||||
`mj_name2id` function. Adding support for these features is a high priority,
|
||||
as it affects user code written in JavaScript.
|
||||
|
||||
2. **Improve the developer experience.**
|
||||
There is still work to be done to improve the developer experience when
|
||||
developing the WASM bindings. The most obvious issue is that bindings
|
||||
generation is not yet fully automated. As a result, it is currently less
|
||||
convenient than we'd like to identify and apply the changes needed to update
|
||||
the bindings. The goal is to eventually automate all binding code generation
|
||||
and clearly communicate what changes are required in the WASM bindings as a
|
||||
result of C++ updates. This problem should only affect developers working on
|
||||
the MuJoCo engine in C++, not end users writing JavaScript.
|
||||
|
||||
3. **Improve the documentation.**
|
||||
The documentation in this README will eventually be merged into the main
|
||||
MuJoCo documentation once the bindings are complete and named access is
|
||||
implemented. We also intend to review the bindings APIs and make adjustments
|
||||
to minimize differences with the Python bindings (while respecting language
|
||||
idioms) to reduce the amount of additional documentation required.
|
||||
|
||||
4. **Improve the [example](#example-application).**
|
||||
We aim to provide an example application that can be easily modified and
|
||||
embedded into a paper project page (see [this example](https://kzakka.com/robopianist/)).
|
||||
This could be achieved by extending the Three.js example or by compiling the
|
||||
MuJoCo toolbox C++ code using the Emscripten toolchain. Community suggestions
|
||||
and contributions are welcome!
|
||||
|
||||
Reference in New Issue
Block a user