131 lines
5.6 KiB
Markdown
131 lines
5.6 KiB
Markdown
# HaMeR: Hand Mesh Recovery
|
|
Code repository for the paper:
|
|
**Reconstructing Hands in 3D with Transformers**
|
|
|
|
[Georgios Pavlakos](https://geopavlakos.github.io/), [Dandan Shan](https://ddshan.github.io/), [Ilija Radosavovic](https://people.eecs.berkeley.edu/~ilija/), [Angjoo Kanazawa](https://people.eecs.berkeley.edu/~kanazawa/), [David Fouhey](https://cs.nyu.edu/~fouhey/), [Jitendra Malik](http://people.eecs.berkeley.edu/~malik/)
|
|
|
|
[](https://arxiv.org/pdf/2312.05251.pdf) [](https://geopavlakos.github.io/hamer/) [](https://colab.research.google.com/drive/1rQbQzegFWGVOm1n1d-S6koOWDo7F2ucu?usp=sharing) [](https://huggingface.co/spaces/geopavlakos/HaMeR)
|
|
|
|
|
|
|
|
## News
|
|
|
|
- [2026/02] Check out our [new work](https://gkarv.github.io/hand-texture-module/) using learned texture priors to improve HaMeR.
|
|
- [2024/06] HaMeR received the 2nd place award in the Ego-Pose Hands task of the Ego-Exo4D Challenge! Please check the [validation report](https://www.cs.utexas.edu/~pavlakos/hamer/resources/egoexo4d_challenge.pdf).
|
|
- [2024/05] We have released the evaluation pipeline!
|
|
- [2024/05] We have released the HInt dataset annotations! Please check [here](https://github.com/ddshan/hint).
|
|
- [2023/12] Original release!
|
|
|
|
## Installation
|
|
First you need to clone the repo:
|
|
```
|
|
git clone --recursive https://github.com/geopavlakos/hamer.git
|
|
cd hamer
|
|
```
|
|
|
|
We recommend creating a virtual environment for HaMeR. You can use venv:
|
|
```bash
|
|
python3.10 -m venv .hamer
|
|
source .hamer/bin/activate
|
|
```
|
|
|
|
or alternatively conda:
|
|
```bash
|
|
conda create --name hamer python=3.10
|
|
conda activate hamer
|
|
```
|
|
|
|
Then, you can install the rest of the dependencies. This is for CUDA 11.7, but you can adapt accordingly:
|
|
```bash
|
|
pip install torch torchvision --index-url https://download.pytorch.org/whl/cu117
|
|
pip install -e .[all]
|
|
pip install -v -e third-party/ViTPose
|
|
```
|
|
|
|
You also need to download the trained models:
|
|
```bash
|
|
bash fetch_demo_data.sh
|
|
```
|
|
|
|
Besides these files, you also need to download the MANO model. Please visit the [MANO website](https://mano.is.tue.mpg.de) and register to get access to the downloads section. We only require the right hand model. You need to put `MANO_RIGHT.pkl` under the `_DATA/data/mano` folder.
|
|
|
|
### Docker Compose
|
|
|
|
If you wish to use HaMeR with Docker, you can use the following command:
|
|
|
|
```
|
|
docker compose -f ./docker/docker-compose.yml up -d
|
|
```
|
|
|
|
After the image is built successfully, enter the container and run the steps as above:
|
|
|
|
```
|
|
docker compose -f ./docker/docker-compose.yml exec hamer-dev /bin/bash
|
|
```
|
|
|
|
Continue with the installation steps:
|
|
|
|
```bash
|
|
bash fetch_demo_data.sh
|
|
```
|
|
|
|
## Demo
|
|
```bash
|
|
python demo.py \
|
|
--img_folder example_data --out_folder demo_out \
|
|
--batch_size=48 --side_view --save_mesh --full_frame
|
|
```
|
|
|
|
## HInt Dataset
|
|
We have released the annotations for the HInt dataset. Please follow the instructions [here](https://github.com/ddshan/hint)
|
|
|
|
## Training
|
|
First, download the training data to `./hamer_training_data/` by running:
|
|
```
|
|
bash fetch_training_data.sh
|
|
```
|
|
|
|
Then you can start training using the following command:
|
|
```
|
|
python train.py exp_name=hamer data=mix_all experiment=hamer_vit_transformer trainer=gpu launcher=local
|
|
```
|
|
Checkpoints and logs will be saved to `./logs/`.
|
|
|
|
## Evaluation
|
|
Download the [evaluation metadata](https://www.dropbox.com/scl/fi/7ip2vnnu355e2kqbyn1bc/hamer_evaluation_data.tar.gz?rlkey=nb4x10uc8mj2qlfq934t5mdlh) to `./hamer_evaluation_data/`. Additionally, download the FreiHAND, HO-3D, and HInt dataset images and update the corresponding paths in `hamer/configs/datasets_eval.yaml`.
|
|
|
|
Run evaluation on multiple datasets as follows, results are stored in `results/eval_regression.csv`.
|
|
```bash
|
|
python eval.py --dataset 'FREIHAND-VAL,HO3D-VAL,NEWDAYS-TEST-ALL,NEWDAYS-TEST-VIS,NEWDAYS-TEST-OCC,EPICK-TEST-ALL,EPICK-TEST-VIS,EPICK-TEST-OCC,EGO4D-TEST-ALL,EGO4D-TEST-VIS,EGO4D-TEST-OCC'
|
|
```
|
|
|
|
Results for HInt are stored in `results/eval_regression.csv`. For [FreiHAND](https://github.com/lmb-freiburg/freihand) and [HO-3D](https://codalab.lisn.upsaclay.fr/competitions/4318) you get as output a `.json` file that can be used for evaluation using their corresponding evaluation processes.
|
|
|
|
## Acknowledgements
|
|
Parts of the code are taken or adapted from the following repos:
|
|
- [4DHumans](https://github.com/shubham-goel/4D-Humans)
|
|
- [SLAHMR](https://github.com/vye16/slahmr)
|
|
- [ProHMR](https://github.com/nkolot/ProHMR)
|
|
- [SPIN](https://github.com/nkolot/SPIN)
|
|
- [SMPLify-X](https://github.com/vchoutas/smplify-x)
|
|
- [HMR](https://github.com/akanazawa/hmr)
|
|
- [ViTPose](https://github.com/ViTAE-Transformer/ViTPose)
|
|
- [Detectron2](https://github.com/facebookresearch/detectron2)
|
|
|
|
Additionally, we thank [StabilityAI](https://stability.ai/) for a generous compute grant that enabled this work.
|
|
|
|
## Open-Source Contributions
|
|
- [Wentao Hu](https://vincenthu19.github.io/) integrated the hand parameters predicted by HaMeR into SMPL-X - [Mano2Smpl-X](https://github.com/VincentHu19/Mano2Smpl-X)
|
|
|
|
## Citing
|
|
If you find this code useful for your research, please consider citing the following paper:
|
|
|
|
```bibtex
|
|
@inproceedings{pavlakos2024reconstructing,
|
|
title={Reconstructing Hands in 3{D} with Transformers},
|
|
author={Pavlakos, Georgios and Shan, Dandan and Radosavovic, Ilija and Kanazawa, Angjoo and Fouhey, David and Malik, Jitendra},
|
|
booktitle={CVPR},
|
|
year={2024}
|
|
}
|
|
```
|