liyang/hand-motion-pipeline
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Initial snapshot: hand motion pipeline (Dyn-HaMR + dex-retargeting + SPIDER)

Browse Source 
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
liyang
2026-09-15 14:13:49 +08:00
commit 7e4ef6f98b
7002 changed files with 1452249 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
third_party/hamer/third-party/ViTPose/tools/webcam/configs/background/README.md
Vendored
+73
View File
@@ -0,0 +1,73 @@
# Matting Effects
We can apply background matting to the videos.
## Instruction
### Get started
Launch the demo from the mmpose root directory:
```shell
python tools/webcam/run_webcam.py --config tools/webcam/configs/background/background.py
```
### Hotkeys
| Hotkey | Function |
| -- | -- |
| b | Toggle the background matting effect on/off. |
| h | Show help information. |
| m | Show the monitoring information. |
| q | Exit. |
Note that the demo will automatically save the output video into a file `record.mp4`.
### Configuration
- **Choose a detection model**
Users can choose detection models from the [MMDetection Model Zoo](https://mmdetection.readthedocs.io/en/v2.20.0/model_zoo.html). Just set the `model_config` and `model_checkpoint` in the detector node accordingly, and the model will be automatically downloaded and loaded.
Note that in order to perform background matting, the model should be able to produce segmentation masks.
```python
# 'DetectorNode':
# This node performs object detection from the frame image using an
# MMDetection model.
dict(
type='DetectorNode',
name='Detector',
model_config='demo/mmdetection_cfg/mask_rcnn_r50_fpn_2x_coco.py',
model_checkpoint='https://download.openmmlab.com/'
'mmdetection/v2.0/mask_rcnn/mask_rcnn_r50_fpn_2x_coco/'
'mask_rcnn_r50_fpn_2x_coco_bbox_mAP-0.392'
'__segm_mAP-0.354_20200505_003907-3e542a40.pth',
input_buffer='_input_', # `_input_` is a runner-reserved buffer
output_buffer='det_result'),
```
- **Run the demo without GPU**
If you don't have GPU and CUDA in your device, the demo can run with only CPU by setting `device='cpu'` in all model nodes. For example:
```python
dict(
type='DetectorNode',
name='Detector',
model_config='demo/mmdetection_cfg/mask_rcnn_r50_fpn_2x_coco.py',
model_checkpoint='https://download.openmmlab.com/'
'mmdetection/v2.0/mask_rcnn/mask_rcnn_r50_fpn_2x_coco/'
'mask_rcnn_r50_fpn_2x_coco_bbox_mAP-0.392'
'__segm_mAP-0.354_20200505_003907-3e542a40.pth',
device='cpu',
input_buffer='_input_', # `_input_` is a runner-reserved buffer
output_buffer='det_result'),
```
- **Debug webcam and display**
You can launch the webcam runner with a debug config:
```shell
python tools/webcam/run_webcam.py --config tools/webcam/configs/examples/test_camera.py
```
third_party/hamer/third-party/ViTPose/tools/webcam/configs/background/background.py
Vendored
+93
View File
@@ -0,0 +1,93 @@
# Copyright (c) OpenMMLab. All rights reserved.
runner = dict(
# Basic configurations of the runner
name='Matting Effects',
camera_id=0,
camera_fps=10,
synchronous=False,
# Define nodes.
# The configuration of a node usually includes:
# 1. 'type': Node class name
# 2. 'name': Node name
# 3. I/O buffers (e.g. 'input_buffer', 'output_buffer'): specify the
# input and output buffer names. This may depend on the node class.
# 4. 'enable_key': assign a hot-key to toggle enable/disable this node.
# This may depend on the node class.
# 5. Other class-specific arguments
nodes=[
# 'DetectorNode':
# This node performs object detection from the frame image using an
# MMDetection model.
dict(
type='DetectorNode',
name='Detector',
model_config='demo/mmdetection_cfg/mask_rcnn_r50_fpn_2x_coco.py',
model_checkpoint='https://download.openmmlab.com/'
'mmdetection/v2.0/mask_rcnn/mask_rcnn_r50_fpn_2x_coco/'
'mask_rcnn_r50_fpn_2x_coco_bbox_mAP-0.392'
'__segm_mAP-0.354_20200505_003907-3e542a40.pth',
input_buffer='_input_', # `_input_` is a runner-reserved buffer
output_buffer='det_result'),
# 'TopDownPoseEstimatorNode':
# This node performs keypoint detection from the frame image using an
# MMPose top-down model. Detection results is needed.
dict(
type='TopDownPoseEstimatorNode',
name='Human Pose Estimator',
model_config='configs/wholebody/2d_kpt_sview_rgb_img/'
'topdown_heatmap/coco-wholebody/'
'vipnas_mbv3_coco_wholebody_256x192_dark.py',
model_checkpoint='https://openmmlab-share.oss-cn-hangz'
'hou.aliyuncs.com/mmpose/top_down/vipnas/vipnas_mbv3_co'
'co_wholebody_256x192_dark-e2158108_20211205.pth',
cls_names=['person'],
input_buffer='det_result',
output_buffer='human_pose'),
# 'ModelResultBindingNode':
# This node binds the latest model inference result with the current
# frame. (This means the frame image and inference result may be
# asynchronous).
dict(
type='ModelResultBindingNode',
name='ResultBinder',
frame_buffer='_frame_', # `_frame_` is a runner-reserved buffer
result_buffer='human_pose',
output_buffer='frame'),
# 'MattingNode':
# This node draw the matting visualization result in the frame image.
# mask results is needed.
dict(
type='BackgroundNode',
name='Visualizer',
enable_key='b',
enable=True,
frame_buffer='frame',
output_buffer='vis_bg',
cls_names=['person']),
# 'NoticeBoardNode':
# This node show a notice board with given content, e.g. help
# information.
dict(
type='NoticeBoardNode',
name='Helper',
enable_key='h',
frame_buffer='vis_bg',
output_buffer='vis',
content_lines=[
'This is a demo for background changing effects. Have fun!',
'', 'Hot-keys:', '"b": Change background',
'"h": Show help information',
'"m": Show diagnostic information', '"q": Exit'
],
),
# 'MonitorNode':
# This node show diagnostic information in the frame image. It can
# be used for debugging or monitoring system resource status.
dict(
type='MonitorNode',
name='Monitor',
enable_key='m',
enable=False,
frame_buffer='vis',
output_buffer='_display_') # `_frame_` is a runner-reserved buffer
])