Initial snapshot: hand motion pipeline (Dyn-HaMR + dex-retargeting + SPIDER)

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-09-15 14:13:49 +08:00
commit 7e4ef6f98b
7002 changed files with 1452249 additions and 0 deletions
@@ -0,0 +1,110 @@
+# Pose Estimation Demo
+
+This demo performs human bounding box and keypoint detection, and visualizes results.
+
+<div align="center">
+    <img src="https://user-images.githubusercontent.com/15977946/148911749-57b263c0-1075-4a65-af53-b51fc815da68.gif" width="600px" alt><br>
+</div>
+
+## Instruction
+
+### Get started
+
+Launch the demo from the mmpose root directory:
+
+```shell
+python tools/webcam/run_webcam.py --config tools/webcam/configs/examples/pose_estimation.py
+```
+
+### Hotkeys
+
+| Hotkey | Function |
+| -- | -- |
+| v | Toggle the pose visualization on/off. |
+| h | Show help information. |
+| m | Show the monitoring information. |
+| q | Exit. |
+
+Note that the demo will automatically save the output video into a file `record.mp4`.
+
+### Configuration
+
+- **Choose a detection model**
+
+Users can choose detection models from the [MMDetection Model Zoo](https://mmdetection.readthedocs.io/en/v2.20.0/model_zoo.html). Just set the `model_config` and `model_checkpoint` in the detector node accordingly, and the model will be automatically downloaded and loaded.
+
+```python
+# 'DetectorNode':
+    # This node performs object detection from the frame image using an
+    # MMDetection model.
+dict(
+    type='DetectorNode',
+    name='Detector',
+    model_config='demo/mmdetection_cfg/'
+    'ssdlite_mobilenetv2_scratch_600e_coco.py',
+    model_checkpoint='https://download.openmmlab.com'
+    '/mmdetection/v2.0/ssd/'
+    'ssdlite_mobilenetv2_scratch_600e_coco/ssdlite_mobilenetv2_'
+    'scratch_600e_coco_20210629_110627-974d9307.pth',
+    input_buffer='_input_',
+    output_buffer='det_result')
+```
+
+- **Choose a or more pose models**
+
+In this demo we use two [top-down](https://github.com/open-mmlab/mmpose/tree/master/configs/body/2d_kpt_sview_rgb_img/topdown_heatmap) pose estimation models for humans and animals respectively. Users can choose models from the [MMPose Model Zoo](https://mmpose.readthedocs.io/en/latest/modelzoo.html). To apply different pose models on different instance types, you can add multiple pose estimator nodes with `cls_names` set accordingly.
+
+```python
+# 'TopDownPoseEstimatorNode':
+# This node performs keypoint detection from the frame image using an
+# MMPose top-down model. Detection results is needed.
+dict(
+    type='TopDownPoseEstimatorNode',
+    name='Human Pose Estimator',
+    model_config='configs/wholebody/2d_kpt_sview_rgb_img/'
+    'topdown_heatmap/coco-wholebody/'
+    'vipnas_mbv3_coco_wholebody_256x192_dark.py',
+    model_checkpoint='https://openmmlab-share.oss-cn-hangz'
+    'hou.aliyuncs.com/mmpose/top_down/vipnas/vipnas_mbv3_co'
+    'co_wholebody_256x192_dark-e2158108_20211205.pth',
+    cls_names=['person'],
+    input_buffer='det_result',
+    output_buffer='human_pose'),
+dict(
+    type='TopDownPoseEstimatorNode',
+    name='Animal Pose Estimator',
+    model_config='configs/animal/2d_kpt_sview_rgb_img/topdown_heatmap'
+    '/animalpose/hrnet_w32_animalpose_256x256.py',
+    model_checkpoint='https://download.openmmlab.com/mmpose/animal/'
+    'hrnet/hrnet_w32_animalpose_256x256-1aa7f075_20210426.pth',
+    cls_names=['cat', 'dog', 'horse', 'sheep', 'cow'],
+    input_buffer='human_pose',
+    output_buffer='animal_pose')
+```
+
+- **Run the demo without GPU**
+
+If you don't have GPU and CUDA in your device, the demo can run with only CPU by setting `device='cpu'` in all model nodes. For example:
+
+```python
+dict(
+    type='DetectorNode',
+    name='Detector',
+    model_config='demo/mmdetection_cfg/'
+    'ssdlite_mobilenetv2_scratch_600e_coco.py',
+    model_checkpoint='https://download.openmmlab.com'
+    '/mmdetection/v2.0/ssd/'
+    'ssdlite_mobilenetv2_scratch_600e_coco/ssdlite_mobilenetv2_'
+    'scratch_600e_coco_20210629_110627-974d9307.pth',
+    device='cpu',
+    input_buffer='_input_',
+    output_buffer='det_result')
+```
+
+- **Debug webcam and display**
+
+You can lanch the webcam runner with a debug config:
+
+```shell
+python tools/webcam/run_webcam.py --config tools/webcam/configs/examples/test_camera.py
+```
@@ -0,0 +1,115 @@
+# Copyright (c) OpenMMLab. All rights reserved.
+runner = dict(
+    # Basic configurations of the runner
+    name='Pose Estimation',
+    camera_id=0,
+    camera_fps=20,
+    synchronous=False,
+    # Define nodes.
+    # The configuration of a node usually includes:
+    #   1. 'type': Node class name
+    #   2. 'name': Node name
+    #   3. I/O buffers (e.g. 'input_buffer', 'output_buffer'): specify the
+    #       input and output buffer names. This may depend on the node class.
+    #   4. 'enable_key': assign a hot-key to toggle enable/disable this node.
+    #       This may depend on the node class.
+    #   5. Other class-specific arguments
+    nodes=[
+        # 'DetectorNode':
+        # This node performs object detection from the frame image using an
+        # MMDetection model.
+        dict(
+            type='DetectorNode',
+            name='Detector',
+            model_config='demo/mmdetection_cfg/'
+            'ssdlite_mobilenetv2_scratch_600e_coco.py',
+            model_checkpoint='https://download.openmmlab.com'
+            '/mmdetection/v2.0/ssd/'
+            'ssdlite_mobilenetv2_scratch_600e_coco/ssdlite_mobilenetv2_'
+            'scratch_600e_coco_20210629_110627-974d9307.pth',
+            input_buffer='_input_',  # `_input_` is a runner-reserved buffer
+            output_buffer='det_result'),
+        # 'TopDownPoseEstimatorNode':
+        # This node performs keypoint detection from the frame image using an
+        # MMPose top-down model. Detection results is needed.
+        dict(
+            type='TopDownPoseEstimatorNode',
+            name='Human Pose Estimator',
+            model_config='configs/wholebody/2d_kpt_sview_rgb_img/'
+            'topdown_heatmap/coco-wholebody/'
+            'vipnas_mbv3_coco_wholebody_256x192_dark.py',
+            model_checkpoint='https://download.openmmlab.com/mmpose/top_down/'
+            'vipnas/vipnas_mbv3_coco_wholebody_256x192_dark'
+            '-e2158108_20211205.pth',
+            cls_names=['person'],
+            input_buffer='det_result',
+            output_buffer='human_pose'),
+        dict(
+            type='TopDownPoseEstimatorNode',
+            name='Animal Pose Estimator',
+            model_config='configs/animal/2d_kpt_sview_rgb_img/topdown_heatmap'
+            '/animalpose/hrnet_w32_animalpose_256x256.py',
+            model_checkpoint='https://download.openmmlab.com/mmpose/animal/'
+            'hrnet/hrnet_w32_animalpose_256x256-1aa7f075_20210426.pth',
+            cls_names=['cat', 'dog', 'horse', 'sheep', 'cow'],
+            input_buffer='human_pose',
+            output_buffer='animal_pose'),
+        # 'ModelResultBindingNode':
+        # This node binds the latest model inference result with the current
+        # frame. (This means the frame image and inference result may be
+        # asynchronous).
+        dict(
+            type='ModelResultBindingNode',
+            name='ResultBinder',
+            frame_buffer='_frame_',  # `_frame_` is a runner-reserved buffer
+            result_buffer='animal_pose',
+            output_buffer='frame'),
+        # 'PoseVisualizerNode':
+        # This node draw the pose visualization result in the frame image.
+        # Pose results is needed.
+        dict(
+            type='PoseVisualizerNode',
+            name='Visualizer',
+            enable_key='v',
+            frame_buffer='frame',
+            output_buffer='vis'),
+        # 'NoticeBoardNode':
+        # This node show a notice board with given content, e.g. help
+        # information.
+        dict(
+            type='NoticeBoardNode',
+            name='Helper',
+            enable_key='h',
+            enable=True,
+            frame_buffer='vis',
+            output_buffer='vis_notice',
+            content_lines=[
+                'This is a demo for pose visualization and simple image '
+                'effects. Have fun!', '', 'Hot-keys:',
+                '"v": Pose estimation result visualization',
+                '"s": Sunglasses effect B-)', '"b": Bug-eye effect 0_0',
+                '"h": Show help information',
+                '"m": Show diagnostic information', '"q": Exit'
+            ],
+        ),
+        # 'MonitorNode':
+        # This node show diagnostic information in the frame image. It can
+        # be used for debugging or monitoring system resource status.
+        dict(
+            type='MonitorNode',
+            name='Monitor',
+            enable_key='m',
+            enable=False,
+            frame_buffer='vis_notice',
+            output_buffer='display'),
+        # 'RecorderNode':
+        # This node save the output video into a file.
+        dict(
+            type='RecorderNode',
+            name='Recorder',
+            out_video_file='record.mp4',
+            frame_buffer='display',
+            output_buffer='_display_'
+            # `_display_` is a runner-reserved buffer
+        )
+    ])
@@ -0,0 +1,19 @@
+# Copyright (c) OpenMMLab. All rights reserved.
+runner = dict(
+    name='Debug CamRunner',
+    camera_id=0,
+    camera_fps=20,
+    nodes=[
+        dict(
+            type='MonitorNode',
+            name='Monitor',
+            enable_key='m',
+            frame_buffer='_frame_',
+            output_buffer='display'),
+        dict(
+            type='RecorderNode',
+            name='Recorder',
+            out_video_file='webcam_output.mp4',
+            frame_buffer='display',
+            output_buffer='_display_')
+    ])