Initial snapshot: hand motion pipeline (Dyn-HaMR + dex-retargeting + SPIDER)

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-09-15 14:13:49 +08:00
commit 7e4ef6f98b
7002 changed files with 1452249 additions and 0 deletions
@@ -0,0 +1,96 @@
+# Super Saiyan Effects
+
+We can apply fun effects on videos with pose estimation results, like Super Saiyan transformation.
+
+https://user-images.githubusercontent.com/11788150/150138076-2192079f-068a-4d43-bf27-2f1fd708cabc.mp4
+
+## Instruction
+
+### Get started
+
+Launch the demo from the mmpose root directory:
+
+```shell
+python tools/webcam/run_webcam.py --config tools/webcam/configs/supersaiyan/saiyan.py
+```
+
+### Hotkeys
+
+| Hotkey | Function |
+| -- | -- |
+| s | Toggle the Super Saiyan effect on/off. |
+| h | Show help information. |
+| m | Show the monitoring information. |
+| q | Exit. |
+
+Note that the demo will automatically save the output video into a file `record.mp4`.
+
+### Configuration
+
+- **Choose a detection model**
+
+Users can choose detection models from the [MMDetection Model Zoo](https://mmdetection.readthedocs.io/en/v2.20.0/model_zoo.html). Just set the `model_config` and `model_checkpoint` in the detector node accordingly, and the model will be automatically downloaded and loaded.
+
+```python
+# 'DetectorNode':
+# This node performs object detection from the frame image using an
+# MMDetection model.
+dict(
+    type='DetectorNode',
+    name='Detector',
+    model_config='demo/mmdetection_cfg/mask_rcnn_r50_fpn_2x_coco.py',
+    model_checkpoint='https://download.openmmlab.com/'
+    'mmdetection/v2.0/mask_rcnn/mask_rcnn_r50_fpn_2x_coco/'
+    'mask_rcnn_r50_fpn_2x_coco_bbox_mAP-0.392'
+    '__segm_mAP-0.354_20200505_003907-3e542a40.pth',
+    input_buffer='_input_',  # `_input_` is a runner-reserved buffer
+    output_buffer='det_result'),
+```
+
+- **Choose a or more pose models**
+
+In this demo we use two [top-down](https://github.com/open-mmlab/mmpose/tree/master/configs/body/2d_kpt_sview_rgb_img/topdown_heatmap) pose estimation models for humans and animals respectively. Users can choose models from the [MMPose Model Zoo](https://mmpose.readthedocs.io/en/latest/modelzoo.html). To apply different pose models on different instance types, you can add multiple pose estimator nodes with `cls_names` set accordingly.
+
+```python
+# 'TopDownPoseEstimatorNode':
+# This node performs keypoint detection from the frame image using an
+# MMPose top-down model. Detection results is needed.
+dict(
+    type='TopDownPoseEstimatorNode',
+    name='Human Pose Estimator',
+    model_config='configs/wholebody/2d_kpt_sview_rgb_img/'
+    'topdown_heatmap/coco-wholebody/'
+    'vipnas_mbv3_coco_wholebody_256x192_dark.py',
+    model_checkpoint='https://openmmlab-share.oss-cn-hangz'
+    'hou.aliyuncs.com/mmpose/top_down/vipnas/vipnas_mbv3_co'
+    'co_wholebody_256x192_dark-e2158108_20211205.pth',
+    cls_names=['person'],
+    input_buffer='det_result',
+    output_buffer='human_pose')
+```
+
+- **Run the demo without GPU**
+
+If you don't have GPU and CUDA in your device, the demo can run with only CPU by setting `device='cpu'` in all model nodes. For example:
+
+```python
+dict(
+    type='DetectorNode',
+    name='Detector',
+    model_config='demo/mmdetection_cfg/mask_rcnn_r50_fpn_2x_coco.py',
+    model_checkpoint='https://download.openmmlab.com/'
+    'mmdetection/v2.0/mask_rcnn/mask_rcnn_r50_fpn_2x_coco/'
+    'mask_rcnn_r50_fpn_2x_coco_bbox_mAP-0.392'
+    '__segm_mAP-0.354_20200505_003907-3e542a40.pth',
+    device='cpu',
+    input_buffer='_input_',  # `_input_` is a runner-reserved buffer
+    output_buffer='det_result'),
+```
+
+- **Debug webcam and display**
+
+You can launch the webcam runner with a debug config:
+
+```shell
+python tools/webcam/run_webcam.py --config tools/webcam/configs/examples/test_camera.py
+```
@@ -0,0 +1,93 @@
+# Copyright (c) OpenMMLab. All rights reserved.
+runner = dict(
+    # Basic configurations of the runner
+    name='Super Saiyan Effects',
+    camera_id=0,
+    camera_fps=30,
+    synchronous=False,
+    # Define nodes.
+    # The configuration of a node usually includes:
+    #   1. 'type': Node class name
+    #   2. 'name': Node name
+    #   3. I/O buffers (e.g. 'input_buffer', 'output_buffer'): specify the
+    #       input and output buffer names. This may depend on the node class.
+    #   4. 'enable_key': assign a hot-key to toggle enable/disable this node.
+    #       This may depend on the node class.
+    #   5. Other class-specific arguments
+    nodes=[
+        # 'DetectorNode':
+        # This node performs object detection from the frame image using an
+        # MMDetection model.
+        dict(
+            type='DetectorNode',
+            name='Detector',
+            model_config='demo/mmdetection_cfg/mask_rcnn_r50_fpn_2x_coco.py',
+            model_checkpoint='https://download.openmmlab.com/'
+            'mmdetection/v2.0/mask_rcnn/mask_rcnn_r50_fpn_2x_coco/'
+            'mask_rcnn_r50_fpn_2x_coco_bbox_mAP-0.392'
+            '__segm_mAP-0.354_20200505_003907-3e542a40.pth',
+            input_buffer='_input_',  # `_input_` is a runner-reserved buffer
+            output_buffer='det_result'),
+        # 'TopDownPoseEstimatorNode':
+        # This node performs keypoint detection from the frame image using an
+        # MMPose top-down model. Detection results is needed.
+        dict(
+            type='TopDownPoseEstimatorNode',
+            name='Human Pose Estimator',
+            model_config='configs/wholebody/2d_kpt_sview_rgb_img/'
+            'topdown_heatmap/coco-wholebody/'
+            'vipnas_mbv3_coco_wholebody_256x192_dark.py',
+            model_checkpoint='https://openmmlab-share.oss-cn-hangz'
+            'hou.aliyuncs.com/mmpose/top_down/vipnas/vipnas_mbv3_co'
+            'co_wholebody_256x192_dark-e2158108_20211205.pth',
+            cls_names=['person'],
+            input_buffer='det_result',
+            output_buffer='human_pose'),
+        # 'ModelResultBindingNode':
+        # This node binds the latest model inference result with the current
+        # frame. (This means the frame image and inference result may be
+        # asynchronous).
+        dict(
+            type='ModelResultBindingNode',
+            name='ResultBinder',
+            frame_buffer='_frame_',  # `_frame_` is a runner-reserved buffer
+            result_buffer='human_pose',
+            output_buffer='frame'),
+        # 'SaiyanNode':
+        # This node draw the Super Saiyan effect in the frame image.
+        # Pose results is needed.
+        dict(
+            type='SaiyanNode',
+            name='Visualizer',
+            enable_key='s',
+            cls_names=['person'],
+            enable=True,
+            frame_buffer='frame',
+            output_buffer='vis_saiyan'),
+        # 'NoticeBoardNode':
+        # This node show a notice board with given content, e.g. help
+        # information.
+        dict(
+            type='NoticeBoardNode',
+            name='Helper',
+            enable_key='h',
+            frame_buffer='vis_saiyan',
+            output_buffer='vis',
+            content_lines=[
+                'This is a demo for super saiyan effects. Have fun!', '',
+                'Hot-keys:', '"s": Saiyan effect',
+                '"h": Show help information',
+                '"m": Show diagnostic information', '"q": Exit'
+            ],
+        ),
+        # 'MonitorNode':
+        # This node show diagnostic information in the frame image. It can
+        # be used for debugging or monitoring system resource status.
+        dict(
+            type='MonitorNode',
+            name='Monitor',
+            enable_key='m',
+            enable=False,
+            frame_buffer='vis',
+            output_buffer='_display_')  # `_frame_` is a runner-reserved buffer
+    ])