liyang/hand-motion-pipeline
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Initial snapshot: hand motion pipeline (Dyn-HaMR + dex-retargeting + SPIDER)

Browse Source 
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
liyang
2026-09-15 14:13:49 +08:00
commit 7e4ef6f98b
7002 changed files with 1452249 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
third_party/Dyn-HaMR/third-party/Hand-BMC-pytorch-main/LICENSE
Vendored
+21
View File
@@ -0,0 +1,21 @@
MIT License
Copyright (c) 2021 Hao Meng
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
third_party/Dyn-HaMR/third-party/Hand-BMC-pytorch-main/README.md
Vendored
+141
View File
@@ -0,0 +1,141 @@
# Hand Biomechanical Constraints Pytorch
**Unofficial** PyTorch reimplementation of [Hand-Biomechanical-Constraints](https://www.ecva.net/papers/eccv_2020/papers_ECCV/papers/123620205.pdf) (ECCV2020).
This project reimplement following components :
1. 3 kinds of biomechanical soft constraints
1. integrate BMC into training procedure (PyTorch version)
## Usage
- Retrieve the code
```sh
git clone https://github.com/MengHao666/Hand-BMC-pytorch
cd Hand-BMC-pytorch
```
- Create and activate the virtual environment with python dependencies
```
conda env create --file=environment.yml
conda activate bmc
```
## Download data
Download 3D joint location data `joints.zip` [Google Drive](https://drive.google.com/file/d/1_wV8QjmsVCMBEBhm56gFA2XTyU8VEHzk/view?usp=sharing) or
[Baidu Pan](https://pan.baidu.com/s/1klLI8H7EojRdQFgoyQNhJg) (`2pip`), and . These statistics are from following datasets:
* [Rendered Handpose Dataset](https://lmb.informatik.uni-freiburg.de/resources/datasets/RenderedHandposeDataset.en.html)
* [GANerated Hands Dataset](https://handtracker.mpi-inf.mpg.de/projects/GANeratedHands/GANeratedDataset.htm)
* [STB Dataset](https://github.com/zhjwustc/icip17_stereo_hand_pose_dataset)
* [Freihand Dataset](https://lmb.informatik.uni-freiburg.de/resources/datasets/FreihandDataset.en.html)
**Note the data from these datasets under their own licenses.**
## Calculate BMC
![BMC](assets/BMC.png)
### Run the code
```
python calculate_bmc.py
```
You will get
- `bone_len_max.npy` `bone_len_min.npy` for bone length limits
- `curvatures_max.npy` `curvatures_min.npy` for Root bones' curvatures
- `PHI_max.npy` `PHI_min.npy` for Root bones' angular distance
- `joint_angles.npy` for Joint angles
And if u want to check the coordinate system, run the code
```
cd utils
python calculate_joint_angles.py
```
- red ,green, blue arrows refer to X,Y,Z of local coordinate system respectively;
- dark arrows refer to bones;
- pink arrows refer to bone projection into X-Z plane of local coordinate system;
One view | Another view
:-------------------------:|:-------------------------:
![](assets/coordinate1.png) | ![](assets/coordinate2.png)
### Run the code
```
python calculate_convex_hull.py
```
You will get `CONVEX_HULLS.npy`, i.e. convex hulls to encircle the anatomically plausible joint angles.
And you will also see every convex hull like following figure:
![BMC](assets/0_convex_hull.png)
- **"Bone PIP"** means the bone from MCP joint to PIP joint in thumb
- **flexion** and **abduction** is two kinds of angle describing joint rotation
- **"ori_convex_hull"** means the original convex hull calculated from all joint angle points
- **"rdp_convex_hull"** means convex hull simplified by the Ramer-Douglas-Peucker algorithm, a polygon simplification algorithm
- **"del_convex_hull"** means convex hull further simplified by a greedy algorithm
- **"rectangle"** means the minimal rectangle to surround all joint angle points
### Run the code
```
python plot.py
```
You will see all the convex hulls
![BMC](assets/convex_hulls.png)
## Integrate BMC into training (PyTorch version)
Run the code
```
python weakloss.py
```
## Experiment results
To check influence of BMC, instead of reimplementing the network of origin paper, I integrate BMC into my own [project](https://github.com/MengHao666/Minimal-Hand-pytorch),
### Train and evaluation curve
(AUC means 3D PCK, and ACC_HM means 2D PCK)
![teaser](assets/train_eval_curve.png)
### 3D PCK AUC Diffenence
| Dataset | DetNet | DetNet+BMC |
| :-----: | :------------: | :-----: |
| **RHD** | 0.9339 | 0.9364 |
| **STB** | 0.8744 | 0.8778 |
| **DO** | 0.9378 | 0.9475 |
| **EO** | 0.9270 | 0.9182 |
### Note
- Adjusting training parameters carefully, longer training time might further boost accuracy.
- **As BMC is a weakly supervised method, it may only make predictions more physically plausible,but cannot boost AUC performance strongly when strong supervision is used.**
## Limitation
- Due to time limitation, I didn't reimplement the network and experiments of original paper.
- There is a little difference between original paper and my reimplementation. But most of them match.
## Citation
This is the **unofficial** pytorch reimplementation of the paper "Weakly supervised 3d hand pose estimation via biomechanical constraints (ECCV 2020).
If you find the project helpful, please star this project and cite them:
```
@article{spurr2020weakly,
title={Weakly supervised 3d hand pose estimation via biomechanical constraints},
author={Spurr, Adrian and Iqbal, Umar and Molchanov, Pavlo and Hilliges, Otmar and Kautz, Jan},
journal={arXiv preprint arXiv:2003.09282},
volume={8},
year={2020},
publisher={Springer}
}
```
third_party/Dyn-HaMR/third-party/Hand-BMC-pytorch-main/__init__.py
Vendored
View File
third_party/Dyn-HaMR/third-party/Hand-BMC-pytorch-main/calculate_bmc.py
Vendored
+127
View File
@@ -0,0 +1,127 @@
# Copyright (c) Hao Meng. All Rights Reserved.
import os
import numpy as np
from tqdm import tqdm
import config as cfg
from utils import calculate_Joint_angles
def load_joints_(set, mode):
path = os.path.join("joints", "{}_{}.npy".format(set, mode))
joints = np.load(path)
return joints
def normalize(vec):
len = np.linalg.norm(vec, axis=-1, keepdims=True)
return vec / len
def load_joints(dict):
joints_all = []
for set, mode in dict.items():
for m in mode:
path = os.path.join("joints", "{}_{}.npy".format(set, m))
joints = np.load(path)
joints_all.append(joints)
joints_all = np.concatenate(joints_all, axis=0).astype(np.float32) # (N, 21,3)
return joints_all
if __name__ == '__main__':
path = "BMC"
if not os.path.isdir(path):
os.makedirs(path)
# original 3D joints in minimal-hand's setting
th_dict = {"rhd": ["train", "test"], "gan": ["train"]}
th_joints = load_joints(th_dict)
joints = th_joints.copy() # (N, 21, 3)
# all joints available setting
# all_dict = {"rhd": ["train", "test"], "gan": ["train"], "stb": ["train", "test"], "fh": ["train"]}
# all_joints = load_joints(all_dict)
# joints = all_joints.copy() # (N, 21, 3)
# root-relative
joints_root = np.expand_dims(joints[:, cfg.JOINT_ROOT_IDX, :], 1)
joints = joints - joints_root
# scale-invariant
ref_bones = np.linalg.norm(joints[:, cfg.REF_BONE_LINK[0]] - joints[:, cfg.REF_BONE_LINK[1]], axis=1)
ref_bones = np.expand_dims(ref_bones, [1, 2])
joints = joints / ref_bones
# loop
# for i in tqdm(range(0, joints.shape[0], 5000)):
# joints_i = joints[i]
# # visualization
# vis.plot3d(joints_i)
# bones = bone.caculate_length(joints_i, label="all")
# print(bones[9])
# calculate bone length limits
kin_chain = [
joints[:, i] - joints[:, cfg.SNAP_PARENT[i]]
for i in range(1, 21)
]
kin_chain = np.array(kin_chain)
kin_chain = kin_chain.swapaxes(1, 0) # (N*20*3)
bone_lens = np.linalg.norm(
kin_chain, ord=2, axis=-1, keepdims=True
) # (N*20)
bone_lens = np.squeeze(bone_lens)
max_bone_len = np.max(bone_lens, axis=0) # (20,)
min_bone_len = np.min(bone_lens, axis=0) # (20,)
np.save("BMC/bone_len_min.npy", min_bone_len)
np.save("BMC/bone_len_max.npy", max_bone_len)
# calculate root bone limits
root_bones = kin_chain[:, [0, 4, 8, 12, 16], :]
normals = normalize(np.cross(root_bones[:, 1:], root_bones[:, :-1]))
edge_normals = np.zeros_like(root_bones)
edge_normals[:, 0] = normals[:, 0]
edge_normals[:, 4] = normals[:, 3]
edge_normals[:, 1:4] = normalize(normals[:, 1:] + normals[:, :-1])
curvatures = np.zeros([joints.shape[0], 4])
PHI = np.zeros([joints.shape[0], 4])
for i in range(4):
e_tmp = edge_normals[:, i + 1] - edge_normals[:, i]
b_tmp = root_bones[:, i + 1] - root_bones[:, i]
b_tmp_norm = np.linalg.norm(
b_tmp, ord=2, axis=-1 # N
)
curvatures[:, i] = np.sum(e_tmp * b_tmp, axis=-1) / (b_tmp_norm ** 2)
PHI[:, i] = np.sum(root_bones[:, i] * root_bones[:, i + 1], axis=-1)
tmp1 = np.linalg.norm(root_bones[:, i], ord=2, axis=-1)
tmp2 = np.linalg.norm(root_bones[:, i + 1], ord=2, axis=-1)
PHI[:, i] /= (tmp1 * tmp2)
PHI[:, i] = np.arccos(PHI[:, i])
max_curvatures = np.max(curvatures, axis=0) # (4,)
min_curvatures = np.min(curvatures, axis=0) # (4,)
np.save("BMC/curvatures_max.npy", max_curvatures)
np.save("BMC/curvatures_min.npy", min_curvatures)
max_PHI = np.max(PHI, axis=0) # (4,)
min_PHI = np.min(PHI, axis=0) # (4,)
np.save("BMC/PHI_max.npy", max_PHI)
np.save("BMC/PHI_min.npy", min_PHI)
jas = []
for i in tqdm(range(joints.shape[0])):
joint = joints[i]
ja = calculate_Joint_angles.caculate_ja(joint)
jas.append(ja)
jas = np.array(jas) # (N, 15, 2)
np.save("BMC/joint_angles.npy", jas)
third_party/Dyn-HaMR/third-party/Hand-BMC-pytorch-main/calculate_convex_hull.py
Vendored
+173
View File
@@ -0,0 +1,173 @@
import argparse
import os
import matplotlib.pyplot as plt
import numpy as np
from rdp import rdp
from utils import hull
FINGER_NAMES = ["Thumb", "Index", "Middle", "Ring", "Pinky(Little)"]
BONE_NAMES = ["PIP", "DIP", "TIP"]
def count_num_in_convex_hull(points_, convex_hull_):
'''
:param points_: points in plane, N*2
:param convex_hull_: list of points to construct a convex hull , M*2
:return: numbers of points in convex hull
'''
points = points_.copy()
convex_hull = convex_hull_.copy()
convex_hull = np.append(convex_hull, [convex_hull[0]], axis=0)
v = convex_hull[1:] - convex_hull[:-1] # NUM*2
v = np.tile(v, (points_.shape[0], 1, 1))
w = -np.tile(convex_hull[:-1], (points_.shape[0], 1, 1)) + points[:, np.newaxis, :] # N*NUM*2
# 2d cross product (w1, w2) x (v1, v2) := w1*v2 - w2*v1
cross_product_2d = w[:, :, 0] * v[:, :, 1] - w[:, :, 1] * v[:, :, 0]
tmp = (cross_product_2d < 1e-6).all(axis=-1).sum()
return tmp
def calculate_convex_hull(joint_angles, args):
all_del_hulls = []
for ID in range(15):
print('*' * 40)
ja = joint_angles[:, ID]
ja_list = [tuple(x) for x in ja.tolist()]
# original convex hull
convex = hull.convex(ja_list)
convex = np.array(convex)
print("ori_hull.shape=", convex.shape)
convex_list = convex.tolist()
# convex hull simplified by the Ramer-Douglas-Peucker algorithm, a polygon simplification algorithm
rdp_convex = rdp(convex_list, epsilon=args.epsilon)
rdp_convex = np.array(rdp_convex)
print("rdp_hull.shape=", rdp_convex.shape)
# convex hull further simplified by a greedy algorithm
dep_hull = rdp_convex.copy()
for i in range(rdp_convex.shape[0]):
tmp_index = np.argwhere(dep_hull == rdp_convex[i])[0][0]
tmp_num = count_num_in_convex_hull(ja, np.delete(dep_hull, tmp_index, axis=0))
tmp_ratio = tmp_num / ja.shape[0]
print("tmp_ratio=", tmp_ratio)
if tmp_ratio > args.ratio:
dep_hull = np.delete(dep_hull, tmp_index, axis=0)
elif args.ratio - args.delta <= tmp_ratio <= args.ratio:
dep_hull = np.delete(dep_hull, tmp_index, axis=0)
break
else:
break
print("dep_hull.shape=", dep_hull.shape)
all_del_hulls.append(dep_hull)
if args.visualize:
fig = plt.figure()
figManager = plt.get_current_fig_manager()
figManager.window.showMaximized()
ax = fig.add_subplot(111)
ax.scatter(ja[:, 0], ja[:, 1], s=0.05, c='r')
ax.set_xlabel("flexion")
ax.set_ylabel("abduction")
ax.set_title('ID:{} Finger:{} Bone:{}'.format(ID, FINGER_NAMES[ID % 5], BONE_NAMES[int(ID / 5)]))
plt.plot(np.append(convex[:, 0], convex[0, 0]), np.append(convex[:, 1], convex[0, 1]), 'ro--', linewidth=3,
label='ori_convex_hull')
plt.plot(np.append(rdp_convex[:, 0], rdp_convex[0, 0]), np.append(rdp_convex[:, 1], rdp_convex[0, 1]),
'gv--',
linewidth=2, label='rdp_convex_hull')
plt.plot(np.append(dep_hull[:, 0], dep_hull[0, 0]),
np.append(dep_hull[:, 1], dep_hull[0, 1]),
'-b*', linewidth=1, label='del_convex_hull')
ja_min = np.min(ja, axis=0)
ja_max = np.max(ja, axis=0)
plt.gca().add_patch(
plt.Rectangle((ja_min[0], ja_min[1]), ja_max[0] - ja_min[0], ja_max[1] - ja_min[1], edgecolor="yellow",
fill=False, linewidth=2, label='rectangle'))
plt.xticks(np.arange(-3, 4, 1))
plt.yticks(np.arange(-2, 2, 0.5))
plt.legend(title='Convex hull category:')
plt.show()
# check if hull_test is in counter-clockwise order
# hull_test = convex
# for END in range(
# hull_test.shape[0]):
# fig = plt.figure()
# figManager = plt.get_current_fig_manager()
# figManager.window.showMaximized()
# ax = fig.add_subplot(111)
# ax.scatter(ja[:, 0], ja[:, 1], s=0.1, c='r')
# ax.set_xlabel("flexion")
# ax.set_ylabel("abduction")
#
# plt.plot(hull_test[:END, 0], hull_test[:END, 1], 'y--', linewidth=1)
# plt.xticks(np.arange(-3, 4, 1))
# plt.yticks(np.arange(-2, 2, 0.5))
# plt.show()
all_del_hulls = np.array(all_del_hulls, dtype=object)
np.save("BMC/CONVEX_HULLS.npy", all_del_hulls)
print("all_del_hulls.shape=", all_del_hulls.shape)
def main(args):
joint_angles = np.load(os.path.join(args.path, "joint_angles.npy"))
print("joint_angles.shape=", joint_angles.shape)
calculate_convex_hull(joint_angles, args)
if __name__ == '__main__':
parser = argparse.ArgumentParser(
description='calculate convex hull for joint angles ')
parser.add_argument(
'-p',
'--path',
default='BMC',
type=str,
metavar='data_root',
help='directory')
parser.add_argument(
'-vis',
'--visualize',
action='store_true',
help='visualize reconstruction result',
default=True
)
parser.add_argument(
'--epsilon',
type=float,
default=5e-4,
help='epsilon0.'
)
parser.add_argument(
'--ratio',
type=float,
default=0.9995,
help='ration0.'
)
parser.add_argument(
'--delta',
type=float,
default=0.0005,
help='ration_delta.'
)
main(parser.parse_args())
third_party/Dyn-HaMR/third-party/Hand-BMC-pytorch-main/config.py
Vendored
+34
View File
@@ -0,0 +1,34 @@
import numpy as np
SNAP_PARENT = [
0, # 0's parent
0, # 1's parent
1,
2,
3,
0, # 5's parent
5,
6,
7,
0, # 9's parent
9,
10,
11,
0, # 13's parent
13,
14,
15,
0, # 17's parent
17,
18,
19,
]
JOINT_ROOT_IDX = 9
REF_BONE_LINK = (0, 9) # mid mcp
# bone indexes in 20 bones setting
ID_ROOT_bone = np.array([0, 4, 8, 12, 16]) # ROOT_bone from wrist to MCP
ID_PIP_bone = np.array([1, 5, 9, 13, 17]) # PIP_bone from MCP to PIP
ID_DIP_bone = np.array([2, 6, 10, 14, 18]) # DIP_bone from PIP to DIP
ID_TIP_bone = np.array([3, 7, 11, 15, 19]) # TIP_bone from DIP to TIP
third_party/Dyn-HaMR/third-party/Hand-BMC-pytorch-main/environment.yml
Vendored
+78
View File
@@ -0,0 +1,78 @@
name: bmc
channels:
- pytorch
- defaults
dependencies:
- _libgcc_mutex=0.1=main
- blas=1.0=mkl
- ca-certificates=2021.1.19=h06a4308_0
- certifi=2020.12.5=py37h06a4308_0
- cffi=1.14.5=py37h261ae71_0
- cudatoolkit=10.0.130=0
- cycler=0.10.0=py37_0
- dbus=1.13.18=hb2f20db_0
- expat=2.2.10=he6710b0_2
- fontconfig=2.13.1=h6c09931_0
- freetype=2.10.4=h5ab3b9f_0
- glib=2.67.4=h36276a3_1
- gst-plugins-base=1.14.0=h8213a91_2
- gstreamer=1.14.0=h28cd5cc_2
- icu=58.2=he6710b0_3
- intel-openmp=2020.2=254
- jpeg=9b=h024ee3a_2
- kiwisolver=1.3.1=py37h2531618_0
- lcms2=2.11=h396b838_0
- ld_impl_linux-64=2.33.1=h53a641e_7
- libedit=3.1.20191231=h14c3975_1
- libffi=3.3=he6710b0_2
- libgcc-ng=9.1.0=hdf63c60_0
- libpng=1.6.37=hbc83047_0
- libstdcxx-ng=9.1.0=hdf63c60_0
- libtiff=4.1.0=h2733197_1
- libuuid=1.0.3=h1bed415_2
- libxcb=1.14=h7b6447c_0
- libxml2=2.9.10=hb55368b_3
- lz4-c=1.9.3=h2531618_0
- matplotlib=3.3.4=py37h06a4308_0
- matplotlib-base=3.3.4=py37h62a2d02_0
- mkl=2020.2=256
- mkl-service=2.3.0=py37he8ac12f_0
- mkl_fft=1.3.0=py37h54f3939_0
- mkl_random=1.1.1=py37h0573a6f_0
- ncurses=6.2=he6710b0_1
- ninja=1.10.2=py37hff7bd54_0
- numpy=1.19.2=py37h54aff64_0
- numpy-base=1.19.2=py37hfa32c7d_0
- olefile=0.46=py_0
- openssl=1.1.1j=h27cfd23_0
- pcre=8.44=he6710b0_0
- pillow=8.1.0=py37he98fc37_0
- pip=21.0.1=py37h06a4308_0
- pycparser=2.20=py_2
- pyparsing=2.4.7=pyhd3eb1b0_0
- pyqt=5.9.2=py37h05f1152_2
- python=3.7.10=hdb3f193_0
- python-dateutil=2.8.1=pyhd3eb1b0_0
- qt=5.9.7=h5867ecd_1
- readline=8.1=h27cfd23_0
- setuptools=52.0.0=py37h06a4308_0
- sip=4.19.8=py37hf484d3e_0
- six=1.15.0=pyhd3eb1b0_0
- sqlite=3.33.0=h62c20be_0
- tk=8.6.10=hbc83047_0
- tornado=6.1=py37h27cfd23_0
- tqdm=4.56.0=pyhd3eb1b0_0
- wheel=0.36.2=pyhd3eb1b0_0
- xz=5.2.5=h7b6447c_0
- zlib=1.2.11=h7b6447c_3
- zstd=1.4.5=h9ceee32_0
- pytorch=1.2.0=py3.7_cuda10.0.130_cudnn7.6.2_0
- torchvision=0.4.0=py37_cu100
- pip:
- art==3.7
- coverage==4.5.3
- mxnet==1.6.0
- rdp==0.8
- torch==1.2.0
third_party/Dyn-HaMR/third-party/Hand-BMC-pytorch-main/plot.py
Vendored
+59
View File
@@ -0,0 +1,59 @@
import matplotlib.pyplot as plt
import numpy as np
from rdp import rdp
from tqdm import tqdm
import utils.hull as hull
joint_angles = np.load("BMC/joint_angles.npy")
print(joint_angles.shape)
EPSILON = 0.0005
FINGER_NAMES = ["Thumb", "Index", "Middle", "Ring", "Pinky(Little)"]
BONE_NAMES = ["PIP", "DIP", "TIP"] # i.e. PIP bone means the bone from MCP joint to PIP joint
del_rdp_hulls = np.load("BMC/CONVEX_HULLS.npy", allow_pickle=True)
fig = plt.figure()
plt.suptitle("CONVEX HULLS of JOINT ANGLES")
for ID in tqdm(range(15)):
ja = joint_angles[:, ID]
ja_list = [tuple(x) for x in ja.tolist()]
convex = hull.convex(ja_list)
convex = np.array(convex)
convex_list = convex.tolist()
rdp_convex = rdp(convex_list, epsilon=EPSILON)
rdp_convex = np.array(rdp_convex)
del_rdp_hull = del_rdp_hulls[ID]
ax = fig.add_subplot(3, 5, ID + 1)
ax.scatter(ja[:, 0], ja[:, 1], s=0.05, c='r')
ax.set_xlabel("flexion")
ax.set_ylabel("abduction")
ax.set_title('ID:{} Finger:{} Bone:{}'.format(ID, FINGER_NAMES[ID % 5], BONE_NAMES[int(ID / 5)]))
plt.plot(np.append(convex[:, 0], convex[0, 0]), np.append(convex[:, 1], convex[0, 1]), '--ro', linewidth=3,label='ori_convex_hull')
plt.plot(np.append(rdp_convex[:, 0], rdp_convex[0, 0]), np.append(rdp_convex[:, 1], rdp_convex[0, 1]), '--gv',
linewidth=2, label='rdp_convex_hull')
plt.plot(np.append(del_rdp_hull[:, 0], del_rdp_hull[0, 0]),
np.append(del_rdp_hull[:, 1], del_rdp_hull[0, 1]),
'-b*', linewidth=1, label='del_convex_hull')
plt.xticks(np.arange(-3, 4, 1))
plt.yticks(np.arange(-2, 2, 0.5))
ja_min = np.min(ja, axis=0)
ja_max = np.max(ja, axis=0)
plt.gca().add_patch(
plt.Rectangle((ja_min[0], ja_min[1]), ja_max[0] - ja_min[0], ja_max[1] - ja_min[1], edgecolor="yellow",
fill=False, linewidth=2, label='rectangle'))
plt.legend(title='Convex hull category:')
figManager = plt.get_current_fig_manager()
figManager.window.showMaximized()
plt.subplots_adjust(left=0.125, bottom=0.05, right=0.9, top=0.9, wspace=0.3, hspace=0.3)
plt.show()
third_party/Dyn-HaMR/third-party/Hand-BMC-pytorch-main/utils/__init__.py
Vendored
+1
View File
@@ -0,0 +1 @@
import config as cfg
third_party/Dyn-HaMR/third-party/Hand-BMC-pytorch-main/utils/calculate_Joint_angles.py
Vendored
+262
View File
@@ -0,0 +1,262 @@
import matplotlib.pyplot as plt
import numpy as np
import config as cfg
def angle_between(v1, v2):
'''
:param v1: B*3
:param v2: B*3
:return: B
'''
v1_u = normalize(v1.copy())
v2_u = normalize(v2.copy())
inner_product = np.sum(v1_u * v2_u, axis=-1)
tmp = np.clip(inner_product, -1.0, 1.0)
tmp = np.arccos(tmp)
return tmp
def normalize(vec_):
'''
:param vec: B*3
:return: B*1
'''
vec = vec_.copy()
len = calcu_len(vec) + 1e-8
return vec / len
def axangle2mat(axis, angle, is_normalized=False):
'''
:param axis: B*3
:param angle: B*1
:param is_normalized:
:return: B*3*3
'''
if not is_normalized:
axis = normalize(axis)
x = axis[:, 0];
y = axis[:, 1];
z = axis[:, 2]
c = np.cos(angle);
s = np.sin(angle);
C = 1 - c
xs = x * s;
ys = y * s;
zs = z * s
xC = x * C;
yC = y * C;
zC = z * C
xyC = x * yC;
yzC = y * zC;
zxC = z * xC
Q = np.array([
[x * xC + c, xyC - zs, zxC + ys],
[xyC + zs, y * yC + c, yzC - xs],
[zxC - ys, yzC + xs, z * zC + c]])
Q = Q.transpose(2, 0, 1)
return Q
def calcu_len(vec):
'''
calculate length of vector
:param vec: B*3
:return: B*1
'''
return np.linalg.norm(vec, axis=-1, keepdims=True)
def caculate_ja(joint, vis=False):
'''
:param joint: 21*3
:param vis:
:return: 15*2
'''
ALL_bones = np.array([
joint[i] - joint[cfg.SNAP_PARENT[i]]
for i in range(1, 21)
])
ROOT_bones = ALL_bones[cfg.ID_ROOT_bone] # FROM THUMB TO LITTLE FINGER
PIP_bones = ALL_bones[cfg.ID_PIP_bone]
DIP_bones = ALL_bones[cfg.ID_DIP_bone]
TIP_bones = ALL_bones[cfg.ID_TIP_bone]
ALL_Z_axis = normalize(ALL_bones)
PIP_Z_axis = ALL_Z_axis[cfg.ID_ROOT_bone]
DIP_Z_axis = ALL_Z_axis[cfg.ID_PIP_bone]
TIP_Z_axis = ALL_Z_axis[cfg.ID_DIP_bone]
normals = normalize(np.cross(ROOT_bones[1:5], ROOT_bones[0:4]))
# ROOT bones
PIP_X_axis = np.zeros([5, 3]) # (5,3)
PIP_X_axis[[0, 1, 4], :] = -normals[[0, 1, 3], :]
PIP_X_axis[2:4] = -normalize(normals[2:4] + normals[1:3])
PIP_Y_axis = normalize(np.cross(PIP_Z_axis, PIP_X_axis))
tmp = np.sum(PIP_bones * PIP_Y_axis, axis=-1, keepdims=True)
PIP_bones_xz = PIP_bones - tmp * PIP_Y_axis
PIP_theta_flexion = angle_between(PIP_bones_xz, PIP_Z_axis) # in global coordinate
PIP_theta_abduction = angle_between(PIP_bones_xz, PIP_bones) # in global coordinate
# x-component of the bone vector
tmp = np.sum((PIP_bones * PIP_X_axis), axis=-1)
tmp_index = np.where(tmp < 0)
PIP_theta_flexion[tmp_index] = -PIP_theta_flexion[tmp_index]
# y-component of the bone vector
tmp = np.sum((PIP_bones * PIP_Y_axis), axis=-1)
tmp_index = np.where(tmp < 0)
PIP_theta_abduction[tmp_index] = -PIP_theta_abduction[tmp_index]
temp_axis = normalize(np.cross(PIP_Z_axis, PIP_bones))
temp_alpha = angle_between(PIP_Z_axis, PIP_bones) # alpha belongs to [0, pi]
temp_R = axangle2mat(axis=temp_axis, angle=temp_alpha, is_normalized=True)
# DIP bones
DIP_X_axis = np.matmul(temp_R, PIP_X_axis[:, :, np.newaxis])
DIP_Y_axis = np.matmul(temp_R, PIP_Y_axis[:, :, np.newaxis])
DIP_X_axis = np.squeeze(DIP_X_axis)
DIP_Y_axis = np.squeeze(DIP_Y_axis)
tmp = np.sum(DIP_bones * DIP_Y_axis, axis=-1, keepdims=True)
DIP_bones_xz = DIP_bones - tmp * DIP_Y_axis
DIP_theta_flexion = angle_between(DIP_bones_xz, DIP_Z_axis) # in global coordinate
DIP_theta_abduction = angle_between(DIP_bones_xz, DIP_bones) # in global coordinate
# x-component of the bone vector
tmp = np.sum((DIP_bones * DIP_X_axis), axis=-1)
tmp_index = np.where(tmp < 0)
DIP_theta_flexion[tmp_index] = -DIP_theta_flexion[tmp_index]
# y-component of the bone vector
tmp = np.sum((DIP_bones * DIP_Y_axis), axis=-1)
tmp_index = np.where(tmp < 0)
DIP_theta_abduction[tmp_index] = -DIP_theta_abduction[tmp_index]
temp_axis = normalize(np.cross(DIP_Z_axis, DIP_bones))
temp_alpha = angle_between(DIP_Z_axis, DIP_bones) # alpha belongs to [pi/2, pi]
temp_R = axangle2mat(axis=temp_axis, angle=temp_alpha, is_normalized=True)
# TIP bones
TIP_X_axis = np.matmul(temp_R, DIP_X_axis[:, :, np.newaxis])
TIP_Y_axis = np.matmul(temp_R, DIP_Y_axis[:, :, np.newaxis])
TIP_X_axis = np.squeeze(TIP_X_axis)
TIP_Y_axis = np.squeeze(TIP_Y_axis)
tmp = np.sum(TIP_bones * TIP_Y_axis, axis=-1, keepdims=True)
TIP_bones_xz = TIP_bones - tmp * TIP_Y_axis
TIP_theta_flexion = angle_between(TIP_bones_xz, TIP_Z_axis) # in global coordinate
TIP_theta_abduction = angle_between(TIP_bones_xz, TIP_bones) # in global coordinate
# x-component of the bone vector
tmp = np.sum((TIP_bones * TIP_X_axis), axis=-1)
tmp_index = np.where(tmp < 0)
TIP_theta_flexion[tmp_index] = -TIP_theta_flexion[tmp_index]
# y-component of the bone vector
tmp = np.sum((TIP_bones * TIP_Y_axis), axis=-1)
tmp_index = np.where(tmp < 0)
TIP_theta_abduction[tmp_index] = -TIP_theta_abduction[tmp_index]
if vis:
fig = plt.figure(figsize=[50, 50])
ax = fig.gca(projection='3d')
plt.plot(joint[[0, 1, 2, 3, 5, 6, 7, 9, 10, 11, 13, 14, 15, 17, 18, 19], 0],
joint[[0, 1, 2, 3, 5, 6, 7, 9, 10, 11, 13, 14, 15, 17, 18, 19], 1],
joint[[0, 1, 2, 3, 5, 6, 7, 9, 10, 11, 13, 14, 15, 17, 18, 19], 2], 'yo', label='keypoint')
plt.plot(joint[:5, 0], joint[:5, 1],
joint[:5, 2],
'--y', )
# label='thumb')
plt.plot(joint[[0, 5, 6, 7, 8, ], 0], joint[[0, 5, 6, 7, 8, ], 1],
joint[[0, 5, 6, 7, 8, ], 2],
'--y',
)
plt.plot(joint[[0, 9, 10, 11, 12, ], 0], joint[[0, 9, 10, 11, 12], 1],
joint[[0, 9, 10, 11, 12], 2],
'--y',
)
plt.plot(joint[[0, 13, 14, 15, 16], 0], joint[[0, 13, 14, 15, 16], 1],
joint[[0, 13, 14, 15, 16], 2],
'--y',
)
plt.plot(joint[[0, 17, 18, 19, 20], 0], joint[[0, 17, 18, 19, 20], 1],
joint[[0, 17, 18, 19, 20], 2],
'--y',
)
plt.plot(joint[4][0], joint[4][1], joint[4][2], 'rD', label='thumb')
plt.plot(joint[8][0], joint[8][1], joint[8][2], 'r*', label='index')
plt.plot(joint[12][0], joint[12][1], joint[12][2], 'r+', label='middle')
plt.plot(joint[16][0], joint[16][1], joint[16][2], 'rx', label='ring')
plt.plot(joint[20][0], joint[20][1], joint[20][2], 'ro', label='pinky')
ax.set_xlabel('x')
ax.set_ylabel('y')
ax.set_zlabel('z')
parent = np.array(cfg.SNAP_PARENT[1:])
x, y, z = joint[parent, 0], joint[parent, 1], joint[parent, 2]
u, v, w = ALL_bones[:, 0], ALL_bones[:, 1], ALL_bones[:, 2],
ax.quiver(x, y, z, u, v, w, length=0.25, color="black", normalize=True)
ALL_X_axis = np.stack((PIP_X_axis, DIP_X_axis, TIP_X_axis), axis=0).reshape(-1, 3)
ALL_Y_axis = np.stack((PIP_Y_axis, DIP_Y_axis, TIP_Y_axis), axis=0).reshape(-1, 3)
ALL_Z_axis = np.stack((PIP_Z_axis, DIP_Z_axis, TIP_Z_axis), axis=0).reshape(-1, 3)
ALL_Bone_xz = np.stack((PIP_bones_xz, DIP_bones_xz, TIP_bones_xz), axis=0).reshape(-1, 3)
ALL_joints_ID = np.array([cfg.ID_PIP_bone, cfg.ID_DIP_bone, cfg.ID_TIP_bone]).flatten()
jx, jy, jz = joint[ALL_joints_ID, 0], joint[ALL_joints_ID, 1], joint[ALL_joints_ID, 2]
ax.quiver(jx, jy, jz, ALL_X_axis[:, 0], ALL_X_axis[:, 1], ALL_X_axis[:, 2], length=0.05, color="r",
normalize=True)
ax.quiver(jx, jy, jz, ALL_Y_axis[:, 0], ALL_Y_axis[:, 1], ALL_Y_axis[:, 2], length=0.10, color="g",
normalize=True)
ax.quiver(jx, jy, jz, ALL_Z_axis[:, 0], ALL_Z_axis[:, 1], ALL_Z_axis[:, 2], length=0.10, color="b",
normalize=True)
ax.quiver(jx, jy, jz, ALL_Bone_xz[:, 0], ALL_Bone_xz[:, 1], ALL_Bone_xz[:, 2], length=0.25, color="pink",
normalize=True)
plt.legend()
ax.view_init(-180, 90)
plt.show()
ALL_theta_flexion = np.stack((PIP_theta_flexion, DIP_theta_flexion, TIP_theta_flexion), axis=0).flatten() # (15,)
ALL_theta_abduction = np.stack((PIP_theta_abduction, DIP_theta_abduction, TIP_theta_abduction),
axis=0).flatten() # (15,)
ALL_theta = np.stack((ALL_theta_flexion, ALL_theta_abduction), axis=1) # (15, 2)
return ALL_theta
if __name__ == '__main__':
joint = np.array([[-0.6114823, 0.70412624, -0.36096475],
[-0.38049987, 0.303464, -0.58379203],
[-0.22365516, -0.05147859, -0.7509124],
[-0.04204354, -0.33151808, -0.92656446],
[0.09191624, -0.6089648, -1.0512774],
[0.13345791, -0.02334917, -0.24654008],
[0.34996158, -0.41111353, -0.16837479],
[0.4534959, -0.69078416, -0.12496376],
[0.49604133, -0.96323794, -0.10438757],
[0., 0., 0.],
[0.3559839, -0.37889394, 0.13638118],
[0.48572803, -0.69607633, 0.13675757],
[0.5390761, -0.9938516, 0.09033547],
[-0.14901564, 0.00647402, 0.16235268],
[0.19227624, -0.34850615, 0.29296255],
[0.37767693, -0.57762665, 0.36711285],
[0.27133223, -0.7816264, 0.20363101],
[-0.3334502, 0.0463345, 0.27828288],
[-0.2731263, -0.21098317, 0.49082187],
[-0.22576298, -0.40466458, 0.6499127],
[-0.16024478, -0.58365273, 0.8177859]])
caculate_ja(joint, vis=True)
third_party/Dyn-HaMR/third-party/Hand-BMC-pytorch-main/utils/hull.py
Vendored
+207
View File
@@ -0,0 +1,207 @@
import math
def distance(p1, p2):
"""
Calculates the distance between two points.
:param p1, p2: points
:return: distance between points
"""
p1x, p1y = p1
p2x, p2y = p2
return math.sqrt((p1x - p2x) ** 2 + (p2y - p2y) ** 2)
def knn(points, p, k):
"""
Calculates the k nearest neighbours of a point.
:param points: list of points
:param p: reference point
:param k: amount of neighbours
:return: list of k neighbours
"""
return sorted(points, key=lambda x: distance(p, x))[:k]
def intersects(p1, p2, p3, p4):
"""
Checks if the lines [p1, p2] and [p3, p4] intersect.
:param p1, p2: line
:param p3, p4: line
:return: lines intersect
"""
p0x, p0y = p1
p1x, p1y = p2
p2x, p2y = p3
p3x, p3y = p4
s10x = p1x - p0x
s10y = p1y - p0y
s32x = p3x - p2x
s32y = p3y - p2y
denom = s10x * s32y - s32x * s10y
if denom == 0:
return False
denom_positive = denom > 0
s02x = p0x - p2x
s02y = p0y - p2y
s_numer = s10x * s02y - s10y * s02x
if (s_numer < 0) == denom_positive:
return False
t_numer = s32x * s02y - s32y * s02x
if (t_numer < 0) == denom_positive:
return False
if (s_numer > denom) == denom_positive or (t_numer > denom) == denom_positive:
return False
t = t_numer / denom
x = p0x + (t * s10x)
y = p0y + (t * s10y)
return (x, y) not in [p1, p2, p3, p4]
def angle(p1, p2, previous=0):
"""
Calculates the angle between two points.
:param p1, p2: points
:param previous: previous angle
:return: angle
"""
p1x, p1y = p1
p2x, p2y = p2
return (math.atan2(p1y - p2y, p1x - p2x) - previous) % (math.pi * 2) - math.pi
def point_in_polygon(point, polygon):
"""
Checks if a point is inside a polygon.
:param point: point
:param polygon: polygon
:return: point is inside polygon
"""
px, py = point
size = len(polygon)
for i in range(size):
p1x, p1y = polygon[i]
p2x, p2y = polygon[(i + 1) % size]
if min(p1x, p2x) < px <= max(p1x, p2x):
p = p1y - p2y
q = p1x - p2x
y = (px - p1x) * p / q + p1y
if y < py:
return True
return False
def concave(points, k=3):
"""
Calculates the concave hull for a list of points. Each point is a tuple
containing the x- and y-coordinate. k defines the number of considered
neighbours.
:param points: list of points
:param k: considered neighbours
:return: concave hull
"""
dataset = list(set(points)) # Remove duplicates
if len(dataset) < 3:
raise Exception("Dataset length cannot be smaller than 3")
if len(dataset) == 3:
return dataset # Points are a polygon already
k = min(max(k, 3), len(dataset) - 1) # Make sure that k neighbours can be found
first = current = min(dataset, key=lambda x: x[1])
hull = [first] # Initialize hull
dataset.remove(first) # Remove processed point
previous_angle = 0
while (current != first or len(hull) == 1) and len(dataset) > 0:
if len(hull) == 3:
dataset.append(first) # Add first point again
neighbours = knn(dataset, current, k)
c_points = sorted(neighbours, key=lambda x: -angle(x, current, previous_angle))
its = True
i = -1
while its and i < len(c_points) - 1:
i += 1
last_point = 1 if c_points[i] == first else 0
j = 1
its = False
while not its and j < len(hull) - last_point:
its = intersects(hull[-1], c_points[i], hull[-j - 1], hull[-j])
j += 1
if its: # All points intersect, try again with higher a number of neighbours
return concave(points, k + 1)
previous_angle = angle(c_points[i], current)
current = c_points[i]
hull.append(current) # Valid candidate was found
dataset.remove(current)
for point in dataset:
if not point_in_polygon(point, hull):
return concave(points, k + 1)
return hull
def cross(o, a, b):
"""
Calculates cross between two vectors.
:param o, a: vector
:param o, b: vector
:return: cross product
"""
ox, oy = o
ax, ay = a
bx, by = b
return (ax - ox) * (by - oy) - (ay - oy) * (bx - ox)
def convex(points):
"""
Calculates the concave hull for a list of points. Each point is a tuple
containing the x- and y-coordinate.
:param points: list of points
:return: convex hull
"""
dataset = sorted(set(points)) # Remove duplicates
if len(dataset) <= 1:
return dataset
# Build lower hull
lower = []
for p in dataset:
while len(lower) >= 2 and cross(lower[-2], lower[-1], p) <= 0:
lower.pop()
lower.append(p)
# Build upper hull
upper = []
for p in reversed(dataset):
while len(upper) >= 2 and cross(upper[-2], upper[-1], p) <= 0:
upper.pop()
upper.append(p)
return lower[:-1] + upper[:-1]
third_party/Dyn-HaMR/third-party/Hand-BMC-pytorch-main/weakloss.py
Vendored
+370
View File
@@ -0,0 +1,370 @@
import os
import matplotlib.pyplot as plt
import numpy as np
import torch
import torch.nn.functional as torch_f
import config as cfg
def plot_hull(theta, hull):
del_rdp_hull = hull.detach().cpu().numpy()
theta = theta.detach().cpu().numpy()
fig = plt.figure()
figManager = plt.get_current_fig_manager()
figManager.window.showMaximized()
ax = fig.add_subplot(111)
ax.scatter(theta[:, 0], theta[:, 1], s=10, c='r')
ax.set_xlabel("flexion")
ax.set_ylabel("abduction")
plt.plot(del_rdp_hull[:, 0],
del_rdp_hull[:, 1],
'-yo', linewidth=2)
plt.xticks(np.arange(-3, 4, 1))
plt.yticks(np.arange(-2, 2, 0.5))
plt.show()
def two_norm(a):
'''
Args:
a: B*M*2 or B*M*3
Returns:
'''
return torch.norm(a, dim=-1)
def one_norm(a):
'''
Args:
a: B*M*2 or B*M*3
Returns:
'''
return torch.norm(a, dim=-1, p=1)
def calculate_joint_angle_loss(thetas, hulls):
'''
Args:
Theta: B*15*2
hulls: list
Returns:
'''
loss = torch.Tensor([0]).cuda()
for i in range(15):
# print("i=",i)
hull = hulls[i] # (M*2)
theta = thetas[:, i] # (B*2)
hull = torch.cat((hull, hull[0].unsqueeze(0)), dim=0)
v = (hull[1:] - hull[:-1]).unsqueeze(0) # (M-1)*2
w = - hull[:-1].unsqueeze(0) + theta.unsqueeze(1).repeat(1, hull[:-1].shape[0], 1) # B*(M-1)*2
cross_product_2d = w[:, :, 0] * v[:, :, 1] - w[:, :, 1] * v[:, :, 0]
tmp = torch.sum(cross_product_2d < 1e-6, dim=-1)
is_outside = (tmp != (hull.shape[0] - 1))
if not torch.sum(is_outside):
sub_loss = torch.Tensor([0]).cuda()
else:
outside_theta = theta[is_outside]
outside_theta = outside_theta.unsqueeze(1).repeat(1, hull[:-1].shape[0], 1)
w_outside = - hull[:-1].unsqueeze(0) + outside_theta # B*(M-1)*2
t = torch.clamp(inner_product(w_outside, v) / (two_norm(v) ** 2), min=0, max=1).unsqueeze(2)
p = hull[:-1] + t * v
D = one_norm(torch.cos(outside_theta) - torch.cos(p)) + one_norm(torch.sin(outside_theta) - torch.sin(p))
sub_loss = torch.sum(torch.min(D, dim=-1)[0])
vis = 0
if vis:
print(theta)
plot_hull(theta, hull)
loss += sub_loss
loss /= (15 * thetas.shape[0])
return loss
def angle_between(v1, v2):
epsilon = 1e-7
cos = torch_f.cosine_similarity(v1, v2, dim=-1).clamp(-1 + epsilon, 1 - epsilon) # (B)
theta = torch.acos(cos) # (B)
return theta
def normalize(vec):
return torch_f.normalize(vec, p=2, dim=-1)
def inner_product(x1, x2):
return torch.sum(x1 * x2, dim=-1)
def cross_product(x1, x2):
return torch.cross(x1, x2, dim=-1)
def axangle2mat_torch(axis, angle, is_normalized=False):
""" Rotation matrix for rotation angle `angle` around `axis`
Parameters
----------
axis : [B,M, 3] element sequence
vector specifying axis for rotation.
angle :[B,M, ] scalar
angle of rotation in radians.
is_normalized : bool, optional
True if `axis` is already normalized (has norm of 1). Default False.
Returns
-------
mat : array shape (B, M,3,3)
rotation matrix for specified rotation
Notes
-----
From: http://en.wikipedia.org/wiki/Rotation_matrix#Axis_and_angle
"""
B = axis.shape[0]
M = axis.shape[1]
if not is_normalized:
norm_axis = axis.norm(p=2, dim=-1, keepdim=True)
normed_axis = axis / norm_axis
else:
normed_axis = axis
x, y, z = normed_axis[:, :, 0], normed_axis[:, :, 1], normed_axis[:, :, 2]
c = torch.cos(angle)
s = torch.sin(angle)
C = 1 - c
xs = x * s;
ys = y * s;
zs = z * s # noqa
xC = x * C;
yC = y * C;
zC = z * C # noqa
xyC = x * yC;
yzC = y * zC;
zxC = z * xC # noqa
TMP = torch.stack([x * xC + c, xyC - zs, zxC + ys, xyC + zs, y * yC + c, yzC - xs, zxC - ys, yzC + xs, z * zC + c],
dim=-1)
return TMP.reshape(B, M, 3, 3)
def interval_loss(value, min, max):
'''
calculate interval loss
Args:
value: B*M
max: M
min: M
Returns:
'''
batch_3d_size = value.shape[0]
min = min.repeat(value.shape[0], 1)
max = max.repeat(value.shape[0], 1)
loss1 = torch.max(min - value, torch.Tensor([0]).cuda())
loss2 = torch.max(value - max, torch.Tensor([0]).cuda())
loss = (loss1 + loss2).sum()
loss /= (batch_3d_size * value.shape[1])
return loss
class BMCLoss:
def __init__(
self,
lambda_bl=0.,
lambda_rb=0.,
lambda_a=0.,
):
self.lambda_bl = lambda_bl
self.lambda_rb = lambda_rb
self.lambda_a = lambda_a
# self.lp = "../BMC"
self.lp = "BMC"
self.bone_len_max = np.load(os.path.join(self.lp, "bone_len_max.npy"))
self.bone_len_min = np.load(os.path.join(self.lp, "bone_len_min.npy"))
self.rb_curvatures_max = np.load(os.path.join(self.lp, "curvatures_max.npy"))
self.rb_curvatures_min = np.load(os.path.join(self.lp, "curvatures_min.npy"))
self.rb_PHI_max = np.load(os.path.join(self.lp, "PHI_max.npy"))
self.rb_PHI_min = np.load(os.path.join(self.lp, "PHI_min.npy"))
self.joint_angle_limit = np.load(os.path.join(self.lp, "CONVEX_HULLS.npy"),
allow_pickle=True)
LEN_joint_angle_limit = len(self.joint_angle_limit)
self.bl_max = torch.from_numpy(self.bone_len_max).float().cuda()
self.bl_min = torch.from_numpy(self.bone_len_min).float().cuda()
self.rb_curvatures_max = torch.from_numpy(self.rb_curvatures_max).float().cuda()
self.rb_curvatures_min = torch.from_numpy(self.rb_curvatures_min).float().cuda()
self.rb_PHI_max = torch.from_numpy(self.rb_PHI_max).float().cuda()
self.rb_PHI_min = torch.from_numpy(self.rb_PHI_min).float().cuda()
self.joint_angle_limit = [torch.from_numpy(self.joint_angle_limit[i]).float().cuda() for i in
range(LEN_joint_angle_limit)]
def compute_loss(self, joints):
'''
Args:
joints: B*21*3
Returns:
'''
batch_size = joints.shape[0]
final_loss = torch.Tensor([0]).cuda()
BMC_losses = {"bmc_bl": torch.Tensor([0]).cuda(), "bmc_rb": torch.Tensor([0]).cuda(), "bmc_a": torch.Tensor([0]).cuda()}
if (self.lambda_bl < 1e-6) and (self.lambda_rb < 1e-6) and (self.lambda_a < 1e-6):
return final_loss, BMC_losses
ALL_bones = [
(
joints[:, i, :] -
joints[:, cfg.SNAP_PARENT[i], :]
) for i in range(21)
]
ALL_bones = torch.stack(ALL_bones[1:], dim=1) # (B,20,3)
ROOT_bones = ALL_bones[:, cfg.ID_ROOT_bone] # (B,5,3)
PIP_bones = ALL_bones[:, cfg.ID_PIP_bone]
DIP_bones = ALL_bones[:, cfg.ID_DIP_bone]
TIP_bones = ALL_bones[:, cfg.ID_TIP_bone]
ALL_Z_axis = normalize(ALL_bones)
PIP_Z_axis = ALL_Z_axis[:, cfg.ID_ROOT_bone]
DIP_Z_axis = ALL_Z_axis[:, cfg.ID_PIP_bone]
TIP_Z_axis = ALL_Z_axis[:, cfg.ID_DIP_bone]
normals = normalize(cross_product(ROOT_bones[:, 1:], ROOT_bones[:, :-1]))
# compute loss of bone length
bl_loss = torch.Tensor([0]).cuda()
if self.lambda_bl:
bls = two_norm(ALL_bones) # (B,20,1)
bl_loss = interval_loss(value=bls, min=self.bl_min, max=self.bl_max)
final_loss += self.lambda_bl * bl_loss
BMC_losses["bmc_bl"] = bl_loss
# compute loss of Root bones
rb_loss = torch.Tensor([0]).cuda()
if self.lambda_rb:
edge_normals = torch.zeros_like(ROOT_bones).cuda() # (B,5,3)
edge_normals[:, [0, 4]] = normals[:, [0, 3]]
edge_normals[:, 1:4] = normalize(normals[:, 1:4] + normals[:, :3])
curvatures = inner_product(edge_normals[:, 1:] - edge_normals[:, :4],
ROOT_bones[:, 1:] - ROOT_bones[:, :4]) / \
(two_norm(ROOT_bones[:, 1:] - ROOT_bones[:, :4]) ** 2)
PHI = angle_between(ROOT_bones[:, :4], ROOT_bones[:, 1:]) # (B)
rb_loss = interval_loss(value=curvatures, min=self.rb_curvatures_min, max=self.rb_curvatures_max) + \
interval_loss(value=PHI, min=self.rb_PHI_min, max=self.rb_PHI_max)
final_loss += self.lambda_rb * rb_loss
BMC_losses["bmc_rb"] = rb_loss
# compute loss of Joint angles
a_loss = torch.Tensor([0]).cuda()
if self.lambda_a:
# PIP bones
PIP_X_axis = torch.zeros([batch_size, 5, 3]).cuda() # (B,5,3)
PIP_X_axis[:, [0, 1, 4], :] = -normals[:, [0, 1, 3]]
PIP_X_axis[:, 2:4] = -normalize(normals[:, 2:4] + normals[:, 1:3]) # (B,2,3)
PIP_Y_axis = normalize(cross_product(PIP_Z_axis, PIP_X_axis)) # (B,5,3)
PIP_bones_xz = PIP_bones - inner_product(PIP_bones, PIP_Y_axis).unsqueeze(2) * PIP_Y_axis
PIP_theta_flexion = angle_between(PIP_bones_xz, PIP_Z_axis) # in global coordinate (B)
PIP_theta_abduction = angle_between(PIP_bones_xz, PIP_bones) # in global coordinate (B)
# x-component of the bone vector
tmp = inner_product(PIP_bones, PIP_X_axis)
PIP_theta_flexion = torch.where(tmp < 1e-6, -PIP_theta_flexion, PIP_theta_flexion)
# y-component of the bone vector
tmp = torch.sum((PIP_bones * PIP_Y_axis), dim=-1)
PIP_theta_abduction = torch.where(tmp < 1e-6, -PIP_theta_abduction, PIP_theta_abduction)
temp_axis = normalize(cross_product(PIP_Z_axis, PIP_bones))
temp_alpha = angle_between(PIP_Z_axis, PIP_bones) # alpha belongs to [pi/2, pi]
temp_R = axangle2mat_torch(axis=temp_axis, angle=temp_alpha, is_normalized=True)
# DIP bones
DIP_X_axis = torch.matmul(temp_R, PIP_X_axis.unsqueeze(3)).squeeze()
DIP_Y_axis = torch.matmul(temp_R, PIP_Y_axis.unsqueeze(3)).squeeze()
DIP_bones_xz = DIP_bones - inner_product(DIP_bones, DIP_Y_axis).unsqueeze(2) * DIP_Y_axis
DIP_theta_flexion = angle_between(DIP_bones_xz, DIP_Z_axis) # in global coordinate
DIP_theta_abduction = angle_between(DIP_bones_xz, DIP_bones) # in global coordinate
# x-component of the bone vector
tmp = inner_product(DIP_bones, DIP_X_axis)
DIP_theta_flexion = torch.where(tmp < 0, -DIP_theta_flexion, DIP_theta_flexion)
# y-component of the bone vector
tmp = inner_product(DIP_bones, DIP_Y_axis)
DIP_theta_abduction = torch.where(tmp < 0, -DIP_theta_abduction, DIP_theta_abduction)
temp_axis = normalize(cross_product(DIP_Z_axis, DIP_bones))
temp_alpha = angle_between(DIP_Z_axis, DIP_bones) # alpha belongs to [pi/2, pi]
temp_R = axangle2mat_torch(axis=temp_axis, angle=temp_alpha, is_normalized=True)
# TIP bones
TIP_X_axis = torch.matmul(temp_R, DIP_X_axis.unsqueeze(3)).squeeze()
TIP_Y_axis = torch.matmul(temp_R, DIP_Y_axis.unsqueeze(3)).squeeze()
TIP_bones_xz = TIP_bones - inner_product(TIP_bones, TIP_Y_axis).unsqueeze(2) * TIP_Y_axis
TIP_theta_flexion = angle_between(TIP_bones_xz, TIP_Z_axis) # in global coordinate
TIP_theta_abduction = angle_between(TIP_bones_xz, TIP_bones) # in global coordinate
# x-component of the bone vector
tmp = inner_product(TIP_bones, TIP_X_axis)
TIP_theta_flexion = torch.where(tmp < 1e-6, -TIP_theta_flexion, TIP_theta_flexion)
# y-component of the bone vector
tmp = inner_product(TIP_bones, TIP_Y_axis)
TIP_theta_abduction = torch.where(tmp < 1e-6, -TIP_theta_abduction, TIP_theta_abduction)
# ALL
ALL_theta_flexion = torch.cat((PIP_theta_flexion, DIP_theta_flexion, TIP_theta_flexion), dim=-1)
ALL_theta_abduction = torch.cat((PIP_theta_abduction, DIP_theta_abduction, TIP_theta_abduction), dim=-1)
ALL_theta = torch.stack((ALL_theta_flexion, ALL_theta_abduction), dim=-1)
a_loss = calculate_joint_angle_loss(ALL_theta, self.joint_angle_limit)
final_loss += self.lambda_a * a_loss
BMC_losses["bmc_a"] = a_loss
return final_loss, BMC_losses
if __name__ == '__main__':
bmc = BMCLoss(lambda_bl=1, lambda_rb=1, lambda_a=1)
joints = torch.rand(10 * 63).reshape(-1, 21, 3).float().cuda() # (100,21,3)
loss_total, loss_dict = bmc.compute_loss(joints)
print("loss_total=", loss_total)
print("loss_dict=", loss_dict)