Skip to content

Latest commit

 

History

History
107 lines (72 loc) · 4.48 KB

README.md

File metadata and controls

107 lines (72 loc) · 4.48 KB

🚀 SynthPose Update!

SynthPose models are now available on Hugging Face Transformers 🤗, and you can test it in a few clicks in a dedicated Hugging Face Space!

OpenCapBench

OpenCapBench is a benchmark designed to bridge the gap between pose estimation and biomechanics. It evaluates pose estimation models under physiological constraints using consistent kinematic metrics computed via OpenSim.

Pipeline Overview

Features

  • Unified evaluation benchmark for biomechanics and pose estimation.
  • Integration with OpenSim for joint angle computations.
  • Fine-tuning models with SynthPose, enabling dense keypoint predictions for accurate kinematic analysis.
  • Tools to benchmark custom models on clinically relevant datasets.

Installation

Usage

Replace the example values with your values and run the following.

python benchmarking/benchmark.py \
    --model_config_pose "mmpose_dir"/configs/body_2d_keypoint/topdown_heatmap/.../your_mmpose_model_config.py \
    --model_ckpt_pose  "your_mmpose_weights" \
    --dataDir "your_dataDir" \
    --dataName "hrnet48_final"

SynthPose: Fine-tuning Pose Models

SynthPose fine-tunes pre-trained pose estimation models using synthetic datasets to predict arbitrarily dense sets of keypoints.

SynthPose Pipeline

Installation

  • Clone this repository.
  • Download the synthetic data that you want to use to finetune your model (e.g BEDLAM, VisionFit etc.).
  • Download the SMPL-X model and place in in a folder named models at the root of this repository.
  • Install pytorch, smplx, pycocotools python packages.

Usage

  • Customize the SMPL/X vertices you want to finetune your model on in synthpose/resource/vertices_keypoints_corr.csv.
  • Use the "generate_dataset.py" scripts in synthpose/"dataset_name" to generate the 2D keypoints annotations corresponding to the vertices chosen in the previous step.
  • Finetune a pose estimation model on this generated dataset. For an example of how to do this, you can take a look at my fork of mmpose where I created an "Infinity" dataset where the keypoints correspond to anatomical markers. The mmpose documentation is great to learn how to finetune a 2D pose estimation models.

Key Results

Models finetuned with SynthPose to predict anatomical/MoCap markers demonstrate superior performance for kinematics compare to models predicting COCO keypoints or SMPL mesh.
Here are the results on OpenCapBench:

Results Comparison

Visual Results

Examples of marker predictions with a SynthPose model predicting anatomical/MoCap markers on OpenCap dataset subjects:

SynthPose Visualizations
SynthPose Visualizations

Citation

If you find OpenCapBench useful in your research, please cite:

@misc{gozlan2024opencapbenchbenchmarkbridgepose,
      title={OpenCapBench: A Benchmark to Bridge Pose Estimation and Biomechanics}, 
      author={Yoni Gozlan and Antoine Falisse and Scott Uhlrich and Anthony Gatti and Michael Black and Akshay Chaudhari},
      year={2024},
      eprint={2406.09788},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2406.09788}, 
}

Links

License

This project is licensed under the MIT License.