Edge Weight Prediction For Category-Agnostic Pose Estimation

Or Hirschorn, Shai Avidan
Tel Aviv University
arXiv Code HuggingFace Demo
EdgeCape teaser.


Given only one example image and skeleton,
our method refines the skeleton to enhance pose estimation on unseen categories.

Abstract

Category-Agnostic Pose Estimation (CAPE) localizes keypoints across diverse object categories with a single model, using one or a few annotated support images. Recent works have shown that using a pose graph (i.e., treating keypoints as nodes in a graph rather than isolated points) helps handle occlusions and break symmetry. However, these methods assume a static pose graph with equal-weight edges, leading to suboptimal results.
We introduce EdgeCape, a novel framework that overcomes these limitations by predicting the graph's edge weights which optimizes localization. To further leverage structural priors, we propose integrating Markovian Structural Bias, which modulates the self-attention interaction between nodes based on the number of hops between them. We show that this improves the model’s ability to capture global spatial dependencies. Evaluated on the MP-100 benchmark, which includes 100 categories and over 20K images, EdgeCape achieves state-of-the-art results in the 1-shot setting and leads among similar-sized methods in the 5-shot setting, significantly improving keypoint localization accuracy.

Results

Qualitative Results

Using our method, given a support image and skeleton we can refine the structure for better pose estimation on images from unseen categories.

Predicted Graphs

Our model predicts the best weighted graph structure for localization.
Edge weights are shown in the graph edges, with thicker edges indicating higher weights.

BibTeX

If you find this research useful, please cite the following:


@misc{hirschorn2024edgeweightpredictioncategoryagnostic,
      title={Edge Weight Prediction For Category-Agnostic Pose Estimation}, 
      author={Or Hirschorn and Shai Avidan},
      year={2024},
      eprint={2411.16665},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2411.16665}, 
}