Benchmarks and Challenges in Pose Estimation for Egocentric Hand Interactions with Objects

arXiv (Cornell University)

Abstract

We interact with the world with our hands and see it through our own (egocentric) perspective. A holistic 3Dunderstanding of such interactions from egocentric views is important for tasks in robotics, AR/VR, action recognition and motion generation. Accurately reconstructing such interactions in 3D is challenging due to heavy occlusion, viewpoint bias, camera distortion, and motion blur from the head movement. To this end, we designed the HANDS23 challenge based on the AssemblyHands and ARCTIC datasets with carefully designed training and testing splits. Based on the results of the top submitted methods and more recent baselines on the leaderboards, we perform a thorough analysis on 3D hand(-object) reconstruction tasks. Our analysis demonstrates the effectiveness of addressing distortion specific to egocentric cameras, adopting high-capacity transformers to learn complex hand-object interactions, and fusing predictions from different views. Our study further reveals challenging scenarios intractable with state-of-the-art methods, such as fast hand motion, object reconstruction from narrow egocentric views, and close contact between two hands and objects. Our efforts will enrich the community's knowledge foundation and facilitate future hand studies on egocentric hand-object interactions.

Authors 22

1. Zicong Fan
2. Takehiko Ohkawa

   The University of Tokyo

   Affiliation as printed

   Equal

   The University of Tokyo ,

3. Linlin Yang
   Affiliation as printed

   Communication Univer- sity of China ,

   Equal

4. Lin Nie

   The University of Tokyo

   Affiliation as printed

   The University of Tokyo ,

5. Zhishan Zhou
   Affiliation as printed

   Jiiov ,

6. Shihao Zhou
   Affiliation as printed

   Jiiov ,

7. Jiajun Liang
   Affiliation as printed

   Jiiov ,

8. Zhong Gao
   Affiliation as printed

   Bytedance ,

9. Xuanyang Zhang

   Fujitsu (Japan)

   Affiliation as printed

   Bytedance ,

   Fujitsu Research & Development Center Co., Ltd. ,

10. Fei Li

    Fujitsu (Japan)

    Affiliation as printed

    Fujitsu Research & Development Center Co., Ltd. ,

11. Zheng Liu

    Beihang University

    Affiliation as printed

    Beihang University ,

12. Feng Lu

    Beihang University

    Affiliation as printed

    Beihang University ,

13. Bastian Leibe Aachen

    RWTH Aachen University

    Affiliation as printed

    RWTH Aachen University , 10 UNIST , 11

14. Jeongwan On
    Affiliation as printed

    UIUC , 12 Meta ,

15. Seungryul Baek
    Affiliation as printed

    UIUC , 12 Meta ,

16. Aditya Prakash
    Affiliation as printed

    UIUC , 12 Meta ,

17. Saurabh Gupta
    Affiliation as printed

    UIUC , 12 Meta ,

18. Kun He
    Affiliation as printed

    UIUC , 12 Meta ,

19. Yoichi Sato

    The University of Tokyo

    Affiliation as printed

    The University of Tokyo ,

20. Otmar Hilliges

    ETH Zurich

    Affiliation as printed

    ETH Zürich ,

21. Hyung Jin Chang

    University of Birmingham

    Affiliation as printed

    University of Birmingham ,

22. Angela Yao

    National University of Singapore

    Affiliation as printed

    National University of Singapore