Lightning Pose: improved animal pose estimation via semi-supervised learning, Bayesian ensembling, and cloud-native open-source tools

bioRxiv (Cold Spring Harbor Laboratory)

Abstract

Contemporary pose estimation methods enable precise measurements of behavior via supervised deep learning with hand-labeled video frames. Although effective in many cases, the supervised approach requires extensive labeling and often produces outputs that are unreliable for downstream analyses. Here, we introduce "Lightning Pose," an efficient pose estimation package with three algorithmic contributions. First, in addition to training on a few labeled video frames, we use many unlabeled videos and penalize the network whenever its predictions violate motion continuity, multiple-view geometry, and posture plausibility (semi-supervised learning). Second, we introduce a network architecture that resolves occlusions by predicting pose on any given frame using surrounding unlabeled frames. Third, we refine the pose predictions post-hoc by combining ensembling and Kalman smoothing. Together, these components render pose trajectories more accurate and scientifically usable. We release a cloud application that allows users to label data, train networks, and predict new videos directly from the browser.

Authors 19

1. Dan Biderman corresponding

   Columbia University

   Affiliation as printed

   Columbia University, New York, USA

2. Matthew R Whiteway corresponding

   Columbia University

   Affiliation as printed

   Columbia University, New York, USA

3. Cole Hurwitz

   Columbia University

   Affiliation as printed

   Columbia University, New York, USA

4. Nicholas Greenspan

   Columbia University

   Affiliation as printed

   Columbia University, New York, USA

5. Ankit Vishnubhotla

   Columbia University

   Affiliation as printed

   Columbia University, New York, USA

6. Richard A. Warren

   Columbia University

   Affiliation as printed

   Columbia University, New York, USA

7. Federico Pedraja

   Columbia University

   Affiliation as printed

   Columbia University, New York, USA

8. Dillon Noone

   Columbia University

   Affiliation as printed

   Columbia University, New York, USA

9. Michael M. Schartner

   Champalimaud Foundation

   Affiliation as printed

   Champalimaud Centre for the Unknown, Lisbon, Portugal

10. Julia M. Huntenburg

    Max Planck Institute for Biological Cybernetics

    Affiliation as printed

    Max Planck Institute for Biological Cybernetics, Tübingen, Germany

11. Anup Khanal

    University of California, Los Angeles

    Affiliation as printed

    University of California Los Angeles, Los Angeles, USA

12. Guido T. Meijer

    Champalimaud Foundation

    Affiliation as printed

    Champalimaud Centre for the Unknown, Lisbon, Portugal

13. Jean‐Paul Noel

    New York University

    Affiliation as printed

    New York University, New York, USA

14. Alejandro Pan-Vazquez

    Princeton University

    Affiliation as printed

    Princeton University, Princeton, USA

15. Karolina Z. Socha

    University College London

    Affiliation as printed

    University College London, London, United Kingdom

16. Anne E Urai Aachen

    Leiden University

    Affiliation as printed

    Leiden University, Leiden, The Netherlands

17. John P. Cunningham

    Columbia University

    Affiliation as printed

    Columbia University, New York, USA

18. Nathaniel B. Sawtell

    Columbia University

    Affiliation as printed

    Columbia University, New York, USA

19. Liam Paninski

    Columbia University

    Affiliation as printed

    Columbia University, New York, USA