Partitioning variability in animal behavioral videos using semi-supervised variational autoencoders
PLoS Computational Biology, vol. 17, pp. e1009439
Abstract
Recent neuroscience studies demonstrate that a deeper understanding of brain function requires a deeper understanding of behavior. Detailed behavioral measurements are now often collected using video cameras, resulting in an increased need for computer vision algorithms that extract useful information from video data. Here we introduce a new video analysis tool that combines the output of supervised pose estimation algorithms (e.g. DeepLabCut) with unsupervised dimensionality reduction methods to produce interpretable, low-dimensional representations of behavioral videos that extract more information than pose estimates alone. We demonstrate this tool by extracting interpretable behavioral features from videos of three different head-fixed mouse preparations, as well as a freely moving mouse in an open field arena, and show how these interpretable features can facilitate downstream behavioral and neural analyses. We also show how the behavioral features produced by our model improve the precision and interpretation of these downstream analyses compared to using the outputs of either fully supervised or fully unsupervised methods alone.
Authors 17
-
Matthew R Whiteway corresponding
Mortimer B. Zuckerman Mind Brain Behavior Institute · Columbia University
Affiliation as printed
Center for Theoretical Neuroscience, Columbia University, New York, New York, United States of America
Department of Neuroscience, Columbia University, New York, New York, United States of America
Department of Statistics, Columbia University, New York, New York, United States of America
Grossman Center for the Statistics of Mind, Columbia University, New York, New York, United States of America
Mortimer B. Zuckerman Mind Brain Behavior Institute, Columbia University, New York, New York, United States of America
-
Mortimer B. Zuckerman Mind Brain Behavior Institute · Columbia University
Affiliation as printed
Center for Theoretical Neuroscience, Columbia University, New York, New York, United States of America
Department of Neuroscience, Columbia University, New York, New York, United States of America
Department of Statistics, Columbia University, New York, New York, United States of America
Grossman Center for the Statistics of Mind, Columbia University, New York, New York, United States of America
Mortimer B. Zuckerman Mind Brain Behavior Institute, Columbia University, New York, New York, United States of America
-
Massachusetts Institute of Technology · Columbia University
Affiliation as printed
Center for Theoretical Neuroscience, Columbia University, New York, New York, United States of America
Department of Brain and Cognitive Sciences, Massachusetts Institute of Technology, Boston, Massachusetts, United States of America
-
Mortimer B. Zuckerman Mind Brain Behavior Institute · Columbia University
Affiliation as printed
Center for Theoretical Neuroscience, Columbia University, New York, New York, United States of America
Mortimer B. Zuckerman Mind Brain Behavior Institute, Columbia University, New York, New York, United States of America
-
Mortimer B. Zuckerman Mind Brain Behavior Institute · Columbia University
Affiliation as printed
Center for Theoretical Neuroscience, Columbia University, New York, New York, United States of America
Department of Neuroscience, Columbia University, New York, New York, United States of America
Department of Statistics, Columbia University, New York, New York, United States of America
Grossman Center for the Statistics of Mind, Columbia University, New York, New York, United States of America
Mortimer B. Zuckerman Mind Brain Behavior Institute, Columbia University, New York, New York, United States of America
-
Mortimer B. Zuckerman Mind Brain Behavior Institute · Columbia University
Affiliation as printed
Center for Theoretical Neuroscience, Columbia University, New York, New York, United States of America
Department of Neuroscience, Columbia University, New York, New York, United States of America
Department of Statistics, Columbia University, New York, New York, United States of America
Grossman Center for the Statistics of Mind, Columbia University, New York, New York, United States of America
Mortimer B. Zuckerman Mind Brain Behavior Institute, Columbia University, New York, New York, United States of America
-
Affiliation as printed
Department of Computer Science, Columbia University, New York, New York, United States of America
-
Affiliation as printed
Champalimaud Centre for the Unknown, Lisbon, Portugal
-
Sainsbury Wellcome Centre · University College London
Affiliation as printed
Sainsbury-Wellcome Centre for Neural Circuits and Behavior, University College London, London, United Kingdom
-
Affiliation as printed
Center for Neural Science, New York University, New York, New York, United States of America
-
Mortimer B. Zuckerman Mind Brain Behavior Institute · Columbia University
Affiliation as printed
Department of Neuroscience, Columbia University, New York, New York, United States of America
Mortimer B. Zuckerman Mind Brain Behavior Institute, Columbia University, New York, New York, United States of America
-
Affiliation as printed
Champalimaud Centre for the Unknown, Lisbon, Portugal
-
Affiliation as printed
Institute of Ophthalmology, University College London, London, United Kingdom
-
Affiliation as printed
Cognitive Psychology Unit, Leiden University, Leiden, The Netherlands
-
New York State Psychiatric Institute · Mortimer B. Zuckerman Mind Brain Behavior Institute · Columbia University
Affiliation as printed
Department of Neuroscience, Columbia University, New York, New York, United States of America
Department of Psychiatry, Columbia University, New York, New York, United States of America
Kavli Institute for Brain Sciences, New York, New York, United States of America
Mortimer B. Zuckerman Mind Brain Behavior Institute, Columbia University, New York, New York, United States of America
New York State Psychiatric Institute, New York, New York, United States of America
-
Mortimer B. Zuckerman Mind Brain Behavior Institute · Columbia University
Affiliation as printed
Center for Theoretical Neuroscience, Columbia University, New York, New York, United States of America
Department of Statistics, Columbia University, New York, New York, United States of America
Grossman Center for the Statistics of Mind, Columbia University, New York, New York, United States of America
Mortimer B. Zuckerman Mind Brain Behavior Institute, Columbia University, New York, New York, United States of America
-
Mortimer B. Zuckerman Mind Brain Behavior Institute · Columbia University
Affiliation as printed
Center for Theoretical Neuroscience, Columbia University, New York, New York, United States of America
Department of Neuroscience, Columbia University, New York, New York, United States of America
Department of Statistics, Columbia University, New York, New York, United States of America
Grossman Center for the Statistics of Mind, Columbia University, New York, New York, United States of America
Mortimer B. Zuckerman Mind Brain Behavior Institute, Columbia University, New York, New York, United States of America
Cited by 35 stored of 34
No patents citing this paper on Lens.org (checked 2026-10-11).