STEP: Segmenting and Tracking Every Pixel

arXiv (Cornell University)

Abstract

The task of assigning semantic classes and track identities to every pixel in a video is called video panoptic segmentation. Our work is the first that targets this task in a real-world setting requiring dense interpretation in both spatial and temporal domains. As the ground-truth for this task is difficult and expensive to obtain, existing datasets are either constructed synthetically or only sparsely annotated within short video clips. To overcome this, we introduce a new benchmark encompassing two datasets, KITTI-STEP, and MOTChallenge-STEP. The datasets contain long video sequences, providing challenging examples and a test-bed for studying long-term pixel-precise segmentation and tracking under real-world conditions. We further propose a novel evaluation metric Segmentation and Tracking Quality (STQ) that fairly balances semantic and tracking aspects of this task and is more appropriate for evaluating sequences of arbitrary length. Finally, we provide several baselines to evaluate the status of existing methods on this new challenging dataset. We have made our datasets, metric, benchmark servers, and baselines publicly available, and hope this will inspire future research.

Authors 12

1. Jun Xie

   Google (United States) · University of Washington

   Affiliation as printed

   Google Research ,

   University of Washington, Seattle, United States

2. Maxwell D. Collins

   Google (United States)

   Affiliation as printed

   Google Research ,

   Google (United States), Mountain View, United States

3. Yukun Zhu

   Google (United States)

   Affiliation as printed

   Google Research ,

   Google (United States), Mountain View, United States

4. Paul Voigtlaender Aachen

   Google (United States) · RWTH Aachen University

   Affiliation as printed

   Google Research ,

   RWTH Aachen University ,

   RWTH Aachen University, Aachen, Germany

5. Hartwig Adam

   Google (United States)

   Affiliation as printed

   Google Research ,

   Google (United States), Mountain View, United States

6. Bradley Green

   Google (United States)

   Affiliation as printed

   Google Research ,

   Google (United States), Mountain View, United States

7. Andreas Geiger

   Max Planck Society · University of Tübingen

   Affiliation as printed

   MPI-IS and University of Tübingen

   Max Planck Society, Munich, Germany

8. Bastian Leibe Aachen

   RWTH Aachen University

   Affiliation as printed

   RWTH Aachen University ,

   RWTH Aachen University, Aachen, Germany

9. Daniel Cremers

   Technical University of Munich

   Affiliation as printed

   Technical University Munich ,

   Technical University of Munich, Munich, Germany

10. Aljoša Ošep

    Technical University of Munich

    Affiliation as printed

    Technical University Munich ,

    Technical University of Munich, Munich, Germany

11. Laura Leal-Taixé

    Technical University of Munich

    Affiliation as printed

    Technical University Munich ,

    Technical University of Munich, Munich, Germany

12. Liang-Chieh Chen

    Google (United States)

    Affiliation as printed

    Google Research ,

    Google (United States), Mountain View, United States