TarViS: A Unified Approach for Target-Based Video Segmentation

IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings, pp. 18738–18748

Abstract

The general domain of video segmentation is currently fragmented into different tasks spanning multiple benchmarks. Despite rapid progress in the state-of-the-art, current methods are overwhelmingly task-specific and cannot conceptually generalize to other tasks. Inspired by recent approaches with multi-task capability, we propose TarViS: a novel, unified network architecture that can be applied to any task that requires segmenting a set of arbitrarily defined ‘targets' in video. Our approach is flexible with respect to how tasks define these targets, since it models the latter as abstract ‘queries' which are then used to predict pixel-precise target masks. A single TarViS model can be trained jointly on a collection of datasets spanning different tasks, and can hot-swap between tasks during inference without any task-specific retraining. To demonstrate its effectiveness, we apply TarViS to four different tasks, namely Video Instance Segmentation (VIS), Video Panoptic Segmentation (VPS), Video Object Segmentation (VOS) and Point Exemplar-guided Tracking (PET). Our unified, jointly trained model achieves state-of-the-art performance on 5/7 benchmarks spanning these four tasks, and competitive performance on the remaining two. Code and model weights are available at: https://github.com/Ali2500/TarViS

Authors 5

1. Ali Athar Aachen

   RWTH Aachen University

   Affiliation as printed

   RWTH Aachen University,Germany

   RWTH Aachen University, Germany

2. Alexander Hermans Aachen

   RWTH Aachen University

   Affiliation as printed

   RWTH Aachen University,Germany

   RWTH Aachen University, Germany

3. Jonathon Luiten Aachen

   Carnegie Mellon University · RWTH Aachen University

   Affiliation as printed

   RWTH Aachen University,Germany

   Carnegie Mellon University, USA

   RWTH Aachen University, Germany

4. Deva Ramanan

   Carnegie Mellon University

   Affiliation as printed

   Carnegie Mellon University,USA

   Carnegie Mellon University, USA

5. Bastian Leibe Aachen

   RWTH Aachen University

   Affiliation as printed

   RWTH Aachen University,Germany

   RWTH Aachen University, Germany