SEVERE++: Evaluating Benchmark Sensitivity in Generalization of Video Representation Learning

arXiv (Cornell University)

Abstract

Continued advances in self-supervised learning have led to significant progress in video representation learning, offering a scalable alternative to supervised approaches by removing the need for manual annotations. Despite strong performance on standard action recognition benchmarks, video self-supervised learning methods are largely evaluated under narrow protocols, typically pretraining on Kinetics-400 and fine-tuning on similar datasets, limiting our understanding of their generalization in real world scenarios. In this work, we present a comprehensive evaluation of modern video self-supervised models, focusing on generalization across four key downstream factors: domain shift, sample efficiency, action granularity, and task diversity. Building on our prior work analyzing benchmark sensitivity in CNN-based contrastive learning, we extend the study to cover state-of-the-art transformer-based video-only and video-text models. Specifically, we benchmark 12 transformer-based methods (7 video-only, 5 video-text) and compare them to 10 CNN-based methods, totaling over 1100 experiments across 8 datasets and 7 downstream tasks. Our analysis shows that, despite architectural advances, transformer-based models remain sensitive to downstream conditions. No method generalizes consistently across all factors, video-only transformers perform better under domain shifts, CNNs outperform for fine-grained tasks, and video-text models often underperform despite large scale pretraining. We also find that recent transformer models do not consistently outperform earlier approaches. Our findings provide a detailed view of the strengths and limitations of current video SSL methods and offer a unified benchmark for evaluating generalization in video representation learning.

Authors 7

1. Fida Mohammad Thoker

   King Abdullah University of Science and Technology

   Affiliation as printed

   CEMSE , King Abdullah University of Science and Technology , Makkah , KSA

2. Letian Jiang

   King Abdullah University of Science and Technology

   Affiliation as printed

   CEMSE , King Abdullah University of Science and Technology , Makkah , KSA

3. Chen Zhao

   King Abdullah University of Science and Technology

   Affiliation as printed

   CEMSE , King Abdullah University of Science and Technology , Makkah , KSA

4. Piyush Bagad

   University of Oxford

   Affiliation as printed

   VGG , University of Oxford , Oxford , England

5. Hazel Doughty Aachen

   Leiden University

   Affiliation as printed

   LIACS , Leiden University , Leiden , Netherlands

6. Bernard Ghanem

   King Abdullah University of Science and Technology

   Affiliation as printed

   CEMSE , King Abdullah University of Science and Technology , Makkah , KSA

7. Cees G. M. Snoek

   University of Amsterdam

   Affiliation as printed

   Video & Image Sense Lab , University of Amsterdam , Amsterdam , Netherlands