LiteVSR: Efficient Visual Speech Recognition by Learning from Speech Representations of Unlabeled Data

arXiv (Cornell University)

Abstract

This paper proposes a novel, resource-efficient approach to Visual Speech Recognition (VSR) leveraging speech representations produced by any trained Automatic Speech Recognition (ASR) model. Moving away from the resource-intensive trends prevalent in recent literature, our method distills knowledge from a trained Conformer-based ASR model, achieving competitive performance on standard VSR benchmarks with significantly less resource utilization. Using unlabeled audio-visual data only, our baseline model achieves a word error rate (WER) of 47.4% and 54.7% on the LRS2 and LRS3 test benchmarks, respectively. After fine-tuning the model with limited labeled data, the word error rate reduces to 35% (LRS2) and 45.7% (LRS3). Our model can be trained on a single consumer-grade GPU within a few days and is capable of performing real-time end-to-end VSR on dated hardware, suggesting a path towards more accessible and resource-efficient VSR methodologies.

Authors 6

1. Hendrik Laux Aachen

   Universitätsklinikum Aachen · RWTH Aachen University

   Affiliation as printed

   University Hospital RWTH Aachen , Germany

2. Emil Mededovic Aachen

   Universitätsklinikum Aachen · RWTH Aachen University

   Affiliation as printed

   University Hospital RWTH Aachen , Germany

3. Ahmed Hallawa
   Affiliation as printed

   Clinomic Medical GmbH , Aachen , Germany

4. Lukas Märtin
   Affiliation as printed

   Clinomic Medical GmbH , Aachen , Germany

5. Arne Peine
   Affiliation as printed

   Clinomic Medical GmbH , Aachen , Germany

6. Anke Schmeink Aachen Chair of Information Theory and Data Analytics

   RWTH Aachen University

   Affiliation as printed

   Chair of Information Theory and Data Analytics , RWTH Aachen University , Germany