LITEVSR: Efficient Visual Speech Recognition by Learning from Speech Representations of Unlabeled Data

IEEE International Conference on Acoustics Speech and Signal Processing, pp. 10391–10395

Abstract

This paper proposes a novel, resource-efficient approach to Visual Speech Recognition (VSR) leveraging speech representations produced by any trained Automatic Speech Recognition (ASR) model. Moving away from the resource-intensive trends prevalent in recent literature, our method distills knowledge from a trained Conformer-based ASR model, achieving competitive performance on standard VSR benchmarks with significantly less resource utilization. Using unlabeled audio-visual data only, our baseline model achieves a word error rate (WER) of 47.4% and 54.7% on the LRS2 and LRS3 test benchmarks, respectively. After fine-tuning the model with limited labeled data, the word error rate reduces to 35% (LRS2) and 45.7% (LRS3). Our model can be trained on a single consumer-grade GPU within a few days and is capable of performing real-time end-to-end VSR on dated hardware, suggesting a path towards more accessible and resource-efficient VSR methodologies.

Authors 6

1. Hendrik Laux Aachen

   Universitätsklinikum Aachen · RWTH Aachen University

   Affiliation as printed

   University Hospital RWTH Aachen,Germany

   University Hospital RWTH Aachen, Germany

2. Emil Mededovic Aachen

   Universitätsklinikum Aachen · RWTH Aachen University

   Affiliation as printed

   University Hospital RWTH Aachen,Germany

   University Hospital RWTH Aachen, Germany

3. Ahmed Hallawa
   Affiliation as printed

   Clinomic Medical GmbH,Aachen,Germany

   Clinomic Medical GmbH, Aachen, Germany

4. Lukas Märtin
   Affiliation as printed

   Clinomic Medical GmbH,Aachen,Germany

   Clinomic Medical GmbH, Aachen, Germany

5. Arne Peine
   Affiliation as printed

   Clinomic Medical GmbH,Aachen,Germany

   Clinomic Medical GmbH, Aachen, Germany

6. Anke Schmeink Aachen Chair of Information Theory and Data Analytics

   RWTH Aachen University

   Affiliation as printed

   RWTH Aachen University,Chair of Information Theory and Data Analytics,Germany

   Chair of Information Theory and Data Analytics, RWTH Aachen University, Germany