Two-stage visual speech recognition for intensive care patients

Scientific Reports, vol. 13, pp. 928

Abstract

In this work, we propose a framework to enhance the communication abilities of speech-impaired patients in an intensive care setting via reading lips. Medical procedure, such as a tracheotomy, causes the patient to lose the ability to utter speech with little to no impact on the habitual lip movement. Consequently, we developed a framework to predict the silently spoken text by performing visual speech recognition, i.e., lip-reading. In a two-stage architecture, frames of the patient's face are used to infer audio features as an intermediate prediction target, which are then used to predict the uttered text. To the best of our knowledge, this is the first approach to bring visual speech recognition into an intensive care setting. For this purpose, we recorded an audio-visual dataset in the University Hospital of Aachen's intensive care unit (ICU) with a language corpus hand-picked by experienced clinicians to be representative of their day-to-day routine. With a word error rate of 6.3%, the trained system reaches a sufficient overall performance to significantly increase the quality of communication between patient and clinician or relatives.

Authors 6

1. Hendrik Laux corresponding Aachen Department of Intensive Care and Intermediate Care

   Universitätsklinikum Aachen · RWTH Aachen University

   Affiliation as printed

   Department of Intensive Care and Intermediate Care, University Hospital RWTH Aachen, Pauwelsstreet 30, 52072, Aachen, Germany. hendrik.laux@rwth-aachen.de

   Department of Intensive Care and Intermediate Care, University Hospital RWTH Aachen, Pauwelsstreet 30, 52072, Aachen, Germany

2. Ahmed Hallawa Aachen Department of Intensive Care and Intermediate Care

   Universitätsklinikum Aachen · RWTH Aachen University

   Affiliation as printed

   Department of Intensive Care and Intermediate Care, University Hospital RWTH Aachen, Pauwelsstreet 30, 52072, Aachen, Germany

3. Julio Cesar Sevarolli Assis Aachen

   RWTH Aachen University

   Affiliation as printed

   Research Area Information Theory and Systematic Design of Communication Systems, RWTH Aachen University, Kopernikusstreet 16, 52074, Aachen, Germany

4. Anke Schmeink Aachen

   RWTH Aachen University

   Affiliation as printed

   Research Area Information Theory and Systematic Design of Communication Systems, RWTH Aachen University, Kopernikusstreet 16, 52074, Aachen, Germany

5. Lukas Märtin Aachen Department of Intensive Care and Intermediate Care

   Universitätsklinikum Aachen · RWTH Aachen University

   Affiliation as printed

   Department of Intensive Care and Intermediate Care, University Hospital RWTH Aachen, Pauwelsstreet 30, 52072, Aachen, Germany

6. Arne Peine Aachen Department of Intensive Care and Intermediate Care

   Universitätsklinikum Aachen · RWTH Aachen University

   Affiliation as printed

   Department of Intensive Care and Intermediate Care, University Hospital RWTH Aachen, Pauwelsstreet 30, 52072, Aachen, Germany