AVT-ECoClass-VR: An open-source audiovisual 360° video and immersive CGI multi-talker dataset to evaluate cognitive performance

International Conference on Quality of Multimedia Experience (QoMEX), pp. 207–213

Abstract

The paper is part of a project to assess how complex visual and acoustic scenes affect cognitive performance in classroom scenarios, across age groups from children to adults. Here, the potential of audiovisual virtual environments for systematic user studies is explored. As of now, most studies have examined rather simple acoustic and visual representations, which do not reflect the reality of school children. An adapted version of the audiovisual scene analysis paradigm is presented, focusing on the localization and identification of talkers within a scene. The dataset includes two audiovisual scenarios (360° video and computer-generated imagery) and two implementations for dataset playback. The paper details the recording and post-processing of the content. The 360° video part of the dataset features 200 video and single-channel audio recordings of 20 speakers reading ten stories, and 20 videos of speakers in silence, resulting in a total of 220 video and 200 audio recordings. The dataset also includes one 360° background image of a real primary school classroom scene, targeting young school children for subsequent subjective tests. All stories were recorded in the German language with native German speakers. The second part of the dataset comprises 20 different 3D models of the speakers and a computer-generated classroom scene, along with an immersive audiovisual virtual environment implementation that can be interacted with using an HTC Vive controller. Both implementations also include a Unity plugin to connect and interact with the Virtual Acoustics auralization software. As a proof of concept, the dataset includes example output data collected from ongoing perception tests. There, subjects have the task of identifying which talker in the scene is reading out which story, using the story-to-speaker mapping input system developed within this paper.

Authors 6

1. Stephan Fremerey

   Technische Universität Ilmenau

   Affiliation as printed

   Technische Universität,Audiovisual Technology Group,Ilmenau,Germany

2. Carolin Breuer Aachen Institute for Hearing Technology and Acoustics

   RWTH Aachen University

   Affiliation as printed

   RWTH Aachen University,Institute for Hearing Technology and Acoustics,Germany

3. Larissa Leist

   University of Kaiserslautern · Rheinland-Pfälzische Technische Universität Kaiserslautern-Landau

   Affiliation as printed

   RPTU Kaiserslautern,Center for Cognitive Science,Landau,Germany

4. Maria Klatte

   University of Kaiserslautern · Rheinland-Pfälzische Technische Universität Kaiserslautern-Landau

   Affiliation as printed

   RPTU Kaiserslautern,Center for Cognitive Science,Landau,Germany

5. Janina Fels Aachen Institute for Hearing Technology and Acoustics

   RWTH Aachen University

   Affiliation as printed

   RWTH Aachen University,Institute for Hearing Technology and Acoustics,Germany

6. Alexander Raake

   Technische Universität Ilmenau

   Affiliation as printed

   Technische Universität,Audiovisual Technology Group,Ilmenau,Germany