Adaptive Multimodal Emotion Detection Architecture for Social Robots

IEEE Access, vol. 10, pp. 20727–20744

Abstract

Emotion recognition is a strategy for social robots used to implement better Human-Robot Interaction and model their social behaviour. Since human emotions can be expressed in different ways (e.g., face, gesture, voice), multimodal approaches are useful to support the recognition process. However, although there exist studies dealing with multimodal emotion recognition for social robots, they still present limitations in the fusion process, dropping their performance if one or more modalities are not present or if modalities have different qualities. This is a common situation in social robotics, due to the high variety of the sensory capacities of robots; hence, more flexible multimodal models are needed. In this context, we propose an adaptive and flexible emotion recognition architecture able to work with multiple sources and modalities of information and manage different levels of data quality and missing data, to lead robots to better understand the mood of people in a given environment and accordingly adapt their behaviour. Each modality is analyzed independently to then aggregate the partial results with a previous proposed fusion method, called EmbraceNet+, which is adapted and integrated to our proposed framework. We also present an extensive review of state-of-the-art studies dealing with fusion methods for multimodal emotion recognition approaches. We evaluate the performance of our proposed architecture by performing different tests in which several modalities are combined to classify emotions using four categories (i.e., happiness, neutral, sadness, and anger). Results reveal that our approach is able to adapt to the quality and presence of modalities. Furthermore, results obtained are validated and compared with other similar proposals, obtaining competitive performance with state-of-the-art models.

Authors 7

1. Juanpablo Heredia

   Universidad Católica San Pablo

   Affiliation as printed

   Computer Science Department, Universidad Católica San Pablo, Arequipa, Peru

2. Edmundo Lopes-Silva

   Instituto Federal da Bahia

   Affiliation as printed

   Department of Electrical Engineering, Instituto Federal da Bahia, Vitoria da Conquista, Brazil

3. Yudith Cardinale

   Universidad Católica San Pablo · Simón Bolívar University

   Affiliation as printed

   Departamento de Computación y T.I, Universidad Simón Bolívar, Caracas, Venezuela

   Electrical and Electronics Engineering Department, Universidad Católica San Pablo, Arequipa, Peru

4. José Díaz-Amado

   Instituto Federal da Bahia · Universidad Católica San Pablo

   Affiliation as printed

   Department of Electrical Engineering, Instituto Federal da Bahia, Vitoria da Conquista, Brazil

   Electrical and Electronics Engineering Department, Universidad Católica San Pablo, Arequipa, Peru

5. Irvin Dongo

   Université de Bordeaux · École Supérieure des Technologies Industrielles Avancées · Universidad Católica San Pablo

   Affiliation as printed

   Electrical and Electronics Engineering Department, Universidad Católica San Pablo, Arequipa, Peru

   University of Bordeaux, Estia Institute of Technology, Bordeaux, Bidart, France

6. Wilfredo Graterol

   Simón Bolívar University

   Affiliation as printed

   Departamento de Computación y T.I, Universidad Simón Bolívar, Caracas, Venezuela

7. Ana Aguilera Aachen

   Inform (Germany) · University of Valparaíso

   Affiliation as printed

   Escuela de Ingeniería Informática, Facultad de Ingeniería, Universidad de Valparaíso, Valparaíso, Chile