The Embodied Crossmodal Self Forms Language and Interaction: A Computational Cognitive Review

Abstract

Human language is inherently embodied and grounded in sensorimotor representations of the self and the world around it. This suggests that the body schema and ideomotor action-effect associations play an important role in language understanding, language generation, and verbal/physical interaction with others. There are computational models that focus purely on non-verbal interaction between humans and robots, and there are computational models for dialog systems that focus only on verbal interaction. However, there is a lack of research that integrates these approaches. We hypothesize that the development of computational models of the self is very appropriate for considering joint verbal and physical interaction. Therefore, they provide the substantial potential to foster the psychological and cognitive understanding of language grounding, and they have significant potential to improve human-robot interaction methods and applications. This review is a first step toward developing models of the self that integrate verbal and non-verbal communication. To this end, we first analyze the relevant findings and mechanisms for language grounding in the psychological and cognitive literature on ideomotor theory. Second, we identify the existing computational methods that implement physical decision-making and verbal interaction. As a result, we outline how the current computational methods can be used to create advanced computational interaction models that integrate language grounding with body schemas and self-representations.

Authors 5

1. Frank Röder corresponding Aachen

   Leiden University · Universität Hamburg · University of Calabria · Carnegie Mellon University

   Affiliation as printed

   Carnegie Mellon University , United States

   Carnegie Mellon University, United States

   Knowledge Technology, Department of Informatics, University of Hamburg, Hamburg, Germany

   Leiden University, Netherlands

   University of Calabria, Italy

2. Ozan Özdemir Aachen

   Leiden University · Universität Hamburg · University of Calabria · Carnegie Mellon University

   Affiliation as printed

   Knowledge Technology , Department of Informatics , University of Hamburg , Hamburg , Germany

   Carnegie Mellon University, United States

   Knowledge Technology, Department of Informatics, University of Hamburg, Hamburg, Germany

   Leiden University, Netherlands

   University of Calabria, Italy

3. Phuong D. H. Nguyen Aachen

   Leiden University · Universität Hamburg · University of Calabria · Carnegie Mellon University

   Affiliation as printed

   Knowledge Technology , Department of Informatics , University of Hamburg , Hamburg , Germany

   Carnegie Mellon University, United States

   Knowledge Technology, Department of Informatics, University of Hamburg, Hamburg, Germany

   Leiden University, Netherlands

   University of Calabria, Italy

4. Stefan Wermter Aachen

   Leiden University · Universität Hamburg · University of Calabria · Carnegie Mellon University

   Affiliation as printed

   Knowledge Technology , Department of Informatics , University of Hamburg , Hamburg , Germany

   Carnegie Mellon University, United States

   Knowledge Technology, Department of Informatics, University of Hamburg, Hamburg, Germany

   Leiden University, Netherlands

   University of Calabria, Italy

5. Manfred Eppe Aachen

   Leiden University · Universität Hamburg · University of Calabria · Carnegie Mellon University

   Affiliation as printed

   Knowledge Technology , Department of Informatics , University of Hamburg , Hamburg , Germany

   Carnegie Mellon University, United States

   Knowledge Technology, Department of Informatics, University of Hamburg, Hamburg, Germany

   Leiden University, Netherlands

   University of Calabria, Italy