Collecting Mementos: A Multimodal Dataset for Context-Sensitive Modeling of Affect and Memory Processing in Responses to Videos

IEEE Transactions on Affective Computing, vol. 14, pp. 1249–1266

Abstract

In this article we introduceMementos: the first multimodal corpus for computational modeling of affect and memory processing in response to video content. It was collected online via crowdsourcing and captures 1995 individual responses collected from 297 unique viewers responding to 42 different segments of music videos. Apart from webcam recordings of their upper-body behavior (totaling 2012 minutes) and self-reports of their emotional experience, it contains detailed descriptions of the occurrence and content of 989 personal memories triggered by the video content. Finally, the dataset includes self-report measures related to individual differences in participants’ background and situation (Demographics,Personality, andMood), thereby facilitating the exploration of important contextual factors in research using the dataset. We describe 1) the construction and contents of the corpus itself, 2) analyse thevalidityof its content by investigating biases and consistency with existing research on affect and memory processing, 3) review previously published work that demonstrates theusefulnessof the multimodal data in the corpus for research on automated detection and prediction tasks, and 4) provide suggestions for how the dataset can be used in future research on modelingVideo-Induced Emotions,Memory-Associated Affect, andMemory Evocation.

Authors 4

1. Bernd Dudzik

   Delft University of Technology

   Affiliation as printed

   Department of Intelligent Systems (INSY), Delft University of Technology, Delft, The Netherlands

2. Hayley Hung

   Delft University of Technology

   Affiliation as printed

   Department of Intelligent Systems (INSY), Delft University of Technology, Delft, The Netherlands

3. Mark A. Neerincx

   Delft University of Technology

   Affiliation as printed

   Department of Intelligent Systems (INSY), Delft University of Technology, Delft, The Netherlands

4. Joost Broekens Aachen Leiden Institute of Advanced Computer Science

   Leiden University

   Affiliation as printed

   Leiden Institute of Advanced Computer Science (LIACS), Leiden University, Leiden, The Netherlands