Task Learnability Modulates Surprise but Not Valence Processing for Reinforcement Learning in Probabilistic Choice Tasks

Journal of Cognitive Neuroscience, vol. 34, pp. 34–53

Abstract

The goal of temporal difference (TD) reinforcement learning is to maximize outcomes and improve future decision-making. It does so by utilizing a prediction error (PE), which quantifies the difference between the expected and the obtained outcome. In gambling tasks, however, decision-making cannot be improved because of the lack of learnability. On the basis of the idea that TD utilizes two independent bits of information from the PE (valence and surprise), we asked which of these aspects is affected when a task is not learnable. We contrasted behavioral data and ERPs in a learning variant and a gambling variant of a simple two-armed bandit task, in which outcome sequences were matched across tasks. Participants were explicitly informed that feedback could be used to improve performance in the learning task but not in the gambling task, and we predicted a corresponding modulation of the aspects of the PE. We used a model-based analysis of ERP data to extract the neural footprints of the valence and surprise information in the two tasks. Our results revealed that task learnability modulates reinforcement learning via the suppression of surprise processing but leaves the processing of valence unaffected. On the basis of our model and the data, we propose that task learnability can selectively suppress TD learning as well as alter behavioral adaptation based on a flexible cost-benefit arbitration.

Authors 6

1. Franz Wurm Aachen Leiden Institute for Brain and Cognition

   Leiden University · Catholic University of Eichstätt-Ingolstadt

   Affiliation as printed

   Catholic University of Eichstätt-Ingolstadt, Germany

   Leiden Institute for Brain and Cognition

   Leiden University

2. Wioleta Walentowska

   Jagiellonian University · Ghent University

   Affiliation as printed

   Ghent University

   Jagiellonian University, Krakow, Poland

3. Benjamin Ernst

   Catholic University of Eichstätt-Ingolstadt

   Affiliation as printed

   Catholic University of Eichstätt-Ingolstadt, Germany

4. Mario Carlo Severo

   Ghent University

   Affiliation as printed

   Ghent University

5. Gilles Pourtois

   Ghent University

   Affiliation as printed

   Ghent University

6. Marco Steinhauser

   Catholic University of Eichstätt-Ingolstadt

   Affiliation as printed

   Catholic University of Eichstätt-Ingolstadt, Germany