Surprise-minimization as a solution to the structural credit assignment problem

Abstract

In naturalistic environments, causal relationships between actions and their consequences are often hidden from direct observation. To overcome this structural credit-assignment problem, agents have to infer causal structures from experience. Here, we developed a computational model which formalizes action selection as the competition between structural representations, while action control is arbitrated towards the representation that minimizes surprise over time. To validate this model, we recorded behavioral and electrophysiological data from human participants in a novel task in which independent decisions are followed by outcomes, whereby the decision-outcome mapping is unknown. The model could account for patterns of choice behavior revealing clear hallmarks of credit assignment. Model-based analysis of EEG activity confirmed central model characteristics of concurrent prediction errors and a signature of evidence accumulation driving arbitration between representations. These findings highlight a key role of surprise minimization for both value and representation learning and reveal neural correlates of credit assignment.

Authors 3

1. Franz Wurm corresponding Aachen Leiden Institute for Brain and Cognition

   Leiden University · Catholic University of Eichstätt-Ingolstadt

   Affiliation as printed

   Catholic University of Eichstätt-Ingolstadt, Germany;

   Leiden Institute for Brain and Cognition, Leiden University, Leiden, the Netherlands

   Leiden University, the Netherlands;

2. Benjamin Ernst

   Catholic University of Eichstätt-Ingolstadt

   Affiliation as printed

   Catholic University of Eichstätt-Ingolstadt, Germany;

3. Marco Steinhauser

   Catholic University of Eichstätt-Ingolstadt

   Affiliation as printed

   Catholic University of Eichstätt-Ingolstadt, Germany;