Attribution-based Explanations for Markov Decision Processes

Abstract

Attribution techniques explain the outcome of an AI model by assigning a numerical score to its inputs. So far, these techniques have mainly focused on attributing importance to static input features at a single point in time, and thus fail to generalize to sequential decision-making settings. This paper fills this gap by introducing techniques to generate attribution-based explanations for Markov Decision Processes (MDPs). We give a formal characterization of what attributions should represent in MDPs, focusing on explanations that assign importance scores to both individual states and execution paths. We show how importance scores can be computed by leveraging techniques for strategy synthesis, enabling the efficient computation of these scores despite the non-determinism inherent in an MDP. We evaluate our approach on five case-studies, demonstrating its utility in providing interpretable insights into the logic of sequential decision-making agents.

Authors 6

1. Paul Kobialka

   University of Oslo

   Affiliation as printed

   University of Oslo

2. Andrea Pferscher

   University of Oslo

   Affiliation as printed

   University of Oslo

3. Francesco Leofante

   Imperial College London

   Affiliation as printed

   Imperial College London

4. Erika Ábrahám Aachen

   RWTH Aachen University

   Affiliation as printed

   RWTH Aachen University

5. Silvia Lizeth Tapia Tarifa

   University of Oslo

   Affiliation as printed

   University of Oslo

6. Einar Broch Johnsen

   University of Oslo

   Affiliation as printed

   University of Oslo