“Even explanations will not help in trusting [this] fundamentally biased system”: A Predictive Policing Case-Study

ACM Conference on User Modeling, Adaptation and Personalization (UMAP), pp. 51–62

Abstract

In today's society, where Artificial Intelligence (AI) has gained a vital role, concerns regarding user's trust have garnered significant attention.The use of AI systems in high-risk domains have often led users to either under-trust it, potentially causing inadequate reliance or over-trust it, resulting in over-compliance.Therefore, users must maintain an appropriate level of trust.Past research has indicated that explanations provided by AI systems can enhance user understanding of when to trust or not trust the system.However, the utility of presentation of different explanations forms still remains to be explored especially in high-risk domains.Therefore, this study explores the impact of different explanation types (text, visual, and hybrid) and user expertise (retired police officers and lay users) on establishing appropriate trust in AI-based predictive policing.While we observed that the hybrid form of explanations increased the subjective trust in AI for expert users, it did not led to better decision-making.Furthermore, no form of explanations helped build appropriate trust.The findings of our study emphasize the importance of re-evaluating the use of explanations to build [appropriate] trust in AI based systems especially when the system's use is questionable.Finally, we synthesize potential challenges and policy recommendations based on our results to design for appropriate trust in high-risk based AI-based systems.

Authors 6

1. Siddharth Mehrotra

   Delft University of Technology

   Affiliation as printed

   TU Delft, Delft, Netherlands

2. Ujwal Gadiraju

   Delft University of Technology

   Affiliation as printed

   Web Information Systems, Software Technology Department, TU Delft, Delft, Netherlands

3. Eva Alice Christiane Bittner

   Universität Hamburg

   Affiliation as printed

   University of Hamburg, Hamburg, Germany

4. Folkert van Delden

   Delft University of Technology

   Affiliation as printed

   TU Delft, Delft, Netherlands

5. Catholijn M. Jonker Aachen TU Delft

   Leiden University · Delft University of Technology

   Affiliation as printed

   TU Delft, Delft, Netherlands and Leiden University, Leiden, Netherlands

6. Myrthe Lotte Tielman

   Delft University of Technology

   Affiliation as printed

   TU Delft, Delft, Netherlands