A Language Prior Based Focal Loss for Visual Question Answering

IEEE International Conference on Multimedia and Expo (ICME), pp. 1–6

Abstract

According to current research, one of the major challenges in Visual Question Answering (VQA) models is the overdependence on language priors (and neglect of the visual modality). VQA models tend to predict answers only based on superficial correlations between the first few words in question and frequency of related answer candidates. To address this issue, we propose a novel Language Prior based Focal Loss (LP-Focal Loss) by rescaling the standard cross entropy loss. Specifically, we employ a question-only branch to capture the language biases for each answer candidate based on the corresponding question input. Then, the LP-Focal Loss dynamically assigns lower weights to biased answers when computing the training loss, thereby reducing the contribution of more-biased instances in the train split. Extensive experiments show that the LP-Focal Loss can be generally applied to common baseline VQA models, and achieves significantly better performance on the VQA-CP v2 dataset, with an overall 18% accuracy boost over benchmark models.

Authors 4

1. Mingrui Lao Aachen LIACS Media Lab

   Leiden University

   Affiliation as printed

   Leiden University,LIACS Media Lab,the Netherlands

   LIACS Media Lab, Leiden University, the Netherlands

2. Yanming Guo

   National University of Defense Technology

   Affiliation as printed

   National University of Defense Technology,College of Systems Engineering,China

   College of Systems Engineering, National University of Defense Technology, China

3. Yu Liu

   Dalian University of Technology

   Affiliation as printed

   Dalian University of Technology,International School of Information Science & Engineering,China

4. Michael S. Lew Aachen LIACS Media Lab

   Leiden University

   Affiliation as printed

   Leiden University,LIACS Media Lab,the Netherlands

   LIACS Media Lab, Leiden University, the Netherlands