VQA-BC: Robust Visual Question Answering Via Bidirectional Chaining

IEEE International Conference on Acoustics Speech and Signal Processing, pp. 4833–4837

Abstract

Current VQA models are suffering from the problem of overdependence on language bias, which severely reduces their robustness in real-world scenarios. In this paper, we analyze VQA models from the view of forward/backward chaining in the inference engine, and propose to enhance their robustness via a novel Bidirectional Chaining (VQA-BC) framework. Specifically, we introduce a backward chaining with hardnegative contrastive learning to reason from the consequence (answers) to generate crucial known facts (question-related visual region features). Furthermore, to alleviate the overconfident problem in answer prediction (forward chaining), we present a novel introspective regularization to connect forward and backward chaining with label smoothing. Extensive experiments verify that VQA-BC not only effectively overcomes language bias on out-of-distribution dataset, but also alleviates the over-correct problem caused by ensemble-based method on in-distribution dataset. Compared with competitive debiasing strategies, our method achieves state-of-the-art performance to reduce language bias on VQA-CP v2 dataset.

Authors 5

1. Mingrui Lao Aachen LIACS Medialab

   Leiden University

   Affiliation as printed

   Leiden University,LIACS Medialab

   LIACS Medialab, Leiden University

2. Yanming Guo

   National University of Defense Technology

   Affiliation as printed

   National University of Defense Technology,College of Systems Engineering

   College of Systems Engineering, National University of Defense Technology

3. Wei Chen Aachen LIACS Medialab

   Leiden University

   Affiliation as printed

   Leiden University,LIACS Medialab

   LIACS Medialab, Leiden University

4. Nan Pu Aachen LIACS Medialab

   Leiden University

   Affiliation as printed

   Leiden University,LIACS Medialab

   LIACS Medialab, Leiden University

5. Michael S. Lew Aachen LIACS Medialab

   Leiden University

   Affiliation as printed

   Leiden University,LIACS Medialab

   LIACS Medialab, Leiden University