KM-BART: Knowledge Enhanced Multimodal BART for Visual Commonsense Generation

arXiv (Cornell University)

Abstract

We present Knowledge Enhanced Multimodal BART (KM-BART), which is a Transformer-based sequence-to-sequence model capable of reasoning about commonsense knowledge from multimodal inputs of images and texts. We adapt the generative BART architecture to a multimodal model with visual and textual inputs. We further develop novel pretraining tasks to improve the model performance on the Visual Commonsense Generation (VCG) task. In particular, our pretraining task of Knowledge-based Commonsense Generation (KCG) boosts model performance on the VCG task by leveraging commonsense knowledge from a large language model pretrained on external commonsense knowledge graphs. To the best of our knowledge, we are the first to propose a dedicated task for improving model performance on the VCG task. Experimental results show that our model reaches state-of-the-art performance on the VCG task by applying these novel pretraining tasks.

Authors 6

1. Yiran Xing Aachen

   ETH Zurich · Ludwig-Maximilians-Universität München · RWTH Aachen University

   Affiliation as printed

   LMU Munich, Germany

   RWTH Aachen, Germany

   ♣ ETH Zurich, Switzerland

2. Zai Shi
3. Meng Zhao
4. Gerhard Lakemeyer Aachen

   ETH Zurich · Ludwig-Maximilians-Universität München · RWTH Aachen University

   Affiliation as printed

   LMU Munich, Germany

   RWTH Aachen, Germany

   ♣ ETH Zurich, Switzerland

5. Yunpu Ma
6. Roger P. Wattenhofer