Leveraging attention‐based visual clue extraction for image classification

IET Image Processing, vol. 15, pp. 2937–2947

Abstract

Abstract Deep learning‐based approaches have made considerable progress in image classification tasks, but most of the approaches lack interpretability, especially in revealing the decisive information causing the categorization of images. This paper seeks to answer the question of what clues encode the discriminative visual information between image categories and can help improve the classification performance. To this end, an attention‐based clue extraction network (ACENet) is introduced to mine the decisive local visual information for image classification. ACENet constructs a clue‐attention mechanism, that is global‐local attention, between the image and visual clue proposals extracted from it and then introduces a contrastive loss defined over the achieved discrete attention distribution to increase the discriminability of clue proposals. The loss encourages considerable attention to be devoted to discriminative clue proposals, that is those similar within the same category and dissimilar across categories. The experimental results for the Negative Web Image (NWI) dataset and the public ImageNet2012 dataset demonstrate that ACENet can extract true clues to improve the image classification performance and outperforms the baselines and the state‐of‐the‐art methods.

Authors 5

1. Yunbo Cui

   Xi'an Jiaotong University

   Affiliation as printed

   Faculty of Electronic and Information Engineering Xi'an Jiaotong University No. 28, Xianning West Road Xi'an China

   Faculty of Electronic and Information Engineering, Xi'an Jiaotong University, No. 28, Xianning West Road, Xi'an, China

2. Youtian Du corresponding

   Xi'an Jiaotong University

   Affiliation as printed

   Faculty of Electronic and Information Engineering Xi'an Jiaotong University No. 28, Xianning West Road Xi'an China

   Faculty of Electronic and Information Engineering, Xi'an Jiaotong University, No. 28, Xianning West Road, Xi'an, China

3. Xue Wang Aachen Institute of Advanced Computer Science Leiden University Leiden Netherlands Institute of Advanced Computer Science

   Leiden University · Xi'an Jiaotong University

   Affiliation as printed

   Faculty of Electronic and Information Engineering Xi'an Jiaotong University No. 28, Xianning West Road Xi'an China

   Institute of Advanced Computer Science Leiden University Leiden Netherlands

   Faculty of Electronic and Information Engineering, Xi'an Jiaotong University, No. 28, Xianning West Road, Xi'an, China

   Institute of Advanced Computer Science, Leiden University, Leiden, Netherlands

4. Hang Wang

   Xi'an Jiaotong University

   Affiliation as printed

   Faculty of Electronic and Information Engineering Xi'an Jiaotong University No. 28, Xianning West Road Xi'an China

   Faculty of Electronic and Information Engineering, Xi'an Jiaotong University, No. 28, Xianning West Road, Xi'an, China

5. Chang Su

   Cornell University · Weill Cornell Medicine

   Affiliation as printed

   Department of Healthcare Policy and Research at Weill Cornell Medicine Cornell University Ithaca, New York USA

   Department of Healthcare Policy and Research at Weill Cornell Medicine, Cornell University, Ithaca, New York, USA