ITACLIP: Boosting Training-Free Semantic Segmentation with Image, Text, and Architectural Enhancements

IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pp. 4142–4152

Abstract

Recent advances in foundational Vision Language Models (VLMs) have reshaped the evaluation paradigm in computer vision tasks. These foundational models, especially CLIP, have accelerated research in open-vocabulary computer vision tasks, including Open-Vocabulary Semantic Segmentation (OVSS). Although the initial results are promising, the dense prediction capabilities of VLMs still require further improvement. In this study, we enhance the semantic segmentation performance of CLIP by introducing new modules and modifications: 1) architectural changes in the last layer of ViT and the incorporation of attention maps from the middle layers with the last layer, 2) Image Engineering: applying data augmentations to enrich input image representations, and 3) using Large Language Models (LLMs) to generate definitions and synonyms for each class name to leverage CLIP's open-vocabulary capabilities. Our training-free method, ITACLIP, outperforms current state-of-the-art approaches on five popular segmentation benchmarks. Our code is available at https://github.com/m-arda-aydn/ITACLIP.

Authors 5

1. M. Arda Aydın

   Bilkent University

   Affiliation as printed

   Bilkent University

2. Efe Mert Çırpar Aachen

   RWTH Aachen University

   Affiliation as printed

   RWTH Aachen University

3. Elvin Abdinli

   Technical University of Munich

   Affiliation as printed

   Technical University of Munich

4. Gozde Una

   Istanbul Technical University

   Affiliation as printed

   Istanbul Technical University

5. Yusuf H. Şahin

   Istanbul Technical University

   Affiliation as printed

   Istanbul Technical University