Conformal Cross-Modal Active Learning

arXiv (Cornell University)

Abstract

Foundation models for vision have transformed visual recognition with powerful pretrained representations and strong zero-shot capabilities, yet their potential for data-efficient learning remains largely untapped. Active Learning (AL) aims to minimize annotation costs by strategically selecting the most informative samples for labeling, but existing methods largely overlook the rich multimodal knowledge embedded in modern vision-language models (VLMs). We introduce Conformal Cross-Modal Acquisition (CCMA), a novel AL framework that bridges vision and language modalities through a teacher-student architecture. CCMA employs a pretrained VLM as a teacher to provide semantically grounded uncertainty estimates, conformally calibrated to guide sample selection for a vision-only student model. By integrating multimodal conformal scoring with diversity-aware selection strategies, CCMA achieves superior data efficiency across multiple benchmarks. Our approach consistently outperforms state-of-the-art AL baselines, demonstrating clear advantages over methods relying solely on uncertainty or diversity metrics.

Authors 6

1. Huy Hoang Nguyen

   AIT Austrian Institute of Technology GmbH

   Affiliation as printed

   AIT Austrian Institute of Technology

2. Cédric Jung

   AIT Austrian Institute of Technology GmbH · Institute of Automation

   Affiliation as printed

   AIT Austrian Institute of Technology

   Automation & Control Institute , Technical University of Vienna

3. Shirin Salehi Aachen Chair of Information Theory and Data Analytics

   RWTH Aachen University

   Affiliation as printed

   Chair of Information Theory and Data Analytics (INDA) , RWTH Aachen University

4. Tobias Glück

   AIT Austrian Institute of Technology GmbH

   Affiliation as printed

   AIT Austrian Institute of Technology

5. Anke Schmeink Aachen Chair of Information Theory and Data Analytics

   RWTH Aachen University

   Affiliation as printed

   Chair of Information Theory and Data Analytics (INDA) , RWTH Aachen University

6. Andreas Kugi

   AIT Austrian Institute of Technology GmbH · Institute of Automation

   Affiliation as printed

   AIT Austrian Institute of Technology

   Automation & Control Institute , Technical University of Vienna