MISHAP-Bench: A Hallucination Benchmark for Large Audio-Language Models

arXiv (Cornell University)

Abstract

Large audio-language models (LALMs) produce fluent responses about audio but often hallucinate by making plausible yet ungrounded claims. Existing audio hallucination benchmarks mainly measure response correctness, leaving it unclear whether an LALM hallucinates or simply fails to understand the audio. We challenge correctness-based evaluation by defining two hallucination categories: (i) context, where claims are not grounded in the audio; and (ii) knowledge, where claims about audio-related topics lack support from externally verifiable facts. We introduce MISHAP-Bench, a comprehensive benchmark with 12,000 challenging open-ended question-audio pairs and a rigorous evaluation pipeline covering both categories. To evaluate open-ended responses, we propose a groundedness judge that uses reference rubrics and judge prompts guided by human annotations. We extensively evaluate ten state-of-the-art LALMs and show that hallucination remains substantial. Even a frontier model such as Gemini 3.7 Flash reaches a hallucination rate of 36.5%. We further adapt and benchmark four mitigation methods from multiple domains for LALMs. Despite some improvements, effective hallucination mitigation remains an open challenge. Finally, we call on the community to evaluate hallucination and benchmark mitigation methods with MISHAP-Bench.

Authors 4

1. Zhi Wen Soi

   TU Dortmund University · University of Neuchâtel

   Affiliation as printed

   TU Dortmund University

   University of Neuchâtel

2. Giulio Segalini

   University of Neuchâtel

   Affiliation as printed

   University of Neuchâtel

3. Jian-Jia Chen Aachen

   TU Dortmund University · RWTH Aachen University

   Affiliation as printed

   RWTH Aachen University

   TU Dortmund University

4. Lydia Y. Chen

   University of Neuchâtel · Delft University of Technology

   Affiliation as printed

   Delft University of Technology

   University of Neuchâtel