Facts are Harder Than Opinions -- A Multilingual, Comparative Analysis of LLM-Based Fact-Checking Reliability

arXiv (Cornell University)

Abstract

The proliferation of misinformation necessitates scalable, automated fact-checking solutions. Yet, current benchmarks often overlook multilingual and topical diversity. This paper introduces a novel, dynamically extensible data set that includes 61,514 claims in multiple languages and topics, extending existing datasets up to 2024. Through a comprehensive evaluation of five prominent Large Language Models (LLMs), including GPT-4o, GPT-3.5 Turbo, LLaMA 3.1, and Mixtral 8x7B, we identify significant performance gaps between different languages and topics. While overall GPT-4o achieves the highest accuracy, it declines to classify 43% of claims. Across all models, factual-sounding claims are misclassified more often than opinions, revealing a key vulnerability. These findings underscore the need for caution and highlight challenges in deploying LLM-based fact-checking systems at scale.

Authors 4

1. Lorraine Saju Aachen Chair for Computational Social Science

   RWTH Aachen University · GESIS - Leibniz Institute for the Social Sciences

   Affiliation as printed

   GESIS -Leibniz Institute for the Social Sciences , Unter Sachsenhausen 6-8 , 50667 , North Rhine-Westphalia , Germany

   RWTH Aachen University , Chair for Computational Social Science , Templergraben 55 , 52062 , North Rhine-Westphalia , Germany

2. Arnim Bleier

   GESIS - Leibniz Institute for the Social Sciences

   Affiliation as printed

   GESIS -Leibniz Institute for the Social Sciences , Unter Sachsenhausen 6-8 , 50667 , North Rhine-Westphalia , Germany

3. Jana Lasser Aachen Chair for Computational Social Science

   RWTH Aachen University

   Affiliation as printed

   RWTH Aachen University , Chair for Computational Social Science , Templergraben 55 , 52062 , North Rhine-Westphalia , Germany

   University of Graz , Complexity Science Hub , Auenbruggerplatz 2 , 8036 , Styria , Austria

4. Claudia Wagner Aachen Chair for Computational Social Science

   RWTH Aachen University · GESIS - Leibniz Institute for the Social Sciences

   Affiliation as printed

   GESIS -Leibniz Institute for the Social Sciences , Unter Sachsenhausen 6-8 , 50667 , North Rhine-Westphalia , Germany

   RWTH Aachen University , Chair for Computational Social Science , Templergraben 55 , 52062 , North Rhine-Westphalia , Germany