Diagnosing LLM Hallucinations in Process Mining Tasks: a Taxonomy and a Benchmark

Abstract

Large language models (LLMs) are increasingly used to support process mining tasks, yet their answers may contain hallucinations, content that is unfaithful to the prompt, unsupported by evidence, or invalid with respect to PM formalisms. The PM-LLM-Benchmark evaluates model answers with an expert LLM-as-a-judge that assigns a numeric score and a short explanation. This paper introduces a complementary hallucination audit that reads the judge's explanation and converts it into structured annotations about the answering model. We map issues mentioned by the judge onto a PM-tailored taxonomy (four families, twelve sub-types) with severities, aggregate them per model and task family, and analyze their relationships with benchmark performance and model characteristics. Across models, higher PM-LLM-Benchmark scores align with fewer hallucinations; reasoning-oriented and newer models show more favorable profiles; and hallucination families tend to co-occur rather than trade off. We discuss safeguards (reinforce context fidelity, disciplined reasoning, and validate structured outputs), that directly target the dominant failure modes.

Authors 2

1. Alessandro Berti Aachen

   RWTH Aachen University

   Affiliation as printed

   RWTH Aachen University

2. Humam Kourani Aachen

   RWTH Aachen University · Fraunhofer Institute for Applied Information Technology

   Affiliation as printed

   Fraunhofer Institute for Applied Information Technology FIT, Schloss Birlinghoven

   RWTH Aachen University