Knowledge-Driven Hallucination in Large Language Models: An Empirical Study on Process Modeling

Abstract

The utility of Large Language Models (LLMs) in analytical tasks is rooted in their vast pre-trained knowledge, which allows them to interpret ambiguous inputs and infer missing information. However, this same capability introduces a critical risk of what we term knowledgedriven hallucination: a phenomenon where the model's output contradicts explicit source evidence because it is overridden by the model's generalized internal knowledge. This paper investigates this phenomenon by evaluating LLMs on the task of automated process modeling, where the goal is to generate a formal business process model from a given source artifact. The domain of Business Process Management (BPM) provides an ideal context for this study, as many core business processes follow standardized patterns, making it likely that LLMs possess strong pre-trained schemas for them. We conduct a controlled experiment designed to create scenarios with deliberate conflict between provided evidence and the LLM's background knowledge. We use inputs describing both standard and deliberately atypical process structures to measure the LLM's fidelity to the provided evidence. Our work provides a methodology for assessing this critical reliability issue and raises awareness of the need for rigorous validation of AI-generated artifacts in any evidence-based domain.

Authors 3

1. Alessandro Berti Aachen

   RWTH Aachen University · Fraunhofer Institute for Applied Information Technology

   Affiliation as printed

   Fraunhofer Institute for Applied Information Technology FIT, Schloss Birlinghoven

   RWTH Aachen University

2. Humam Kourani Aachen

   RWTH Aachen University · Fraunhofer Institute for Applied Information Technology

   Affiliation as printed

   Fraunhofer Institute for Applied Information Technology FIT, Schloss Birlinghoven

   RWTH Aachen University

3. Wil M. P. van der Aalst Aachen

   RWTH Aachen University

   Affiliation as printed

   RWTH Aachen University