Evaluating Large Language Models on Business Process Modeling: Framework, Benchmark, and Self-Improvement Analysis

arXiv (Cornell University)

Abstract

Large Language Models (LLMs) are rapidly transforming various fields, and their potential in Business Process Management (BPM) is substantial. This paper assesses the capabilities of LLMs on business process modeling using a framework for automating this task, a comprehensive benchmark, and an analysis of LLM self-improvement strategies. We present a comprehensive evaluation of 16 state-of-the-art LLMs from major AI vendors using a custom-designed benchmark of 20 diverse business processes. Our analysis highlights significant performance variations across LLMs and reveals a positive correlation between efficient error handling and the quality of generated models. It also shows consistent performance trends within similar LLM groups. Furthermore, we investigate LLM self-improvement techniques, encompassing self-evaluation, input optimization, and output optimization. Our findings indicate that output optimization, in particular, offers promising potential for enhancing quality, especially in models with initially lower performance. Our contributions provide insights for leveraging LLMs in BPM, paving the way for more advanced and automated process modeling techniques.

Authors 4

1. Humam Kourani Aachen

   RWTH Aachen University · Fraunhofer Institute for Applied Information Technology

   Affiliation as printed

   Fraunhofer Institute for Applied Information Technology FIT , Schloss Birlinghoven , Sankt Augustin , 53757 , Germany

   RWTH Aachen University , Ahornstraße 55 , Aachen , 52074 , Germany

2. Alessandro Berti Aachen

   RWTH Aachen University

   Affiliation as printed

   RWTH Aachen University , Ahornstraße 55 , Aachen , 52074 , Germany

3. Daniel Schuster Aachen

   RWTH Aachen University

   Affiliation as printed

   Process Intelligence Solutions , Kurfürstenstraße 5 , Aachen , 52066 , Germany

   RWTH Aachen University , Ahornstraße 55 , Aachen , 52074 , Germany

4. Wil M. P. van der Aalst Aachen

   RWTH Aachen University

   Affiliation as printed

   RWTH Aachen University , Ahornstraße 55 , Aachen , 52074 , Germany