Unintended Changes: How LLMs Corrupt and Correct Textual Models

ACM/IEEE International Conference on Model Driven Engineering Languages and Systems Companion (MODELS-C), pp. 636–645

Abstract

This work evaluates to what degree AI-based modeldriven software development approaches may unintentionally alter parts of a model, potentially affecting its semantics, particularly when the model is defined in a domain-specific language unfamiliar to the LLM, and discusses to what degree the perplexity metric can be used to indicate susceptible modeling languages. Although much work has focused on evaluating the correctness of AI-based model creation, it remains important to assess whether and to what extent LLMs introduce unintended modifications in parts of the input model that they are merely expected to reproduce. Previous research has primarily focused on the success of programming and modeling tasks using AI, such as achieving syntactic and semantic correctness. In contrast, this work investigates the side effects of AI usage, specifically the introduction of unintended modifications during iterative modeling processes. We evaluate the likelihood of error introduction in an AI-based modeling approach for uncommon DSLs and use the perplexity-metric to identify error prone elements in the model. We use our own modeling language for class diagrams CD4A, an LLM that is not trained on this DSL and prompt it with simple modeling tasks. Next, we measure additional changes to the model that are not related to the given modeling task. We show that AI-driven Modeling approaches need to also focus on side effects of iterative LLM usage with unfamiliar modeling languages, and demonstrate how these risks can be identified.

Authors 4

1. Lukas Netz Aachen

   RWTH Aachen University

   Affiliation as printed

   RWTH Aachen University,Software Engineering,Aachen,Germany

2. Finn Kampe Aachen

   RWTH Aachen University

   Affiliation as printed

   RWTH Aachen University,Aachen,Germany

3. Jan Reimer Aachen

   RWTH Aachen University

   Affiliation as printed

   RWTH Aachen University,Software Engineering,Aachen,Germany

4. Bernhard Rumpe⋆ Aachen

   RWTH Aachen University

   Affiliation as printed

   RWTH Aachen University,Software Engineering,Aachen,Germany