Supporting Artifact Evaluation with LLMs: A Study with Published Security Research Papers

IEEE International Conference on Big Data (BigData), pp. 5077–5085

Abstract

Artifact Evaluation (AE) is essential for ensuring the transparency and reliability of research, closing the gap between exploratory work and real-world deployment is particularly important in cybersecurity, particularly in IoT and CPSs, where large-scale, heterogeneous, and privacy-sensitive data meet safety-critical actuation. Yet, manual reproducibility checks are time-consuming and do not scale with growing submission volumes. In this work, we demonstrate that Large Language Models (LLMs) can provide powerful support for AE tasks: (i) text-based reproducibility rating, (ii) autonomous sandboxed execution environment preparation, and (iii) assessment of methodological pitfalls. Our reproducibility-assessment toolkit yields an accuracy of over 72% and autonomously sets up execution environments for 28% of runnable cybersecurity artifacts. Our automated pitfall assessment detects seven prevalent pitfalls with high accuracy ($F_1$ > 92%). Hence, the toolkit significantly reduces reviewer effort and, when integrated into established AE processes, could incentivize authors to submit higher-quality and more reproducible artifacts. IoT, CPS, and cybersecurity conferences and workshops may integrate the toolkit into their peer-review processes to support reviewers' decisions on awarding artifact badges, improving the overall sustainability of the process.

Authors 8

1. David Heye Aachen

   RWTH Aachen University

   Affiliation as printed

   RWTH Aachen University,Communication and Distributed Systems,Germany

2. Karl Kindermann Aachen

   RWTH Aachen University

   Affiliation as printed

   RWTH Aachen University,Communication and Distributed Systems,Germany

3. Robin Decker Aachen

   RWTH Aachen University

   Affiliation as printed

   RWTH Aachen University,Communication and Distributed Systems,Germany

4. Johannes Lohmöller Aachen

   RWTH Aachen University

   Affiliation as printed

   RWTH Aachen University,Communication and Distributed Systems,Germany

5. Anastasiia Belova Aachen

   RWTH Aachen University

   Affiliation as printed

   RWTH Aachen University,Data Stream Management and Analysis,Germany

6. Sandra Geisler Aachen

   RWTH Aachen University

   Affiliation as printed

   RWTH Aachen University,Data Stream Management and Analysis,Germany

7. Klaus Wehrle Aachen

   RWTH Aachen University

   Affiliation as printed

   RWTH Aachen University,Communication and Distributed Systems,Germany

8. Jan Pennekamp Aachen

   RWTH Aachen University

   Affiliation as printed

   RWTH Aachen University,Communication and Distributed Systems,Germany