Using Code Coverage to Assess Feature Gaps in MPI Correctness Tool Classification Tests

Abstract

We examine the code generator-based MPI correctness benchmark MPI-BugBench (MBB) by analyzing the code coverage it triggers in three tools: MUST, PARCOACH, and clang-tidy. We present our analysis as a complement to MBB’s original design, which is based on pruning the potentially exhaustive test set based on real-world MPI usage patterns. Our assessment identifies two key limitations in MBB’s generated tests: incomplete coverage of MPI features, such as varying-count collectives, and limited structural diversity, such as lack of loops and array-based MPI handles. In addition, the current strategy of MBB’s code generation by increasing test volume alone offers limited benefit for exercising the analysis of these tools in our assessment. To address these gaps, we implemented 34 new tests covering missing MPI features and more varied code structures. The tests exercise previously uncovered analysis code, where a single varying-count collectives test adds 770 covered code lines in MUST.

Authors 6

1. Alexander Hück

   Technische Universität Darmstadt

   Affiliation as printed

   Technical University of Darmstadt, Darmstadt, Germany

2. Simon Schwitanski Aachen

   RWTH Aachen University

   Affiliation as printed

   RWTH Aachen University, Aachen, Germany

3. Tim Jammer

   Technische Universität Darmstadt

   Affiliation as printed

   Technical University of Darmstadt, Darmstadt, Germany

4. Joachim Jenke Aachen

   RWTH Aachen University

   Affiliation as printed

   RWTH Aachen University, Aachen, Germany

5. Yussur Mustafa Oraji

   Technische Universität Darmstadt

   Affiliation as printed

   Technical University of Darmstadt, Darmstadt, Germany

6. Christian H Bischof

   Technische Universität Darmstadt

   Affiliation as printed

   Technical University of Darmstadt, Darmstadt, Germany