Which bug reports are valid and why? Using the BERT transformer to classify bug reports and explain their validity

Abstract

Defects and other issues regarding quality and performance of software systems are reported and stored in defect-tracking systems. For software developers, classifying valid bug reports in large defect repositories is challenging. Partly, this is because bug reports commonly contain noise and domain-specific terms. Deep learning, with its strong learning ability from complex text, offers a solution to this issue. However, merely focusing on the performance of the model is not sufficient - clarifying the classification decisions of the system holds equal significance. Our method uses BERT to perform the bug report validity classification task, and we test various mechanisms to explain the classification. Through rigorous evaluation on five open source datasets and benchmarking BERT’s performance against CNN, we demonstrate significant improvements in recall, precision, and F1 score. Importantly, through the evaluation of results from three explanation techniques, our method effectively identifies key features essential to the validity and quality of bug reports.

Authors 2

1. Qianru Meng Aachen [LIACS

   Leiden University

   Affiliation as printed

   LIACS (Leiden Institute of Advanced Computer Science), Leiden University, Netherlands

2. Joost Visser Aachen [LIACS

   Leiden University

   Affiliation as printed

   LIACS (Leiden Institute of Advanced Computer Science), Leiden University, Netherlands