An Empirical Study of Learning to Rank Techniques for Effort-Aware Defect Prediction.
Abstract
Effort-Aware Defect Prediction (EADP) ranks software modules based on the possibility of these modules being defective, their predicted number of defects, or defect density by using learning to rank algorithms. Prior empirical studies compared a few learning to rank algorithms considering small number of datasets, evaluating with inappropriate or one type of performance measure, and non-robust statistical test techniques. To address these concerns and investigate the impact of learning to rank algorithms on the performance of EADP models, we examine the practical effects of 23 learning to rank algorithms on 41 available defect datasets from the PROMISE repository using a module-based effort-aware performance measure (FPA) and a source lines of code (SLOC) based effort-aware performance measure (Norm(Popt)). In addition, we compare the performance of these algorithms when they are trained on a more relevant feature subset selected by the Information Gain feature selection method. In terms of FPA and Norm(Popt), statistically significant differences are observed among these algorithms with BRR (Bayesian Ridge Regression) performing best in terms of FPA, and BRR and LTR (Learning-to-Rank) performing best in terms of Norm(Popt). When these algorithms are trained on a more relevant feature subset selected by Information Gain, LTR and BRR still perform best with significant differences in terms of FPA and Norm(Popt). Therefore, we recommend BRR and LTR for building the EADP model in order to find more defects by inspecting a certain number of modules or lines of codes. Index Terms-effort-aware defect prediction; learning to rank; empirical study; Scott-Knott ESD test. I. INTRODUCTION Software defect prediction (SDP) has been an active research area in the field of software engineering attracting more and more attention from both industry and academia [1], [2]. SDP models predict whether a software module is defective based on some software features such as source lines of codes (SLOC) and McCabe's cyclomatic complexity. Accurate prediction results can help allocate limited testing resources by suggesting that software testers pay more attention on those predicted de