Empirical Evaluation of Cross-Release Effort-Aware Defect Prediction Models.
Abstract
To prioritize quality assurance efforts, various fault prediction models have been proposed. However, the best performing fault prediction model is unknown due to three major drawbacks: (1) comparison of few fault prediction models considering small number of data sets, (2) use of evaluation measures that ignore testing efforts and (3) use of n-fold crossvalidation instead of the more practical cross-release validation. To address these concerns, we conducted cross-release evaluation of 11 fault density prediction models using data sets collected from 2 releases of 25 open source software projects with an effort-aware performance measure known as Norm(Popt ). Our result shows that, whilst M5 and K* had the best performances; they were greatly influenced by the percentage of faulty modules present and size of data set. Using Norm(Popt ) produced an overall average performance of more than 50% across all the selected models clearly indicating the importance of considering testing efforts in building fault-prone prediction models. Key words : fault-density estimation, empirical study, open source project, cross version prediction, Dem̆sar's significance diagram among different research papers as observed by Myrtveit et al. [7]. • Cost-effectiveness (hereby refered to as testing effort in this paper) of applying fault-prone prediction models during evaluation and comparison of their prediction performance is ignored in most research studies as pointed out by Arisholm et al. [8]. Several previous studies predict the presence or absence of faults in a module, and evaluate the prediction performance by precision/recall/Fmeasure/ROC-Curve and others [9], [10], [8]. However, most of these performance measures ignores effort needed to fix faults, that is, they do not distinguish between a predicted fault in a small module and a predicted fault in a large module. From the viewpoint of finding more bugs with less testing effort (cost of additional quality assurance such as verification cost which includes code inspections, coverage testing and others) on selected faulty modules, it would be better to predict high faultdensity modules, which contain many faults in a few lines o