Impact of the Distribution Parameter of Data Sampling Approaches on Software Defect Prediction Models.
Abstract
Sampling methods are known to impact defect prediction performance. These sampling methods have configurable parameters that can significantly affect the prediction performance. It is however, impractical to assess the effect of all the possible different settings in the parameter space for all the several existing sampling methods. A constant and easy to tweak parameter present in all sampling methods is the distribution of the defective and non-defective modules in the dataset known as Pfp (% of fault-prone modules). In this paper, we investigate and assess the performance of defect prediction models where the Pfp parameter of sampling methods are tweaked. An empirical experiment and assessment of seven sampling methods on five prediction models over 20 releases of 10 static metric projects indicate that (1) Area Under the Receiver Operating Characteristics Curve (AUC) performance is not improved after tweaking the Pfp parameter, (2) pf (false alarms) performance degrades as the Pfp is increased. (3) a stable predictor is difficult to achieve across different Pfp rates. Hence, we conclude that the Pfp parameter setting can have a large impact on the performance (except AUC) of defect prediction models. We thus recommend researchers experiment with the Pfp parameter of the sampling method since the distribution of training datasets vary. Keywords - Search based SE, Defect prediction, Sampling methods, Imbalanced Data, Preprocessing, Empirical software engineering. the training data. Different sampling methods have different configurable parameters and it is impractical to configure all these parameters of the various sampling methods during model construction. Nevertheless, a constant parameter for all sampling method is the amount of resampling or percentage of fault-prone modules (Pfp) which controls the amount of data added to or removed from the minority class samples or the majority class samples respectively. Our literature analysis reveal that prior defect prediction studies that apply sampling methods as preprocessing techniques do not consider the impact of these parameters and its effect on prediction performance. Agrawal and Menzies [1] note that when th