Assessing the Significant Impact of Concept Drift in Software Defect Prediction.
Abstract
Concept drift is a known phenomenon in software data analytics. It refers to the changes in the data distribution over time. The performance of analytic and prediction models degrades due to the changes in the data over time. To improve prediction performance, most studies propose that the prediction model be updated when concept drift occurs. In this work, we investigate the existence of concept drift and its associated effects on software defect prediction performance. We adopt the strategy of an empirically proven method DDM (Drift Detection Method) and evaluate its statistical significance using the chisquare test with Yates continuity correction. The objective is to empirically determine the concept drift and to calibrate the base model accordingly. The empirical study indicates that the concept drift occurs in software defect datasets, and its existence subsequently degrades the performance of prediction models. Two types of concept drifts (gradual and sudden drifts) were identified using the chi-square test with Yates continuity correction in the software defect datasets studied. We suggest concept drift should be considered by software quality assurance teams when building prediction models. Keywords-concept drift detection; defect prediction; empirical software engineering; software quality; streaming data I. I NTRODUCTION Software defect prediction models are developed from historical software data to improve the quality by fixing the bugs of the software [1], [2]. These types of models have been developed over the years to identify bugs in the source code to reduce the development cost [9]. If the historical data (i.e., fixed distributions) changes over time, the prediction model becomes outdated [6]. Krishna and Menzies [2] caution that if the prediction model changes because of the changing historical data, managers will have less faith in the potential of the trained prediction model leading to inefficient use of scarce testing resources. The characteristics of data distribution changes over time i.e. referred to as concept drift. Recently, several defect prediction models based on historical data (fixed distribution) [1]-[4], [6]-[8], [10] have been p