A Stratification and Sampling Model for Bellwether Moving Window.
Abstract
An effective method for finding the relevant number (window size) and the elapsed time (window age) of recently completed projects has proven elusive in software effort estimation. Although these two parameters significantly affect the prediction accuracy, there is no effective method to stratify and sample chronological projects to improve prediction performance of software effort estimation models. Exemplary projects (Bellwether) representing the training set have been empirically validated to improve the prediction accuracy in the domain of software defect prediction. However, the concept of Bellwether and its effect have not been empirically proven in software effort estimation as a method of selecting exemplary/relevant projects with defined window size and age. In view of this, we introduce a novel method for selecting relevant and recently completed projects referred to as Bellwether moving window for improving the software effort prediction accuracy. We first sort and cluster a pool of N projects and apply statistical stratification based on Markov chain modeling to select the Bellwether moving window. We evaluate the proposed approach using the baseline Automatically Transformed Linear Model on the ISBSG dataset. Results show that (1) Bellwether effect exist in software effort estimation dataset, (2) the Bellwether moving window with a window size of 82 to 84 projects and window age of 1.5 to 2 years resulted in an improved prediction accuracy than the traditional approach. Keywords-Bellwether effect; Markov chains; Window age; Window size; Chronological dataset. 1. INTRODUCTION In the construction of a robust and accurate prediction model, there is the need to consider the best selection of training and validation sets prior to modeling. According to Lokan and Mendes [5][12], selection of training and validation sets from a subset of chronologically arranged projects (referred to as moving window) can further improve the prediction accuracy of effort estimation models. The moving window theory is based on the assumption that recent projects are likely to share similar characteristics with new projects. Recent studies [10][2][7][17] support the theo