Finding conclusion stability for selecting the best effort predictor in software effort estimation.
Abstract
Background: Conclusion Instability in software effort estimation (SEE) refers to the inconsistent results produced by a diversity of predictors using different datasets. This is largely due to the "ranking instability" problem, which is highly related to the evaluation criteria and the subset of the data being used. Aim: To determine stable rankings of different predictors. Method: 90 predictors are used with 20 datasets and evaluated using 7 performance measures, whose results are subject to Wilcoxon rank test (95 %). These results are called the "aggregate results". The aggregate results are challenged by a sanity check, which focuses on a single error measure (MRE) and uses a newly developed evaluation algorithm called CLUSTER. These results are called the "specific results." Results: Aggregate results show that: (1) It is now possible to draw stable conclusions about the relative performance of SEE predictors; (2) Regression trees or analogy-based methods are the best performers. The aggregate results are also confirmed by the specific results of the sanity check. Conclusion: This study offers means to address the conclusion instability issue in SEE, which is an important finding for empirical software engineering. Keywords Effort estimation · Data mining · Stability · Linear regression · Regression trees · Neural nets · Analogy · MMRE · Evaluation criteria J. Keung Department of Computing, The Hong Kong Polytechnic University, Kowloon, Hong Kong e-mail: Jacky.Keung@comp.polyu.edu.hk E. Kocaguneli () · T. Menzies Lane Department of Computer Science and Electrical Engineering, West Virginia University, Morgantown, WV 26505, USA e-mail: ekocagun@mix.wvu.edu T. Menzies e-mail: tim@menzies.us 544 Autom Softw Eng (2013) 20:543-567 1 Introduction Being able to choose the most appropriate software development effort predictor for the local software projects remains elusive for many project managers and researchers. For decades, researchers have been seeking for the "best" software effort predictor. At the time of writing, there is no such a commonly agreed "best" predictor found, which provides consistently the most accurate estimate. The usual conclusion from