Exploiting the Essential Assumptions of Analogy-Based Effort Estimation.
Abstract
Background: There are too many design options for software effort estimators. How can we best explore them all? Aim: We seek aspects on general principles of effort estimation that can guide the design of effort estimators. Method: We identified the essential assumption of analogy-based effort estimation, i.e., the immediate neighbors of a project offer stable conclusions about that project. We test that assumption by generating a binary tree of clusters of effort data and comparing the variance of supertrees versus smaller subtrees. Results: For 10 data sets (from Coc81, Nasa93, Desharnais, Albrecht, ISBSG, and data from Turkish companies), we found: 1) The estimation variance of cluster subtrees is usually larger than that of cluster supertrees; 2) if analogy is restricted to the cluster trees with lower variance, then effort estimates have a significantly lower error (measured using MRE, AR, and Pred(25) with a Wilcoxon test, 95 percent confidence, compared to nearest neighbor methods that use neighborhoods of a fixed size). Conclusion: Estimation by analogy can be significantly improved by a dynamic selection of nearest neighbors, using only the project data from regions with small variance. Index Terms-Software cost estimation, analogy, k-NN. Ç 1 INTRODUCTION S OFTWARE effort estimates are often wrong by a factor of four [1] or even more [2]. As a result, the allocated funds may be inadequate to develop the required project. In the worst case, overrunning projects are canceled and the entire development effort is wasted. For example, NASA canceled its incomplete Check-out Launch Control System project after the initial $200 M estimate was exceeded by another $200 M [3]. It is clear that we need better ways to generate project effort estimates. However, it is not clear how to do that. For example, later in this paper, we document thousands of variations for analogy-based effort estimation (ABE). Effort estimation is an active area of research [4], [5], [6], [7] and more variations are constantly being developed. We expect many more variations of ABE and other effort estimation methods to appear in the very near future. Recent publications propose data m