Active Learning and Effort Estimation: Finding the Essential Content of Software Effort Estimation Data.
Abstract
Background: Do we always need complex methods for software effort estimation (SEE)? Aim: To characterize the essential content of SEE data, i.e., the least number of features and instances required to capture the information within SEE data. If the essential content is very small, then 1) the contained information must be very brief and 2) the value added of complex learning schemes must be minimal. Method: Our QUICK method computes the euclidean distance between rows (instances) and columns (features) of SEE data, then prunes synonyms (similar features) and outliers (distant instances), then assesses the reduced data by comparing predictions from 1) a simple learner using the reduced data and 2) a state-of-the-art learner (CART) using all data. Performance is measured using hold-out experiments and expressed in terms of mean and median MRE, MAR, PRED(25), MBRE, MIBRE, or MMER. Results: For 18 datasets, QUICK pruned 69 to 96 percent of the training data (median = 89 percent). K ¼ 1 nearest neighbor predictions (in the reduced data) performed as well as CART's predictions (using all data). Conclusion: The essential content of some SEE datasets is very small. Complex estimation methods may be overelaborate for such datasets and can be simplified. We offer QUICK as an example of such a simpler SEE method. Index Terms-Software cost estimation, active learning, analogy, k-NN Ç 1 INTRODUCTION A CCURATE software effort estimation (SEE) is needed for many business tasks such as project plans, iteration plans, budgets, investment analysis, pricing processes, and bidding rounds. Such estimates can be generated via expertbased methods that use human expertise (possibly augmented with process guidelines, checklists, and data) to generate predictions [1], [2]. Alternatively, model-based methods can summarize old data with data miners that make predictions about new projects [3], [4]. The SEE literature is particularly interested in the summarization of old data by various model-based methods. The SEE literature review of Jorgensen and Shepperd [5] reports that 61 percent of the selected studies deal with the introduction of new methods and their comparison to old ones.