Assessing the Representativeness of Open Source Projects in Empirical Software Engineering Studies.
Abstract
BACKGROUND: Software engineering researchers have carried out many empirical studies on open source software (OSS) projects to understand the OSS phenomenon, and to develop better software engineering techniques. Many of these studies typically use only a few successful projects as study subjects. Recently, these studies have received criticisms and challenges on their representativeness on OSS projects. AIM: First, we aim to examine whether successful OSS projects are representative to the whole population of OSS. If successful projects are the minority of OSS population, conclusions from existing studies may only partially reflect the OSS phenomenon. Second, we aim to examine to what extent data extracted from successful projects are different from data extracted from the majority. If data extracted from successful projects are different from data extracted from the majority, approaches from existing studies may be ineffective on unsuccessful projects. METHODOLOGY: We selected 11,684 OSS projects on SourceForge, and classified these projects into four categories based their attractiveness to users and developers. We use the K-means clustering technique to produce combined result. Furthermore, we selected eight indicators that are used in many existing studies (e.g., team sizes), and compared indicators that are extracted from different categories to investigate their differences. RESULT: For the first research aim, the result shows that 66.1% projects are under developing; 14.7% projects are user preference; 14.2% projects are developer preference; and only 5.0% projects are considered successful. For the second research aim, the result shows that all the eight analyzed indicators are highly imbalanced with the gamma distribution, and the result reveals that users and developers of SourceForge have different perceptions on the development status defined by SourceForge. CONCLUSION: We conclude that successful projects are not representative to the whole population of OSS, and data extracted from successful projects are quite different from data extracted from the majority. The result implies that conclusions drawn from only a few successful projects may be inadequ