Evaluating guidelines for empirical software engineering studies.
Abstract
Categories and Subject Descriptors Background. Several researchers have criticized the standards of performing and reporting empirical studies in software engineering. In order to address this problem, Andreas Jedlitschka and Dietmar Pfahl have produced reporting guidelines for controlled experiments in software engineering. They pointed out that their guidelines needed evaluation. We agree that guidelines need to be evaluated before they can be widely adopted. If guidelines are flawed, they will cause more problems that they solve. K.6.3 [Software Software process. Engineering]: Software Management - General Terms: Management, Experimentation. Keywords: Controlled experiments, software engineering, guidelines, perspective-based inspection 1. INTRODUCTION Aim. The aim of this paper is to present the method we used to evaluate the guidelines and report the results of our evaluation exercise. We suggest our evaluation process may be of more general use if reporting guidelines for other types of empirical study are developed. This paper reports an exercise undertaken by staff and students in the Empirical Software Engineering (ESE) group at the National ICT Australia to evaluate the reporting guidelines for controlled experiments proposed by Jedlitschka and Pfahl [6]. In spite of the existence of a specialist book to help software engineers conduct experiments [14], software engineering experiments are still subject to criticism. The guidelines were developed in response to general criticisms of current standards of performing and reporting empirical studies (Kitchenham et al. [7]), and more specific criticisms that the lack of reporting standards is causing problems when researchers attempt to aggregate empirical evidence because important information is not reported or is reported in an inconsistent fashion (e.g. [9], [13]). Method. We used perspective-based inspections to perform a theoretical evaluation of the guidelines. A separate inspection was performed for each perspective. The perspectives used were: Researcher, Practitioner/Consultant, Meta-analyst, Replicator, Reviewer and Author. Apart from the Author perspective, the inspections were based