Evaluating guidelines for reporting empirical software engineering studies.
Abstract
Background Several researchers have criticized the standards of performing and reporting empirical studies in software engineering. In order to address this problem, Jedlitschka and Pfahl have produced reporting guidelines for controlled experiments in software engineering. They pointed out that their guidelines needed evaluation. We agree that guidelines need to be evaluated before they can be widely adopted. M. Ali Babar was working with the National ICT Australia when the reported work was performed. H. Al-Khilidar : M. Berry : K. Cox : J. Keung : F. Kurniawati : M. Staples : H. Zhang : L. Zhu National ICT Australia Ltd, Sydney 1466 NSW, Australia K. Cox e-mail: karl.cox@nicta.com.au J. Keung e-mail: jkeung@cse.unsw.edu.au F. Kurniawati e-mail: felicia.kurniawata@nicta.com.au M. Staples e-mail: mark.staples@nicta.com.au L. Zhu e-mail: liming.zhu@nicta.com.au M. A. Babar Lero, The Irish Software Engineering Research Centre, University of Limerick, Limerick, Ireland e-mail: malibaba@lero.ie H. Al-Khilidar : M. Berry : J. Keung : H. Zhang School of Computer Science & Engineering, University of New South Wales, Sydney 2052 NSW, Australia B. Kitchenham (*) Keele University, School of Computing and Mathematics, Keele, Staffordshire, ST5 5BG, UK e-mail: Barbara.Kitchenham@cs.keele.ac.uk 98 Empir Software Eng (2008) 13:97-121 Aim The aim of this paper is to present the method we used to evaluate the guidelines and report the results of our evaluation exercise. We suggest our evaluation process may be of more general use if reporting guidelines for other types of empirical study are developed. Method We used a reading method inspired by perspective-based and checklist-based reviews to perform a theoretical evaluation of the guidelines. The perspectives used were: Researcher, Practitioner/Consultant, Meta-analyst, Replicator, Reviewer and Author. Apart from the Author perspective, the reviews were based on a set of questions derived by brainstorming. A separate review was performed for each perspective. The review using the Author perspective considered each section of the guidelines sequentially. Results The reviews detected 44 issues where the guidelines would