Abstract
1. Introduction There is a reasonably well-established tradition of conducting empirical studies in software engineering [1], most notably in the form of experiments and quasi-experiments [2]. However, one of the criticisms of such studies as conducted by software engineering researchers is that they are often too small to have the necessary degree of statistical power to detect the levels of effect that are likely to be produced by software engineering practices [3], [4], [5]. (The statistical power is the probability that a given test will correctly reject the null hypothesis.) The empirical research community has also begun to recognise the value of performing replicated studies, although there is still debate about how best to organise these [6], [7], [8], [9]. Both of these issues are increasingly important because such ‘primary' studies provide the inputs for the secondary studies that form the key tool of the evidence-based paradigm [10]. Because secondary studies aim to synthesise the results from many primary studies, they need a sufficient number of these, and also need them to be of good quality and meaningful (which implies having a good level of statistical power) [3], [5]. Context: Due to the lack of suitably skilled participants, software engineering experiments often lack the statistical power needed to detect the levels of effect that may be encountered. Aim: To investigate whether this can be remedied by running an experiment across multiple sites, organised as a single study rather than as a set of replications. Method: We performed a ‘trial' of the idea using a topic (structured abstracts) that some of us had studied previously and which required no participant training. We used five sites, each with 16 participants. Results: We were able to demonstrate the benefits of increased statistical power (and of structured abstracts). We report on our experiences with designing and conducting the study and identify some key lessons about how future studies of this form might be organised. Conclusions: The distributed model offers a flexible, robust form that is capable of delivering better statistical power than would be achieved by running a set of par