Can Results from Software Engineering Experiments be Safely Combined?

Deriving reliable empirical results from a single experiment is an unlikely event. Hence to progress multiple experiments must be undertaken per hypothesis and the subsequent results effectively combined to produce a single reliable conclusion. Other disciplines use meta-analytic techniques to achieve this result. The treatise of this paper is: can meta-analysis be successfully applied to current Software Engineering experiments?The question is investigated by examining a series of experiments, which themselves investigate - which defect detection technique is best? Applying meta-analysis techniques to the Software Engineering data is relatively straightforward, but unfortunately the results are highly unstable, as the meta-analysis shows that the results are highly disparate and don't lead to a single reliable conclusion.The reason for this deficiency is the excessive variation within various components of the experiments. Finally the paper describes a number of recommendations for controlling and reporting empirical work to advance the discipline towards a position where meta-analysis can be profitably employed.