Guidelines for Promoting Software Product Line Experiments

The importance of experimentation for Software Engineering research has been notably established in the last years. The software engineering community has discussed how to proper report and evaluate experiments using different approaches, such as quality criteria, scales, and checklists. Nevertheless, there are no guidelines to support researchers and practitioners active in straightforward software engineering research areas, as in Software Product Lines (SPL), at conducting experiments. We hypothesize that experimentation guidelines may aid such a specific area by providing advice and actual excerpts reflecting good practices of SPL experimentation, thus experimentally evolving this area. Therefore, the goal of this paper is to provide guidelines for properly reporting and promoting SPL experiments. We defined such guidelines based on well-known software engineering experiment reports, quality evaluation checklists, and data extracted from 211 SPL experiments identified in a systematic mapping study. We evaluated the guidelines with a qualitative study with SPL and experimentation experts applying open and axial coding procedures. The evaluation enabled us to improve the guidelines. The resulting guidelines contain specific advice to researchers active in SPL and provide examples taken from published SPL experiments. The experts’ positive points indicate that the proposed guidelines can aid SPL researchers and practitioners. Sharing the resulting guidelines could support conducting SPL experiments and allow further area evolution based on prospective experiment replications and reproductions from well-designed and reported experiments.