The Price of Using Students

arXiv (Cornell University), 2018

In a recent article, Falessi et al. (2017) call for a deeper understanding of the pros and cons of using students and professionals in experiments. The authors state: we have observed too many times that our papers were rejected because we used students as subjects. Good experiments with students are certainly a valuable asset in the body of research in software engineering. Papers should thus not be rejected solely on the ground that the subjects are students. However, the distribution in skill is different for students and professionals. Since previous studies have shown that skill may have a moderating effect on the treatment of participants, we are concerned that studies involving developers with only low to medium skill (i.e., students) may result in wrong inferences about which technology, method or tool is better in the software industry. We therefore provide suggestions for how experiments with students can be improved and also comment on some of the alleged drawbacks of using professionals that Falessi et al. point out.

Empirical Software Engineering

Empirical Software Engineering, 2018

Proceedings of the 15th ACM / IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM), 2021

2003

Abstract Several empirical studies have been carried out with college students as subjects in the last few years. Researchers often use these studies to pilot experiments before they are carried out in industrial environments. Reports on these studies usually focus on the results obtained and issues such as their external validity. However, the effects and value of empirical studies with students may go beyond the contribution to scientific literature.

Context: Researchers perform experiments to check their proposals under controlled conditions. Thus, experiments are an important category of empirical studies and are the classical approach for identifying cause-effect relationships. Goal: Quantitatively characterize and analyze the controlled experiments in software engineering published in journal and conference proceedings over the decade 2003-2013. Method: We performed a systematic mapping study that includes all full papers published at EASE, ESEM and ESEJ. Results: We obtained 110 papers that report controlled experiments. In these experiments we obtained quantitative data about authors and institutions, subjects, tasks, environment, replication and threats to validity. Conclusions: The main contribution of this work is to show the amount of experiments published in the three main venues of Empirical Software Engineering between the years 2003-2013 and also how these experiments have being reported and executed.

Empirical Software Engineering, 2013

Journal of Systems and Software, 2015

… Engineering, 2008. ICSE …, 2009