research-article

Sampling in software engineering research: a critical review and guidelines

Authors:

Sebastian Baltes,

Paul RalphAuthors Info & Claims

Empirical Software Engineering, Volume 27, Issue 4

https://doi.org/10.1007/s10664-021-10072-8

Published: 01 July 2022 Publication History

Abstract

Representative sampling appears rare in empirical software engineering research. Not all studies need representative samples, but a general lack of representative sampling undermines a scientific field. This article therefore reports a critical review of the state of sampling in recent, high-quality software engineering research. The key findings are: (1) random sampling is rare; (2) sophisticated sampling strategies are very rare; (3) sampling, representativeness and randomness often appear misunderstood. These findings suggest that software engineering research has a generalizability crisis. To address these problems, this paper synthesizes existing knowledge of sampling into a succinct primer and proposes extensive guidelines for improving the conduct, presentation and evaluation of sampling in software engineering research. It is further recommended that while researchers should strive for more representative samples, disparaging non-probability sampling is generally capricious and particularly misguided for predominately qualitative research.

References

[1]

Amir B, Ralph P (2018) There is no random sampling in software engineering research. In: Proceedings of the 40th international conference on software engineering: companion proceeedings, pp 344–345

[2]

Arnett JJ The neglected 95%: why American psychology needs to become less American Am Psychol 2008 63 7 602

[3]

Baltes S, Diehl S (2016) Worse than spam: issues in sampling software developers. In: Genero M, Jedlitschka A, Jorgensen M (eds) 10th international symposium on Empirical Software Engineering and Measurement (ESEM 2016), ACM, Ciudad Real, Spain, pp 52:1–52:6, DOI, (to appear in print)

[4]

Baltes S, Ralph P (2020) Sampling in software engineering research supplementary material [data set].

[5]

Baltes S, Dumani L, Treude C, Diehl S (2018) SOTOrrent: reconstructing and analyzing the evolution stack overflow posts. In: Zaidman A, Hill E, Kamei Y (eds) 15th international conference on Mining Software Repositories (MSR 2018), ACM, Gothenburg, Sweden, pp 319–330

[6]

Beecham S, Baddoo N, Hall T, Robinson H, and Sharp H Motivation in software engineering: a systematic literature review Inf Softw Technol 2008 50 9 860-878

[7]

Breckenridge J and Jones D Demystifying theoretical sampling in grounded theory research Grounded Theory Rev 2009 8 2 112-126

[8]

Caine K (2016) Local standards for sample size at chi. In: Proceedings of the 2016 CHI conference on human factors in computing systems, ACM, New York, NY, USA, CHI ’16, pp 981–992

[9]

Charmaz K Constructing grounded theory 2014 London Sage

[10]

Checkland P and Holwell S Action research: its nature and validity Syst Pract Action Res 1998 11 1 9-21

[11]

Cochran WG (2007) Sampling techniques. John Wiley & Sons

[12]

Cohen J Statistical power analysis for the behavioral sciences 1988 Hillsdale Lawrence Erlbaum Associates

[13]

Cosentino V, Izquierdo JLC, Cabot J (2016) Findings from github: methods, datasets and limitations. In: 2016 IEEE/ACM 13th working conference on Mining Software Repositories (MSR), IEEE, pp 137–141

[14]

Daniel J (2011) Sampling essentials: practical guidelines for making sampling choices. Sage Publications

[15]

De Mello RM, Travassos GH (2016) Surveys in software engineering: Identifying representative samples. In: Proceedings of the 10th ACM/IEEE international symposium on empirical software engineering and measurement, pp 1–6

[16]

Dillman DA, Smyth JD, and Christian LM Internet, phone, mail and mixed-mode surveys: the tailored design method 2014 4th edn. Hoboken John Wiley & Sons

[17]

Draucker CB, Martsolf DS, Ross R, and Rusk TB Theoretical sampling and category development in grounded theory Qual Health Res 2007 17 8 1137-1148

[18]

Duignan B (2019) Postmodernism. In: Encyclopedia Britannica, Encyclopedia Britannica, Inc. https://www.britannica.com/topic/postmodernism-philosophy

[19]

Easterbrook S, Singer J, Storey MA, Damian D (2008) Selecting empirical methods for software engineering research. In: Guide to advanced empirical software engineering, Springer, pp 285–311

[20]

Easton G (2010) One case study is enough. Lancaster University technical report https://eprints.lancs.ac.uk/id/eprint/49016/

[21]

Falessi D, Juristo N, Wohlin C, Turhan B, Münch J, Jedlitschka A, and Oivo M Empirical software engineering experts on the use of students and professionals in experiments Empir Softw Eng 2018 23 1 452-489

[22]

Faugier J and Sargeant M Sampling hard to reach populations J Adv Nurs 1997 26 4 790-797

[23]

Feldt R, Zimmermann T, Bergersen GR, Falessi D, Jedlitschka A, Juristo N, Münch J, Oivo M, Runeson P, Shepperd M, Sjøberg DIK, and Turhan B Four commentaries on the use of students and professionals in empirical software engineering experiments Empir Softw Eng 2018 23 6 3801-3820

[24]

Fitts PM The information capacity of the human motor system in controlling the amplitude of movement J Exp Psychol 1954 47 6 381-391

[25]

Foster E (2014) Software engineering: a methodical approach apress. New York, USA

[26]

Gentles SJ, Charles C, Ploeg J, and McKibbon KA Sampling in qualitative research: insights from an overview of the methods literature Qual Rep 2015 20 11 1772-1789

[27]

Glaser BG, Strauss AL (2017) Discovery of grounded theory: strategies for qualitative research. Routledge

[28]

Goel S and Salganik MJ Assessing respondent-driven sampling Proceedings of the National Academy of Sciences 2010 107 15 6743-6747

[29]

Gousios G (2013) The GHTorrent dataset and tool suite. In: Zimmermann T, Di Penta M, Kim S (eds) 10Th international working conference on Mining Software Repositories (MSR, vol 2013. IEEE, San Francisco, CA, USA, pp 233–236

[30]

Guba EG and Lincoln YS Epistemological and methodological bases of naturalistic inquiry Educ Commun Technol J 1982 30 4 233-252

[31]

Heckathorn DD Respondent-driven sampling: a new approach to the study of hidden populations Soc Probl 1997 44 2 174-199

[32]

Henrich J, Heine SJ, and Norenzayan A The weirdest people in the world? Behav Brain Sci 2010 33 2-3 61-83

[33]

Henry GT (1990) Practical sampling. Sage

[34]

van Hoeven LR, Janssen MP, Roes KC, and Koffijberg H Aiming for a representative sample: simulating random versus purposive strategies for hospital selection BMC Med Res Methodol 2015 15 1 90

[35]

Huang X, Zhang H, Zhou X, Babar MA, Yang S (2018) Synthesizing qualitative research in software engineering: a critical review. In: Proceedings of the 40th international conference on software engineering, pp 1207–1218

[36]

Humbatova N, Jahangirova G, Bavota G, Riccio V, Stocco A, Tonella P (2020) Taxonomy of real faults in deep learning systems. In: 2020 IEEE/ACM 42nd international conference on software engineering (ICSE), IEEE

[37]

Ingram C, Drachen A (2020) How software practitioners use informal local meetups to share software engineering knowledge. In: 2020 IEEE/ACM 42nd international conference on software engineering (ICSE), IEEE

[38]

Johnston LG and Sabin K Sampling hard-to-reach populations with respondent driven sampling Methodological Innovations Online 2010 5 2 38-48

[39]

Kitchenham B, Charters S (2007) Guidelines for performing systematic literature reviews in software engineering. Tech. rep., Keele University and University of Durham

[40]

Kitchenham B and Pfleeger SL Principles of survey research: Part 5: populations and samples ACM SIGSOFT Softw Eng Notes 2002 27 5 17-20

[41]

Kitchenham BA, Pfleeger SL (2008) Personal opinion surveys. In: Guide to advanced empirical software engineering, Springer, pp 63–92

[42]

Kruskal W and Mosteller F Representative sampling, i: mon-scientific literature Int Stat Rev 1979 47 1 13-24

[43]

Kruskal W and Mosteller F Representative sampling, iii: the current statistical literature Int Stat Rev 1979 47 3 245-265

[44]

Landon EL Jr and Banks SK Relative efficiency and bias of Plus-One telephone sampling J Mark Res 1977 14 3 294

[45]

Lee AS and Baskerville RL Generalizing generalizability in information systems research Inf Syst Res 2003 14 3 221-243

[46]

Maalej W and Robillard MP Patterns of knowledge in api reference documentation IEEE Trans Softw Eng 2013 39 9 1264-1282

[47]

Malekinejad M, Johnston LG, Kendall C, Kerr LRFS, Rifkin MR, and Rutherford GW Using respondent-driven sampling methodology for HIV biological and behavioral surveillance in international settings: a systematic review AIDS and Behavior 2008 12 1 105-130

[48]

van Manen M (2016) Phenomenology of practice: meaning-giving methods in phenomenological research and writing. Routledge

[49]

McElreath R (2020) Statistical rethinking: a Bayesian course with examples in R and Stan. CRC press

[50]

de Mello RM, Travassos GH (2015) Characterizing sampling frames in software engineering surveys. In: Proceedings of the Ibero-American conference on sofware engineering (CibSE)

[51]

de Mello RM, Da Silva PC, and Travassos GH Investigating probabilistic sampling approaches for large-scale surveys in software engineering J Softw Eng Res Dev 2015 3 1 1-26

[52]

Miles MB, Huberman AM, and Saldaña J Qualitative data analysis: a methods sourcebook 2014 4th edn. Thousand Oaks, California, USA Sage

[53]

Mohanani R, Turhan B, Ralph P (2019) Requirements framing affects design creativity. IEEE Trans Softw Eng

[54]

Moher D, Liberati A, Tetzlaff J, Altman DG, Group P, et al. Preferred reporting items for systematic reviews and meta-analyses: the prisma statement PLoS Med 2009 6 7 e1000097

[55]

Mullinix KJ, Leeper TJ, Druckman JN, and Freese J The generalizability of survey experiments J Exp Polit Sci 2015 2 2 109-138

[56]

Nagappan M, Zimmermann T, Bird C (2013) Diversity in software engineering research. In: Proceedings of the 9th joint meeting on foundations of software engineering, ACM, pp 466–476

[57]

Patton MQ (2014) Qualitative research & evaluation methods: integrating theory and practice. Sage Publications

[58]

Paulson JW, Succi G, and Eberlein A An empirical study of open-source and closed-source software products IEEE Trans Softw Eng 2004 30 4 246-256

[59]

Ralph P Toward methodological guidelines for process theories and taxonomies in software engineering IEEE Trans Softw Eng 2019 45 7 712-735

[60]

Ralph P, Ali Nb, Baltes S, Bianculli D, Diaz J, Dittrich Y, Ernst N, Felderer M, Feldt R, Filieri A et al (2020a) Empirical standards for software engineering research. arXiv:201003525

[61]

Ralph P, Baltes S, Adisaputri G, Torkar R, Kovalenko V, Kalinowski M, Novielli N, Yoo S, Devroey X, Tan X et al (2020b) Pandemic programming: how covid-19 affects software developers and how their organizations can help. Empir Softw Eng

[62]

Russo D, Stol K (in press) Gender differences in personality traits of software engineers. IEEE Trans Softw Eng

[63]

Salleh N, Hoda R, Su MT, Kanij T, and Grundy J Recruitment, engagement and feedback in empirical software engineering studies in industrial contexts Inform Software Technol 2018 98 161-172

[64]

Sax LJ, Gilmartin SK, and Bryant AN Assessing response rates and nonresponse bias in web and paper surveys Res High Educ 2003 44 4 409-432

[65]

Sedano T, Ralph P, Péraire C (2019) The product backlog. In: 2019 IEEE/ACM 41St international conference on software engineering (ICSE), IEEE, pp 200–211

[66]

Sjøberg D, Anda B, Arisholm E, Dyba T, Jørgensen M, Karahasanovic A, Koren EF, Vokac M (2002) Conducting realistic experiments in software engineering. In: 2002 international symposium on empirical software engineering. IEEE, Nara, Japan, pp 17–26, DOI, (to appear in print)

[67]

Stol KJ and Fitzgerald B The abc of software engineering research ACM Transactions on Software Engineering and Methodology (TOSEM) 2018 27 3 11

[68]

Stol KJ, Ralph P, Fitzgerald B (2016) Grounded theory in software engineering research: a critical review and guidelines. In: Proceedings of the international conference on software engineering, IEEE, Austin, TX, USA, pp 120–131

[69]

Tempero E, Anslow C, Dietrich J, Han T, Li J, Lumpe M, Melton H, Noble J (2010) The qualitas corpus: a curated collection of java code for empirical studies. In: Proceedings of the 17th Asia Pacific software engineering conference. IEEE, Sydney, Australia, pp 336–345, DOI, (to appear in print)

[70]

Theisen C, Dunaiski M, Williams L, and Visser W Software engineering research at the international conference on software engineering in 2016 ACM SIGSOFT Software Engineering Notes 2018 42 4 1-7

[71]

Thomas G, Myers K (2015) The anatomy of the case study. Sage

[72]

Thompson SK Adaptive cluster sampling J Am Stat Assoc 1990 85 412 1050-1059

[73]

Toepoel V (2012) Effects of incentives in surveys. In: Gideon L (ed) Handbook of survey methodology for the social sciences, springer, pp 209–223

[74]

Torchiano M, Fernández DM, Travassos GH, de Mello RM (2017) Lessons learnt in conducting survey research. In: 2017 IEEE/ACM 5th international workshop on Conducting Empirical Studies in Industry (CESI), IEEE, pp 33–39

[75]

Trochim WM and Donnelly JP Research methods knowledge base, vol 2 2001 Cincinnati, OH, USA Atomic Dog Publishing

[76]

Trost JE Statistically nonrepresentative stratified sampling: a sampling technique for qualitative studies Qual Sociol 1986 9 1 54-57

[77]

Turk P and Borkowski JJ A review of adaptive cluster sampling: 1990–2003 Environ Ecol Stat 2005 12 1 55-94

[78]

Valliant R, Dever JA, Kreuter F (2018) Designing multistage samples. In: Practical tools for designing and weighting survey samples, Springer, pp 209–264

[79]

Vasilescu B, Posnett D, Ray B, van den Brand MG, Serebrenik A, Devanbu P, Filkov V (2015) Gender and Tenure Diversity in GitHub Teams. In: Proceedings of the 33rd annual ACM conference on human factors in computing systems - CHI ’15. ACM Press, Seoul, Republic of Korea, pp 3789–3798, DOI, (to appear in print)

[80]

Wohlin C, Runeson P, Höst M, Ohlsson MC, Regnell B, Wesslén A (2012) Experimentation in software engineering. Springer Science & Business Media

[81]

Yin RK Case study research: Design and methods 2018 6th edn. California Sage, Thousand Oaks

[82]

Zannier C, Melnik G, Maurer F (2006) On the success of empirical studies in the international conference on software engineering. In: Proceedings of the 28th international conference on software engineering, pp 341–350

[83]

Zhang H, Huang X, Zhou X, Huang H, Babar MA (2019) Ethnographic research in software engineering: a critical review and checklist. In: Proceedings of the 2019 27th ACM joint meeting on European software engineering conference and symposium on the foundations of software engineering, pp 659–670

Cited By

Habiba UHabib MBogner JFritzsch JWagner S(2025)How do ML practitioners perceive explainability? an interview study of practices and challengesEmpirical Software Engineering10.1007/s10664-024-10565-230:1Online publication date: 1-Feb-2025
https://dl.acm.org/doi/10.1007/s10664-024-10565-2
Butler AFilkov VRay BZhou M(2024)Software Supply Chain Risk: Characterization, Measurement & AttenuationProceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering10.1145/3691620.3695608(2506-2509)Online publication date: 27-Oct-2024
https://dl.acm.org/doi/10.1145/3691620.3695608
van Heijningen SWiik TNeto FGay GViggedal KFriberg DFilkov VRay BZhou M(2024)Integrating Mutation Testing Into Developer Workflow: An Industrial Case StudyProceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering10.1145/3691620.3695273(2110-2120)Online publication date: 27-Oct-2024
https://dl.acm.org/doi/10.1145/3691620.3695273
Show More Cited By

Recommendations

There is no random sampling in software engineering research
ICSE '18: Proceedings of the 40th International Conference on Software Engineering: Companion Proceeedings

Representative sampling is considered crucial for predominately quantitative, positivist research. Researchers typically argue that a sample is representative when items are selected randomly from a population. However, random sampling is rare in ...
Applying Statistical Sampling for Fast and Efficient Simulation of Commercial Workloads

Commercial workloads form an important class of applications and have performance characteristics that are distinct from scientific and technical benchmarks such as SPEC CPU. However, due to the prohibitive simulation time of commercial workloads, it is ...
Sampling improvement in software engineering surveys
ESEM '14: Proceedings of the 8th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement

Context: Small and non-probabilistic samples represent relevant issues when discussing the external validity of empirical studies in Software Engineering. Goal: To investigate alternatives to improve the quality of samples (size, heterogeneity and level ...

Comments

Information & Contributors

Information

Published In

cover image Empirical Software Engineering

Empirical Software Engineering Volume 27, Issue 4

Jul 2022

848 pages

ISSN:1382-3256

Issue’s Table of Contents

© The Author(s), under exclusive licence to Springer Science+Business Media, LLC, part of Springer Nature 2022.

Publisher

Kluwer Academic Publishers

United States

Publication History

Published: 01 July 2022

Accepted: 28 October 2021

Author Tags

Qualifiers

Research-article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

57
Total Citations
View Citations
0
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 13 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Habiba UHabib MBogner JFritzsch JWagner S(2025)How do ML practitioners perceive explainability? an interview study of practices and challengesEmpirical Software Engineering10.1007/s10664-024-10565-230:1Online publication date: 1-Feb-2025
https://dl.acm.org/doi/10.1007/s10664-024-10565-2
Butler AFilkov VRay BZhou M(2024)Software Supply Chain Risk: Characterization, Measurement & AttenuationProceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering10.1145/3691620.3695608(2506-2509)Online publication date: 27-Oct-2024
https://dl.acm.org/doi/10.1145/3691620.3695608
van Heijningen SWiik TNeto FGay GViggedal KFriberg DFilkov VRay BZhou M(2024)Integrating Mutation Testing Into Developer Workflow: An Industrial Case StudyProceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering10.1145/3691620.3695273(2110-2120)Online publication date: 27-Oct-2024
https://dl.acm.org/doi/10.1145/3691620.3695273
Gordon CEdwards JTaeumel M(2024)The Linguistics of ProgrammingProceedings of the 2024 ACM SIGPLAN International Symposium on New Ideas, New Paradigms, and Reflections on Programming and Software10.1145/3689492.3689806(162-182)Online publication date: 17-Oct-2024
https://dl.acm.org/doi/10.1145/3689492.3689806
Berndt ABach TBaltes S(2024)Do Test and Environmental Complexity Increase Flakiness? An Empirical Study of SAP HANAProceedings of the 18th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement10.1145/3674805.3695407(572-581)Online publication date: 24-Oct-2024
https://dl.acm.org/doi/10.1145/3674805.3695407
de Souza Santos Rde Lima LBaldassarre MSpínola R(2024)Preliminary Insights on Industry Practices for Addressing Fairness DebtProceedings of the 18th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement10.1145/3674805.3695406(566-571)Online publication date: 24-Oct-2024
https://dl.acm.org/doi/10.1145/3674805.3695406
Gorostidi JAit ACabot JCanovas Izquierdo J(2024)On the Creation of Representative Samples of Software RepositoriesProceedings of the 18th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement10.1145/3674805.3690747(434-439)Online publication date: 24-Oct-2024
https://dl.acm.org/doi/10.1145/3674805.3690747
Silva KBarcomb ASantos Rd'Amorim M(2024)Paths to Testing: Why Women Enter and Remain in Software Testing?Companion Proceedings of the 32nd ACM International Conference on the Foundations of Software Engineering10.1145/3663529.3663822(4-9)Online publication date: 10-Jul-2024
https://dl.acm.org/doi/10.1145/3663529.3663822
Robillard MArya DErnst NGuo JLamothe MNassif MNovielli NSerebrenik ASteinmacher IStol K(2024)Communicating Study Design Trade-offs in Software EngineeringACM Transactions on Software Engineering and Methodology10.1145/364959833:5(1-10)Online publication date: 4-Jun-2024
https://dl.acm.org/doi/10.1145/3649598
Le TDu XBabar MSpinellis DConstantinou EBacchelli A(2024)Are Latent Vulnerabilities Hidden Gems for Software Vulnerability Prediction? An Empirical StudyProceedings of the 21st International Conference on Mining Software Repositories10.1145/3643991.3644919(716-727)Online publication date: 15-Apr-2024
https://dl.acm.org/doi/10.1145/3643991.3644919
Show More Cited By

View Options

View options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Media

Figures

Other

Tables

View Issue’s Table of Contents