research-article

Sampling improvement in software engineering surveys

Authors:

Rafael Maiani de Mello,

Pedro Correa da Silva,

Guilherme Horta TravassosAuthors Info & Claims

ESEM '14: Proceedings of the 8th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement

Article No.: 13, Pages 1 - 4

https://doi.org/10.1145/2652524.2652566

Published: 18 September 2014 Publication History

Get Access

Abstract

Context: Small and non-probabilistic samples represent relevant issues when discussing the external validity of empirical studies in Software Engineering. Goal: To investigate alternatives to improve the quality of samples (size, heterogeneity and level of confidence). Method: To replicate a survey on characteristics of agility in software processes by applying a systematic recruitment strategy over a professional social network. Results: It resulted in a sampling frame composed by 19 groups stratified according two perspectives: sharing of groups' members and main software engineering skills reported by the subjects. In total, 7,745 subjects were randomly recruited, resulting in 291 contributions. Conclusions: This sample was significantly larger, more heterogeneous and presents some strata with higher confidence levels than previous trials samples.

References

[1]

Sjøberg, D. I. et al. 2005. A survey of controlled experiments in software engineering. IEEE Transactions on Software Engineering, 31(9): 733--753.

Digital Library

Google Scholar

[2]

Dybå, T., Kampenes, V. B. and Sjøberg, D. I. K. 2007. A systematic review of statistical power in software engineering experiments. Inf. and Soft. Tech. 48: 745--755.

Crossref

Google Scholar

[3]

Conradi, R. et al. 2005. Reflections on conducting an international survey of Software Engineering. In Intl. Symposium on Empirical Software Engineering, 214--223.

Google Scholar

[4]

Santos, G. et al. 2012. MPS. BR Program and MPS Model: Main Results, Benefits and Beneficiaries of Software Process Improvement in Brazil. In Proc. of Intl. Conf. on Quality of Inf. and Comm. Technology (QUATIC), 137--142. IEEE.

Digital Library

Google Scholar

[5]

de Mello, R. M. and Travassos, G. H. 2103. Would Sociable Software Engineers Observe Better? In 7th ESEM, IEEE.

Google Scholar

[6]

Abrantes, J. F. and Travassos, G. H. 2013. Towards Pertinent Characteristics of Agility and Agile Practices for Software Processes. CLEI Electronic Journal 16(1).

Google Scholar

[7]

Fernández-Sáez, A. M. et al. 2013. Are forward designed or reverse-engineered UML diagrams more helpful for code maintenance?: a controlled experiment. In: Proc. of 17th EASE, 60--71, ACM.

Digital Library

Google Scholar

[8]

de Mello, R. M., da Silva, P. C. and Travassos, G. H. 2014. Investigating Probabilistic Sampling Approaches for Large-Scale Surveys in Software Engineering. In: Proc. of 11th Workshop on Experimental Software Engineering(ESELAW).

Google Scholar

[9]

Lahlou, Saadi. 1996. A method to extract social representations from linguistic corpora. Japanese journal of experimental social psychology, 35 (3). pp. 278--391.

Google Scholar

[10]

Bauer, M. W., Gaskell, G. 1999. Towards a paradigm for research on social representations. Journal for the theory of social behaviour, 29(2), 163--186.

Crossref

Google Scholar

[11]

Hair, J. F. et al. 2009. Multivariate Data Analysis, 7^th ed. Prentice Hall.

Google Scholar

Cited By

View all

Davis MAghayi ELatoza TWang XMyers BSunshine J(2023)What’s (Not) Working in Programmer User Studies?ACM Transactions on Software Engineering and Methodology10.1145/358715732:5(1-32)Online publication date: 24-Jul-2023
https://dl.acm.org/doi/10.1145/3587157
Elfadil NIbrahim I(2022)Embedded System Design Student’s Learning Readiness Instruments: Systematic Literature ReviewFrontiers in Education10.3389/feduc.2022.7996837Online publication date: 18-Feb-2022
https://doi.org/10.3389/feduc.2022.799683
Garousi VFelderer MKuhrmann MHerkiloğlu KEldh S(2020)Exploring the industry's challenges in software testingJournal of Software: Evolution and Process10.1002/smr.225132:8Online publication date: 3-Aug-2020
https://dl.acm.org/doi/10.1002/smr.2251
Show More Cited By

Index Terms

Sampling improvement in software engineering surveys
1. Theory of computation
  1. Semantics and reasoning
    1. Program reasoning
      1. Program verification

Recommendations

Surveys in Software Engineering: Identifying Representative Samples
ESEM '16: Proceedings of the 10th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement

Context: The representativeness of samples in Software Engineering primary studies is still a great challenge, mainly when identifying available sources for establishing adequate sampling frames, characterizing subjects and stimulating their ...
Towards a framework to support large scale sampling in software engineering surveys
ESEM '14: Proceedings of the 8th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement

Context: The low quality and small size of samples in empirical studies in software engineering hamper the interpretation and generalization of their results. Therefore, enlarging sample sizes and improving their quality represent an important research ...
Sampling in software engineering research: a critical review and guidelines
Abstract
Representative sampling appears rare in empirical software engineering research. Not all studies need representative samples, but a general lack of representative sampling undermines a scientific field. This article therefore reports a critical ...

Comments

Information & Contributors

Information

Published In

ESEM '14: Proceedings of the 8th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement

September 2014

461 pages

ISBN:9781450327749

DOI:10.1145/2652524

General Chair:
Maurizio Morisio
Politecnico di Torino, Italy
,
Program Chairs:
Tore Dybå
Sintef, Norway
,
Marco Torchiano
Politecnico di Torino, Italy

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 18 September 2014

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

ESEM '14

Sponsor:

SIGSOFT

ESEM '14: 2014 ACM-IEEE International Symposium on Empirical Software Engineering and Measurement

September 18 - 19, 2014

Torino, Italy

Acceptance Rates

ESEM '14 Paper Acceptance Rate 23 of 123 submissions, 19%;

Overall Acceptance Rate 130 of 594 submissions, 22%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

9
Total Citations
View Citations
228
Total Downloads

Downloads (Last 12 months)13
Downloads (Last 6 weeks)1

Reflects downloads up to 04 Sep 2024

Other Metrics

View Author Metrics

Citations

Cited By

View all

Davis MAghayi ELatoza TWang XMyers BSunshine J(2023)What’s (Not) Working in Programmer User Studies?ACM Transactions on Software Engineering and Methodology10.1145/358715732:5(1-32)Online publication date: 24-Jul-2023
https://dl.acm.org/doi/10.1145/3587157
Elfadil NIbrahim I(2022)Embedded System Design Student’s Learning Readiness Instruments: Systematic Literature ReviewFrontiers in Education10.3389/feduc.2022.7996837Online publication date: 18-Feb-2022
https://doi.org/10.3389/feduc.2022.799683
Garousi VFelderer MKuhrmann MHerkiloğlu KEldh S(2020)Exploring the industry's challenges in software testingJournal of Software: Evolution and Process10.1002/smr.225132:8Online publication date: 3-Aug-2020
https://dl.acm.org/doi/10.1002/smr.2251
Venson EAlfayez RGomes MFigueiredo RBoehm B(2019)The Impact of Software Security Practices on Development Effort: An Initial Survey2019 ACM/IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM)10.1109/ESEM.2019.8870153(1-12)Online publication date: Sep-2019
https://doi.org/10.1109/ESEM.2019.8870153
de Mello RUchôa AOliveira ROliveira DFonseca BGarcia Ade Mello FDittrich YFagerholm FHoda RSocha DSteinmacher I(2019)Investigating the social representations of code smell identificationProceedings of the 12th International Workshop on Cooperative and Human Aspects of Software Engineering10.1109/CHASE.2019.00022(53-60)Online publication date: 27-May-2019
https://dl.acm.org/doi/10.1109/CHASE.2019.00022
Garousi VPfahl DFernandes JFelderer MMäntylä MShepherd DArcuri ACoşkunçay ATekinerdogan B(2019)Characterizing industry-academia collaborations in software engineeringEmpirical Software Engineering10.1007/s10664-019-09711-y24:4(2540-2602)Online publication date: 1-Aug-2019
https://dl.acm.org/doi/10.1007/s10664-019-09711-y
de Mello RTravassos GGenero MJedlitschka AJørgensen MScanniello GSampath SCaivano DPort D(2016)Surveys in Software EngineeringProceedings of the 10th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement10.1145/2961111.2962632(1-6)Online publication date: 8-Sep-2016
https://dl.acm.org/doi/10.1145/2961111.2962632
Molléri JPetersen KMendes EGenero MJedlitschka AJørgensen MScanniello GSampath SCaivano DPort D(2016)Survey Guidelines in Software EngineeringProceedings of the 10th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement10.1145/2961111.2962619(1-6)Online publication date: 8-Sep-2016
https://dl.acm.org/doi/10.1145/2961111.2962619
de Mello RStolee KTravassos G(2015)Investigating Samples Representativeness for an Online Experiment in Java Code Search2015 ACM/IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM)10.1109/ESEM.2015.7321205(1-10)Online publication date: Oct-2015
https://doi.org/10.1109/ESEM.2015.7321205

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Cited By

Index Terms

Recommendations

Surveys in Software Engineering: Identifying Representative Samples

Towards a framework to support large scale sampling in software engineering surveys

Sampling in software engineering research: a critical review and guidelines