research-article

Investigations about replication of empirical studies in software engineering: preliminary findings from a mapping study

Authors:

Cleyton V. C. de Magalhães,

Fabio Q. B. da Silva,

Ronnie E. S. SantosAuthors Info & Claims

EASE '14: Proceedings of the 18th International Conference on Evaluation and Assessment in Software Engineering

Article No.: 37, Pages 1 - 10

https://doi.org/10.1145/2601248.2601289

Published: 13 May 2014 Publication History

Abstract

Context. A recent mapping study intended to verify the current state of replication of empirical studies in Software Engineering (SE) identified two sets of studies: the empirical studies reporting actually replications and a second group of studies that were concerned with concepts, classifications, guidelines, and other themes about replication. Objective. The purpose of this research is to analyze and discuss the content of the second set of studies about replications published between 1996 and 2013. Method. This is a mapping study, in which the primary studies were collected by two previous mapping studies covering the period 1996-2012 complemented by a non-systematic search that collected some articles published in 2013. Results. We analyzed 36 papers reporting empirical and non-empirical studies about replications in SE published in the last 17 years. These papers explored different topics related to concepts and classifications, presented guidelines, and discussed theoretical issues that are relevant for our understanding of replication in our field.

References

[1]

Popper, K. 1959. The Logic of Scientific Discovery. Hutchinson & Co. p. 45.

[2]

Basili, V., Shull, F. and Lanubile, F., 1999. Building knowledge through families of experiments, IEEE Transactions on Software Engineering 25 (1999) 456--473.

Digital Library

[3]

Gómez, O. S., Juristo, N. and Vegas, S. 2010. "Replications Types in Experimental Disciplines" In: Proceedings of the ACM/IEEE 4th International Symposium on Empirical Software Engineering and Measurement, September 16--17, Bolzano-Bozen, Italy. pp. 1--10.

Digital Library

[4]

Shull, F., Carver, J., Vegas, S. and Juristo, N. 2008. The role of replications in empirical Software Engineering. Empirical Software Engineering, 13(2):211--218, 2008. DOI= 10.1007/s10664-008-9060-1.

Digital Library

[5]

Juristo, N and Vegas S. 2011. The role of non-exact replications in Software Engineering experiments. Empirical Software Engineering. 16, 295--324. DOI= 10.1007/s10664-010-9141-9

Digital Library

[6]

Carver, J. Towards Reporting Guidelines for Experimental Replications: A Proposal. 2010. In Proceedings of the 1st International Workshop on Replication in Empirical Software Engineering Research, Cape Town, South Africa.

[7]

De Magalhaes, C. V. C and Da Silva, F. Q. B. 2013. Towards a Taxonomy of Replications in Empirical Software Engineering Research: A Research Proposal. 2013. In: Proceedings of the 3rd International Workshop on Replication in Empirical Software Engineering Research (RESER). DOI= 10.1109/RESER.2013.10.

Digital Library

[8]

Easterbrook, S. M., Singer, J., M.-A. Storey, S. M. and Damian, D. 2007. Selecting empirical methods for Software Engineering research. Guide to Advanced Empirical Software Engineering, pages 285--311. Springer.

[9]

Shull, F., Basili, V., Carver, J., Maldonado, J. C., Travassos, G. H., Mendonça, M. and Fabbri, S. 2002. Replicating Software Engineering experiments: Addressing the tacit knowledge problem. In: Proceedings of Int. Symp. on Empirical Softw. Eng., Washington, DC, USA, IEEE Computer Society

Digital Library

[10]

Lung, J., Aranda J., Easterbrook, S. and Wilson, G. 2008. On the difficulty of replicating human subjects studies in Software Engineering. In Proceedings of the 13th international conference on Software Engineering - ICSE '08, New York, USA: ACM Press, p. 191.

Digital Library

[11]

da Silva, F. Q. B., Suassuna, M., França, A. C. C., Grubb, A. M., Gouveia, T. B., Monteiro, C. V. F., dos Santos, I. E. 2012. Replication of empirical studies in Software Engineering research: a systematic mapping study. Empirical Software Engineering (Dordrecht. Online). DOI= 10.1007/s10664-012-9227-7.

Digital Library

[12]

Arksey, H. and O'Malley, L. 2005. Scoping studies: towards a methodological framework. Int J Social Research Methodology. 8(1):19--32.

[13]

da Silva, F. Q. B., Santos, A. L. M., Soares, S. C. B., França, A. C. C., Monteiro, C. V. F. and Maciel, F. F. 2011. Six years of systematic literature reviews in Software Engineering: An updated tertiary study. Information and Software Technology 53, 899--913.

Digital Library

[14]

Petticrew, M. and Roberts, H. 2006. Systematic Reviews in the Social Sciences. Blackwell Publishing. 336

[15]

Kitchenham, B. and Charters, S. Guidelines for performing systematic literature reviews in Software Engineering. 2007. Technical Report EBSE-2007-01, School of Computer Science and Mathematics, Keele University.

[16]

Almqvist, J. F. P. 2006. Replication of controlled experiments in empirical Software Engineering -- a survey. 2006. Master's thesis, Department of Computer Science, Faculty of Science, Lund University, Sweden.

[17]

Krein, J. L. and Knutson, C. D. 2010. A Case for Replication: Synthesizing Research Methodologies in Software Engineering. In: 1st International Workshop on Replication in Empirical Software Engineering Research, RESER 2010

[18]

{ABO001} Mendonça, M., Maldonado, J., Oliveira, M., Carver, J., Fabbri, S., Shull, F., Travassos, G., Höhn, E., and Basili, V. 2008. A Framework for Software Engineering Experimental Replications. In Proceedings of the 13th International Conference on Engineering Complex Computer Systems (ICECCS 2008). March 31 - April 4, 2008. Belfast, Northern Ireland. p. 203--212.

Digital Library

[19]

{ABO002} Cater-Steel, A., Toleman, M. and Rout, T. 2005. Addressing the challenges of replications of surveys in Software Engineering research. In: International Symposium on Empirical Software Engineering, 17--18 Nov 2005, Noosa Heads, Australia.

[20]

{ABO003} Vegas, S., Juristo N., Moreno, A. M., Solari, M. and Letelier, P. 2006. Analysis of the influence of communication between researchers on experiment replication. In Proceedings of the 2006 ACM/IEEE international symposium on Empirical software engineering: 28--37. New York, NY, USA. DOI= 10.1145/1159733.1159741

Digital Library

[21]

{ABO004} Basili, V., Shull, F. and Lanubile, F., 1999. Building knowledge through families of experiments, IEEE Transactions on Software Engineering. 25 (1999) 456--473.

Digital Library

[22]

{ABO005} Kitchenham, B., Al-Khilidar, H., Ali Babar, M., Berry, M., Cox, C., Keung, J., Kurniawati, F., Staples, M., Zhang, H. and Zhu, L. 2006. Evaluating Guidelines for Empirical Software Engineering Studies; In Proc. of ACM/IEEE Intern. Symposium on Software Engineering, Rio de Janerio, Brazil, Sep 2006, IEEE CS.

Digital Library

[23]

{ABO006} Ohlsson M. C. and Runeson P. 2002. "Experience from replicating empirical studies on prediction models", In: Proceedings of the 8th International Software Metrics Symposium, Ottawa, Ontario, Canada, IEEE Computer Society, June 2002, pp. 217--226.

Digital Library

[24]

{ABO007} Shull, F., Mendonça, M. G., Basili, V., Carver, J., Maldonado, J. C., Fabbri, S., Travassos, G. H., and Ferreira, M. C. 2004. Knowledge-Sharing Issues in Experimental Software Engineering. Empirical Software Engineering, 9(1--2):111--137, March 2004.

Digital Library

[25]

{ABO008} Robles, G. 2010. Replicating MSR: A study of the potential replicability of papers published in the Mining Software Repositories proceedings. In: 7th IEEE Working Conference on Mining Software Repositories (MSR 2010). Cape Town, South Africa: IEEE; p. 171--80

[26]

{ABO009} Mende, T. 2010. Replication of Defect Prediction Studies: Problems, Pitfalls and Recommendations. In: Proceedings of the International Conference on Predictive Models in Software Engineering, ACM Press; p. 1--10.

Digital Library

[27]

{ABO010} Gomez, O. S., Juristo, N. and Vegas, S. 2010. Replications types in experimental disciplines. In: Proceedings of International Symposium on Empirical Software Engineering and Measurement. DOI= 10.1145/1852786.1852790.

Digital Library

[28]

{ABO011} Juristo, N. and Vegas, S. 2011. The role of non-exact replications in Software Engineering experiments. Empirical Software Engineering 16(3): 295--324. DOI= 10.1007/s10664-010-9141-9.

Digital Library

[29]

{ABO012} Shull F., Carver, J. C., Vegas, S. and Juristo, N. 2008. The role of replications in Empirical Software Engineering. Empirical Software Engineering 13(2): 211--218, DOI= 10.1007/s10664-008-9060-1.

Digital Library

[30]

{ABO013} Kitchenham, B. 2008. The role of replications in empirical Software Engineering - a word of warning. Empirical Software Engineering 13(2): 219--221. DOI= 10.1007/s10664-008-9061-0.

Digital Library

[31]

{ABO014} Juristo, N. J. and Vegas, S. 2010. Using differences among replications of Software Engineering experiments to gain knowledge. In: Proceedings of the 2009 3rd International Symposium on Empirical Software Engineering and Measurement, Cape Town, South Africa. DOI= 10.1109/ESEM.2009.5314236.

Digital Library

[32]

{ABO015} Shull, F., Basili, V., Carver, J., Maldonado, J. C., Travassos, G. H., Mendonça, M., and Fabbri, S. 2002. Replicating Software Engineering Experiments: Addressing the Tacit Knowledge Problem. In: Proceedings of the 2002 International Symposium on Empirical Software Engineering, p. 7--16. Nara, Japan, October 3-4 2002.

Digital Library

[33]

{ABO016} Shull, F., Carver, J., Travassos, G., Maldonado, J., Conradi, R., and Basili, V. 2003. Replicated Studies: Building a Body of Knowledge about Software Reading Techniques. Lecture Notes on Empirical Software Engineering. World Scientific.

Digital Library

[34]

{ABO017} Miller, J. 2005. Replicating Software Engineering Experiments: A Poisoned Chalice or the Holy Grail, Information and Software Technology, vol. 47, no. 4, pp. 233--244.

Digital Library

[35]

{ABO018} Carver, J. 2010. Towards Reporting Guidelines for Experimental Replications: A Proposal. In: Proceedings of the 1st International Workshop on Replication in Empirical Software Engineering Research (RESER) {Held during ICSE 2010}. May 4, 2010. Cape Town, South Africa.

[36]

{ABO019} Dieste, O., Fernandez, E., Garcia, R., and Juristo, N. 2010. Hidden evidence behind useless replications. In: Proceedings of the 1st International Workshop on Replication in Empirical Software Engineering Research (RESER) {Held during ICSE 2010}. May 4, 2010. Cape Town, South Africa.

[37]

{ABO020} Ferrari, R., Sudmann, O., Henke, C., Geisler, J., Schäfer, W. and Madhavji, N. H. Transitioning from lab studies to large-scale studies: Emerging results from a literal replication. In Proceedings of the 1st International Workshop on Replication in Empirical Software Engineering Research (RESER) {Held during ICSE 2010}. May 4, 2010. Cape Town, South Africa.

[38]

{ABO021} Gomez O. S., Juristo, N., Vegas, S. 2010. Replication, reproduction and re-analysis: three ways for verifying experimental findings. In: Proceedings of the 1st International Workshop on Replication in Empirical Software Engineering Research (RESER) {Held during ICSE 2010}. May 4, 2010. Cape Town, South Africa.

[39]

{ABO022} Jonathan L. Krein and Charles D. Knutson. 2010. A Case for Replication: Synthesizing Research Methodologies in Software Engineering. In: Proceedings of the 1st International Workshop on Replication in Empirical Software Engineering Research (RESER) {Held during ICSE 2010}. May 4, 2010. Cape Town, South Africa.

[40]

{ABO023} Mäntylä, M. V., Lassenius, C. and Vanhanen, J., 2010. Rethinking Replication in Software Engineering: Can We See the Forest for the Trees? In: Proceedings of the 1st International Workshop on Replication in Empirical Software Engineering Research (RESER) {Held during ICSE 2010}. May 4, 2010. Cape Town, South Africa.

[41]

{ABO024} A. Mockus, B. Anda and D. I. K. Sjøberg. 2010. Experiences from Replicating a Case Study to Investigate Reproducibility of Software Development. In: Proceedings of the 1st International Workshop on Replication in Empirical Software Engineering Research (RESER) {Held during ICSE 2010}. May 4, 2010. Cape Town, South Africa.

[42]

{ABO025} Robles, G. and German, D. M. 2010. Beyond Replication: An example of the potential benefits of replicability in the Mining of Software Repositories Community. In: Proceedings of the 1st International Workshop on Replication in Empirical Software Engineering Research (RESER) {Held during ICSE 2010}. May 4, 2010. Cape Town, South Africa.

[43]

{ABO026} Juristo, N., Vegas, S. 2011. The role of non-exact replications in Software Engineering experiments. Empirical Software Engineering June 2011, Vol. 16, Issue 3, p. 295--324. DOI= 10.1007/s10664-010-9141-9.

Digital Library

[44]

{ABO027} Juristo, N. and Gómez, O. 2012. Replication of Software Engineering Experiments. Empirical Software Engineering and Verification. Lecture Notes in Computer Science Volume 7007, 2012, pp 60--88. DOI= 10.1007/978-3-642-25231-0_2.

Digital Library

[45]

{ABO028} Scatalon, L. P., Garcia R. E., Correia, R. C. M. 2011. Packaging Controlled Experiments Using an Evolutionary Approach Based on Ontology(S). In: Proceedings of the 23rd International Conference on Software Engineering & Knowledge Engineering (SEKE'2011), Eden Roc Renaissance, Miami Beach, USA, July 7--9, 2011

[46]

{ABO029} Gallardo, E. G. E. 2012. Using configuration management and product line software paradigms to support the experimentation process in Software Engineering. In: 6th International Conference on Research Challenges in Information Science (RCIS 2012) p. 1--6, Valencia, Spain.

[47]

{ABO030} De Magalhaes, C. V. C and Da Silva, F. Q. B. 2013. Towards a Taxonomy of Replications in Empirical Software Engineering Research: A Research Proposal. In: Proceedings of the 3rd International Workshop on Replication in Empirical Software Engineering Research (RESER 2013). Baltimore, MD. DOI= 10.1109/RESER.2013.10

Digital Library

[48]

{ABO031} Ghezzi, G. and Gall H. C. 2013. Replicating mining studies with SOFAS. In: Proceedings of the 10th Working Conference on Mining Software Repositories. p. 363--372.

Digital Library

[49]

{ABO032} Callele, D., Wnuk, K. and Borg, M. 2013. Confounding factors when conducting industrial replications in requirements engineering. In: 1st International Workshop on Conducting Empirical Studies in Industry (CESI). p. 55--58. DOI= 10.1109/CESI.2013.6618472

[50]

{ABO033} Squire, M. 2013. A Replicable Infrastructure for Empirical Studies of Email Archives. In: Proceedings of the 3rd International Workshop on Replication in Empirical Software Engineering Research (RESER 2013). Baltimore, MD. p. 43--49. DOI= 10.1109/RESER.2013.11.

Digital Library

[51]

{ABO034} Juristo, N., Vegas, S., Solari, M., Abrahão, S., and Ramos, I. 2013. A process for managing interaction between experimenters to get useful similar replications. Information & Software Technology 55(2): 215--225

Digital Library

[52]

{ABO035} Runeson, P., Stefik, A. and Andrews, A. 2013. Variation factors in the design and analysis of replicated controlled experiments. Empirical Software Engineering. Springer US. DOI= 10.1007/s10664-013-9262-z.

[53]

{ABO036} Brooks, A., Daly, J., Miller, J., Roper, M. and Wood, M. 1996. Replication of Experimental Results in Software Engineering. Technical Report ISERN-96-10, Department of Computer Science, University of Strathclyde, Glasgow, Scotland, UK, 1996

Cited By

Panach JDieste OMarin BEspana SVegas SPastor OJuristo N(2021)Evaluating Model-Driven Development Claims with Respect to Quality: A Family of ExperimentsIEEE Transactions on Software Engineering10.1109/TSE.2018.288470647:1(130-145)Online publication date: 1-Jan-2021
https://doi.org/10.1109/TSE.2018.2884706
Charalampidou SAmpatzoglou AKarountzos EAvgeriou P(2021)Empirical studies on software traceabilityJournal of Software: Evolution and Process10.1002/smr.229433:2Online publication date: 14-Feb-2021
https://dl.acm.org/doi/10.1002/smr.2294
Kochhar PLo DMendes ECounsell SPetersen K(2017)Revisiting Assert Use in GitHub ProjectsProceedings of the 21st International Conference on Evaluation and Assessment in Software Engineering10.1145/3084226.3084259(298-307)Online publication date: 15-Jun-2017
https://dl.acm.org/doi/10.1145/3084226.3084259
Show More Cited By

Index Terms

Investigations about replication of empirical studies in software engineering: preliminary findings from a mapping study
1. Software and its engineering

Recommendations

Support mechanisms to conduct empirical studies in software engineering
ESEM '14: Proceedings of the 8th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement

Context: Empirical studies are gaining recognition in the Software Engineering (SE) research community. In order to foster empirical research, it is essential understand the environments, guidelines, process, and other mechanisms available to support ...
Replication of Empirical Studies in Software Engineering: Preliminary Findings from a Systematic Mapping Study
RESER '11: Proceedings of the 2011 Second International Workshop on Replication in Empirical Software Engineering Research

Our goal in this study is to review the research related to the replication of empirical studies in software engineering in terms of replications of empirical studies and conceptual or theoretical work about replications. In this article we present the ...
Investigations about replication of empirical studies in software engineering

ContextTwo recent mapping studies which were intended to verify the current state of replication of empirical studies in Software Engineering (SE) identified two sets of studies: empirical studies actually reporting replications (published between 1994 ...

Comments

Information & Contributors

Information

Published In

cover image ACM Other conferences

EASE '14: Proceedings of the 18th International Conference on Evaluation and Assessment in Software Engineering

May 2014

486 pages

ISBN:9781450324762

DOI:10.1145/2601248

General Chair:
Martin Shepperd
Brunel University, UK
,
Program Chairs:
Tracy Hall
Brunel University
,
Ingunn Myrtveit
The Norwegian Business School

Copyright © 2014 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Brunel University: Brunel University

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 13 May 2014

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

Conference

EASE '14

Sponsor:

Brunel University

EASE '14: 18th International Conference on Evaluation and Assessment in Software Engineering

May 13 - 14, 2014

England, London, United Kingdom

Acceptance Rates

Overall Acceptance Rate 71 of 232 submissions, 31%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

8
Total Citations
View Citations
238
Total Downloads

Downloads (Last 12 months)18
Downloads (Last 6 weeks)1

Reflects downloads up to 24 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Panach JDieste OMarin BEspana SVegas SPastor OJuristo N(2021)Evaluating Model-Driven Development Claims with Respect to Quality: A Family of ExperimentsIEEE Transactions on Software Engineering10.1109/TSE.2018.288470647:1(130-145)Online publication date: 1-Jan-2021
https://doi.org/10.1109/TSE.2018.2884706
Charalampidou SAmpatzoglou AKarountzos EAvgeriou P(2021)Empirical studies on software traceabilityJournal of Software: Evolution and Process10.1002/smr.229433:2Online publication date: 14-Feb-2021
https://dl.acm.org/doi/10.1002/smr.2294
Kochhar PLo DMendes ECounsell SPetersen K(2017)Revisiting Assert Use in GitHub ProjectsProceedings of the 21st International Conference on Evaluation and Assessment in Software Engineering10.1145/3084226.3084259(298-307)Online publication date: 15-Jun-2017
https://dl.acm.org/doi/10.1145/3084226.3084259
Alexandru CPanichella SGall H(2017)Reducing redundancies in multi-revision code analysis2017 IEEE 24th International Conference on Software Analysis, Evolution and Reengineering (SANER)10.1109/SANER.2017.7884617(148-159)Online publication date: Feb-2017
https://doi.org/10.1109/SANER.2017.7884617
Jongeling RSarkar PDatta SSerebrenik A(2017)On negative results when using sentiment analysis tools for software engineering researchEmpirical Software Engineering10.1007/s10664-016-9493-x22:5(2543-2584)Online publication date: 1-Oct-2017
https://dl.acm.org/doi/10.1007/s10664-016-9493-x
Zhu JZhou MMockus AZimmermann TCleland-Huang JSu Z(2016)Effectiveness of code contribution: from patch-based to pull-request-based toolsProceedings of the 2016 24th ACM SIGSOFT International Symposium on Foundations of Software Engineering10.1145/2950290.2950364(871-882)Online publication date: 1-Nov-2016
https://dl.acm.org/doi/10.1145/2950290.2950364
Barn BClark TAli AArif RSarkar SSureka ACotroneo DSinha NSinha VVenkatasubramanyam RJoshi PNaik RSingh PLalchandani J(2016)A Systematic Mapping Study of the Current Practice of Indian Software EngineeringProceedings of the 9th India Software Engineering Conference10.1145/2856636.2856644(89-98)Online publication date: 18-Feb-2016
https://dl.acm.org/doi/10.1145/2856636.2856644
Falcao LFerreira WBorges ANepomuceno VSoares SBaldassare M(2015)An Analysis of Software Engineering Experiments Using Human Subjects2015 ACM/IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM)10.1109/ESEM.2015.7321185(1-4)Online publication date: Oct-2015
https://doi.org/10.1109/ESEM.2015.7321185

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten