research-article

Research synthesis in software engineering

Authors:

Daniela S. Cruzes,

Tore DybåAuthors Info & Claims

Information and Software Technology, Volume 53, Issue 5

Pages 440 - 455

https://doi.org/10.1016/j.infsof.2011.01.004

Published: 01 May 2011 Publication History

Abstract

ContextComparing and contrasting evidence from multiple studies is necessary to build knowledge and reach conclusions about the empirical support for a phenomenon. Therefore, research synthesis is at the center of the scientific enterprise in the software engineering discipline. ObjectiveThe objective of this article is to contribute to a better understanding of the challenges in synthesizing software engineering research and their implications for the progress of research and practice. MethodA tertiary study of journal articles and full proceedings papers from the inception of evidence-based software engineering was performed to assess the types and methods of research synthesis in systematic reviews in software engineering. ResultsAs many as half of the 49 reviews included in the study did not contain any synthesis. Of the studies that did contain synthesis, two thirds performed a narrative or a thematic synthesis. Only a few studies adequately demonstrated a robust, academic approach to research synthesis. ConclusionWe concluded that, despite the focus on systematic reviews, there is limited attention paid to research synthesis in software engineering. This trend needs to change and a repertoire of synthesis methods needs to be an integral part of systematic reviews to increase their significance and utility for research and practice.

References

[1]

Anderson S, Allen P, Peckham S, Goodwin N. (2008) "Asking the right questions: scoping studies in the commissioning of research on the organisation and delivery of health services,"Health Res Policy Syst. Jul 9; 6:7.

[2]

Arksey, H. and O'Malley, L., Scoping studies: towards a methodological framework. Int. J. Social Res. Methodol. v8 i1. 19-32.

[3]

Braun, V. and Clarke, V., Using thematic analysis in psychology. Qual. Res. Psychol. v3. 77-101.

[4]

CASP, Critical Appraisals Skills Programme, NSH, UK, 2006. <http://www.sph.nhs.uk/what-we-do/public-health-workforce/resources/critical-appraisals-skills-programme>.

[5]

Cohen, B.P., Developing Sociological Knowledge: Theory and Method. 1989. second Ed. Nelson-Hall, Chicago.

[6]

Cohen, J., Statistical Power Analysis for the Behavioral Sciences. 1988. second ed. Laurence Erlbaum.

[7]

. In: Cooper, H., Hedges, L.V., Valentine, J.C. (Eds.), The Handbook of Research Synthesis and Meta-Analysis, Russell Sage Foundation.

Digital Library

[8]

Corbin, J. and Strauss, A., Basics of Qualitative Research. 2007. third ed. Sage.

[9]

D.S. Cruzes, M.G. Mendonça, V.R. Basili, F. Shull, M. Jino, Extracting Information from Experimental Software Engineering Papers, Proc. SCCC'07, 2007, pp. 105-114.

Digital Library

[10]

D.S. Cruzes, T. Dybå, Synthesizing Evidence in Software Engineering Research," Proceedings of the 4th International Symposium on Empirical Software Engineering and Measurement (ESEM 2010), Bolzano-Bozen, Italy, 16-17 September, 2010.

Digital Library

[11]

D.S. Cruzes, V.R. Basili, F. Shull, M. Jino, Using Context Distance Measurement to Analyze Results across Studies, Proc. ESEM'07, 2007, pp. 235-244.

Digital Library

[12]

Davies, K., Drey, N. and Gould, D., What are scoping studies? A review of the nursing literature. Int. J. Nurs. Stud. v46. 1386-1400.

[13]

Dixon-Woods, M., Agarwal, S., Jones, D., Young, B. and Sutton, A., Synthesising qualitative and quantitative evidence: a review of possible methods. J. Health Ser. Res. Policy. v10 i1. 45-53.

[14]

Dixon-Woods, M., Booth, A. and Sutton, A., Synthesizing qualitative research: a review of published reports. Qual. Res. v7 i3. 375-422.

[15]

Dybå, T., Improvisation in small software organizations. IEEE Soft. v17 i5. 82-87.

Digital Library

[16]

T. Dybå, T. Dingsøyr, Strength of Evidence in Systematic Reviews in Software Engineering, Proceedings of the 2nd International Symposium on Empirical Software Engineering and Measurement (ESEM'08), Kaiserslautern, Germany, 9-10 October, ACM Press, 2008, pp. 178-187.

Digital Library

[17]

T. Dybå, T. Dingsøyr, G.K. Hanssen, Applying Systematic Reviews to Diverse Study Types: An Experience Report, Proc ESEM'07, 2007, pp. 225-234.

Digital Library

[18]

Dybå, T., Kitchenham, B.A. and Jørgensen, M., Evidence-based software engineering for practitioners. IEEE Soft. v22 i1. 58-65.

Digital Library

[19]

Estabrooks, C.A., Field, P.A. and Morse, J.M., Aggregating qualitative findings: an approach to theory development. Qual. Health Res. v4 i4. 503-511.

[20]

Franzosi, R., Quantitative Narrative Analysis. 2010. Sage.

[21]

Glaser, and Strauss, The Discovery of Grounded Theory: Strategies for Qualitative Research. 1967. Aldine Transaction.

[22]

Glass, G.V., Primary, secondary, and meta-analysis of research. Educ. Res. v5 i10. 3-8.

[23]

M. Hammersley, Systematic or Unsystematic, is that the Question? Some Reflections on the Science, Art and Politics of Reviewing Research Evidence, London: Health Development Agency Public Health Steering Group, 2002.

[24]

J.P.T. Higgins, S. Green (Eds.), Cochrane Handbook for Systematic Reviews of Interventions, Version 5.0.2, The Cochrane Collaboration, 2009. <http://www.cochrane-handbook.org> (updated September).

[25]

M. Jørgensen, T. Dybå, B. Kitchenham, Teaching Evidence-Based Software Engineering to University Students, Proceedings of the 11th International Software Metrics Symposium (Metrics 2005), Como, Italy, 19-22 September, 2005.

Digital Library

[26]

Kitchenham, B.A., Pfleeger, S.L., Pickard, L.M., Jones, P.W., Hoaglin, D.C., Emam, K.E. and Rosenberg, J., Preliminary guidelines for empirical research in software engineering. IEEE Trans. Softw. Eng. v28 i8. 721-734.

Digital Library

[27]

Kitchenham, B., Pearl Brereton, O., Budgen, D., Turner, M., Bailey, J. and Linkman, S., Systematic literature reviews in software engineering - a systematic literature review. Inf. Softw. Technol. v51 i1. 7-15.

Digital Library

[28]

Kitchenham, B., Pretorius, R., Budgen, D., Pearl Brereton, O., Turner, M., Niazi, M. and Linkman, S., Systematic literature reviews in software engineering - a tertiary study. Inf. Softw. Technol. v52 i8. 792-805.

Digital Library

[29]

B. Kitchenham, D.I. Sjøberg, O.P. Brereton, D. Budgen, T. Dybå, M. Höst, D. Pfahl, P. Runeson, Can we evaluate the quality of software engineering experiments?. In: Proceedings of the 2010 ACM-IEEE International Symposium on Empirical Software Engineering and Measurement (Bolzano-Bozen, Italy, September 16-17, 2010). ESEM '10. ACM, New York, NY, 2010, 1-8.

Digital Library

[30]

B.A. Kitchenham, Procedures for Performing Systematic Reviews, Keele University, Technical Report TR/SE-0401 and NICTA Technical Report 0400011T.1, 2004.

[31]

B.A. Kitchenham, S. Charters, Guidelines for performing Systematic Literature Reviews in Software Engineering, Version 2.3, Keele University, EBSE Technical Report, EBSE-2007-01, 2007.

[32]

B.A. Kitchenham, T. Dybå, M. Jørgensen, Evidence-based Software Engineering, Proc. ICSE'04, Edinburgh, Scotland, 23-28 May, 2004, pp. 273-281.

Digital Library

[33]

Lipsey, M.W. and Wilson, D.B., Practical Meta - Analysis. 2001. Sage.

[34]

Long, A.F. and Godfrey, M., An evaluation tool to assess the quality of qualitative research studies. Int. J. Soc. Res. Meth. v7. 181-196.

[35]

Mays, N., Pope, C. and Popay, J., Systematically reviewing qualitative and quantitative evidence to inform management and policy-making in the health field. J. Health Serv. Res. Policy. v10 iSupplement 1. 6-20.

[36]

Miles, M.B. and Huberman, A.M., Qualitative Data Analysis: An Expanded Source Book. 1994. Sage.

[37]

Miller, J., Applying meta-analytical procedures to software engineering experiments. J. Syst. Soft. v54. 29-39.

Digital Library

[38]

. In: Mulrow, C., Cook, D. (Eds.), Systematic Re-views: Synthesis of Best Evidence for Health Care Decisions, Am. College of Physicians, Philadelphia.

[39]

Noblit, G.W. and Hare, R.D., Meta-ethnography: Synthesizing Qualitative Studies. 1988. Sage.

[40]

Ogawa, R.T. and Malen, B., Towards rigor in reviews of multivocal literatures: applying the exploratory case study method. Rev. Educ. Res. v61 i3. 265-286.

[41]

Paterson, B.L., Thorne, S.E., Canam, C. and Jillings, C., Meta-study of Qualitative Health Research: A Practical Guide to Meta-analysis and Meta-synthesis. 2001. Sage.

[42]

Pawson, R., Evidence-based Policy: A Realist Perspective. 2006. Sage.

[43]

Pawson, R., Greenhalgh, T., Harvey, G. and Walshe, K., Realist review - a new method of systematic review designed for complex policy interventions. J. Health Serv. Res. Policy. v10 i1. 21-34.

[44]

Pope, C., Mays, N. and Popay, J., Synthesizing Qualitative and Quantitative Health Evidence: A Guide to Methods. 2007. Open University Press.

[45]

Ragin, C.C., The Comparative Method: Moving Beyond Qualitative and Quantitative Strategies. 1987. University of California Press.

[46]

Robson, C., Real World Research. 2002. second ed. Blackwell.

[47]

Rodgers, M., Sowden, A., Petticrew, M., Arai, L., Roberts, H., Britten, N. and Popay, J., Testing methodological guidance on the conduct of narrative synthesis in systematic reviews. Evaluation. v15 i1. 49-74.

[48]

Rosenthal, R. and DiMatteo, D.M., Meta-analysis: recent development in quantitative methods for literature reviews. Ann. Rev. Psychol. v52. 59-82.

[49]

Runeson, P. and Höst, M., Guidelines for conducting and reporting case study research in software engineering. Empirical Softw. Eng. v14 i2. 131-164.

Digital Library

[50]

Sandelowski, M. and Barroso, J., Handbook for Synthesizing Qualitative Research. 2007. Springer.

[51]

Sandelowski, M., Docherty, S. and Emden, C., Qualitative metasynthesis: issues techniques. Res. Nurs. Health. v20 i4. 365-371.

[52]

Schwandt, T.A., Farewell to criteriology. Qual. Inquiry. v2 i1. 58-72.

[53]

D.I.K. Sjøberg, T. Dybå, M. Jørgensen, The Future of Empirical Methods in Software Engineering Research, Proc. FOSE'07, 2007, pp. 358-378.

Digital Library

[54]

Spencer, L., Ritchie, J., Lewis, J. and Dillon, L., Quality in Qualitative Evaluation: A Framework for Assessing Research Evidence. 2003. Cabinet Office, London.

[55]

Walsh, D. and Downe, S., Appraising the quality of qualitative research. Midwifery. v22. 108-119.

[56]

Yin, R.K. and Heald, K.A., Using the case survey method to analyze policy studies. Admin. Sci. Quart. v20. 371-381.

Cited By

Wyrich MMunoz Baron MBogner J(2024)Apples, Oranges, and Software Engineering: Study Selection Challenges for Secondary Research on Latent VariablesProceedings of the 1st IEEE/ACM International Workshop on Methodological Issues with Empirical Studies in Software Engineering10.1145/3643664.3648213(42-47)Online publication date: 16-Apr-2024
https://dl.acm.org/doi/10.1145/3643664.3648213
Zhao XClear TLal R(2024)Identifying the primary dimensions of DevSecOpsJournal of Systems and Software10.1016/j.jss.2024.112063214:COnline publication date: 1-Aug-2024
https://dl.acm.org/doi/10.1016/j.jss.2024.112063
Foidl HGolendukhina VRamler RFelderer M(2024)Data pipeline qualityJournal of Systems and Software10.1016/j.jss.2023.111855207:COnline publication date: 1-Jan-2024
https://dl.acm.org/doi/10.1016/j.jss.2023.111855
Show More Cited By

Recommendations

Using ethnographic methods in software engineering research
ICSE '10: Proceedings of the 32nd ACM/IEEE International Conference on Software Engineering - Volume 2

This tutorial provides an overview of the role of ethnography in Software Engineering research. It describes the use of ethnographic methods as a means to provide an in-depth understanding of the socio-technological realities surrounding everyday ...
Engaging the net generation with evidence-based software engineering through a community-driven web database

Software engineering faculty face the challenge of educating future researchers and industry practitioners regarding the generation of empirical software engineering studies and their use in evidence-based software engineering. In order to engage the ...
Empirical studies of agile software development: A systematic review

Agile software development represents a major departure from traditional, plan-based approaches to software engineering. A systematic review of empirical studies of agile software development up to and including 2005 was conducted. The search strategy ...

Comments

Information & Contributors

Information

Published In

Copyright © Elsevier B.V.

Publisher

Butterworth-Heinemann

United States

Publication History

Published: 01 May 2011

Author Tags

Qualifiers

Research-article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

99
Total Citations
View Citations
0
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 14 Oct 2024

Other Metrics

View Author Metrics

Citations

Cited By

Wyrich MMunoz Baron MBogner J(2024)Apples, Oranges, and Software Engineering: Study Selection Challenges for Secondary Research on Latent VariablesProceedings of the 1st IEEE/ACM International Workshop on Methodological Issues with Empirical Studies in Software Engineering10.1145/3643664.3648213(42-47)Online publication date: 16-Apr-2024
https://dl.acm.org/doi/10.1145/3643664.3648213
Zhao XClear TLal R(2024)Identifying the primary dimensions of DevSecOpsJournal of Systems and Software10.1016/j.jss.2024.112063214:COnline publication date: 1-Aug-2024
https://dl.acm.org/doi/10.1016/j.jss.2024.112063
Foidl HGolendukhina VRamler RFelderer M(2024)Data pipeline qualityJournal of Systems and Software10.1016/j.jss.2023.111855207:COnline publication date: 1-Jan-2024
https://dl.acm.org/doi/10.1016/j.jss.2023.111855
Khoshnevis SArdestani O(2024)Search-based approaches to optimizing software product line architecturesInformation and Software Technology10.1016/j.infsof.2024.107446170:COnline publication date: 1-Jun-2024
https://dl.acm.org/doi/10.1016/j.infsof.2024.107446
Hannousse AYahiouche SNait-Hamoud M(2024)Twenty-two years since revealing cross-site scripting attacksComputer Science Review10.1016/j.cosrev.2024.10063452:COnline publication date: 1-May-2024
https://dl.acm.org/doi/10.1016/j.cosrev.2024.100634
Alam MShahid MMustajab S(2024)Security challenges for workflow allocation model in cloud computing environment: a comprehensive survey, framework, taxonomy, open issues, and future directionsThe Journal of Supercomputing10.1007/s11227-023-05873-180:8(11491-11555)Online publication date: 1-May-2024
https://dl.acm.org/doi/10.1007/s11227-023-05873-1
Agh HAzamnouri AWagner S(2024)Software product line testing: a systematic literature reviewEmpirical Software Engineering10.1007/s10664-024-10516-x29:6Online publication date: 2-Sep-2024
https://doi.org/10.1007/s10664-024-10516-x
Pretorius CRazavian MEling KLangerak F(2024)When rationality meets intuitionJournal of Software: Evolution and Process10.1002/smr.266436:9Online publication date: 16-Sep-2024
https://dl.acm.org/doi/10.1002/smr.2664
Andrade ÁVeloso ESantos G(2023)What We Know About Software Dependability in DevOps - A Tertiary StudyProceedings of the XXII Brazilian Symposium on Software Quality10.1145/3629479.3629502(178-187)Online publication date: 7-Nov-2023
https://dl.acm.org/doi/10.1145/3629479.3629502
Mbaka W(2023)Towards Unveiling Effects Of Human Factors Within Security Risk AssessmentACM SIGSOFT Software Engineering Notes10.1145/3573074.357309248:1(70-75)Online publication date: 17-Jan-2023
https://dl.acm.org/doi/10.1145/3573074.3573092
Show More Cited By

View Options

View options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Media

Figures

Other

Tables

View Issue’s Table of Contents