research-article

Guidelines for snowballing in systematic literature studies and a replication in software engineering

Author:

Claes WohlinAuthors Info & Claims

EASE '14: Proceedings of the 18th International Conference on Evaluation and Assessment in Software Engineering

Article No.: 38, Pages 1 - 10

https://doi.org/10.1145/2601248.2601268

Published: 13 May 2014 Publication History

Abstract

Background: Systematic literature studies have become common in software engineering, and hence it is important to understand how to conduct them efficiently and reliably.

Objective: This paper presents guidelines for conducting literature reviews using a snowballing approach, and they are illustrated and evaluated by replicating a published systematic literature review.

Method: The guidelines are based on the experience from conducting several systematic literature reviews and experimenting with different approaches.

Results: The guidelines for using snowballing as a way to search for relevant literature was successfully applied to a systematic literature review.

Conclusions: It is concluded that using snowballing, as a first search strategy, may very well be a good alternative to the use of database searches.

References

[1]

Kitchenham, B. A., Dybå, T. and Jørgensen, M. 2004. Evidence-based software engineering. In Proceedings of 27th IEEE International Software Engineering Conference, 273--281, IEEE Computer Society, 2004.

Digital Library

[2]

Webster, J. and Watson, R. T. 2002. Analyzing the past to prepare for the future: Writing a literature review. MIS Quarterly 26, 2, xiii--xxiii.

Digital Library

[3]

Pickard, L., Kitchenham B. A. and Jones P. 1998. Combining empirical results in software engineering. Information & Software Technology 40, 14, 811--821.

Digital Library

[4]

Miller, J. 1999. Can results from software engineering experiments be safely combined?. In Proceedings IEEE 6th International Symposium on Software Metrics, 152--158.

Digital Library

[5]

Hayes, W. 1999. Research synthesis in software engineering: A case for meta-analysis. In Proceedings 6th IEEE International Software Metrics Symposium, 143--151.

Digital Library

[6]

Basili V. R., Shull F. and Lanubile F. 1999. Building knowledge through families of experiments. IEEE Transactions on Software Engineering 25, 4, 456--473.

Digital Library

[7]

Kitchenham B. A. and Charters S. 2007. Guidelines for performing systematic literature reviews in software engineering. Version 2.3, EBSE Technical Report, EBSE-2007-01, Keele University.

[8]

Petersen K., Feldt R., Mujtaba S. and Mattsson M. 2008. Systematic mapping studies in software engineering. In Proceedings 12^th International Conference on Evaluation and Assessment in Software Engineering.

Digital Library

[9]

Kitchenham B. A., Budgen, D. and Brereton, O. P. 2011. Using mapping studies as the basis for further research -- a participant-observer case study. Information and Software Technology 53, 6, 638--651.

Digital Library

[10]

Wohlin C. and Prikladnicki R. 2013. Systematic literature reviews in software engineering. Information and Software Technology 55, 6, 919--920.

Digital Library

[11]

MacDonell, S., Shepperd, M., Kitchenham, B. A. and Mendes, E. 2010. How reliable are systematic reviews in empirical software engineering?. IEEE Transactions on Software Engineering 36, 5, 676--687.

Digital Library

[12]

Wohlin, C., Runeson, P., da Mota Silveira Neto, P. A., Engström, E., do Carmo Machado, I. and de Almeida, E. S. 2013. On the reliability of mapping studies in software engineering. Journal of Systems and Software 86, 10, 2594--2610.

[13]

Kitchenham, B. A., Brereton, O. P., Budgen, D., Turner, M., Bailey, J. and Linkman, S. 2009. Systematic literature reviews in software engineering: A systematic literature review. Information and Software Technology 51, 1, 7--15.

Digital Library

[14]

Kitchenham, B. A., Pretorius, R., Budgen, D., Brereton, O. P., Turner, M., Niazi, M. and Linkman, S. 2010. Systematic literature reviews in software engineering: A tertiary study. Information and Software Technology 52, 8, 792--805.

Digital Library

[15]

Greenhalgh T. and Peacock, R. 2005. Effectiveness and efficiency of search methods in systematic reviews of complex evidence: Audit of primary sources. BMJ 331, 7524, 1064--1065.

[16]

Skoglund, M. and Runeson, P. 2009. Reference-based search strategies in systematic reviews. In Proceedings 13th Evaluation and Assessment in Software Engineering, 31-40.

Digital Library

[17]

Kitchenham, B. A., Brereton, O. P., Li, Z., Budgen, D. and Burn, A. 2011. Repeatability of systematic literature reviews. In Proceedings of the 15th International Conference on Evaluation and Assessment in Software Engineering, 46--55.

[18]

Zhang, H., Babar, M. A. and Tell, P. 2011. Identifying relevant studies in software engineering. Information and Software Technology 53, 6, 625--637.

Digital Library

[19]

Jalali, S. and Wohlin, C. 2012. Systematic literature studies: Database searches vs. backward snowballing. In Proceedings 6th International Symposium on Empirical Software Engineering and Measurement, 29--38.

Digital Library

[20]

Jalali, S. and Wohlin, C. 2012. Global software engineering and agile practices: A systematic review. Journal of Software: Evolution and Process 24, 6, 643--659.

[21]

Smite, D., Wohlin, C., Galvina, Z. and Prikladnicki, R. 2012. An empirically based terminology and taxonomy for global software engineering. Empirical Software Engineering: An International Journal 19, 1, 105--153.

Digital Library

Cited By

Leal LVieira RMarques IFilho ÁImbiriba KPacheco FGomes S(2025)A indústria hoteleira e seus impactos para uma economia circular: uma discussão quanto as estratégias de expansãoCuadernos de Educación y Desarrollo10.55905/cuadv17n1-12217:1(e7299)Online publication date: 17-Jan-2025
https://doi.org/10.55905/cuadv17n1-122
Seila FSelim GNewisar M(2025)A Systematic Review of Factors Contributing to Ineffective Cultural Heritage ManagementSustainability10.3390/su1701036617:1(366)Online publication date: 6-Jan-2025
https://doi.org/10.3390/su17010366
Silva ILapa NRibeiro HDuarte E(2025)Pig Slurry Anaerobic Digestion: The Role of Biochar as an Additive Towards Biogas and Digestate ImprovementApplied Sciences10.3390/app1503103715:3(1037)Online publication date: 21-Jan-2025
https://doi.org/10.3390/app15031037
Show More Cited By

Index Terms

Guidelines for snowballing in systematic literature studies and a replication in software engineering

Recommendations

Experiences from using snowballing and database searches in systematic literature studies
EASE '15: Proceedings of the 19th International Conference on Evaluation and Assessment in Software Engineering

Background: Systematic literature studies are commonly used in software engineering. There are two main ways of conducting the searches for these type of studies; they are snowballing and database searches. In snowballing, the reference list (backward ...
Second-generation systematic literature studies using snowballing
EASE '16: Proceedings of the 20th International Conference on Evaluation and Assessment in Software Engineering

Systematic literature studies have become standard practice in software engineering to synthesize evidence in different areas of the discipline. As more such studies are published, there is also a need to extend previously published systematic ...
Systematic literature studies: database searches vs. backward snowballing
ESEM '12: Proceedings of the ACM-IEEE international symposium on Empirical software engineering and measurement

Systematic studies of the literature can be done in different ways. In particular, different guidelines propose different first steps in their recommendations, e.g. start with search strings in different databases or start with the reference lists of a ...

Comments

Information & Contributors

Information

Published In

cover image ACM Other conferences

EASE '14: Proceedings of the 18th International Conference on Evaluation and Assessment in Software Engineering

May 2014

486 pages

ISBN:9781450324762

DOI:10.1145/2601248

General Chair:
Martin Shepperd
Brunel University, UK
,
Program Chairs:
Tracy Hall
Brunel University
,
Ingunn Myrtveit
The Norwegian Business School

Copyright © 2014 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Brunel University: Brunel University

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 13 May 2014

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

Lunds Universitet

Conference

EASE '14

Sponsor:

Brunel University

EASE '14: 18th International Conference on Evaluation and Assessment in Software Engineering

May 13 - 14, 2014

England, London, United Kingdom

Acceptance Rates

Overall Acceptance Rate 71 of 232 submissions, 31%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

2,037
Total Citations
View Citations
17,999
Total Downloads

Downloads (Last 12 months)2,436
Downloads (Last 6 weeks)207

Reflects downloads up to 29 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Leal LVieira RMarques IFilho ÁImbiriba KPacheco FGomes S(2025)A indústria hoteleira e seus impactos para uma economia circular: uma discussão quanto as estratégias de expansãoCuadernos de Educación y Desarrollo10.55905/cuadv17n1-12217:1(e7299)Online publication date: 17-Jan-2025
https://doi.org/10.55905/cuadv17n1-122
Seila FSelim GNewisar M(2025)A Systematic Review of Factors Contributing to Ineffective Cultural Heritage ManagementSustainability10.3390/su1701036617:1(366)Online publication date: 6-Jan-2025
https://doi.org/10.3390/su17010366
Silva ILapa NRibeiro HDuarte E(2025)Pig Slurry Anaerobic Digestion: The Role of Biochar as an Additive Towards Biogas and Digestate ImprovementApplied Sciences10.3390/app1503103715:3(1037)Online publication date: 21-Jan-2025
https://doi.org/10.3390/app15031037
Khalafi MRosenkranz SGhasemi FKheradmand SHabibi Maleki AKorivi MTsao J(2025)Efficacy of intermittent fasting on improving liver function in individuals with metabolic disorders: a systematic review and meta-analysisNutrition & Metabolism10.1186/s12986-024-00885-x22:1Online publication date: 6-Jan-2025
https://doi.org/10.1186/s12986-024-00885-x
Mane SKundu SSharma R(2025)A Survey on Online Aggression: Content Detection and Behavioural Analysis on Social Media PlatformsACM Computing Surveys10.1145/3711125Online publication date: 4-Jan-2025
https://dl.acm.org/doi/10.1145/3711125
Standen MKim JSzabo C(2025)Adversarial Machine Learning Attacks and Defences in Multi-Agent Reinforcement LearningACM Computing Surveys10.1145/370832057:5(1-35)Online publication date: 24-Jan-2025
https://dl.acm.org/doi/10.1145/3708320
Busch NZalewski A(2025)A Systematic Literature Review of Enterprise Architecture Evaluation MethodsACM Computing Surveys10.1145/370658257:5(1-36)Online publication date: 9-Jan-2025
https://dl.acm.org/doi/10.1145/3706582
Begum MCrossley JStrömbäck FAkrida EAlpizar-Chacon IEvans AGross JHaglund PLonati VSatyavolu CThorgeirsson SMonga MLonati VBarendsen ESheard JPaterson J(2025)A Pedagogical Framework for Developing Abstraction Skills2024 Working Group Reports on Innovation and Technology in Computer Science Education10.1145/3689187.3709613(258-299)Online publication date: 22-Jan-2025
https://dl.acm.org/doi/10.1145/3689187.3709613
Guan WCao JZhao HGu YQian S(2025)Survey and Benchmark of Anomaly Detection in Business ProcessesIEEE Transactions on Knowledge and Data Engineering10.1109/TKDE.2024.348415937:1(493-512)Online publication date: Jan-2025
https://doi.org/10.1109/TKDE.2024.3484159
Siino MFalco MCroce DRosso P(2025)Exploring LLMs Applications in Law: A Literature Review on Current Legal NLP ApproachesIEEE Access10.1109/ACCESS.2025.353321713(18253-18276)Online publication date: 2025
https://doi.org/10.1109/ACCESS.2025.3533217
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten