research-article

The effects of continuous integration on software development: a systematic literature review

Authors:

Eliezio Soares,

Gustavo Sizilio,

Daniel Alencar da Costa,

Uirá KuleszaAuthors Info & Claims

Empirical Software Engineering, Volume 27, Issue 3

https://doi.org/10.1007/s10664-021-10114-1

Published: 01 May 2022 Publication History

Abstract

Context

Continuous integration (CI) is a software engineering technique that proclaims frequent activities to assure the software product health. Researchers and practitioners mention several benefits related to CI. However, no systematic study surveys state of the art regarding such benefits or cons.

Objective

This study aims to identify and interpret empirical evidence regarding how CI impacts software development.

Method

Through a Systematic Literature Review, we search for studies in six digital libraries. Starting from 479 studies, we select 101 empirical studies that evaluate CI in the context of software development. We thoroughly read and extract information regarding (i) the CI environment, (ii) findings related to the effects of CI, and (iii) the employed research methods. We apply a thematic synthesis to group and summarize the findings.

Results

Existing research has explored the positive effects of CI, such as better cooperation, or negative effects, such as adding technical and process challenges. From our thematic synthesis, we identify six themes: development activities, software process, quality assurance, integration patterns, issues & defects, and build patterns.

Conclusions

Empirical research in CI has been increasing over recent years. We found that much of the existing research reveals that CI brings positive effects to software development. However, CI may also bring technical challenges to software development teams. Despite the overall positive outlook regarding CI, we still find room for improvements in the existing empirical research that evaluates the effects of CI.

References

[1]

Beck K and Andres C Extreme programming explained: embrace change 2005 Boston Addison-Wesley

[2]

Bernardo JH, Costa DAD, Kulesza U (2018) Studying the impact of adopting continuous integration on the delivery time of pull requests. In: Proceedings of the 15th international conference on mining software repositories—MSR 18.

[3]

Beller M, Gousios G, Zaidman A (2017) TravisTorrent: synthesizing travis CI and GitHub for full-stack research on continuous integration. In: Proceedings of the 14th working conference on mining software repositories. http://www.st.ewi.tudelft.nl/~mbeller/publications/2017_beller_gousios_zaidman_travistorrent_synthesizing_travis_ci_and_github_for_full-stack_research_on_continuous_integration.pdf Accessed 22 July 2020

[4]

Chen L, Babar MA, Zhang H (2010) Towards an evidence-based understanding of electronic data sources.

[5]

CI Theatre (2017) In: ThoughtWorks. https://www.thoughtworks.com/radar/techniques/ci-theatre. Accessed 25 Aug 2020

[6]

Cohen JWeighted kappa: nominal scale agreement provision for scaled disagreement or partial creditPsychol Bull196870213-220https://doi.org/10.1037/h0026256

[7]

Cruzes DS, Dybå T (2011) Recommended steps for thematic synthesis in software engineering. In: 2011 International symposium on empirical software engineering and measurement.

[8]

Debbiche A, Dienér M, Svensson RB (2014) Challenges when adopting continuous integration: a case study. In: Product-focused software process improvement lecture notes in computer science., pp 17–32

[9]

Dikert K, Paasivaara M, and Lassenius CChallenges and success factors for large-scale agile transformations: a systematic literature reviewJ Syst Softw201611987-108https://doi.org/10.1016/j.jss.2016.06.013

[10]

Duvall PM, Matyas S, and Glover A Continuous integration improving software quality and reducing risk 2013 Upper Saddle River Addison-Wesley

[11]

Duvall PM (2018) Continuous delivery—patterns and anti-patterns in the software lifecycle. In: dzone.com. https://dzone.com/refcardz/continuous-delivery-patterns. Accessed 7 Feb 2021

[12]

Dybå T, Dingsøyr T (2008) Strength of evidence in systematic reviews in software engineering. In: Proceedings of the second ACM-IEEE international symposium on empirical software engineering and measurement—ESEM 08.

[13]

Dybå T, Dingsoyr T, Hanssen GK (2007) Applying systematic reviews to diverse study types: an experience report. In: First international symposium on empirical software engineering and measurement (ESEM 2007).

[14]

Easterbrook S, Singer J, Storey MA, Damian D (2008) Selecting empirical methods for software engineering research. In: Guide to advanced empirical software engineering. Springer, London, pp 285–311

[15]

Embury SM, Page C (2019) Effect of continuous integration on build health in undergraduate team projects. In: Software engineering aspects of continuous development and new paradigms of software production and deployment lecture notes in computer science.

[16]

Felidre W, Furtado L, Costa DAD et al (2019) Continuous integration theater. In: 2019 ACM/IEEE international symposium on empirical software engineering and measurement (ESEM).

[17]

Fitzgerald B and Stol K-JContinuous software engineering: a roadmap and agendaJ Syst Softw2017123176-189https://doi.org/10.1016/j.jss.2015.06.063

[18]

Fowler M (2006) Continuous integration. In: Martinfowler.com. https://martinfowler.com/articles/continuousIntegration.html. Accessed 29 Jun 2020

[19]

Fowler M (2013) Extremeprogramming. In: Martinfowler.com. https://martinfowler.com/bliki/ExtremeProgramming.html

[20]

Fowler M (2017) Continuousintegrationcertification. In: Martinfowler.com. https://martinfowler.com/bliki/ContinuousIntegrationCertification.html. Accessed: 26 Jun 2020

[21]

Ghaleb TA, Costa DAD, and Zou YAn empirical study of the long duration of continuous integration buildsEmpir Softw Eng2019242102-2139https://doi.org/10.1007/s10664-019-09695-9

[22]

Hilton M, Tunnell T, Huang K, et al. (2016) Usage, costs, and benefits of continuous integration in open-source projects. In: Proceedings of the 31st IEEE/ACM international conference on automated software engineering—ASE 2016.

[23]

Holmstrom H, Conchuir E, Agerfalk P, Fitzgerald B (2006) Global software development challenges: a case study on temporal, geographical and socio-cultural distance. In: 2006 IEEE international conference on global software engineering (ICGSE06).

[24]

Johanssen JO, Kleebaum A, Paech B, Bruegge B (2018) Practitioners eye on continuous software engineering. In: Proceedings of the 2018 international conference on software and system process—ICSSP 18.

[25]

Kitchenham B, Charters S (2007) Guidelines for performing systematic literature reviews in software engineering. https://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.117.471. Accessed 26 June 2020

[26]

Kitchenham BA, Tore D, Magne J (2004) Evidence-based software engineering.

[27]

Laukkanen E, Itkonen J, and Lassenius C Problems, causes and solutions when adopting continuous delivery—a systematic literature review Inf Softw Technol 2017 82 55-79

[28]

Leppanen M, Makinen S, Pagels M, et al.The highways and country roads to continuous deploymentIEEE Softw20153264-72https://doi.org/10.1109/ms.2015.50

[29]

Meedeniya DA, D I, Perera I (2019) Software artefacts consistency management towards continuous integration: a roadmap. Int J Adva Comput Sci Appl.

[30]

Pinto G, Castor F, Bonifacio R, and Rebouças MWork practices and challenges in continuous integration: a survey with Travis CI usersSoftw: Pract Exp2018482223-2236https://doi.org/10.1002/spe.2637

[31]

Rausch T, Hummer W, Leitner P, Schulte S (2017) An empirical analysis of build failures in the continuous integration workflows of Java-based open-source software. In: 2017 IEEE/ACM 14th international conference on mining software repositories (MSR).

[32]

Robles G (2010) Replicating MSR: a study of the potential replicability of papers published in the mining software repositories proceedings. In: 2010 7th IEEE working conference on mining software repositories (MSR 2010).

[33]

Rodríguez-Pérez G, Robles G, and González-Barahona JMReproducibility and credibility in empirical software engineering: a case study based on a systematic literature review of the use of the SZZ algorithmInf Softw Technol201899164-176https://doi.org/10.1016/j.infsof.2018.03.009

[34]

Rogers RO (2004) Scaling continuous integration. In: Extreme programming and agile processes in software engineering lecture notes in computer science, pp 68–76

[35]

Shahin M, Babar MA, and Zhu L Continuous integration, delivery and deployment: a systematic review on approaches, tools, challenges and practices IEEE Access 2017 5 3909-3943

[36]

Soares E, Sizilio G, Santos J, Alencar D, Kulesza U (2021) SLR artifacts—continuous integration quality impacts (v.1.0.2) [Data set]. In: Zenodo. Accessed: 10 Aug 2021

[37]

Ståhl D, Bosch J (2013) Experienced benefits of continuous integration in industry software product development: a case study. Artificial intelligence and applications/794: modelling, identification and control/795: parallel and distributed computing and networks/796: software engineering/792: web-based education.

[38]

Ståhl D and Bosch JModeling continuous integration practice differences in industry software developmentJ Syst Softw20148748-59https://doi.org/10.1016/j.jss.2013.08.032

[39]

Ståhl D, Bosch J (2014b) Automated software integration flows in industry: a multiple-case study. In: Companion proceedings of the 36th international conference on software engineering—ICSE Companion 2014.

[40]

Vasilescu B, Yu Y, Wang H, et al. (2015) Quality and productivity outcomes relating to continuous integration in GitHub. In: Proceedings of the 2015 10th joint meeting on foundations of software engineering—ESEC/FSE 2015.

[41]

Vassallo C, Palomba F, Gall HC (2018) Continuous refactoring in CI: a preliminary study on the perceived advantages and barriers. In: 2018 IEEE international conference on software maintenance and evolution (ICSME).

[42]

Vassallo C, Proksch S, Gall HC, Penta MD (2019) Automated reporting of anti-patterns and decay in continuous integration. In: 2019 IEEE/ACM 41st international conference on software engineering (ICSE).

[43]

Viggiato M, Oliveira J, Figueiredo E, et al. (2019) Understanding similarities and differences in software development practices across domains. In: 2019 ACM/IEEE 14th international conference on global software engineering (ICGSE).

[44]

Volf Z, Shmueli E (2017) Screening heuristics for project gating systems. In: Proceedings of the 2017 11th joint meeting on foundations of software engineering—ESEC/FSE 2017.

[45]

Wheelan SA and Hochberger JM Validation studies of the group development questionnaire Small Group Res 1996 27 1 143-170

[46]

Yu Y, Yin G, Wang T et al (2016) Determinants of pull-based development in the context of continuous integration. Sci China Inf Sci.

[47]

Zampetti F, Scalabrino S, Oliveto R et al (2017) How open source projects use static code analysis tools in continuous integration pipelines. In: 2017 IEEE/ACM 14th international conference on mining software repositories (MSR).

[48]

Zhao Y, Serebrenik A, Zhou Y, et al. (2017) The impact of continuous integration on other software development practices: a large-scale empirical study. In: 2017 32nd IEEE/ACM international conference on automated software engineering (ASE).

Cited By

Bhutani VToosi FBuckley J(2024)Analysing the AnalysersApplied Computer Systems10.2478/acss-2024-001329:1(98-111)Online publication date: 15-Aug-2024
https://dl.acm.org/doi/10.2478/acss-2024-0013
Kola-Olawuyi AWeeraddana NNagappan MSpinellis DConstantinou EBacchelli A(2024)The Impact of Code Ownership of DevOps Artefacts on the Outcome of DevOps CI BuildsProceedings of the 21st International Conference on Mining Software Repositories10.1145/3643991.3644924(543-555)Online publication date: 15-Apr-2024
https://dl.acm.org/doi/10.1145/3643991.3644924
Straubinger PFraser GRoychoudhury APaiva AAbreu RStorey MGama KSiegmund J(2024)Gamifying a Software Testing Course with Continuous IntegrationProceedings of the 46th International Conference on Software Engineering: Software Engineering Education and Training10.1145/3639474.3640054(34-45)Online publication date: 14-Apr-2024
https://dl.acm.org/doi/10.1145/3639474.3640054
Show More Cited By

Recommendations

Systematic literature review on the impacts of agile release engineering practices

ContextAgile release engineering (ARE) practices are designed to deliver software faster and cheaper to end users; hence, claims of such impacts should be validated by rigorous and relevant empirical studies. ObjectiveThe study objective was to analyze ...
Systematic literature review on agile practices in global software development
Abstract Context
Developing software in distributed development environments exhibits coordination, control and communication challenges. Agile practices, which demand frequent communication and self-organization between remote sites, are increasingly ...
Software Process Improvement in Agile Software Development A Systematic Literature Review
SEAA '15: Proceedings of the 2015 41st Euromicro Conference on Software Engineering and Advanced Applications

It is recognized the relevance and importance that software process improvement (SPI) and agile development have gained in the field of software engineering. Both are approaches that increase the efficiency and effectiveness of a software development ...

Comments

Information & Contributors

Information

Published In

cover image Empirical Software Engineering

Empirical Software Engineering Volume 27, Issue 3

May 2022

844 pages

ISSN:1382-3256

Issue’s Table of Contents

© The Author(s), under exclusive licence to Springer Science+Business Media, LLC, part of Springer Nature 2022.

Publisher

Kluwer Academic Publishers

United States

Publication History

Published: 01 May 2022

Accepted: 29 December 2021

Author Tags

Qualifiers

Research-article

Funding Sources

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

8
Total Citations
View Citations
0
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 12 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Bhutani VToosi FBuckley J(2024)Analysing the AnalysersApplied Computer Systems10.2478/acss-2024-001329:1(98-111)Online publication date: 15-Aug-2024
https://dl.acm.org/doi/10.2478/acss-2024-0013
Kola-Olawuyi AWeeraddana NNagappan MSpinellis DConstantinou EBacchelli A(2024)The Impact of Code Ownership of DevOps Artefacts on the Outcome of DevOps CI BuildsProceedings of the 21st International Conference on Mining Software Repositories10.1145/3643991.3644924(543-555)Online publication date: 15-Apr-2024
https://dl.acm.org/doi/10.1145/3643991.3644924
Straubinger PFraser GRoychoudhury APaiva AAbreu RStorey MGama KSiegmund J(2024)Gamifying a Software Testing Course with Continuous IntegrationProceedings of the 46th International Conference on Software Engineering: Software Engineering Education and Training10.1145/3639474.3640054(34-45)Online publication date: 14-Apr-2024
https://dl.acm.org/doi/10.1145/3639474.3640054
Freitas GBernardo JSizíLio GCosta DKulesza U(2023)Analyzing the Impact of CI Sub-practices on Continuous Code Quality in Open-Source Projects: An Empirical StudyProceedings of the XXXVII Brazilian Symposium on Software Engineering10.1145/3613372.3613403(1-10)Online publication date: 25-Sep-2023
https://dl.acm.org/doi/10.1145/3613372.3613403
Jin XServant F(2023)HybridCISave: A Combined Build and Test Selection Approach in Continuous IntegrationACM Transactions on Software Engineering and Methodology10.1145/357603832:4(1-39)Online publication date: 26-May-2023
https://dl.acm.org/doi/10.1145/3576038
Narang PMittal P(2023)Continuous Assessment and Improvement of Software Quality with DevOps-Based Hybrid Model of Automation ToolsJournal of Computer and Systems Sciences International10.1134/S106423072302014462:2(412-419)Online publication date: 1-Apr-2023
https://dl.acm.org/doi/10.1134/S1064230723020144
Bernardo Jda Costa DKulesza UTreude C(2023)The impact of a continuous integration service on the delivery time of merged pull requestsEmpirical Software Engineering10.1007/s10664-023-10327-628:4Online publication date: 30-Jun-2023
https://dl.acm.org/doi/10.1007/s10664-023-10327-6
Rostami Mazrae PMens TGolzadeh MDecan A(2023)On the usage, co-usage and migration of CI/CD tools: A qualitative analysisEmpirical Software Engineering10.1007/s10664-022-10285-528:2Online publication date: 7-Mar-2023
https://dl.acm.org/doi/10.1007/s10664-022-10285-5

View Options

View options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Media

Figures

Other

Tables

View Issue’s Table of Contents