research-article

Investigating the Impact of Continuous Integration Practices on the Productivity and Quality of Open-Source Projects

Authors:

Daniel Alencar da Costa,

Uirá KuleszaAuthors Info & Claims

ESEM '22: Proceedings of the 16th ACM / IEEE International Symposium on Empirical Software Engineering and Measurement

Pages 137 - 147

https://doi.org/10.1145/3544902.3546244

Published: 19 September 2022 Publication History

Abstract

Background: Much research has been conducted to investigate the impact of Continuous Integration (CI) on the productivity and quality of open-source projects. Most of studies have analyzed the impact of adopting a CI server service (e.g, Travis-CI) but did not analyze CI sub-practices. Aims: We aim to evaluate the impact of five CI sub-practices with respect to the productivity and quality of GitHub open-source projects. Method: We collect CI sub-practices of 90 relevant open-source projects for a period of 2 years. We use regression models to analyze whether projects upholding the CI sub-practices are more productive and/or generate fewer bugs. We also perform a qualitative document analysis to understand whether CI best practices are related to a higher quality of projects. Results: Our findings reveal a correlation between the Build Activity and Commit Activity sub-practices and the number of merged pull requests. We also observe a correlation between the Build Activity, Build Health and Time to Fix Broken Builds sub-practices and number of bug-related issues. The qualitative analysis reveals that projects with the best values for CI sub-practices face fewer CI-related problems compared to projects that exhibit the worst values for CI sub-practices. Conclusions: We recommend that projects should strive to uphold the several CI sub-practices as they can impact in the productivity and quality of projects.

References

[1]

Miltiadis Allamanis, Hao Peng, and Charles Sutton. 2016. A Convolutional Attention Network for Extreme Summarization of Source Code. CoRR abs/1602.03001(2016). arXiv:1602.03001http://arxiv.org/abs/1602.03001

[2]

João Helis Bernardo, Daniel Alencar da Costa, and Uirá Kulesza. 2018. Studying the Impact of Adopting Continuous Integration on the Delivery Time of Pull Requests. In Proceedings of the 15th International Conference on Mining Software Repositories. Association for Computing Machinery, New York, NY, USA, 131–141. https://doi.org/10.1145/3196398.3196421

Digital Library

[3]

Francesco S Bersani, Daniel Lindqvist, Synthia H Mellon, Elissa S Epel, Rachel Yehuda, Janine Flory, Clare Henn-Hasse, Linda M Bierer, Iouri Makotkine, Duna Abu-Amara, 2016. Association of dimensional psychological health measures with telomere length in male war veterans. Journal of affective disorders 190 (2016), 537–542.

[4]

Glenn Bowen. 2009. Document Analysis as a Qualitative Research Method. Qualitative Research Journal 9 (08 2009), 27–40. https://doi.org/10.3316/QRJ0902027

[5]

Hyunyoung Choi and Hal Varian. 2012. Predicting the present with Google Trends. Economic record 88(2012), 2–9.

[6]

Cuelogic. 2017. The Levenshtein Algorithm. Retrieved december 02, 2021 from https://www.cuelogic.com/blog/the-levenshtein-algorithm

[7]

Marcos César de Oliveira. 2017. DRACO: Discovering Refactorings That Improve Architecture Using Fine-Grained Co-Change Dependencies. In Proceedings of the 2017 11th Joint Meeting on Foundations of Software Engineering (Paderborn, Germany) (ESEC/FSE 2017). Association for Computing Machinery, New York, NY, USA, 1018–1021. https://doi.org/10.1145/3106237.3119872

Digital Library

[8]

Marcos César de Oliveira, Davi Freitas, Rodrigo Bonifácio, Gustavo Pinto, and David Lo. 2019. Finding needles in a haystack: Leveraging co-change dependencies to recommend refactorings. Journal of Systems and Software 158 (2019), 110420. https://doi.org/10.1016/j.jss.2019.110420

Digital Library

[9]

Paul Duvall, Stephen M. Matyas, and Andrew Glover. 2007. Continuous Integration: Improving Software Quality and Reducing Risk (The Addison-Wesley Signature Series). Addison-Wesley Professional.

Digital Library

[10]

Wagner Felidré, Leonardo Furtado, Daniel A. da Costa, Bruno Cartaxo, and Gustavo Pinto. 2019. Continuous Integration Theater. In 2019 ACM/IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM). 1–10. https://doi.org/10.1109/ESEM.2019.8870152

[11]

Martin Fowler. 2006. Continuous Integration. Retrieved november 24, 2021 from https://martinfowler.com/articles/continuousIntegration.html

[12]

GeeksforGeeks. 2021. Jaro and Jaro-Winkler similarity. Retrieved december 02, 2021 from https://www.geeksforgeeks.org/jaro-and-jaro-winkler-similarity/

[13]

Taher Ghaleb, Daniel Costa, and Ying Zou. 2019. An Empirical Study of the Long Duration of Continuous Integration Builds. Empirical Software Engineering 24 (08 2019). https://doi.org/10.1007/s10664-019-09695-9

Digital Library

[14]

Taher Ahmed Ghaleb, Daniel Alencar da Costa, Ying Zou, and Ahmed E Hassan. 2019. Studying the impact of noises in build breakage data. IEEE Transactions on Software Engineering(2019).

[15]

UCLA: Statistical Consulting Group. 2022. ZERO-INFLATED NEGATIVE BINOMIAL REGRESSION | R DATA ANALYSIS EXAMPLES. Retrieved may 02, 2022 from https://stats.oarc.ucla.edu/r/dae/zinb/

[16]

Manjul Gupta, Joey F George, and Weidong Xia. 2019. Relationships between IT department culture and agile software development practices: An empirical investigation. International Journal of Information Management 44 (2019), 13–24.

[17]

Frank Harrell. 2022. ols: Linear Model Estimation Using Ordinary Least Squares. Retrieved may 02, 2022 from https://rdrr.io/cran/rms/man/ols.html

[18]

Michael Hilton, Nicholas Nelson, Timothy Tunnell, Darko Marinov, and Danny Dig. 2017. Trade-Offs in Continuous Integration: Assurance, Security, and Flexibility. Association for Computing Machinery, New York, NY, USA. https://doi.org/10.1145/3106237.3106270

Digital Library

[19]

FRANKLINE KIBUACHA. 2021. How to Determine Sample Size for a Research Study. (2021). https://www.geopoll.com/blog/sample-size-research/

[20]

Eero Laukkanen, Maria Paasivaara, and Teemu Arvonen. 2015. Stakeholder Perceptions of the Adoption of Continuous Integration – A Case Study. In 2015 Agile Conference. 11–20. https://doi.org/10.1109/Agile.2015.15

Digital Library

[21]

Steve McConnell. 2004. Code Complete: A Practical Handbook of Software Construction (2 ed.). Microsoft Press, Redmond, WA. https://www.safaribooksonline.com/library/view/code-complete-second/0735619670/

[22]

Andrew Glover Paul Duvall, Steve Matyas. 2007. Introducing continuous integration. Retrieved june 21, 2022 from https://www.infoworld.com/article/2077731/introducing-continuous-integration.html?page=3

[23]

Wouter Poncin, Alexander Serebrenik, and Mark van den Brand. 2011. Process Mining Software Repositories. In 2011 15th European Conference on Software Maintenance and Reengineering. 5–14. https://doi.org/10.1109/CSMR.2011.5

Digital Library

[24]

Gustavo Sizilio Nery, Daniel Alencar da Costa, and Uirá Kulesza. 2019. An Empirical Study of the Relationship between Continuous Integration and Test Code Evolution. In 2019 IEEE International Conference on Software Maintenance and Evolution (ICSME). 426–436. https://doi.org/10.1109/ICSME.2019.00075

[25]

Eliezio Soares, Gustavo Sizílio, Jadson Santos, Daniel Alencar da Costa, and Uirá Kulesza. 2021. The Effects of Continuous Integration on Software Development: a Systematic Literature Review. CoRR abs/2103.05451(2021). arXiv:2103.05451https://arxiv.org/abs/2103.05451

[26]

Laerd Statistics. 2021. Spearman’s Rank-Order Correlation. Retrieved december 02, 2021 from https://statistics.laerd.com/statistical-guides/spearmans-rank-order-correlation-statistical-guide.php

[27]

Daniel Ståhl and Jan Bosch. 2014. Modeling continuous integration practice differences in industry software development. Journal of Systems and Software 87 (2014). https://doi.org/10.1016/j.jss.2013.08.032

Digital Library

[28]

STHDA Statistical tools for high-throughput data analysis. 2021. Correlation Test Between Two Variables in R. Retrieved december 02, 2021 from http://www.sthda.com/english/wiki/wiki.php?id_contents=7307

[29]

STHDA Statistical tools for high-throughput data analysis. 2021. Normality Test in R. Retrieved december 02, 2021 from http://www.sthda.com/english/wiki/normality-test-in-r

[30]

Bogdan Vasilescu, Yue Yu, Huaimin Wang, Premkumar Devanbu, and Vladimir Filkov. 2015. Quality and Productivity Outcomes Relating to Continuous Integration in GitHub. In Proceedings of the 2015 10th Joint Meeting on Foundations of Software Engineering. Association for Computing Machinery, New York, NY, USA, 805–816. https://doi.org/10.1145/2786805.2786850

Digital Library

[31]

Carmine Vassallo, Fabio Palomba, Alberto Bacchelli, and Harald C. Gall. 2018. Continuous Code Quality: Are We (Really) Doing That?. In Proceedings of the 33rd ACM/IEEE International Conference on Automated Software Engineering (Montpellier, France) (ASE 2018). Association for Computing Machinery, New York, NY, USA, 790–795. https://doi.org/10.1145/3238147.3240729

Digital Library

[32]

David Gray Widder, Michael Hilton, Christian Kästner, and Bogdan Vasilescu. 2019. A Conceptual Replication of Continuous Integration Pain Points in the Context of Travis CI. Association for Computing Machinery, New York, NY, USA. https://doi.org/10.1145/3338906.3338922

Digital Library

[33]

Kazuhiro Yamashita, Shane McIntosh, Yasutaka Kamei, and Naoyasu Ubayashi. 2014. Magnet or sticky? an oss project-by-project typology. In Proceedings of the 11th working conference on mining software repositories. 344–347.

Digital Library

[34]

Yangyang Zhao, Alexander Serebrenik, Yuming Zhou, Vladimir Filkov, and Bogdan Vasilescu. 2017. The impact of continuous integration on other software development practices: A large-scale empirical study. In 2017 32nd IEEE/ACM International Conference on Automated Software Engineering (ASE). 60–71. https://doi.org/10.1109/ASE.2017.8115619

Cited By

Hong YTantithamthavorn CPasuksmit JThongtanunam PFriedman AZhao XKrasikov Ad'Amorim M(2024)Practitioners’ Challenges and Perceptions of CI Build Failure Predictions at AtlassianCompanion Proceedings of the 32nd ACM International Conference on the Foundations of Software Engineering10.1145/3663529.3663856(370-381)Online publication date: 10-Jul-2024
https://dl.acm.org/doi/10.1145/3663529.3663856
Ricós FMarín BVos TNeeft RAho P(2024)Delta GUI change detection using inferred modelsComputer Standards & Interfaces10.1016/j.csi.2024.103925(103925)Online publication date: Sep-2024
https://doi.org/10.1016/j.csi.2024.103925
Freitas GBernardo JSizíLio GCosta DKulesza U(2023)Analyzing the Impact of CI Sub-practices on Continuous Code Quality in Open-Source Projects: An Empirical StudyProceedings of the XXXVII Brazilian Symposium on Software Engineering10.1145/3613372.3613403(1-10)Online publication date: 25-Sep-2023
https://dl.acm.org/doi/10.1145/3613372.3613403
Show More Cited By

Index Terms

Investigating the Impact of Continuous Integration Practices on the Productivity and Quality of Open-Source Projects
1. Software and its engineering
  1. Software creation and management
    1. Software development process management
      1. Software development methods
        Agile software development
    2. Software post-development issues
      1. Software evolution

Recommendations

Usage, costs, and benefits of continuous integration in open-source projects
ASE '16: Proceedings of the 31st IEEE/ACM International Conference on Automated Software Engineering

Continuous integration (CI) systems automate the compilation, building, and testing of software. Despite CI rising as a big success story in automated software engineering, it has received almost no attention from the research community. For example, ...
Analyzing the Impact of CI Sub-practices on Continuous Code Quality in Open-Source Projects: An Empirical Study
SBES '23: Proceedings of the XXXVII Brazilian Symposium on Software Engineering

Continuous Integration (CI) is a widely adopted practice that automates and enhances the frequency of code integration. Previous research has explored the relationship between CI sub-practices (such as frequent commit activity and high test coverage) ...
Analyzing Continuous Integration Bad Practices in Closed-Source Projects: An Initial Study
SBES '20: Proceedings of the XXXIV Brazilian Symposium on Software Engineering

Continuous integration (CI) is a process widely used in projects of different types and sizes, generally aimed at improving development and also ensuring the quality of the software. However, some bad practices can mitigate or invalidate the beneficial ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

ESEM '22: Proceedings of the 16th ACM / IEEE International Symposium on Empirical Software Engineering and Measurement

September 2022

318 pages

ISBN:9781450394277

DOI:10.1145/3544902

Copyright © 2022 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGSOFT: ACM Special Interest Group on Software Engineering

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 19 September 2022

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed limited

Funding Sources

Conselho Nacional de Desenvolvimento Científico e Tecnológico
Coordenação de Aperfeiçoamento de Pessoal de Nível Superior

Conference

ESEM '22

Sponsor:

SIGSOFT

ESEM '22: ACM / IEEE International Symposium on Empirical Software Engineering and Measurement

September 19 - 23, 2022

Helsinki, Finland

Acceptance Rates

Overall Acceptance Rate 130 of 594 submissions, 22%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

6
Total Citations
View Citations
185
Total Downloads

Downloads (Last 12 months)79
Downloads (Last 6 weeks)8

Reflects downloads up to 12 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Hong YTantithamthavorn CPasuksmit JThongtanunam PFriedman AZhao XKrasikov Ad'Amorim M(2024)Practitioners’ Challenges and Perceptions of CI Build Failure Predictions at AtlassianCompanion Proceedings of the 32nd ACM International Conference on the Foundations of Software Engineering10.1145/3663529.3663856(370-381)Online publication date: 10-Jul-2024
https://dl.acm.org/doi/10.1145/3663529.3663856
Ricós FMarín BVos TNeeft RAho P(2024)Delta GUI change detection using inferred modelsComputer Standards & Interfaces10.1016/j.csi.2024.103925(103925)Online publication date: Sep-2024
https://doi.org/10.1016/j.csi.2024.103925
Freitas GBernardo JSizíLio GCosta DKulesza U(2023)Analyzing the Impact of CI Sub-practices on Continuous Code Quality in Open-Source Projects: An Empirical StudyProceedings of the XXXVII Brazilian Symposium on Software Engineering10.1145/3613372.3613403(1-10)Online publication date: 25-Sep-2023
https://dl.acm.org/doi/10.1145/3613372.3613403
Saraiva DDa Costa DKulesza USizílio GNeto JCoelho RNagappan M(2023)Unveiling the Relationship Between Continuous Integration and Code Coverage2023 IEEE/ACM 20th International Conference on Mining Software Repositories (MSR)10.1109/MSR59073.2023.00043(247-259)Online publication date: May-2023
https://doi.org/10.1109/MSR59073.2023.00043
Wróbel MSzymukowicz JWeichbroth P(2023)Using Continuous Integration Techniques in Open Source Projects—An Exploratory StudyIEEE Access10.1109/ACCESS.2023.332453611(113848-113863)Online publication date: 2023
https://doi.org/10.1109/ACCESS.2023.3324536
Bernardo Jda Costa DKulesza UTreude C(2023)The impact of a continuous integration service on the delivery time of merged pull requestsEmpirical Software Engineering10.1007/s10664-023-10327-628:4Online publication date: 30-Jun-2023
https://dl.acm.org/doi/10.1007/s10664-023-10327-6

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

HTML Format

View this article in HTML Format.

Media

Figures

Other

Tables

View Table of Contents