Analyzing the Impact of CI Sub-practices on Continuous Code Quality in Open-Source Projects: An Empirical Study
Authors: 
Guilherme Freitas, JoãO Helis Bernardo, Gustavo SizíLio, Daniel Alencar Da Costa, Uirá KuleszaAuthors Info & Claims
Guilherme Freitas, JoãO Helis Bernardo, Gustavo SizíLio, Daniel Alencar Da Costa, Uirá KuleszaAuthors Info & ClaimsPages 1 - 10
Abstract
Continuous Integration (CI) is a widely adopted practice that automates and enhances the frequency of code integration. Previous research has explored the relationship between CI sub-practices (such as frequent commit activity and high test coverage) and software quality. However, limited knowledge exists regarding the impact of specific CI sub-practices on the Continuous Code Quality (CCQ) inspection outcomes of software projects, such as technical debts, bug density, duplicated lines, and code smells. This paper aims to analyze the extent to which the adoption of CI sub-practices improves CCQ outcome metrics in software projects. First, we investigate the association between Travis CI adoption, Travis CI usage maturity, and improved quality inspection outcomes in a set of 75 popular open-source projects from GitHub. Additionally, we use data from other 97 open-source projects to identify specific CI sub-practices, such as maintaining shorter build durations, that exhibit a strong correlation with enhanced quality inspection outcomes. Our findings reveal that the quality inspection outcomes are improved in projects with higher test coverage and shorter build duration. Merely adopting a CI service is not a guarantee for improved quality outcomes. Therefore, it is essential for developers to consistently adhere to recommended CI sub-practices. By doing so, they can effectively harness the benefits of CI in their projects and achieve improved quality outcomes.
References
[1]
[n. d.]. SonarQube 7.9 Documentation. https://docs.sonarqube.org/latest/. Accessed: 2019-08-21.
[2]
M. Beller, G. Gousios, and A. Zaidman. 2017. Oops, My Tests Broke the Build: An Explorative Analysis of Travis CI with GitHub. In Proceedings of the 14th International Conference on Mining Software Repositories (Buenos Aires, Argentina) (MSR ’17). IEEE Press, Piscataway, NJ, USA, 356–367.
[3]
Moritz Beller, Georgios Gousios, and Andy Zaidman. 2017. TravisTorrent: synthesizing Travis CI and GitHub for full-stack research on continuous integration. In Proceedings of the 14th International Conference on Mining Software Repositories, MSR 2017, Buenos Aires, Argentina, May 20-28, 2017. 447–450.
[4]
João Helis Bernardo, Daniel Alencar da Costa, and Uirá Kulesza. 2018. Studying the Impact of Adopting Continuous Integration on the Delivery Time of Pull Requests. In Proceedings of the 15th International Conference on Mining Software Repositories (Gothenburg, Sweden) (MSR ’18). ACM, New York, NY, USA, 131–141.
[5]
Graham Brooks. 2008. Team pace keeping build times down. In Agile 2008 Conference. IEEE, 294–297.
[6]
Tsun Chow and Dac-Buu Cao. 2008. A survey study of critical success factors in agile software projects. Journal of systems and software 81, 6 (2008), 961–971.
[7]
Kevin Crowston, Hala Annabi, and James Howison. 2003. Defining open source software project success. ICIS 2003 Proceedings (2003), 28.
[8]
G. Digkas, M. Lungu, P. Avgeriou, A. Chatzigeorgiou, and A. Ampatzoglou. 2018. How do developers fix issues and pay back technical debt in the Apache ecosystem?. In 2018 IEEE 25th International Conference on Software Analysis, Evolution and Reengineering (SANER). 153–163.
[9]
Paul M Duvall, Steve Matyas, and Andrew Glover. 2007. Continuous integration: improving software quality and reducing risk. Pearson Education.
[10]
Sebastian Elbaum, David Gable, and Gregg Rothermel. 2001. The impact of software evolution on code coverage information. In Proceedings of the IEEE International Conference on Software Maintenance. IEEE Computer Society, 170.
[11]
Wagner Felidré, Leonardo Furtado, Daniel A da Costa, Bruno Cartaxo, and Gustavo Pinto. 2019. Continuous integration theater. (2019), 1–10.
[12]
Martin Fowler. 2018. Refactoring: improving the design of existing code. Addison-Wesley Professional.
[13]
Martin Fowler and Matthew Foemmel. 2006. Continuous integration. Thought-Works) http://www. thoughtworks. com/Continuous Integration. pdf (2006).
[14]
Keheliya Gallaba, Christian Macho, Martin Pinzger, and Shane McIntosh. 2018. Noise and heterogeneity in historical build data: an empirical study of Travis CI. In Proceedings of the 33rd ACM/IEEE International Conference on Automated Software Engineering. ACM, 87–97.
[15]
T. A. Ghaleb, D. A. da Costa, and Y Zou. 2019. An empirical study of the long duration of continuous integration builds. Empirical Software Engineering (2019), 1–38.
[16]
Michael Hilton, Jonathan Bell, and Darko Marinov. 2018. A large-scale study of test coverage evolution. In Proceedings of the 33rd ACM/IEEE International Conference on Automated Software Engineering. ACM, 53–63.
[17]
Michael Hilton, Nicholas Nelson, Timothy Tunnell, Darko Marinov, and Danny Dig. 2017. Trade-offs in continuous integration: assurance, security, and flexibility. In Proceedings of the 2017 11th Joint Meeting on Foundations of Software Engineering. ACM, 197–207.
[18]
M. Hilton, T. Tunnell, K. Huang, D. Marinov, and D. Dig. 2016. Usage, Costs, and Benefits of Continuous Integration in Open-source Projects. In Proceedings of the 31st IEEE/ACM International Conference on Automated Software Engineering (Singapore, Singapore) (ASE 2016). 426–437.
[19]
Michael Hilton, Timothy Tunnell, Kai Huang, Darko Marinov, and Danny Dig. 2016. Usage, costs, and benefits of continuous integration in open-source projects. In ASE 2016 - Proceedings of the 31st IEEE/ACM International Conference on Automated Software Engineering. Association for Computing Machinery, Inc, 426–437.
[20]
C. Leys, C. Ley, O. Klein, P. Bernard, and L. Licata. 2013. Detecting outliers: Do not use standard deviation around the mean, use absolute deviation around the median. Journal of Experimental Social Psychology 49, 4 (2013), 764–766.
[21]
Welder Pinheiro Luz, Gustavo Pinto, and Rodrigo Bonifácio. 2018. Building a collaborative culture: a grounded theory of well succeeded devops adoption in practice. In Proceedings of the 12th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement, ESEM 2018, Oulu, Finland, October 11-12, 2018. 6:1–6:10.
[22]
T. Mårtensson, D. Ståhl, and J Bosch. 2019. Test activities in the continuous integration and delivery pipeline. Journal of Software: Evolution and Process (2019), e2153.
[23]
K. V. R. Paixão, C. Z. Felício, F. M. Delfim, and M. de A. Maia. 2017. On the Interplay Between Non-functional Requirements and Builds on Continuous Integration. In Proceedings of the 14th International Conference on Mining Software Repositories (Buenos Aires, Argentina) (MSR ’17). 479–482.
[24]
Gustavo Pinto, Marcel Rebouças, and Fernando Castor. 2017. Inadequate Testing, Time Pressure, and (Over) Confidence: A Tale of Continuous Integration Users. In 10th IEEE/ACM International Workshop on Cooperative and Human Aspects of Software Engineering, CHASE@ICSE 2017. 74–77.
[25]
F. Rahman, C. Bird, and P. Devanbu. 2012. Clones: What is that smell?Empirical Software Engineering 17, 4-5 (2012), 503–530.
[26]
John S Reel. 1999. Critical success factors in software projects. IEEE software 16, 3 (1999), 18–23.
[27]
Jeanine Romano, Jeffrey D Kromrey, Jesse Coraggio, and Jeff Skowronek. 2006. Appropriate statistics for ordinal level data: Should we really be using t-test and Cohen’sd for evaluating group differences on the NSSE and other surveys. In annual meeting of the Florida Association of Institutional Research. 1–33.
[28]
Jadson Santos, Daniel Alencar da Costa, and Uirá Kulesza. 2022. Investigating the Impact of Continuous Integration Practices on the Productivity and Quality of Open-Source Projects. In Proceedings of the 16th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement. 137–147.
[29]
D. Saraiva, D. A. Da Costa, U. Kulesza, G. Sizílio, J. G. Neto, R. Coelho, and M Nagappan. 2023. Unveiling the Relationship Between Continuous Integration and Code Coverage. In IEEE/ACM International Conference on Mining Software Repositories (MSR). 247–259.
[30]
Gustavo Sizílio, Daniel Alencar da Costa, and Uirá Kulesza. 2019. An Empirical Study of the Relationship between Continuous Integration and Test Code Evolution. In Proceedings of the 35th International Conferenceon Software Maintenance and Evolution (Cleveland, United States) (ICSME ’19). IEEE, New York, NY, USA.
[31]
Eliezio Soares, Gustavo Sizilio, Jadson Santos, Daniel Alencar da Costa, and Uirá Kulesza. 2022. The effects of continuous integration on software development: a systematic literature review. Empirical Software Engineering 27, 3 (2022), 78.
[32]
Girish Suryanarayana, Ganesh Samarthyam, and Tushar Sharma. 2014. Refactoring for software design smells: managing technical debt. Morgan Kaufmann.
[33]
Michele Tufano, Fabio Palomba, Gabriele Bavota, Rocco Oliveto, Massimiliano Di Penta, Andrea De Lucia, and Denys Poshyvanyk. 2015. When and why your code starts to smell bad. In Proceedings of the 37th International Conference on Software Engineering-Volume 1. IEEE Press, 403–414.
[34]
B. Vasilescu, S. van Schuylenburg, J. Wulms, A. Serebrenik, and M. G. J. van den Brand. 2014. Continuous Integration in a Social-Coding World: Empirical Evidence from GitHub. In Proceedings of the 2014 IEEE International Conference on Software Maintenance and Evolution(ICSME ’14). IEEE Computer Society, Washington, DC, USA, 401–405.
[35]
B. Vasilescu, Y. Yu, H. Wang, P. Devanbu, and V. Filkov. 2015. Quality and Productivity Outcomes Relating to Continuous Integration in GitHub. In Proceedings of the 2015 10th Joint Meeting on Foundations of Software Engineering (Bergamo, Italy) (ESEC/FSE 2015). 805–816.
[36]
B. Vasilescu, Y. Yu, H. Wang, P. Devanbu, and V. Filkov. 2015. Quality and Productivity Outcomes Relating to Continuous Integration in GitHub. In Proceedings of the 2015 10th Joint Meeting on Foundations of Software Engineering (Bergamo, Italy) (ESEC/FSE 2015). ACM, New York, NY, USA, 805–816.
[37]
Carmine Vassallo, Fabio Palomba, Alberto Bacchelli, and Harald C. Gall. 2018. Continuous Code Quality: Are We (Really) Doing That?. In Proceedings of the 33rd ACM/IEEE International Conference on Automated Software Engineering (Montpellier, France) (ASE 2018). ACM, New York, NY, USA, 790–795.
[38]
D. G. Widder, M. Hilton, C. Kästner, and B. Vasilescu. 2019. A conceptual replication of continuous integration pain points in the context of Travis CI. In Proceedings of the ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering. ACM, 647–658.
[39]
Yang Zhang, Bogdan Vasilescu, Huaimin Wang, and Vladimir Filkov. 2018. One size does not fit all: an empirical study of containerized continuous deployment workflows. In Proceedings of the 2018 ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, ESEC/SIGSOFT FSE 2018, Lake Buena Vista, FL, USA, 2018. 295–306.
[40]
Y. Zhao, A. Serebrenik, Y. Zhou, V. Filkov, and B. Vasilescu. 2017. The impact of continuous integration on other software development practices: A large-scale empirical study. In 2017 32nd IEEE/ACM International Conference on Automated Software Engineering (ASE). 60–71.
Index Terms
- Analyzing the Impact of CI Sub-practices on Continuous Code Quality in Open-Source Projects: An Empirical Study
Recommendations
Investigating the Impact of Continuous Integration Practices on the Productivity and Quality of Open-Source Projects
ESEM '22: Proceedings of the 16th ACM / IEEE International Symposium on Empirical Software Engineering and MeasurementBackground: Much research has been conducted to investigate the impact of Continuous Integration (CI) on the productivity and quality of open-source projects. Most of studies have analyzed the impact of adopting a CI server service (e.g, Travis-CI) but ...
Usage, costs, and benefits of continuous integration in open-source projects
ASE '16: Proceedings of the 31st IEEE/ACM International Conference on Automated Software EngineeringContinuous integration (CI) systems automate the compilation, building, and testing of software. Despite CI rising as a big success story in automated software engineering, it has received almost no attention from the research community. For example, ...
Analyzing Continuous Integration Bad Practices in Closed-Source Projects: An Initial Study
SBES '20: Proceedings of the XXXIV Brazilian Symposium on Software EngineeringContinuous integration (CI) is a process widely used in projects of different types and sizes, generally aimed at improving development and also ensuring the quality of the software. However, some bad practices can mitigate or invalidate the beneficial ...
Comments
Information & Contributors
Information
Published In
September 2023
570 pages
ISBN:9798400707872
DOI:10.1145/3613372
Copyright © 2023 ACM.
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 25 September 2023
Check for updates
Author Tags
Qualifiers
- Research-article
- Research
- Refereed limited
Conference
SBES 2023
SBES 2023: XXXVII Brazilian Symposium on Software Engineering
September 25 - 29, 2023
Campo Grande, Brazil
Acceptance Rates
Overall Acceptance Rate 147 of 427 submissions, 34%
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 44Total Downloads
- Downloads (Last 12 months)44
- Downloads (Last 6 weeks)4
Reflects downloads up to 03 Sep 2024
Other Metrics
Citations
Cited By
View allView Options
Get Access
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign inFull Access
View options
View or Download as a PDF file.
PDFeReader
View online with eReader.
eReaderHTML Format
View this article in HTML Format.
HTML Format