research-article

Continuous code quality: are we (really) doing that?

Authors:

Carmine Vassallo,

Alberto Bacchelli,

Harald C. GallAuthors Info & Claims

ASE '18: Proceedings of the 33rd ACM/IEEE International Conference on Automated Software Engineering

Pages 790 - 795

https://doi.org/10.1145/3238147.3240729

Published: 03 September 2018 Publication History

Abstract

Continuous Integration (CI) is a software engineering practice where developers constantly integrate their changes to a project through an automated build process. The goal of CI is to provide developers with prompt feedback on several quality dimensions after each change. Indeed, previous studies provided empirical evidence on a positive association between properly following CI principles and source code quality. A core principle behind CI is Continuous Code Quality (also known as CCQ, which includes automated testing and automated code inspection) may appear simple and effective, yet we know little about its practical adoption. In this paper, we propose a preliminary empirical investigation aimed at understanding how rigorously practitioners follow CCQ. Our study reveals a strong dichotomy between theory and practice: developers do not perform continuous inspection but rather control for quality only at the end of a sprint and most of the times only on the release branch. Preprint [https://doi.org/10.5281/zenodo.1341036]. Data and Materials [http://doi.org/10.5281/zenodo.1341015].

References

[1]

Kent Beck, Mike Beedle, Arie Van Bennekum, Alistair Cockburn, Ward Cunningham, Martin Fowler, James Grenning, Jim Highsmith, Andrew Hunt, Ron Jeffries, et al. 2001. The agile manifesto.

[2]

Moritz Beller, Georgios Gousios, and Andy Zaidman. 2017. Oops, my tests broke the build: An explorative analysis of Travis CI with GitHub. In International Conference on Mining Software Repositories.

Digital Library

[3]

Moritz Beller, Georgios Gousios, and Andy Zaidman. 2017. TravisTorrent: Synthesizing Travis CI and GitHub for Full-Stack Research on Continuous Integration. In Proceedings of the 14th working conference on mining software repositories.

Digital Library

[4]

Nigel Bevan. 1999. Quality in use: Meeting user needs for quality. Journal of systems and software 49, 1 (1999), 89–96.

Digital Library

[5]

Andrea Bonaccorsi and Cristina Rossi Lamastra. 2004. Altruistic individuals, selfish firms? The structure of motivation in Open Source software. (2004).

[6]

Laura Dabbish, Colleen Stuart, Jason Tsay, and Jim Herbsleb. 2012. Social coding in GitHub: transparency and collaboration in an open software repository. In Proceedings of the ACM 2012 conference on Computer Supported Cooperative Work. ACM, 1277–1286.

Digital Library

[7]

Nicolas Ducheneaut. 2005. Socialization in an open source software community: A socio-technical analysis. Computer Supported Cooperative Work (CSCW) 14, 4 (2005), 323–368.

Digital Library

[8]

Paul Duvall, Stephen M. Matyas, and Andrew Glover. 2007. Continuous Integration: Improving Software Quality and Reducing Risk. Addison-Wesley.

Digital Library

[9]

Paul M. Duvall. 2010. Continuous integration. Patterns and Antipatterns. DZone refcard #84 (2010). http://bit.ly/l8rf VS

[10]

Neil A Ernst and John Mylopoulos. 2010. On the perception of software quality requirements during the project lifecycle. In International Working Conference on Requirements Engineering: Foundation for Software Quality. Springer, 143–157.

[11]

Simon Gibbs, Eduardo Casais, Oscar Nierstrasz, Xavier Pintado, and Dennis Tsichritzis. 1990. Class management for software communities. Commun. ACM 33, 9 (1990), 90–103.

Digital Library

[12]

GitHub. 2018. GitHub APIs. https://developer.github.com/v3/. Online; accessed 24 July 2018.

[13]

Michael Hilton, Nicholas Nelson, Timothy Tunnell, Darko Marinov, and Danny Dig. 2017. Trade-Offs in Continuous Integration: Assurance, Security, and Flexibility. In Proceedings of the 25th ACM SIGSOFT International Symposium on Foundations of Software Engineering, FSE 2017. To Appear.

Digital Library

[14]

Michael Hilton, Timothy Tunnell, Kai Huang, Darko Marinov, and Danny Dig. 2016. Usage, costs, and benefits of continuous integration in open-source projects. In Proceedings of the 31st IEEE/ACM International Conference on Automated Software Engineering (ASE). 426–437.

Digital Library

[15]

Jez Humble and David Farley. 2010. Continuous Delivery: Reliable Software Releases Through Build, Test, and Deployment Automation. Addison-Wesley Professional.

Digital Library

[16]

Eirini Kalliamvakou, Georgios Gousios, Kelly Blincoe, Leif Singer, Daniel M German, and Daniela Damian. 2016. An in-depth study of the promises and perils of mining GitHub. Empirical Software Engineering 21, 5 (2016), 2035–2071.

Digital Library

[17]

Foutse Khomh, Bram Adams, Tejinder Dhaliwal, and Ying Zou. 2015. Understanding the impact of rapid releases on software quality - The case of firefox. Empirical Software Engineering 20, 2 (2015), 336–373.

Digital Library

[18]

Seojin Kim, Sungjin Park, Jeonghyun Yun, and Younghoo Lee. 2008. Automated continuous integration of component-based software: An industrial experience. In Proceedings of the 2008 23rd IEEE/ACM International Conference on Automated Software Engineering. IEEE Computer Society, 423–426.

Digital Library

[19]

Irwin Kwan, Adrian Schroter, and Daniela Damian. 2011. Does socio-technical congruence have an effect on software build success? a study of coordination in a software project. IEEE Transactions on Software Engineering 37, 3 (2011), 307–324.

Digital Library

[20]

Lina Lagerberg, Tor Skude, Par Emanuelsson, Kristian Sandahl, and Daniel Stahl. 2013. The impact of agile principles and practices on large-scale software development projects: A multiple-case study of two projects at ericsson. In Empirical Software Engineering and Measurement, 2013 ACM/IEEE International Symposium on. IEEE, 348–356.

[21]

E. Laukkanen, M. Paasivaara, and T. Arvonen. 2015. Stakeholder Perceptions of the Adoption of Continuous Integration – A Case Study. In Agile Conference (AGILE), 2015. 11–20.

Digital Library

[22]

Eero Laukkanen, Maria Paasivaara, and Teemu Arvonen. 2015. Stakeholder Perceptions of the Adoption of Continuous Integration–A Case Study. In Agile Conference (AGILE), 2015. IEEE, 11–20.

Digital Library

[23]

Mikael Lindvall, Dirk Muthig, Aldo Dagnino, Christina Wallin, Michael Stupperich, David Kiefer, John May, and Tuomo Kahkonen. 2004. Agile software development in large organizations. Computer 37, 12 (2004), 26–34.

Digital Library

[24]

Jennifer Marlow, Laura Dabbish, and Jim Herbsleb. 2013. Impression formation in online peer production: activity traces and personal profiles in github. In Proceedings of the 2013 conference on Computer supported cooperative work. ACM, 117–128.

Digital Library

[25]

Fabio Palomba, Gabriele Bavota, Massimiliano Di Penta, Rocco Oliveto, and Andrea De Lucia. 2014. Do they really smell bad? a study on developers’ perception of bad code smells. In Software maintenance and evolution (ICSME), 2014 IEEE international conference on. IEEE, 101–110.

Digital Library

[26]

Gerald Schermann, Jürgen Cito, Philipp Leitner, and Harald C. Gall. 2016. Towards quality gates in continuous delivery and deployment. In International Conference on Program Comprehension.

[27]

SonarCloud. 2018. SonarCloud Web APIs. https://sonarcloud.io/web_api. Online; accessed 24 July 2018.

[28]

SonarSource S.A. 2018. SonarQube. https://www.sonarqube.org.

[29]

Damian A Tamburri, Rick Kazman, and Hamed Fahimi. 2016. The Architect’s Role in Community Shepherding. IEEE Software 33, 6 (2016), 70–79.

Digital Library

[30]

Tiobe. 2018. Tiobe Ranking. https://www.tiobe.com/tiobe-index/. Online; accessed 24 July 2018.

[31]

TravisCI. 2018. Travis CI. https://travis-ci.org. Online; accessed 24 July 2018.

[32]

TravisCI. 2018. Travis CI APIs. https://developer.travis-ci.com. Online; accessed 24 July 2018.

[33]

Michele Tufano, Fabio Palomba, Gabriele Bavota, Massimiliano Di Penta, Rocco Oliveto, Andrea De Lucia, and Denys Poshyvanyk. 2017. There and back again: Can you compile that snapshot? Journal of Software: Evolution and Process 29, 4 (2017).

[34]

Michele Tufano, Fabio Palomba, Gabriele Bavota, Rocco Oliveto, Massimiliano Di Penta, Andrea De Lucia, and Denys Poshyvanyk. 2017. When and why your code starts to smell bad (and whether the smells go away). IEEE Transactions on Software Engineering 43, 11 (2017), 1063–1088.

Digital Library

[35]

Bogdan Vasilescu, Yue Yu, Huaimin Wang, Premkumar T. Devanbu, and Vladimir Filkov. 2015. Quality and productivity outcomes relating to continuous integration in GitHub. In ESEC/SIGSOFT FSE. ACM, 805–816.

Digital Library

[36]

Carmine Vassallo, Fabio Palomba, Alberto Bacchelli, and Harald C. Gall. 2018. Continuous Code Quality: Are We (Really) Doing That? Online Appendix.

[37]

Carmine Vassallo, Sebastiano Panichella, Fabio Palomba, Sebastian Proksch, Andy Zaidman, and Harald C. Gall. 2018. Context is king: The developer perspective on the usage of static analysis tools. In SANER. IEEE Computer Society, 38–49.

[38]

Carmine Vassallo, Gerald Schermann, Fiorella Zampetti, Daniele Romano, Philipp Leitner, Andy Zaidman, Massimiliano Di Penta, and Sebastiano Panichella. 2017.

[39]

A Tale of CI Build Failures: An Open Source and a Financial Organization Perspective. In ICSME. IEEE Computer Society, 183–193.

[40]

Carmine Vassallo, Fiorella Zampetti, Daniele Romano, Moritz Beller, Annibale Panichella, Massimiliano Di Penta, and Andy Zaidman. 2016. Continuous Delivery Practices in a Large Financial Organization. In 32nd IEEE International Conference on Software Maintenance and Evolution (ICSME). 41–50.

Cited By

Bhutani VToosi FBuckley J(2024)Analysing the AnalysersApplied Computer Systems10.2478/acss-2024-001329:1(98-111)Online publication date: 15-Aug-2024
https://dl.acm.org/doi/10.2478/acss-2024-0013
Ricós FMarín BVos TNeeft RAho P(2024)Delta GUI change detection using inferred modelsComputer Standards & Interfaces10.1016/j.csi.2024.103925(103925)Online publication date: Sep-2024
https://doi.org/10.1016/j.csi.2024.103925
Khatami AZaidman A(2024)State‐of‐the‐practice in quality assurance in Java‐based open source software developmentSoftware: Practice and Experience10.1002/spe.332154:8(1408-1446)Online publication date: 4-Mar-2024
https://doi.org/10.1002/spe.3321
Show More Cited By

Index Terms

Continuous code quality: are we (really) doing that?
1. Software and its engineering
  1. Software creation and management
    1. Software post-development issues
      1. Maintaining software

Recommendations

Analyzing the Impact of CI Sub-practices on Continuous Code Quality in Open-Source Projects: An Empirical Study
SBES '23: Proceedings of the XXXVII Brazilian Symposium on Software Engineering

Continuous Integration (CI) is a widely adopted practice that automates and enhances the frequency of code integration. Previous research has explored the relationship between CI sub-practices (such as frequent commit activity and high test coverage) ...
"I know it when I see it" Perceptions of Code Quality: ITiCSE '17 Working Group Report
ITiCSE-WGR '17: Proceedings of the 2017 ITiCSE Conference on Working Group Reports

Context. Code quality is a key issue in software development. The ability to develop high quality software is therefore a key learning goal of computing programs. However, there are no universally accepted measures to assess the quality of code and ...
Quality of Code Can Be Planned and Automatically Controlled
VALID '09: Proceedings of the 2009 First International Conference on Advances in System Testing and Validation Lifecycle

Quality of Code is an important and critical health indicator of any software development project. However, due to the complexity and ambiguousness of calculating this indicator it is rarely used in commercial contracts. As programmers are much more ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

ASE '18: Proceedings of the 33rd ACM/IEEE International Conference on Automated Software Engineering

September 2018

955 pages

ISBN:9781450359375

DOI:10.1145/3238147

General Chair:
Marianne Huchard
University of Montpellier, France
,
Program Chairs:
Christian Kästner
Carnegie Mellon University, USA
,
Gordon Fraser
University of Passau, Germany

Copyright © 2018 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGAI: ACM Special Interest Group on Artificial Intelligence
CNRS: Centre National De La Rechercue Scientifique
SIGSOFT: ACM Special Interest Group on Software Engineering
IEEE-CS: Computer Society

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 03 September 2018

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

ASE '18

Sponsor:

SIGAI
CNRS
SIGSOFT
IEEE-CS

ASE '18: 33rd ACM/IEEE International Conference on Automated Software Engineering

September 3 - 7, 2018

Montpellier, France

Acceptance Rates

Overall Acceptance Rate 82 of 337 submissions, 24%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

29
Total Citations
View Citations
565
Total Downloads

Downloads (Last 12 months)43
Downloads (Last 6 weeks)1

Reflects downloads up to 18 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Bhutani VToosi FBuckley J(2024)Analysing the AnalysersApplied Computer Systems10.2478/acss-2024-001329:1(98-111)Online publication date: 15-Aug-2024
https://dl.acm.org/doi/10.2478/acss-2024-0013
Ricós FMarín BVos TNeeft RAho P(2024)Delta GUI change detection using inferred modelsComputer Standards & Interfaces10.1016/j.csi.2024.103925(103925)Online publication date: Sep-2024
https://doi.org/10.1016/j.csi.2024.103925
Khatami AZaidman A(2024)State‐of‐the‐practice in quality assurance in Java‐based open source software developmentSoftware: Practice and Experience10.1002/spe.332154:8(1408-1446)Online publication date: 4-Mar-2024
https://doi.org/10.1002/spe.3321
Freitas GBernardo JSizíLio GCosta DKulesza U(2023)Analyzing the Impact of CI Sub-practices on Continuous Code Quality in Open-Source Projects: An Empirical StudyProceedings of the XXXVII Brazilian Symposium on Software Engineering10.1145/3613372.3613403(1-10)Online publication date: 25-Sep-2023
https://dl.acm.org/doi/10.1145/3613372.3613403
Pornprasit CTantithamthavorn C(2023)DeepLineDP: Towards a Deep Learning Approach for Line-Level Defect PredictionIEEE Transactions on Software Engineering10.1109/TSE.2022.314434849:1(84-98)Online publication date: 1-Jan-2023
https://doi.org/10.1109/TSE.2022.3144348
Etemadi KHarrand NLarsén SAdzemovic HPhu HVerma AMadeiral FWikström DMonperrus M(2023)Sorald: Automatic Patch Suggestions for SonarQube Static Analysis ViolationsIEEE Transactions on Dependable and Secure Computing10.1109/TDSC.2022.316731620:4(2794-2810)Online publication date: 1-Jul-2023
https://doi.org/10.1109/TDSC.2022.3167316
Shi JZou DXu SDeng XJin H(2023)Does OpenBSD and Firefox’s Security Improve With Time?IEEE Transactions on Dependable and Secure Computing10.1109/TDSC.2022.315332520:4(2781-2793)Online publication date: 1-Jul-2023
https://doi.org/10.1109/TDSC.2022.3153325
Khatami AZaidman A(2023)Quality Assurance Awareness in Open Source Software Projects on GitHub2023 IEEE 23rd International Working Conference on Source Code Analysis and Manipulation (SCAM)10.1109/SCAM59687.2023.00027(174-185)Online publication date: 2-Oct-2023
https://doi.org/10.1109/SCAM59687.2023.00027
Sridharan MRantala LMäntylä M(2023)PENTACET data - 23 Million Contextual Code Comments and 250,000 SATD comments2023 IEEE/ACM 20th International Conference on Mining Software Repositories (MSR)10.1109/MSR59073.2023.00063(412-416)Online publication date: May-2023
https://doi.org/10.1109/MSR59073.2023.00063
Giordano GFesta GCatolino GPalomba FFerrucci FGravino C(2023)On the adoption and effects of source code reuse on defect proneness and maintenance effortEmpirical Software Engineering10.1007/s10664-023-10408-629:1Online publication date: 12-Dec-2023
https://doi.org/10.1007/s10664-023-10408-6
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents