research-article

Open access

A Model to Detect Readability Improvements in Incremental Changes

Authors:

Sarah Fakhoury,

Venera ArnaoudovaAuthors Info & Claims

ICPC '20: Proceedings of the 28th International Conference on Program Comprehension

Pages 25 - 36

https://doi.org/10.1145/3387904.3389255

Published: 12 September 2020 Publication History

Abstract

Identifying source code that has poor readability allows developers to focus maintenance efforts on problematic code. Therefore, the effort to develop models that can quantify the readability of a piece of source code has been an area of interest for software engineering researchers for several years. However, recent research questions the usefulness of these readability models in practice. When applying these models to readability improvements that are made in practice, i.e., commits, they are unable to capture these incremental improvements, despite a clear perceived improvement by the developers. This results in a discrepancy between the models we have built to measure readability, and the actual perception of readability in practice.

In this work, we propose a model that is able to detect incremental readability improvements made by developers in practice with an average precision of 79.2% and an average recall of 67% on an unseen test set. We then investigate the metrics that our model associates with developer perceived readability improvements as well as non-readability changes. Finally, we compare our model to existing state-of-the-art readability models, which our model outperforms by at least 23% in terms of precision and 42% in terms of recall.

References

[1]

[n. d.]. PMD. https://pmd.github.io/latest/index.html, last accessed on January 2020.

[2]

[n. d.]. Resource Standard Metrics (RSM). https://msquaredtechnologies.com/Resource-Standard-Metrics.html, last accessed on January 2020.

[3]

Christophe Ambroise and Geoffrey J McLachlan. 2002. Selection bias in gene extraction on the basis of microarray gene-expression data. Proceedings of the national academy of sciences 99, 10 (2002), 6562--6566.

[4]

Gabriele Bavota, Bogdan Dit, Rocco Oliveto, Massimiliano Di Penta, Denys Poshyvanyk, and Andrea De Lucia. 2013. An empirical study on the developers' perception of software coupling. In Proceedings of the International Conference on Software Engineering (ICSE). 692--701.

[5]

David Binkley, Marcia Davis, Dawn Lawrie, and Christopher Morrell. 2009. To CamelCase or Under_score. In Proceedings of the International Conference on Program Comprehension (ICPC). 158--167.

[6]

Jürgen Börstler, Michael E Caspersen, and Marie Nordström. 2016. Beauty and the Beast: on the readability of object-oriented example programs. Software quality journal 24, 2 (2016), 231--246.

[7]

Oliver Burn. 2005. Checkstyle homepage. URL http://checkstyle.sourceforge.net/. last accessed in March 14 (2005).

[8]

Raymond P.L. Buse and Westley R. Weimer. 2010. Learning a Metric for Code Readability. IEEE Transactions on Software Engineering (TSE) 36, 4 (July 2010), 546--558.

Digital Library

[9]

B. D. Chaudhary and H. V. Sahasrabuddhe. 1980. Meaningfulness As a Factor of Program Complexity. In Proceedings of the ACM Annual Conference. 457--466. http://dl.acm.org/citation.cfm?id=810001&dl=ACM&coll=DL&CFID=953873442&CFTOKEN=18528998.

[10]

Tianqi Chen and Carlos Guestrin. 2016. Xgboost: A scalable tree boosting system. In Proceedings of the 22nd acm sigkdd international conference on knowledge discovery and data mining. 785--794.

Digital Library

[11]

Davide Chicco and Giuseppe Jurman. 2020. The advantages of the Matthews correlation coefficient (MCC) over F1 score and accuracy in binary classification evaluation. BMC genomics 21, 1 (2020), 6.

[12]

Jacob Cohen. 1988. Statistical power analysis for the behavioral sciences (2nd edition ed.). Lawrence Earlbaum Associates.

[13]

T. A. Corbi. 1989. Program understanding: Challenge for the 1990s. IBM Systems Journal 28, 2 (1989), 294--306. http://dl.acm.org/citation.cfm?id=97124.

Digital Library

[14]

Steve Counsell, Stephen Swift, Allan Tucker, and Emilia Mendes. 2006. Object-oriented cohesion subjectivity amongst experienced and novice developers: an empirical study. ACM SIGSOFT Software Engineering Notes 31, 5 (Sept. 2006), 1--10.

Digital Library

[15]

Jonathan Dorn. 2012. A general software readability model. Master's thesis. University of Virginia.

[16]

Sarah Fakhoury, Devjeet Roy, Adnan Hassan, and Vernera Arnaoudova. 2019. Improving source code readability: theory and practice. In 2019 IEEE/ACM 27th International Conference on Program Comprehension (ICPC). IEEE, 2--12.

Digital Library

[17]

Beat Fluri and Harald C. Gall. 2006. Classifying Change Types for Qualifying Change Couplings. In Proceedings of the International Conference on Program Comprehension (ICPC). 35--45.

[18]

Beat Fluri, Michael Wuersch, Martin PInzger, and Harald Gall. 2007. Change distilling: Tree differencing for fine-grained source code change extraction. IEEE Transactions on software engineering 33, 11 (2007).

Digital Library

[19]

Isabelle Guyon, Jason Weston, Stephen Barnhill, and Vladimir Vapnik. 2002. Gene selection for cancer classification using support vector machines. Machine learning 46, 1-3 (2002), 389--422.

[20]

Tibor Gyimóthy, Rudolf Ferenc, and István Siket. 2005. Empirical Validation of Object-Oriented Metrics on Open Source Software for Fault Prediction. IEEE Transactions on Software Engineering (TSE) 31, 10 (October 2005), 897--910.

Digital Library

[21]

Maurice H Halstead. 1977. Elements of software science. (1977).

[22]

Yasutaka Kamei, Shinsuke Matsumoto, Akito Monden, Ken ichi Matsumoto, Bram Adams, and Ahmed E. Hassan. 2010. Revisiting Common Bug Prediction Findings Using Effort-Aware Models. In Proceedings of the International Conference on Software Maintenance (ICSM). 1--10.

[23]

Guolin Ke, Qi Meng, Thomas Finley, Taifeng Wang, Wei Chen, Weidong Ma, Qiwei Ye, and Tie-Yan Liu. 2017. Lightgbm: A highly efficient gradient boosting decision tree. In Advances in neural information processing systems. 3146--3154.

Digital Library

[24]

Michele Lanza and Radu Marinescu. 2007. Object-oriented metrics in practice: using software metrics to characterize, evaluate, and improve the design of object-oriented systems. Springer Science & Business Media.

Digital Library

[25]

Dawn Lawrie, Christopher Morrell, Henry Feild, and David Binkley. 2006. What's in a Name? A Study of Identifiers. In Proceedings of International Conference on Program Comprehension (ICPC). 3--12.

[26]

Dawn Lawrie, Christopher Morrell, Henry Feild, and David Binkley. 2007. Effective Identifier Names for Comprehension and Memory. Innovations in Systems and Software Engineering 3, 4 (December 2007), 303--318.

[27]

FrontEndART Software Ltd. [n. d.]. SourceMeter. https://www.sourcemeter.com/, last accessed on March 15, 2019.

[28]

Scott M Lundberg and Su-In Lee. 2017. A Unified Approach to Interpreting Model Predictions. In Advances in Neural Information Processing Systems 30, I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (Eds.). Curran Associates, Inc., 4765--4774. http://papers.nips.cc/paper/7062-a-unified-approach-to-interpreting-model-predictions.pdf

[29]

Andrian Marcus, Denys Poshyvanyk, and Rudolf Ferenc. 2008. Using the Conceptual Cohesion of Classes for Fault Prediction in Object-Oriented Systems. IEEE Transactions on Software Engineering (TSE) 34, 2 (2008), 287--30.

Digital Library

[30]

Matias Martinez and Martin Monperrus. 2019. Coming: a tool for mining change pattern instances from git commits. In 2019 IEEE/ACM 41st International Conference on Software Engineering: Companion Proceedings (ICSE-Companion). IEEE, 79--82.

Digital Library

[31]

B. W Matthews. 1975. Comparison of the predicted and observed secondary structure of T4 phage lysozyme. Biochimica et Biophysica Acta (BBA 2, 405 (1975), 442--451.

[32]

Thomas J. McCabe. 1976. A Complexity Measure. IEEE Transactions on Software Engineering (TSE) SE-2, 4 (1976), 308--320.

Digital Library

[33]

Thilo Mende and Rainer Koschke. 2009. Revisiting the evaluation of defect prediction models. In Proceedings of the International Conference on Predictor Models in Software Engineering (PROMISE). 7:1--7:10.

Digital Library

[34]

Qing Mi, Jacky Keung, Yan Xiao, Solomon Mensah, and Yujin Gao. 2018. Improving code readability classification using convolutional neural networks. Information and Software Technology 104 (2018), 60--71.

[35]

Mohamed Wiem Mkaouer, Marouane Kessentini, Slim Bechikh, Mel Ó Cinnéide, and Kalyanmoy Deb. 2016. On the use of many quality attributes for software refactoring: a many-objective search-based software engineering approach. Empirical Software Engineering 21, 6 (2016), 2503--2545.

Digital Library

[36]

Naouel Moha, Yann-Gaël Guéhéneuc, Laurence Duchien, and Anne-Françoise Le Meur. 2010. DECOR: A Method for the Specification and Detection of Code and Design Smells. IEEE Transactions on Software Engineering (TSE) 36, 1 (January-February 2010), 20--36.

Digital Library

[37]

Nuthan Munaiah, Steven Kroh, Craig Cabrey, and Meiyappan Nagappan. 2017. Curating GitHub for engineered software projects. Empirical Software Engineering 22, 6 (2017), 3219--3253.

Digital Library

[38]

Jevgenija Pantiuchina, Michele Lanza, and Gabriele Bavota. 2018. Improving Code: The (Mis) perception of Quality Metrics. In 2018 IEEE International Conference on Software Maintenance and Evolution (ICSME). IEEE, 80--91.

[39]

Daryl Posnett, Abram Hindle, and Premkumar Devanbu. 2011. A Simpler Model of Software Readability. In Proceedings of the Working Conference on Mining Software Repositories (MSR). 73--82.

Digital Library

[40]

Kata Praditwong, Mark Harman, and Xin Yao. 2011. Software module clustering as a multi-objective search problem. IEEE Transactions on Software Engineering 37, 2 (2011), 264--282.

Digital Library

[41]

Meghan Revelle, Malcom Gethers, and Denys Poshyvanyk. 2011. Using Structural and Textual Information to Capture Feature Coupling in Object-oriented Software. Empirical Software Engineering (EMSE) 16, 6 (December 2011), 773--811.

[42]

Devjeet Roy. 2020. Online Replication Package. https://github.com/devjeetr/a-model-to-detect-readability-improvements-in-incremental-changes

[43]

Witold R Rudnicki, Marcin Kierczak, Jacek Koronacki, and Jan Komorowski. 2006. A statistical method for determining importance of variables in an information system. In International Conference on Rough Sets and Current Trends in Computing. Springer, 557--566.

Digital Library

[44]

Simone Scalabrino, Gabriele Bavota, Christopher Vendome, Mario Linares-Vasquez, Denys Poshyvanyk, and Rocco Oliveto. 2019. Automatically Assessing Code Understandability. IEEE Transactions on Software Engineering (2019), 1--1. https://doi.org/10.1109/tse.2019.2901468

[45]

Simone Scalabrino, Mario Linares-Vásquez, Rocco Oliveto, and Denys Poshyvanyk. 2018. A comprehensive model for code readability. Journal of Software: Evolution and Process 30, 6 (2018), e1958.

Digital Library

[46]

Simone Scalabrino, Mario Linares-Vasquez, Denys Poshyvanyk, and Rocco Oliveto. 2016. Improving code readability models with textual features. In Proceedings of the International Conference on Program Comprehension (ICPC). 1--10.

[47]

Ben Shneiderman and Richard Mayer. 1975. Towards a cognitive model of progammer behavior. Technical Report 37. Indiana University.

[48]

Thomas A. Standish. 1984. An Essay on Software Reuse. IEEE Transactions on Software Engineering (TSE) 10, 5 (September 1984), 494--497.

[49]

Armstrong A. Takang, Penny A. Grubb, and Robert D. Macredie. 1996. The Effects of Comments and Identifier Names on Program Comprehensibility: An Experiential Study. Journal of Program Languages 4, 3 (1996), 143--167.

[50]

Rebecca Tiarks. 2011. What Maintenance Programmers Really Do: An Observational Study. In Proceedings of the Workshop Software Reengineering (WSR). 36--37.

[51]

Nikolaos Tsantalis, Matin Mansouri, Laleh M Eshkevari, Davood Mazinanian, and Danny Dig. 2018. Accurate and Efficient Refactoring Detection in Commit History. (2018), 483--494.

[52]

Sudhir Varma and Richard Simon. 2006. Bias in error estimation when using cross-validation for model selection. BMC bioinformatics 7, 1 (2006), 91.

[53]

Anneliese von Mayrhauser, A. Marie Vans, and Adele E. Howe. 1997. Program Understanding Behaviour During Enhancement of Large-scale Software. Journal of Software Maintenance: Research and Practice 9, 5 (September 1997), 299--327.

[54]

Elaine J. Weyuker, Thomas J. Ostrand, and Robert M. Bell. 2010. Comparing the effectiveness of several modeling methods for fault prediction. Empirical Software Engineering (EMSE) 15, 3 (June 2010), 277--295.

[55]

Ian R White, Patrick Royston, and Angela M Wood. 2011. Multiple imputation using chained equations: issues and guidance for practice. Statistics in medicine 30, 4 (2011), 377--399.

[56]

Claes Wohlin, Per Runeson, Höst Martin, Magnus C. Ohlsson, Björn Regnell, and Anders Wesslén. 2000. Experimentation in Software Engineering - An Introduction. Kluwer Academic Publishers.

Digital Library

[57]

S. N. Woodfield, H. E. Dunsmore, and V. Y. Shen. 1981. The Effect of Modularization and Comments on Program Comprehension. In Proceedings of the International Conference on Software Engineering (ICSE). 215--223.

[58]

X. Xia, L. Bao, D. Lo, Z. Xing, A. E. Hassan, and S. Li. 2018. Measuring Program Comprehension: A Large-Scale Field Study with Professionals. IEEE Transactions on Software Engineering (TSE) 44, 10 (Oct 2018), 951--976. https://doi.org/10.1109/TSE.2017.2734091

Digital Library

[59]

Robert K. Yin. 1994. Case Study Research: Design and Methods (2nd ed.). Sage Publications.

[60]

Yuming Zhou and Hareton Leung. 2006. Empirical Analysis of Object-Oriented Design Metrics for Predicting High and Low Severity Faults. IEEE Transactions on Software Engineering (TSE) 32, 10 (October 2006), 771--789.

[61]

Thomas Zimmermann and Nachiappan Nagappan. 2008. Predicting defects using network analysis on dependency graphs. In Proceedings of the International Conference on Software Engineering (ICSE). ACM, 531--540.

Digital Library

Cited By

Oliveira DSantos Rde Oliveira BMonperrus MCastor FMadeiral F(2025)Understanding Code Understandability Improvements in Code ReviewsIEEE Transactions on Software Engineering10.1109/TSE.2024.345378351:1(14-37)Online publication date: 1-Jan-2025
https://dl.acm.org/doi/10.1109/TSE.2024.3453783
Karanikiotis TDiamantopoulos TSymeonidis A(2023)Employing Source Code Quality Analytics for Enriching Code Snippets DataData10.3390/data80901408:9(140)Online publication date: 31-Aug-2023
https://doi.org/10.3390/data8090140
Bastías ODíaz JLópez Fenner J(2023)Exploring the Intersection between Software Maintenance and Machine Learning—A Systematic Mapping StudyApplied Sciences10.3390/app1303171013:3(1710)Online publication date: 29-Jan-2023
https://doi.org/10.3390/app13031710
Show More Cited By

Index Terms

A Model to Detect Readability Improvements in Incremental Changes
1. Computing methodologies
  1. Machine learning
2. Software and its engineering
  1. Software creation and management
    1. Software post-development issues
      1. Maintaining software

Recommendations

Improving source code readability: theory and practice
ICPC '19: Proceedings of the 27th International Conference on Program Comprehension

There are several widely accepted metrics to measure code quality that are currently being used in both research and practice to detect code smells and to find opportunities for code improvement. Although these metrics have been proposed as a proxy of ...
A simpler model of software readability
MSR '11: Proceedings of the 8th Working Conference on Mining Software Repositories

Software readability is a property that influences how easily a given piece of code can be read and understood. Since readability can affect maintainability, quality, etc., programmers are very concerned about the readability of code. If automatic ...
Adaptation of Classic Readability Metrics to Czech
Text, Speech, and Dialogue
Abstract
We have fitted four classic readability metrics to Czech, using InterCorp (a parallel corpus with manual sentence alignment), CzEng 2.0 (a large parallel corpus of crawled web texts), and the optimize.curve fit algorithm from the SciPy library. ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

ICPC '20: Proceedings of the 28th International Conference on Program Comprehension

July 2020

481 pages

ISBN:9781450379588

DOI:10.1145/3387904

Copyright © 2020 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 12 September 2020

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed limited

Conference

ICPC '20

Sponsor:

SIGSOFT

ICPC '20: 28th International Conference on Program Comprehension

July 13 - 15, 2020

Seoul, Republic of Korea

Upcoming Conference

ICSE 2025

2025 IEEE/ACM 46th International Conference on Software Engineering

April 26 - May 3, 2025

Ottawa , ON , Canada

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

9
Total Citations
View Citations
577
Total Downloads

Downloads (Last 12 months)161
Downloads (Last 6 weeks)15

Reflects downloads up to 04 Feb 2025

Other Metrics

View Author Metrics

Citations

Cited By

Oliveira DSantos Rde Oliveira BMonperrus MCastor FMadeiral F(2025)Understanding Code Understandability Improvements in Code ReviewsIEEE Transactions on Software Engineering10.1109/TSE.2024.345378351:1(14-37)Online publication date: 1-Jan-2025
https://dl.acm.org/doi/10.1109/TSE.2024.3453783
Karanikiotis TDiamantopoulos TSymeonidis A(2023)Employing Source Code Quality Analytics for Enriching Code Snippets DataData10.3390/data80901408:9(140)Online publication date: 31-Aug-2023
https://doi.org/10.3390/data8090140
Bastías ODíaz JLópez Fenner J(2023)Exploring the Intersection between Software Maintenance and Machine Learning—A Systematic Mapping StudyApplied Sciences10.3390/app1303171013:3(1710)Online publication date: 29-Jan-2023
https://doi.org/10.3390/app13031710
Dantas CRocha AMaia M(2023)How do Developers Improve Code Readability? An Empirical Study of Pull Requests2023 IEEE International Conference on Software Maintenance and Evolution (ICSME)10.1109/ICSME58846.2023.00022(110-122)Online publication date: 1-Oct-2023
https://doi.org/10.1109/ICSME58846.2023.00022
Yang YXia XLo DBi TGrundy JYang X(2022)Predictive Models in Software Engineering: Challenges and OpportunitiesACM Transactions on Software Engineering and Methodology10.1145/350350931:3(1-72)Online publication date: 9-Apr-2022
https://dl.acm.org/doi/10.1145/3503509
Newman CDecker MAlsuhaibani RPeruma AMkaouer MMohapatra SVishnoi TZampieri MSheldon THill E(2022)An Ensemble Approach for Annotating Source Code Identifiers With Part-of-Speech TagsIEEE Transactions on Software Engineering10.1109/TSE.2021.309824248:9(3506-3522)Online publication date: 1-Sep-2022
https://doi.org/10.1109/TSE.2021.3098242
Ufuktepe ETuglular TPalaniappan K(2022)Tracking Code Bug Fix Ripple Effects Based on Change Patterns Using Markov Chain ModelsIEEE Transactions on Reliability10.1109/TR.2022.316794371:2(1141-1156)Online publication date: Jun-2022
https://doi.org/10.1109/TR.2022.3167943
Ufuktepe ETuglular TPalaniappan K(2021)The Relation between Bug Fix Change Patterns and Change Impact Analysis2021 IEEE 21st International Conference on Software Quality, Reliability and Security (QRS)10.1109/QRS54544.2021.00117(1089-1099)Online publication date: Dec-2021
https://doi.org/10.1109/QRS54544.2021.00117
Oliveira D(2021)Recommending Code Understandability Improvements Based on Code Reviews2021 36th IEEE/ACM International Conference on Automated Software Engineering Workshops (ASEW)10.1109/ASEW52652.2021.00035(131-132)Online publication date: Nov-2021
https://doi.org/10.1109/ASEW52652.2021.00035

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Figures

Tables

Media

View Table of Conten