research-article

How Software Refactoring Impacts Execution Time

Authors:

Daniele Di Pompeo,

Simone Scalabrino,

Gabriele Bavota,

Vittorio CortellessaAuthors Info & Claims

ACM Transactions on Software Engineering and Methodology (TOSEM), Volume 31, Issue 2

Article No.: 25, Pages 1 - 23

https://doi.org/10.1145/3485136

Published: 24 December 2021 Publication History

Abstract

Refactoring aims at improving the maintainability of source code without modifying its external behavior. Previous works proposed approaches to recommend refactoring solutions to software developers. The generation of the recommended solutions is guided by metrics acting as proxy for maintainability (e.g., number of code smells removed by the recommended solution). These approaches ignore the impact of the recommended refactorings on other non-functional requirements, such as performance, energy consumption, and so forth. Little is known about the impact of refactoring operations on non-functional requirements other than maintainability.

We aim to fill this gap by presenting the largest study to date to investigate the impact of refactoring on software performance, in terms of execution time. We mined the change history of 20 systems that defined performance benchmarks in their repositories, with the goal of identifying commits in which developers implemented refactoring operations impacting code components that are exercised by the performance benchmarks. Through a quantitative and qualitative analysis, we show that refactoring operations can significantly impact the execution time. Indeed, none of the investigated refactoring types can be considered “safe” in ensuring no performance regression. Refactoring types aimed at decomposing complex code entities (e.g., Extract Class/Interface, Extract Method) have higher chances of triggering performance degradation, suggesting their careful consideration when refactoring performance-critical code.

References

[1]

C. Abid, M. Kessentini, V. Alizadeh, M. Dhouadi, and R. Kazman. 2020. How does refactoring impact security when improving quality? A security-aware refactoring approach. IEEE Transactions on Software Engineering (2020), 1–1. DOI:

[2]

Davide Arcelli, Vittorio Cortellessa, Daniele Di Pompeo, Romina Eramo, and Michele Tucci. 2019. Exploiting architecture/runtime model-driven traceability for performance improvement. In IEEE International Conference on Software Architecture (ICSA’19). IEEE, 81–90. DOI:

[3]

Gabriele Bavota, Andrea De Lucia, Andrian Marcus, and Rocco Oliveto. 2014. Recommending refactoring operations in large software systems. In Recommendation Systems in Software Engineering. Springer, 387–419. DOI:

[4]

Gabriele Bavota, Rocco Oliveto, Malcom Gethers, Denys Poshyvanyk, and Andrea De Lucia. 2014. Methodbook: Recommending move method refactorings via relational topic models. IEEE Transactions on Software Engineering (TSE) 40, 7 (2014), 671–694.

Digital Library

[5]

Lubomír Bulej, Vojtěch Horký, Petr Tuma, François Farquet, and Aleksandar Prokopec. 2020. Duet benchmarking: Improving measurement accuracy in the cloud. In Proceedings of the ACM/SPEC International Conference on Performance Engineering (ICPE’20). Association for Computing Machinery, New York, NY, 100–107. DOI:

Digital Library

[6]

Oscar Chaparro, Gabriele Bavota, Andrian Marcus, and Massimiliano Di Penta. 2014. On the impact of refactoring operations on code quality metrics. In 30th IEEE International Conference on Software Maintenance and Evolution. 456–460.

Digital Library

[7]

Jie Chen, Dongjin Yu, Haiyang Hu, Zhongjin Li, and Hua Hu. 2019. Analyzing performance-aware code changes in software development process. In Proceedings of the 27th International Conference on Program Comprehension (ICPC’19), Yann-Gaël Guéhéneuc, Foutse Khomh, and Federica Sarro (Eds.). IEEE/ACM, 300–310. DOI:

Digital Library

[8]

Michael L. Collard, Michael J. Decker, and Jonathan I. Maletic. 2011. Lightweight transformation and fact extraction with the srcML toolkit. In 2011 IEEE 11th International Working Conference on Source Code Analysis and Manipulation. IEEE, 173–184.

Digital Library

[9]

David Daly, William Brown, Henrik Ingo, Jim O’Leary, and David Bradford. 2020. The use of change point detection to identify software performance regressions in a continuous integration system. In Proceedings of the ACM/SPEC International Conference on Performance Engineering (ICPE’20). Association for Computing Machinery, New York, NY, 67–75. DOI:

Digital Library

[10]

A. C. Davison and D. V. Hinkley. 1997. Bootstrap Methods and their Application. Cambridge University Press. DOI:

[11]

S. Demeyer. 2005. Refactor conditionals into polymorphism: What’s the performance cost of introducing virtual calls? In 21st IEEE International Conference on Software Maintenance (ICSM’05). 627–630.

Digital Library

[12]

Zishuo Ding, Jinfu Chen, and Weiyi Shang. 2020. Towards the use of the readily available tests from the release pipeline as performance tests: Are we there yet? In 42nd International Conference on Software Engineering (ICSE’20), Gregg Rothermel and Doo-Hwan Bae (Eds.). ACM, 1435–1446. DOI:

Digital Library

[13]

Andy Georges, Dries Buytaert, and Lieven Eeckhout. 2007. Statistically rigorous Java performance evaluation. In Proceedings of the 22nd Annual ACM SIGPLAN Conference on Object-Oriented Programming Systems, Languages and Applications (OOPSLA’07). Association for Computing Machinery, New York, NY, 57–76. DOI:

Digital Library

[14]

Andy Georges, Lieven Eeckhout, and Dries Buytaert. 2008. Java performance evaluation through rigorous replay compilation. In Proceedings of the 23rd ACM SIGPLAN Conference on Object-Oriented Programming Systems Languages and Applications (OOPSLA’08). Association for Computing Machinery, New York, NY, 367–384. DOI:

Digital Library

[15]

Holger Giese, Leen Lambers, and Christian Zöllner. 2020. From classic to agile: Experiences from more than a decade of project-based modeling education. In ACM/IEEE 23rd International Conference on Model Driven Engineering Languages and Systems (MODELs’20), Virtual Event, Companion Proceedings, Esther Guerra and Ludovico Iovino (Eds.). ACM, 22:1–22:10. DOI:

Digital Library

[16]

Shi Han, Yingnong Dang, Song Ge, Dongmei Zhang, and Tao Xie. 2012. Performance debugging in the large via mining millions of stack traces. In 2012 34th International Conference on Software Engineering (ICSE’12). 145–155. DOI:ZSCC: 0000165 ISSN: 1558-1225.

[17]

K. Herzig and A. Zeller. 2013. The impact of tangled code changes. In 2013 10th Working Conference on Mining Software Repositories (MSR’13). 121–130.

[18]

Guoliang Jin, Linhai Song, Xiaoming Shi, Joel Scherpelz, and Shan Lu. 2012. Understanding and detecting real-world performance bugs. SIGPLAN Notices 47, 6 (June 2012), 77–88. DOI:

Digital Library

[19]

Tomas Kalibera and Richard Jones. 2013. Rigorous benchmarking in reasonable time. In Proceedings of the 2013 International Symposium on Memory Management (ISMM’13). Association for Computing Machinery, New York, NY, 63–74. DOI:

Digital Library

[20]

Tomas Kalibera and Richard Jones. 2020. Quantifying Performance Changes with Effect Size Confidence Intervals. (2020). arXiv:stat.ME/2007.10899.

[21]

Foutse Khomh, Massimiliano Di Penta, Yann-Gaël Guéhéneuc, and Giuliano Antoniol. 2012. An exploratory study of the impact of antipatterns on class change- and fault-proneness. Empirical Software Engineering 17, 3 (2012), 243–275.

Digital Library

[22]

Christoph Laaber and Philipp Leitner. 2018. An evaluation of open-source software microbenchmark suites for continuous performance assessment. In Proceedings of the 15th International Conference on Mining Software Repositories (MSR’18). Association for Computing Machinery, New York, NY, 119–130. DOI:

Digital Library

[23]

Christoph Laaber, Joel Scheuner, and Philipp Leitner. 2019. Software microbenchmarking in the cloud. How bad is it really?Empirical Software Engineering 24, 4 (Aug. 2019), 2469–2508. DOI:

[24]

Christoph Laaber, Stefan Würsten, Harald C. Gall, and Philipp Leitner. 2020. Dynamically reconfiguring software microbenchmarks: Reducing execution time without sacrificing result quality. In 28th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE’20), Virtual Event, Prem Devanbu, Myra B. Cohen, and Thomas Zimmermann (Eds.). ACM, 989–1001. DOI:

Digital Library

[25]

Philipp Leitner and Cor-Paul Bezemer. 2017. An exploratory study of the state of practice of performance testing in Java-based open source projects. In Proceedings of the 8th ACM/SPEC on International Conference on Performance Engineering (ICPE’17). Association for Computing Machinery, New York, NY, 373–384. DOI:

Digital Library

[26]

Bin Lin, Csaba Nagy, Gabriele Bavota, and Michele Lanza. 2019. On the impact of refactoring operations on code naturalness. In Proceedings of the IEEE 26th International Conference on Software Analysis, Evolution and Reengineering (SANER’19). IEEE, 594–598.

[27]

Qi Luo, Aswathy Nair, Mark Grechanik, and Denys Poshyvanyk. 2017. FOREPOST: Finding performance problems automatically with feedback-directed learning software testing. Empirical Software Engineering 22, 1 (2017), 6–56. DOI:

Digital Library

[28]

Wiem Mkaouer, Marouane Kessentini, Adnan Shaout, Patrice Koligheu, Slim Bechikh, Kalyanmoy Deb, and Ali Ouni. 2015. Many-objective software remodularization using NSGA-III. ACM Transactions on Software Engineering and Methodology (TOSEM) 24, 3 (May 2015), 17:1–17:45.

Digital Library

[29]

Rodrigo Morales, Rubén Saborido, Foutse Khomh, Francisco Chicano, and Giuliano Antoniol. 2018. EARMO: An energy-aware refactoring approach for mobile apps. In Proceedings of the 40th International Conference on Software Engineering (ICSE’18). 59.

Digital Library

[30]

Raimund Moser, Pekka Abrahamsson, Witold Pedrycz, Alberto Sillitti, and Giancarlo Succi. 2007. A case study on the impact of refactoring on quality and productivity in an agile team. In Proceedings of the 2nd IFIP Central and East European Conference on Software Engineering Techniques (CEE-SET’07). Springer, 252–266.

[31]

Todd Mytkowicz, Amer Diwan, Matthias Hauswirth, and Peter F. Sweeney. 2009. Producing wrong data without doing anything obviously wrong!SIGPLAN Notices 44, 3 (March 2009), 265–276. DOI:

Digital Library

[32]

Scott Oaks. 2014. Java Performance: The Definitive Guide (1st ed.). O’Reilly Media, Inc.

[33]

David Georg Reichelt, Stefan Kühne, and Wilhelm Hasselbring. 2019. PeASS: A tool for identifying performance changes at code level. In 34th IEEE/ACM International Conference on Automated Software Engineering (ASE’19). IEEE, 1146–1149. DOI:

Digital Library

[34]

Shiquan Ren, Hong Lai, Wenjing Tong, Mostafa Aminzadeh, Xuezhang Hou, and Shenghan Lai. 2010. Nonparametric bootstrapping for hierarchical data. Journal of Applied Statistics 37, 9 (2010), 1487–1498. DOI: arXiv:https://doi.org/10.1080/02664760903046102

[35]

Cagri Sahin, Lori Pollock, and James Clause. 2014. How do code refactorings affect energy usage? In Proceedings of the 8th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM’14). Article 36, 36:1–36:10 pages.

Digital Library

[36]

Juan Pablo Sandoval Alcocer, Fabian Beck, and Alexandre Bergel. 2019. Performance evolution matrix: Visualizing performance variations along software versions. In 2019 Working Conference on Software Visualization (VISSOFT’19). 1–11. DOI:ZSCC: 0000001.

[37]

Juan Pablo Sandoval Alcocer and Alexandre Bergel. 2015. Tracking down performance variation against source code evolution. ACM SIGPLAN Notices 51, 2 (2015), 129–139. DOI:Number: 2 ZSCC: 0000013.

Digital Library

[38]

Juan Pablo Sandoval Alcocer, Alexandre Bergel, and Marco Tulio Valente. Learning from source code history to identify performance failures. In Proceedings of the 7th ACM/SPEC on International Conference on Performance Engineering (ICPE’16). Association for Computing Machinery, 37–48. DOI:ZSCC: 0000024.

[39]

David J. Sheskin. 2007. Handbook of Parametric and Nonparametric Statistical Procedures (4th ed.). Chapman & Hall/CRC.

Digital Library

[40]

Petr Stefan, Vojtech Horky, Lubomir Bulej, and Petr Tuma. 2017. Unit testing performance in java projects: Are we there yet? In Proceedings of the 8th ACM/SPEC on International Conference on Performance Engineering (ICPE’17). Association for Computing Machinery, New York, NY, 401–412. DOI:

Digital Library

[41]

Gábor Szóke, Gábor Antal, Csaba Nagy, Rudolf Ferenc, and Tibor Gyimóthy. 2014. Bulk fixing coding issues and its effects on software quality: Is it worth refactoring? In Proceedings of the 14th International Working Conference on Source Code Analysis and Manipulation (SCAM’14). IEEE, 95–104.

Digital Library

[42]

Cleiton Tavares, Mariza A. S. Bigonha, and Eduardo Figueiredo. 2020. Quantifying the effects of refactorings on bad smells. In Proceedings of the XXXIV Brazilian Symposium on Software Engineering (SBES’20).

Digital Library

[43]

Luca Della Toffola, Michael Pradel, and Thomas R. Gross. 2018. Synthesizing programs that expose performance bottlenecks. In Proceedings of the 2018 International Symposium on Code Generation and Optimization (CGO’18). Association for Computing Machinery, New York, NY, 314–326. DOI:

Digital Library

[44]

Luca Traini, Daniele Di Pompeo, Michele Tucci, Bin Lin, Simone Scalabrino, Gabriele Bavota, Michele Lanza, Rocco Oliveto, and Vittorio Cortellessa. 2020. How Software Refactoring Impacts Execution Time - Replication Package. https://github.com/SEALABQualityGroup/replicationpackage_refperf.

[45]

Nikolaos Tsantalis and Alexander Chatzigeorgiou. 2009. Identification of move method refactoring opportunities. IEEE Transactions on Software Engineering (TSE) 35, 3 (2009), 347–367.

Digital Library

[46]

Nikolaos Tsantalis, Ameya Ketkar, and Danny Dig. 2020. RefactoringMiner 2.0. IEEE Transactions on Software Engineering (2020), 21. DOI:

[47]

Roberto Verdecchia, René Aparicio Saez, Giuseppe Procaccianti, and Patricia Lago. 2018. Empirical evaluation of the energy impact of refactoring code smells. In Proceedings of the 5th International Conference on Information and Communication Technology for Sustainability (ICT4S’18) (EPiC Series in Computing), Vol. 52. 365–383.

Cited By

Laaber CYue TAli S(2024)Evaluating Search-Based Software Microbenchmark PrioritizationIEEE Transactions on Software Engineering10.1109/TSE.2024.3380836(1-16)Online publication date: 2024
https://doi.org/10.1109/TSE.2024.3380836
Di Menna FCortellessa VLucianelli MSardo LTraini L(2024)RADig-X: a Tool for Regressions Analysis of User Digital Experience2024 IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER)10.1109/SANER60148.2024.00043(359-370)Online publication date: 12-Mar-2024
https://doi.org/10.1109/SANER60148.2024.00043
Dorronsoro BAragón-Jurado JJareño Jde la Torre JRuiz P(2024)A Survey on Automatic Source Code Transformation for Green Software GenerationEncyclopedia of Sustainable Technologies10.1016/B978-0-323-90386-8.00122-4(765-779)Online publication date: 2024
https://doi.org/10.1016/B978-0-323-90386-8.00122-4
Show More Cited By

Index Terms

How Software Refactoring Impacts Execution Time
1. Software and its engineering
  1. Software creation and management
    1. Software post-development issues
      1. Maintaining software
      2. Software evolution
    2. Software verification and validation
      1. Empirical software validation
  2. Software organization and properties
    1. Extra-functional properties
      1. Software performance

Recommendations

Aspect-Oriented Refactoring of Legacy Applications: An Evaluation

The primary claimed benefits of aspect-oriented programming (AOP) are that it improves the understandability and maintainability of software applications by modularizing crosscutting concerns. Before there is widespread adoption of AOP, developers need ...
Refactoring of Crosscutting Concerns with Metaphor-Based Heuristics

It has been advocated that Aspect-Oriented Programming (AOP) is an effective technique to improve software maintainability through explicit support for modularising crosscutting concerns. However, in order to take the advantages of AOP, there is a need ...
Tool-Supported Refactoring of Existing Object-Oriented Code into Aspects

Aspect-Oriented Programming (AOP) provides mechanisms for the separation of crosscutting concerns—functionalities scattered through the system and tangled with the base code. Existing systems are a natural testbed for the AOP approach since they often ...

Comments

Information & Contributors

Information

Published In

cover image ACM Transactions on Software Engineering and Methodology

ACM Transactions on Software Engineering and Methodology Volume 31, Issue 2

April 2022

789 pages

ISSN:1049-331X

EISSN:1557-7392

DOI:10.1145/3492439

Editor:
Mauro Pezzè
USI Università della Svizzera italiana and SIT Schaffhausen Institute of Technology, Switzerland

Issue’s Table of Contents

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 24 December 2021

Accepted: 01 August 2021

Revised: 01 June 2021

Received: 01 December 2020

Published in TOSEM Volume 31, Issue 2

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Refereed

Funding Sources

Swiss National Science foundation

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

9
Total Citations
View Citations
1,390
Total Downloads

Downloads (Last 12 months)410
Downloads (Last 6 weeks)36

Reflects downloads up to 12 Aug 2024

Other Metrics

View Author Metrics

Citations

Cited By

Laaber CYue TAli S(2024)Evaluating Search-Based Software Microbenchmark PrioritizationIEEE Transactions on Software Engineering10.1109/TSE.2024.3380836(1-16)Online publication date: 2024
https://doi.org/10.1109/TSE.2024.3380836
Di Menna FCortellessa VLucianelli MSardo LTraini L(2024)RADig-X: a Tool for Regressions Analysis of User Digital Experience2024 IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER)10.1109/SANER60148.2024.00043(359-370)Online publication date: 12-Mar-2024
https://doi.org/10.1109/SANER60148.2024.00043
Dorronsoro BAragón-Jurado JJareño Jde la Torre JRuiz P(2024)A Survey on Automatic Source Code Transformation for Green Software GenerationEncyclopedia of Sustainable Technologies10.1016/B978-0-323-90386-8.00122-4(765-779)Online publication date: 2024
https://doi.org/10.1016/B978-0-323-90386-8.00122-4
Meng FYu HChong CWang YZhu Z(2024)Automated construction of reference model for software remodularization through software evolutionJournal of Software: Evolution and Process10.1002/smr.2700Online publication date: 19-Jun-2024
https://doi.org/10.1002/smr.2700
Traini LCortellessa V(2023)DeLag: Using Multi-Objective Optimization to Enhance the Detection of Latency Degradation Patterns in Service-Based SystemsIEEE Transactions on Software Engineering10.1109/TSE.2023.326604149:6(3554-3580)Online publication date: 1-Jun-2023
https://dl.acm.org/doi/10.1109/TSE.2023.3266041
Treude C(2023)Navigating Complexity in Software Engineering: A Prototype for Comparing GPT-n Solutions2023 IEEE/ACM 5th International Workshop on Bots in Software Engineering (BotSE)10.1109/BotSE59190.2023.00008(1-5)Online publication date: May-2023
https://doi.org/10.1109/BotSE59190.2023.00008
Stefanakos ICalinescu RGerasimou S(2023)Probabilistic program performance analysis with confidence intervalsInformation and Software Technology10.1016/j.infsof.2022.107143156:COnline publication date: 1-Apr-2023
https://dl.acm.org/doi/10.1016/j.infsof.2022.107143
Traini LCortellessa VDi Pompeo DTucci M(2023)Towards effective assessment of steady state performance in Java software: are we there yet?Empirical Software Engineering10.1007/s10664-022-10247-x28:1Online publication date: 1-Jan-2023
https://dl.acm.org/doi/10.1007/s10664-022-10247-x
Reichelt DKühne SHasselbring W(2022)Automated Identification of Performance Changes at Code Level2022 IEEE 22nd International Conference on Software Quality, Reliability and Security (QRS)10.1109/QRS57517.2022.00096(916-925)Online publication date: Dec-2022
https://doi.org/10.1109/QRS57517.2022.00096

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Full Text

View this article in Full Text.

HTML Format

View this article in HTML Format.

Media

Figures

Other

Tables

View full text|Download PDF

View Issue’s Table of Contents