research-article

Understanding and detecting real-world performance bugs

Authors:

Joel Scherpelz,

Shan LuAuthors Info & Claims

ACM SIGPLAN Notices, Volume 47, Issue 6

Pages 77 - 88

https://doi.org/10.1145/2345156.2254075

Published: 11 June 2012 Publication History

Abstract

Developers frequently use inefficient code sequences that could be fixed by simple patches. These inefficient code sequences can cause significant performance degradation and resource waste, referred to as performance bugs. Meager increases in single threaded performance in the multi-core era and increasing emphasis on energy efficiency call for more effort in tackling performance bugs.

This paper conducts a comprehensive study of 110 real-world performance bugs that are randomly sampled from five representative software suites (Apache, Chrome, GCC, Mozilla, and MySQL). The findings of this study provide guidance for future work to avoid, expose, detect, and fix performance bugs.

Guided by our characteristics study, efficiency rules are extracted from 25 patches and are used to detect performance bugs. 332 previously unknown performance problems are found in the latest versions of MySQL, Apache, and Mozilla applications, including 219 performance problems found by applying rules across applications.

References

[1]

M. K. Aguilera, J. C. Mogul, J. L. Wiener, P. Reynolds, and A. Muthitacharoen. Performance debugging for distributed systems of black boxes. In SOSP, 2003.

Digital Library

[2]

M. Attariyan and J. Flinn. Automating configuration troubleshooting with dynamic information flow analysis. In OSDI, 2010.

Digital Library

[3]

W. Baek and T. M. Chilimbi. Green: a framework for supporting energy-conscious programming using controlled approximation. In PLDI, 2010.

Digital Library

[4]

C. Cadar, D. Dunbar, and D. Engler. Klee: Unassisted and automatic generation of high-coverage tests for complex systems programs. In OSDI, 2008.

Digital Library

[5]

V. Chipounov, V. Kuznetsov, and G. Candea. S2E: a platform for in-vivo multi-path analysis of software systems. In ASPLOS, 2011.

Digital Library

[6]

A. Chou, B. Chelf, D. Engler, and M. Heinrich. Using meta-level compilation to check FLASH protocol code. In ASPLOS, 2000.

Digital Library

[7]

A. Chou, J. Yang, B. Chelf, S. Hallem, and D. R. Engler. An empirical study of operating system errors. In SOSP, 2001.

Digital Library

[8]

I. Cohen, M. Goldszmidt, T. Kelly, J. Symons, and J. S. Chase. Correlating instrumentation data to system states: a building block for automated diagnosis and control. In OSDI, 2004.

Digital Library

[9]

M. Corporation. Performance Testing Guidance for Web Applications. Microsoft Press, 2007.

Digital Library

[10]

A. Diwan, M. Hauswirth, T. Mytkowicz, and P. F. Sweeney. Traceanalyzer: a system for processing performance traces. Softw. Pract. Exper., March 2011.

Digital Library

[11]

B. Dufour, B. G. Ryder, and G. Sevitsky. A scalable technique for characterizing the usage of temporaries in framework-intensive java applications. In FSE, 2008.

Digital Library

[12]

R. F. Dugan. Performance lies my professor told me: the case for teaching software performance engineering to undergraduates. In WOSP, 2004.

Digital Library

[13]

D. Engler, D. Y. Chen, S. Hallem, A. Chou, and B. Chelf. Bugs as deviant behavior: A general approach to inferring errors in systems code. In SOSP, 2001.

Digital Library

[14]

M. Ernst, A. Czeisler, W. G. Griswold, and D. Notkin. Quickly detecting relevant program invariants. In ICSE, 2000.

Digital Library

[15]

Fortify. HP Fortify Static Code Analyzer (SCA). https://www.fortify.com/products/hpfssc/source-code-analyzer.html.

[16]

M. Gabel, J. Yang, Y. Yu, M. Goldszmidt, and Z. Su. Scalable and systematic detection of buggy inconsistencies in source code. In OOPSLA, 2010.

Digital Library

[17]

P. Godefroid, N. Klarlund, and K. Sen. Dart: directed automated random testing. In PLDI, 2005.

Digital Library

[18]

S. Gulwani, K. K. Mehra, and T. Chilimbi. Speed: precise and efficient static estimation of program computational complexity. In POPL, 2009.

Digital Library

[19]

M. Hauswirth, A. Diwan, P. F. Sweeney, and M. C. Mozer. Automating vertical profiling. In OOPSLA, 2005.

Digital Library

[20]

J. K. Hollingsworth, R. B. Irvin, and B. P. Miller. The integration of application and system based metrics in a parallel program performance tool. In PPOPP, 1991.

Digital Library

[21]

D. Hovemeyer and W. Pugh. Finding bugs is easy. In OOPSLA, 2004.

Digital Library

[22]

InfoWorld. Top 10 open source hall of famers. http://www.infoworld.com/d/open-source/top-10-open-source-hall-famers-848.

[23]

R. B. Irvin and B. P. Miller. Mapping performance data for high-level and data views of parallel program performance. In ICS, 1996.

Digital Library

[24]

JMeter. Java desktop application designed for load tests. http://jakarta.apache.org/jmeter.

[25]

M. Jovic, A. Adamoli, and M. Hauswirth. Catch me if you can: performance bug detection in the wild. In OOPSLA, 2011.

Digital Library

[26]

C. Killian, K. Nagaraj, S. Pervez, R. Braud, J. W. Anderson, and R. Jhala. Finding latent performance bugs in systems implementations. In FSE, 2010.

Digital Library

[27]

C. Lattner and V. Adve. LLVM: A compilation framework for lifelong program analysis & transformation. In CGO, 2004.

Digital Library

[28]

Z. Li, S. Lu, S. Myagmar, and Y. Zhou. CP-Miner: A Tool for Finding Copy-paste and Related Bugs in Operating System Code. In OSDI, 2004.

Digital Library

[29]

Z. Li, L. Tan, X. Wang, Y. Zhou, and C. Zhai. Have things changed now?: an empirical study of bug characteristics in modern open source software. In ASID, 2006.

Digital Library

[30]

Z. Li and Y. Zhou. PR-Miner: Automatically extracting implicit programming rules and detecting violations in large software code. In FSE, Sept 2005.

Digital Library

[31]

S. Liu, K. Pattabiraman, T. Moscibroda, and B. G. Zorn. Flikker: saving DRAM refresh-power through critical data partitioning. In ASPLOS, 2011.

Digital Library

[32]

T. Liu and E. D. Berger. Precise detection and automatic mitigation of false sharing. In OOPSLA, 2011.

Digital Library

[33]

V. B. Livshits and T. Zimmermann. Dynamine: Finding common error patterns by mining software revision histories. In FSE, 2005.

Digital Library

[34]

S. Lu, S. Park, E. Seo, and Y. Zhou. Learning from mistakes -- a comprehensive study of real world concurrency bug characteristics. In ASPLOS, 2008.

Digital Library

[35]

J. Mellor-Crummey, R. J. Fowler, G. Marin, and N. Tallent. Hpcview: A tool for top-down analysis of node performance. J. Supercomput., 23(1):81--104, 2002.

Digital Library

[36]

N. Meng, M. Kim, and K. S. McKinley. Systematic editing: generating program transformations from an example. In PLDI, 2011.

Digital Library

[37]

Microsoft. MSDN SAL annotations. http://msdn2.microsoft.com/en-us/library/ms235402.aspx.

[38]

B. P. Miller, M. D. Callaghan, J. M. Cargille, J. K. Hollingsworth, R. B. Irvin, K. L. Karavanic, K. Kunchithapadam, and T. Newhall. The paradyn parallel performance measurement tool. Computer, 28(11), 1995.

Digital Library

[39]

N. Mitchell. The diary of a datum: an approach to modeling runtime complexity in framework-based applications. In ECOOP, 2006.

Digital Library

[40]

I. Molyneaux. The Art of Application Performance Testing: Help for Programmers and Quality Assurance. O'Reilly Media, 2009.

Digital Library

[41]

G. E. Morris. Lessons from the colorado benefits management system disaster. www.ad-mkt-review.com/publichtml/air/ai200411.html, 2004.

[42]

G. Muller, Y. Padioleau, J. L. Lawall, and R. R. Hansen. Semantic patches considered helpful. Operating Systems Review, 40(3):90--92, 2006.

Digital Library

[43]

N. Palix, G. Thomas, S. Saha, C. Calvès, J. Lawall, and G. Muller. Faults in linux: ten years later. In ASPLOS, 2011.

Digital Library

[44]

S. E. Perl and W. E. Weihl. Performance assertion checking. In SOSP, 1993.

Digital Library

[45]

T. Richardson. 1901 census site still down after six months. http://www.theregister.co.uk/2002/07/03/1901_census_site_still_down/.

[46]

C. J. Rossbach, O. S. Hofmann, and E. Witchel. Is transactional programming actually easier? In PPoPP, 2010.

Digital Library

[47]

A. Shankar, M. Arnold, and R. Bodik. Jolt: lightweight dynamic analysis and removal of object churn. In OOPSLA, 2008.

Digital Library

[48]

K. Shen, C. Stewart, C. Li, and X. Li. Reference-driven performance anomaly identification. In SIGMETRICS, 2009.

Digital Library

[49]

R. L. Sites. Identifying dark latency.

[50]

C. U. Smith and L. G. Williams. Software performance antipatterns. In Proceedings of the 2nd international workshop on Software and performance, 2000.

Digital Library

[51]

Stefan Bodewig. Bug 45396: There is no hint in the javadocs. https://issues.apache.org/ bugzilla/show_bug.cgi?id=45396#c4.

[52]

C. Stewart, K. Shen, A. Iyengar, and J. Yin. Entomomodel: Understanding and avoiding performance anomaly manifestations. In MASCOTS, 2010.

Digital Library

[53]

M. Sullivan and R. Chillarege. A comparison of software defects in database management systems and operating systems. In FTCS, 1992.

[54]

L. Torvalds. Sparse - a semantic parser for c. http://www.kernel.org/pub/software/devel/sparse/.

[55]

J. S. Vetter and P. H. Worley. Asserting performance expectations. In Supercomputing, 2002.

Digital Library

[56]

wikipedia. Chi-squared test. http://en.wikipedia.org/wiki/Chi-squared_test.

[57]

G. Xu, M. Arnold, N. Mitchell, A. Rountev, and G. Sevitsky. Go with the flow: profiling copies to find runtime bloat. In PLDI '09, 2009.

Digital Library

[58]

G. Xu, N. Mitchell, M. Arnold, A. Rountev, E. Schonberg, and G. Sevitsky. Finding low-utility data structures. In PLDI, 2010.

Digital Library

[59]

W. Xu, L. Huang, A. Fox, D. Patterson, and M. I. Jordan. Detecting large-scale system problems by mining console logs. In SOSP, 2009.

Digital Library

[60]

A. Zeller. Isolating cause-effect chains from computer programs. In FSE, 2002.

Digital Library

Cited By

Zhang XJiang WShen CLi QWang QLin CGuan X(2025)Deep Learning Library Testing: Definition, Methods and ChallengesACM Computing Surveys10.1145/3716497Online publication date: 5-Feb-2025
https://dl.acm.org/doi/10.1145/3716497
dos Santos Lde Souza ÉEndo ATrubiani CPinciroli RVijaykumar N(2025)Performance regression testing initiativesInformation and Software Technology10.1016/j.infsof.2024.107641179:COnline publication date: 1-Mar-2025
https://dl.acm.org/doi/10.1016/j.infsof.2024.107641
Venkitaraman A(2024)Early Bug Detection through Shift Left TestingInternational Journal of Innovative Science and Research Technology (IJISRT)10.38124/ijisrt/IJISRT24NOV177(185-190)Online publication date: 18-Nov-2024
https://doi.org/10.38124/ijisrt/IJISRT24NOV177
Show More Cited By

Index Terms

Understanding and detecting real-world performance bugs
1. General and reference
  1. Cross-computing tools and techniques
    1. Performance
2. Software and its engineering
  1. Software creation and management
    1. Software verification and validation
      1. Software defect analysis
        Software testing and debugging
  2. Software organization and properties
    1. Extra-functional properties
      1. Software performance

Recommendations

Understanding and detecting real-world performance bugs
PLDI '12: Proceedings of the 33rd ACM SIGPLAN Conference on Programming Language Design and Implementation

Developers frequently use inefficient code sequences that could be fixed by simple patches. These inefficient code sequences can cause significant performance degradation and resource waste, referred to as performance bugs. Meager increases in single ...
A qualitative study on performance bugs
MSR '12: Proceedings of the 9th IEEE Working Conference on Mining Software Repositories

Software performance is one of the important qualities that makes software stand out in a competitive market. However, in earlier work we found that performance bugs take more time to fix, need to be fixed by more experienced developers and require ...
Security versus performance bugs: a case study on Firefox
MSR '11: Proceedings of the 8th Working Conference on Mining Software Repositories

A good understanding of the impact of different types of bugs on various project aspects is essential to improve software quality research and practice. For instance, we would expect that security bugs are fixed faster than other types of bugs due to ...

Comments

Information & Contributors

Information

Published In

cover image ACM SIGPLAN Notices

ACM SIGPLAN Notices Volume 47, Issue 6

PLDI '12

June 2012

534 pages

ISSN:0362-1340

EISSN:1558-1160

DOI:10.1145/2345156

Issue’s Table of Contents

PLDI '12: Proceedings of the 33rd ACM SIGPLAN Conference on Programming Language Design and Implementation
June 2012
572 pages
ISBN:9781450312059
DOI:10.1145/2254064
General Chairs:
Jan Vitek
Purdue University
,
Haibo Lin
Microsoft China
,
Program Chair:
Frank Tip
IBM T.J. Watson Research Center

Copyright © 2012 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 11 June 2012

Published in SIGPLAN Volume 47, Issue 6

Check for updates

Author Tags

Qualifiers

Research-article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

321
Total Citations
View Citations
3,173
Total Downloads

Downloads (Last 12 months)244
Downloads (Last 6 weeks)20

Reflects downloads up to 18 Feb 2025

Other Metrics

View Author Metrics

Citations

Cited By

Zhang XJiang WShen CLi QWang QLin CGuan X(2025)Deep Learning Library Testing: Definition, Methods and ChallengesACM Computing Surveys10.1145/3716497Online publication date: 5-Feb-2025
https://dl.acm.org/doi/10.1145/3716497
dos Santos Lde Souza ÉEndo ATrubiani CPinciroli RVijaykumar N(2025)Performance regression testing initiativesInformation and Software Technology10.1016/j.infsof.2024.107641179:COnline publication date: 1-Mar-2025
https://dl.acm.org/doi/10.1016/j.infsof.2024.107641
Venkitaraman A(2024)Early Bug Detection through Shift Left TestingInternational Journal of Innovative Science and Research Technology (IJISRT)10.38124/ijisrt/IJISRT24NOV177(185-190)Online publication date: 18-Nov-2024
https://doi.org/10.38124/ijisrt/IJISRT24NOV177
Krasniqi R(2024)Exploring the Software Quality Maze: Detecting Scattered and Tangled Crosscutting Quality Concerns in Source Code in Support of Maintenance Tasksundefined10.12794/metadc2332577Online publication date: May-2024
https://doi.org/10.12794/metadc2332577
Cui CFilkov VRay BZhou M(2024)Finding Performance Issues in Rust ProjectsProceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering10.1145/3691620.3695368(2423-2425)Online publication date: 27-Oct-2024
https://dl.acm.org/doi/10.1145/3691620.3695368
Khurram SSabau ALichter HTanachutiwat S(2024)Scenario-based synthetic traffic generation for web applications using workload patternsProceedings of the 2024 6th Asia Pacific Information Technology Conference10.1145/3651623.3651638(22-30)Online publication date: 29-Jan-2024
https://dl.acm.org/doi/10.1145/3651623.3651638
Zhao GGeorgiou SHassan SZou YTruong DCorbin TSpinellis DConstantinou EBacchelli A(2024)Enhancing Performance Bug Prediction Using Performance Code MetricsProceedings of the 21st International Conference on Mining Software Repositories10.1145/3643991.3644920(50-62)Online publication date: 15-Apr-2024
https://dl.acm.org/doi/10.1145/3643991.3644920
Khosravi Tabrizi AEzzati-Jivan NTetreault FBalsamo SKnottenbelt WAbad CShang W(2024)An Adaptive Logging System (ALS): Enhancing Software Logging with Reinforcement Learning TechniquesProceedings of the 15th ACM/SPEC International Conference on Performance Engineering10.1145/3629526.3645033(37-47)Online publication date: 7-May-2024
https://dl.acm.org/doi/10.1145/3629526.3645033
Zhao YXiao LWong S(2024)A Platform-Agnostic Framework for Automatically Identifying Performance Issue Reports with Heuristic Linguistic PatternsIEEE Transactions on Software Engineering10.1109/TSE.2024.3390623(1-22)Online publication date: 2024
https://doi.org/10.1109/TSE.2024.3390623
Al-Bataineh OMoonen LVidziunas L(2024)Extending the range of bugs that automated program repair can handleJournal of Systems and Software10.1016/j.jss.2023.111918209(111918)Online publication date: Mar-2024
https://doi.org/10.1016/j.jss.2023.111918
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Issue’s Table of Contents