research-article

CP-detector: using configuration-related performance properties to expose performance bugs

Authors:

Xiangke LiaoAuthors Info & Claims

ASE '20: Proceedings of the 35th IEEE/ACM International Conference on Automated Software Engineering

Pages 623 - 634

https://doi.org/10.1145/3324884.3416531

Published: 27 January 2021 Publication History

Abstract

Performance bugs are often hard to detect due to their non fail-stop symptoms. Existing debugging techniques can only detect performance bugs with known patterns (e.g., inefficient loops). The key reason behind this incapability is the lack of a general test oracle. Here, we argue that the performance (e.g., throughput, latency, execution time) expectation of configuration can serve as a strong oracle candidate for performance bug detection. First, prior work shows that most performance bugs are related to configurations. Second, the configuration change reflects common expectation on performance changes. If the actual performance is contrary to the expectation, the related code snippet is likely to be problematic.

In this paper, we first conducted a comprehensive study on 173 real-world configuration-related performance bugs (CPBugs) from 12 representative software systems. We then derived seven configuration-related performance properties, which can serve as the test oracle in performance testing. Guided by the study, we designed and evaluated an automated performance testing framework, CP-Detector, for detecting real-world configuration-related performance bugs. CP-Detector was evaluated on 12 open-source projects. The results showed that it detected 43 out of 61 existing bugs and reported 13 new bugs.

References

[1]

Categories and subcategories in computer science. https://en.wikipedia.org/wiki/Category:Computing.

[2]

Fit CDF for normal distribution. https://en.wikipedia.org/wiki/Cumulative_distribution_function.

[3]

spaCy. https://spacy.io.

[4]

Statistical hypothesis testing. https://en.wikipedia.org/wiki/Statistical\_hypothesis_testing.

[5]

IEEE standard glossary of software engineering terminology. IEEE Std 610.12 (1990).

[6]

Linux perf tool. https://perf.wiki.kernel.org/index.php/Main_Page, 2015.

[7]

httpd 33605, Acc.: 2020. https://bz.apache.org/bugzilla/show_bug.cgi?id=33605.

[8]

httpd 42065, Acc.: 2020. https://bz.apache.org/bugzilla/show_bug.cgi?id=42065.

[9]

httpd 50002, Acc.: 2020. https://bz.apache.org/bugzilla/show_bug.cgi?id=50002.

[10]

httpd 58037, Acc.: 2020. https://bz.apache.org/bugzilla/show_bug.cgi?id=58037.

[11]

GCC 17520, Accessed: 2020. https://gcc.gnu.org/bugzilla/show_bug.cgi?id=17520.

[12]

GCC 94957, Accessed: 2020. https://gcc.gnu.org/bugzilla/show_bug.cgi?id=94957.

[13]

MariaDB 5802, Accessed: 2020. https://jira.mariadb.org/browse/MDEV-5802.

[14]

MongoDB 27700, Accessed: 2020. https://jira.mongodb.org/browse/server-27700.

[15]

MongoDB 27753, Accessed: 2020. https://jira.mongodb.org/browse/server-27753.

[16]

MongoDB 30643, Accessed: 2020. https://jira.mongodb.org/browse/server-30643.

[17]

MySQL 21727, Accessed: 2020. https://bugs.mysql.com/bug.php?id=21727.

[18]

MySQL 67432, Accessed: 2020. https://bugs.mysql.com/bug.php?id=67432.

[19]

MySQL 74325, Accessed: 2020. https://bugs.mysql.com/bug.php?id=74325.

[20]

MySQL 77094, Accessed: 2020. https://bugs.mysql.com/bug.php?id=77094.

[21]

Atlee, J.M., Bultan, T., and Whittle, J. View-centric performance optimization for database-backed web applications. In International Conference on Software Engineering (ICSE) (2019).

[22]

Brünink, M., and Rosenblum., D. S. Mining performance specifications. In European Software Engineering Conference and the International Symposium on the Foundations of Software Engineering (ESEC/FSE) (2016).

Digital Library

[23]

Butterfield, A., Ngondi, G. E., and Kerr, A. A Dictionary of Computer Science. Oxford University Press, 2016.

[24]

Chen, B., Liu, Y., and Le., W. Generating performance distributions via probabilistic symbolic execution. In International Conference on Software Engineering (ICSE) (2016).

Digital Library

[25]

Chen, T.-H., Shang, W., Hassan, A. E., Nasser, M., and Flora, P. Cacheoptimizer: Helping developers configure caching frameworks for hibernate-based database-centric web applications. In European Software Engineering Conference and the International Symposium on the Foundations of Software Engineering (ESEC/FSE) (2016).

Digital Library

[26]

Chen, Z., Chen, B., Xiao, L., Wang, X., and Xu, B. Speedoo: prioritizing performance optimization opportunities. In International Conference on Software Engineering (ICSE) (2018).

Digital Library

[27]

Coady, Y., Cox, R., DeTreville, J., Druschel, P., Hellerstein, J., Hume, A., Keeton, K., Nguyen, T., Small, C., Stein, L., and Warfield, A. Falling off the cliff: When systems go nonlinear. In Workshop on Hot Topics in Operating Systems (HotOS) (2005).

[28]

Coppa, E., Demetrescu, C., and Finocchi., I. Input-sensitive profiling. In Conference on Programming Language Design and Implementation (PLDI) (2012).

Digital Library

[29]

Dhok, M., and Ramanathan, M. K. Directed test generation to detect loop inefficiencies. In European Software Engineering Conference and the International Symposium on the Foundations of Software Engineering (ESEC/FSE) (2016).

Digital Library

[30]

Ding, Z., Chen, J., and Shang, W. Towards the use of the readily available tests from the release pipeline as performance tests. are we there yet? In International Conference on Software Engineering (ICSE) (2020).

[31]

GmbH, Y. The industry leader in .NET & Java profiling. https://www.yourkit.com.

[32]

Han, X., Carroll, D., and Yu, T. Reproducing performance bug reports in server applications: The researchers' experiences. Journal of Systems and Software 156 (2019), 268--282.

Digital Library

[33]

Han, X., and Yu, T. An empirical study on performance bugs for highly configurable software systems. In International Symposium on Empirical Software Engineering and Measurement (ESEM) (2016).

Digital Library

[34]

Han, X., Yu, T., and Lo, D. Perflearner: Learning from bug reports to understand and generate performance test frames. In International Conference on Automated Software Engineering (ASE) (2018).

Digital Library

[35]

Jamshidi, P., Siegmund, N., Velez, M., Kästner, C., Patel, A., and Agarwal, Y. Transfer learning for performance modeling of configurable systems: An exploratory analysis. In International Conference on Automated Software Engineering (ASE) (2017).

[36]

Jin, G., Song, L., Shi, X., Scherpelz, J., and Lu, S. Understanding and detecting real-world performance bugs. In Conference on Programming Language Design and Implementation (PLDI) (2012).

Digital Library

[37]

Johnston, O., Jarman, D., Berry, J., Zhou, Z. Q., and Chen, T. Y. Metamorphic relations for detection of performance anomalies. In International Workshop on Metamorphic Testing (MET) (2019).

Digital Library

[38]

Juliana, A. P., Mathieu, A., Hugo, M., and Jean-Marc, J. Sampling effect on performance prediction of configurable systems: A case study. In International Conference on Performance Engineering (ICPE) (2020).

[39]

Kaltenecker, C., Grebhahn, A., Siegmund, N., Guo, J., and Apel, S. Distance-based sampling of software configuration spaces. In International Conference on Software Engineering (ICSE) (2019).

Digital Library

[40]

Leshl, N., Zaki, M. J., and Ogihara3, M. Mining features for sequence classification. In ACM Knowledge Discovery and Data Mining (SIGKDD) (1999).

[41]

Liu, T., and Berger., E. D. Sheriff: Precise detection and automatic mitigation of false sharing. In Conference on Object-Oriented Programming Systems, Languages, and Applications (OOPSLA) (2011).

[42]

Michail, A. Data mining library reuse patterns in user-selected applications. In International Conference on Automated Software Engineering (ASE) (1999).

[43]

Nair, V., Menzies, T., Siegmund, N., and Apel., S. Using bad learners to find good configurations. In European Software Engineering Conference and the International Symposium on the Foundations of Software Engineering (ESEC/FSE) (2017).

Digital Library

[44]

Nguyen, K., and Xu, G. Cachetor: Detecting cacheable data to remove bloat. In European Software Engineering Conference and the International Symposium on the Foundations of Software Engineering (ESEC/FSE) (2013).

Digital Library

[45]

Nistor, A., Chang, P.-C., Radoi, C., and Lu, S. Caramel: Detecting and fixing performance problems that have non-intrusive fixes. In International Conference on Software Engineering (ICSE) (2015).

[46]

Nistor, A., Song, L., Marinov, D., and Lu, S. Toddler: Detecting performance problems via similar memory-access patterns. In International Conference on Software Engineering (ICSE) (2013).

[47]

Oh, J., Batory, D., Myers, M., and Siegmund, N. Finding near-optimal configurations in product lines by random sampling. In European Software Engineering Conference and the International Symposium on the Foundations of Software Engineering (ESEC/FSE) (2017).

Digital Library

[48]

Olivo, O., Dillig, I., and Lin, C. Static detection of asymptotic performance bugs in collection traversals. In Conference on Programming Language Design and Implementation (PLDI) (2015).

Digital Library

[49]

Pradel, M., Huggler, M., and Gross, T. R. Performance regression testing of concurrent classes. In International Symposium on Software Testing & Analysis (ISSTA) (2014).

Digital Library

[50]

Pradel, M., Schuh, P., Necula, G., and Sen, K. Eventbreak: Analyzing the responsiveness of user interfaces through performance-guided test generation. In Conference on Object-Oriented Programming Systems, Languages, and Applications (OOPSLA) (2014).

Digital Library

[51]

Ravjot, S., Cor-Paul, B., Weiyi, S., and E., H. A. Optimizing the performance-related configurations of object-relational mapping frameworks using a multi-objective genetic algorithm. In International Conference on Performance Engineering (ICPE) (2016).

[52]

Sarkar, A., Guo, J., Siegmund, N., Apel, S., and Czarnecki, K. Cost-efficient sampling for performance prediction of configurable systems. In International Conference on Automated Software Engineering (ASE) (2015).

Digital Library

[53]

Segura, S., Troya, J., Durán, A., and Cortés, A. R. Performance metamorphic testing: A proof of concept. Information & Software Technology 98 (2018), 1--4.

Digital Library

[54]

Segura, S., Troya, J., Durán, A., and Ruiz-Cortés, A. Performance metamorphic testing: Motivation and challenges. In International Conference on Software Engineering: New Ideas and Emerging Technologies Results Track (ICSE-NIER), (2017).

Digital Library

[55]

Shen, D., Luo, Q., Poshyvanyk, D., and Grechanik., M. Automating performance bottleneck detection using search-based application profiling. In International Symposium on Software Testing & Analysis (ISSTA) (2015).

Digital Library

[56]

Siegmund, N., Grebhahn, A., Apel, S., and Kästner, C. Performance-influence models for highly configurable systems. In European Software Engineering Conference and the International Symposium on the Foundations of Software Engineering (ESEC/FSE) (2015).

Digital Library

[57]

Sokolovaa, M., and Lapalmeb, G. A systematic analysis of performance measures for classification tasks. Information Processing & Management (2009).

[58]

Song, L., and Lu, S. Statistical debugging for real-world performance problems. In Conference on Object-Oriented Programming Systems, Languages, and Applications (OOPSLA) (2014).

Digital Library

[59]

Song, L., and Lu, S. Performance diagnosis for inefficient loops. In International Conference on Software Engineering (ICSE) (2017).

Digital Library

[60]

Souto, S., d'Amorim, M., and Gheyi, R. Balancing soundness and efficiency for practical testing of configurable systems. In International Conference on Software Engineering (ICSE) (2017).

Digital Library

[61]

Su, P., Wen, S., Yang, H., Chabbi, M., and Liu, X. Redundant loads: A software inefficiency indicator. In International Conference on Software Engineering (ICSE) (2019).

Digital Library

[62]

Toffola, L. D., Pradel, M., and Gross, T. R. Performance problems you can fix: A dynamic analysis of memoization opportunities. In Conference on Object-Oriented Programming Systems, Languages, and Applications (OOPSLA) (2015).

Digital Library

[63]

Xiang, C., Huang, H., Yoo, A., Zhou, Y., and Pasupathy, S. Pracextractor: Extracting configuration good practices from manuals to detect server misconfigurations. In USENIX Annual Technical Conference (ATC) (2020).

[64]

Xu, T., Zhang, J., Huang, P., Zheng, J., Sheng, T., Yuan, D., Zhou, Y., and Pasupathy, S. Do not blame users for misconfigurations. In Symposium on Operating Systems Principles (SOSP) (2013).

Digital Library

[65]

Yang, J., Subramaniam, P., Lu, S., Yan, C., and Cheung, A. How not to structure your database-backed web applications: a study of performance bugs in the wild. In International Conference on Software Engineering (ICSE) (2018).

Digital Library

[66]

Yu, T., and Pradel, M. Syncprof: Detecting, localizing, and optimizing synchronization bottlenecks. In International Symposium on Software Testing & Analysis (ISSTA) (2016).

Digital Library

[67]

Zaparanuks, D., and Hauswirth., M. Algorithmic profiling. In Conference on Programming Language Design and Implementation (PLDI) (2012).

Digital Library

[68]

Zimmermann, T., Weisgerber, P., Diehl, S., and Zeller, A. Mining version histories to guide software changes. In International Conference on Software Engineering (ICSE) (2005).

Digital Library

Cited By

Zhang YYu XLiu JZhang LLi YTan Y(2024)CCE&D: A Configuration Failure Prevention Method for Autonomous Driving SystemsInformation Security and Privacy10.1007/978-981-97-5101-3_16(288-302)Online publication date: 15-Jul-2024
https://doi.org/10.1007/978-981-97-5101-3_16
Du XChen XCao JWen MCheung SJin HChandra SBlincoe KTonella P(2023)Understanding the Bug Characteristics and Fix Strategies of Federated Learning SystemsProceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering10.1145/3611643.3616347(1358-1370)Online publication date: 30-Nov-2023
https://dl.acm.org/doi/10.1145/3611643.3616347
Wang TJia ZLi SZheng SYu YXu EPeng SLiao X(2023)Understanding and Detecting On-The-Fly Configuration Bugs2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE)10.1109/ICSE48619.2023.00062(628-639)Online publication date: May-2023
https://doi.org/10.1109/ICSE48619.2023.00062
Show More Cited By

Index Terms

CP-detector: using configuration-related performance properties to expose performance bugs
1. Software and its engineering
  1. Software organization and properties
    1. Extra-functional properties
      1. Software performance

Recommendations

CP-Miner: a tool for finding copy-paste and related bugs in operating system code
OSDI'04: Proceedings of the 6th conference on Symposium on Operating Systems Design & Implementation - Volume 6

Copy-pasted code is very common in large software because programmers prefer reusing code via copy-paste in order to reduce programming effort. Recent studies show that copy-paste is prone to introducing bugs and a significant portion of operating ...
CP-Miner: Finding Copy-Paste and Related Bugs in Large-Scale Software Code

Recent studies have shown that large software suites contain significant amounts of replicated code. It is assumed that some of this replication is due to copy-and-paste activity and that a significant proportion of bugs in operating systems are due to ...
Statically inferring performance properties of software configurations
EuroSys '20: Proceedings of the Fifteenth European Conference on Computer Systems

Modern software systems often have a huge number of configurations whose performance properties are poorly documented. Unfortunately, obtaining a good understanding of these performance properties is a prerequisite for performance tuning. This paper ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

ASE '20: Proceedings of the 35th IEEE/ACM International Conference on Automated Software Engineering

December 2020

1449 pages

ISBN:9781450367684

DOI:10.1145/3324884

General Chair:
John Grundy,
Program Chairs:
Claire Le Goues,
David Lo

Copyright © 2020 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

In-Cooperation

IEEE CS

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 27 January 2021

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

National Natural Science Foundation of China
National Key R&D Program of China

Conference

ASE '20

Sponsor:

ASE '20: 35th IEEE/ACM International Conference on Automated Software Engineering

December 21 - 25, 2020

Virtual Event, Australia

Acceptance Rates

Overall Acceptance Rate 82 of 337 submissions, 24%

Upcoming Conference

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

9
Total Citations
View Citations
189
Total Downloads

Downloads (Last 12 months)52
Downloads (Last 6 weeks)9

Reflects downloads up to 11 Aug 2024

Other Metrics

View Author Metrics

Citations

Cited By

Zhang YYu XLiu JZhang LLi YTan Y(2024)CCE&D: A Configuration Failure Prevention Method for Autonomous Driving SystemsInformation Security and Privacy10.1007/978-981-97-5101-3_16(288-302)Online publication date: 15-Jul-2024
https://doi.org/10.1007/978-981-97-5101-3_16
Du XChen XCao JWen MCheung SJin HChandra SBlincoe KTonella P(2023)Understanding the Bug Characteristics and Fix Strategies of Federated Learning SystemsProceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering10.1145/3611643.3616347(1358-1370)Online publication date: 30-Nov-2023
https://dl.acm.org/doi/10.1145/3611643.3616347
Wang TJia ZLi SZheng SYu YXu EPeng SLiao X(2023)Understanding and Detecting On-The-Fly Configuration Bugs2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE)10.1109/ICSE48619.2023.00062(628-639)Online publication date: May-2023
https://doi.org/10.1109/ICSE48619.2023.00062
Cao JChen BSun CHu LWu SPeng XRoychoudhury ACadar CKim M(2022)Understanding performance problems in deep learning systemsProceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering10.1145/3540250.3549123(357-369)Online publication date: 7-Nov-2022
https://dl.acm.org/doi/10.1145/3540250.3549123
He JLin YGu XYeh CZhuang ZDwyer MDamian DZeller A(2022)PerfSigProceedings of the 44th International Conference on Software Engineering10.1145/3510003.3510110(1669-1680)Online publication date: 21-May-2022
https://dl.acm.org/doi/10.1145/3510003.3510110
He HJia ZLi SYu YZhou CLiao QWang JLiao XDwyer MDamian DZeller A(2022)Multi-intention-aware configuration selection for performance tuningProceedings of the 44th International Conference on Software Engineering10.1145/3510003.3510094(1431-1442)Online publication date: 21-May-2022
https://dl.acm.org/doi/10.1145/3510003.3510094
Velez MJamshidi PSiegmund NApel SKästner CDwyer MDamian DZeller A(2022)On debugging the performance of configurable software systemsProceedings of the 44th International Conference on Software Engineering10.1145/3510003.3510043(1571-1583)Online publication date: 21-May-2022
https://dl.acm.org/doi/10.1145/3510003.3510043
Avritzer ABritto RTrubiani CCamilli MJanes ARusso Bvan Hoorn AHeinrich RRapp MHenß JChalawadi R(2022)Scalability testing automation using multivariate characterization and detection of software performance antipatternsJournal of Systems and Software10.1016/j.jss.2022.111446(111446)Online publication date: Jul-2022
https://doi.org/10.1016/j.jss.2022.111446
Liu HYu YLi SGeng MMao XLiao X(2021)How to cherry pick the bug report for better summarization?Empirical Software Engineering10.1007/s10664-021-10008-226:6Online publication date: 1-Nov-2021
https://dl.acm.org/doi/10.1007/s10664-021-10008-2

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents