research-article

Reference-driven performance anomaly identification

Authors:

Christopher Stewart,

Chuanpeng Li, and

Xin LiAuthors Info & Claims

SIGMETRICS '09: Proceedings of the eleventh international joint conference on Measurement and modeling of computer systems

June 2009

Pages 85 - 96

https://doi.org/10.1145/1555349.1555360

Published: 15 June 2009 Publication History

Abstract

Complex system software allows a variety of execution conditions on system configurations and workload properties. This paper explores a principled use of reference executions--those of similar execution conditions from the target--to help identify the symptoms and causes of performance anomalies. First, to identify anomaly symptoms, we construct change profiles that probabilistically characterize expected performance deviations between target and reference executions. By synthesizing several single-parameter change profiles, we can scalably identify anomalous reference-to-target changes in a complex system with multiple execution parameters. Second, to narrow the scope of anomaly root cause analysis, we filter anomaly-related low-level system metrics as those that manifest very differently between target and reference executions. Our anomaly identification approach requires little expert knowledge or detailed models on system internals and consequently it can be easily deployed. Using empirical case studies on the Linux I/O subsystem and a J2EE-based distributed online service, we demonstrate our approach's effectiveness in identifying performance anomalies over a wide range of execution conditions as well as multiple system software versions. In particular, we discovered five previously unknown performance anomaly causes in the Linux 2.6.23 kernel. Additionally, our preliminary results suggest that online anomaly detection and system reconfiguration may help evade performance anomalies in complex online systems.

References

[1]

Realistic nonstationary online workloads. http://www.cs.rochester.edu/u/stewart/models.html.

[2]

MySQL JDBC driver. http://www.mysql.com/products/connector.

[3]

R.A. Fisher. The arrangement of field experiments. J. of the Ministry of Agriculture of Great Britain, 33:503--513, 1926.

[4]

M. Grindal, J. Offutt, and S.F. Andler. Combination testing strategies: A survey. Software Testing, Verification and Reliability, 15(3):167--199, Mar. 2005.

[5]

S. Iyer and P. Druschel. Anticipatory scheduling: A disk scheduling framework to overcome deceptive idleness in synchronous I/O. In 18th ACM Symp. on Operating Systems Principles, pages 117--130, Banff, Canada, Oct. 2001.

Digital Library

[6]

N. Joukov, A. Traeger, R. Iyer, C.P. Wright, and E. Zadok. Operating system profiling via latency analysis. In 7th USENIX Symp. on Operating Systems Design and Implementation, pages 89--102, Seattle, WA, Nov. 2006.

Digital Library

[7]

C. Li and K. Shen. Managing prefetch memory for data-intensive online servers. In 4th USENIX Conf. on File and Storage Technologies, pages 253--266, Dec. 2005.

Digital Library

[8]

C. Li, K. Shen, and A. Papathanasiou. Competitive prefetching for concurrent sequential I/O. In Second EuroSys Conf., pages 189--202, Lisbon, Portugal, Mar. 2007.

Digital Library

[9]

Linux kernel bug tracker. http://bugzilla.kernel.org/.

[10]

Linux kernel bug tracker on "many pre-mature anticipation timeouts in anticipatory I/O scheduler". http://bugzilla.kernel.org/show_bug.cgi?id=10756.

[11]

M.P. Mesnier, M. Wachs, R.R. Sambasivan, A.X. Zheng, and G.R. Ganger. Modeling the relative fitness of storage. In ACM SIGMETRICS, pages 37--48, San Diego, CA, June 2007.

Digital Library

[12]

P. Reynolds, C. Killian, J. Wiener, J. Mogul, M. Shah, and A. Vahdat. Pip: Detecting the unexpected in distributed systems. In Third USENIX Symp. on Networked Systems Design and Implementation, San Jose, CA, May 2006.

Digital Library

[13]

RUBiS: Rice University bidding system. http://rubis.objectweb.org.

[14]

Y. Rubner, C. Tomasi, and L.J. Guibas. The earth mover's distance as a metric for image retrieval. Int'l J. of Computer Vision, 40(2):99--121, 2000.

Digital Library

[15]

K. Shen, M. Zhong, and C. Li. I/O system performance debugging using model-driven anomaly characterization. In 4th USENIX Conf. on File and Storage Technologies, pages 309--322, San Francisco, CA, Dec. 2005.

Digital Library

[16]

C. Stewart, T. Kelly, and A. Zhang. Exploiting nonstationarity for performance prediction. In Second EuroSys Conf., pages 31--44, Lisbon, Portugal, Mar. 2007.

Digital Library

[17]

C. Stewart and K. Shen. Performance modeling and system management for multi-component online services. In Second USENIX Symp. on Networked Systems Design and Implementation, pages 71--84, Boston, MA, May 2005.

Digital Library

[18]

E. Thereska and G.R. Ganger. IRONModel: Robust performance models in the wild. In ACM SIGMETRICS, pages 253--264, Annapolis, MD, June 2008.

Digital Library

[19]

A. Traeger, I. Deras, and E. Zadok. DARC: Dynamic analysis of root causes of latency distributions. In ACM SIGMETRICS, pages 277--288, Annapolis, MD, June 2008.

Digital Library

[20]

J. Tucek, S. Lu, C. Huang, S. Xanthos, and Y. Zhou. Triage: Diagnosing production run failures at the user's site. In 21th ACM Symp. on Operating Systems Principles, pages 131--144, Stevenson, WA, Oct. 2007.

Digital Library

[21]

H.J. Wang, J.C. Platt, Y. Chen, R. Zhang, and Y.-M. Wang. Automatic misconfiguration troubleshooting with PeerPressure. In 6th USENIX Symp. on Operating Systems Design and Implementation, pages 245--258, San Francisco, CA, Dec. 2004.

Digital Library

[22]

A. Zeller. Isolating cause-effect chains from computer programs. In 10th ACM Symp. on Foundations of Software Engineering, pages 1--10, Charleston, SC, Nov. 2002.

Digital Library

Cited By

Yang NShi YSu ZWang XYan ZKong F(2023)FSFP: A Fine-Grained Online Service System Performance Fault Prediction Method Based on Cross-attention2023 30th Asia-Pacific Software Engineering Conference (APSEC)10.1109/APSEC60848.2023.00018(81-90)Online publication date: 4-Dec-2023
https://doi.org/10.1109/APSEC60848.2023.00018
Zhao GHassan SZou YTruong DCorbin T(2021)Predicting Performance Anomalies in Software Systems at Run-timeACM Transactions on Software Engineering and Methodology10.1145/344075730:3(1-33)Online publication date: 23-Apr-2021
https://dl.acm.org/doi/10.1145/3440757
Dai TDean DWang PGu XLu S(2019)Hytrace: A Hybrid Approach to Performance Bug Diagnosis in Production Cloud InfrastructuresIEEE Transactions on Parallel and Distributed Systems10.1109/TPDS.2018.285880030:1(107-118)Online publication date: 1-Jan-2019
https://doi.org/10.1109/TPDS.2018.2858800
Show More Cited By

Index Terms

Reference-driven performance anomaly identification
1. General and reference
  1. Cross-computing tools and techniques
    1. Measurement
2. Software and its engineering
  1. Software creation and management
    1. Software development process management
      1. Software development methods

Recommendations

Reference-driven performance anomaly identification
SIGMETRICS '09

Complex system software allows a variety of execution conditions on system configurations and workload properties. This paper explores a principled use of reference executions--those of similar execution conditions from the target--to help identify the ...
Read More
A Performance Anomaly Detection and Analysis Framework for DBMS Development

Detecting performance anomalies and finding their root causes are tedious tasks requiring much manual work. Functionality enhancements in DBMS development as in most software development often introduce performance problems in addition to bugs. To ...
Read More
Performance Anomaly Detection and Bottleneck Identification

In order to meet stringent performance requirements, system administrators must effectively detect undesirable performance behaviours, identify potential root causes, and take adequate corrective measures. The problem of uncovering and understanding ...
Read More

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

SIGMETRICS '09: Proceedings of the eleventh international joint conference on Measurement and modeling of computer systems

June 2009

336 pages

ISBN:9781605585116

DOI:10.1145/1555349

General Chairs:
John Douceur
Microsoft Research, USA
,
Albert Greenberg
Microsoft Research, USA
,
Program Chairs:
Thomas Bonald
Orange Labs, France
,
Jason Nieh
Columbia University, USA

ACM SIGMETRICS Performance Evaluation Review Volume 37, Issue 1
SIGMETRICS '09
June 2009
320 pages
ISSN:0163-5999
DOI:10.1145/2492101
Issue’s Table of Contents

Copyright © 2009 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 15 June 2009

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

SIGMETRICS09

Sponsor:

SIGMETRICS09: ACM SIGMETRICS/PERFORMANCE Joint International Conference on Measurement and Modeling of Computer Systems

June 15 - 19, 2009

WA, Seattle, USA

Acceptance Rates

Overall Acceptance Rate 459 of 2,691 submissions, 17%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

34
Total Citations
View Citations
513
Total Downloads

Downloads (Last 12 months)12
Downloads (Last 6 weeks)1

Other Metrics

View Author Metrics

Citations

Cited By

Yang NShi YSu ZWang XYan ZKong F(2023)FSFP: A Fine-Grained Online Service System Performance Fault Prediction Method Based on Cross-attention2023 30th Asia-Pacific Software Engineering Conference (APSEC)10.1109/APSEC60848.2023.00018(81-90)Online publication date: 4-Dec-2023
https://doi.org/10.1109/APSEC60848.2023.00018
Zhao GHassan SZou YTruong DCorbin T(2021)Predicting Performance Anomalies in Software Systems at Run-timeACM Transactions on Software Engineering and Methodology10.1145/344075730:3(1-33)Online publication date: 23-Apr-2021
https://dl.acm.org/doi/10.1145/3440757
Dai TDean DWang PGu XLu S(2019)Hytrace: A Hybrid Approach to Performance Bug Diagnosis in Production Cloud InfrastructuresIEEE Transactions on Parallel and Distributed Systems10.1109/TPDS.2018.285880030:1(107-118)Online publication date: 1-Jan-2019
https://doi.org/10.1109/TPDS.2018.2858800
Jindal AHu YArpaci-Dusseau AVoelker G(2018)Differential energy profilingProceedings of the 13th USENIX conference on Operating Systems Design and Implementation10.5555/3291168.3291206(511-526)Online publication date: 8-Oct-2018
https://dl.acm.org/doi/10.5555/3291168.3291206
Brocanelli MWang XOliveira RFelber PHu Y(2018)Hang doctorProceedings of the Thirteenth EuroSys Conference10.1145/3190508.3190525(1-15)Online publication date: 23-Apr-2018
https://dl.acm.org/doi/10.1145/3190508.3190525
Gow RRabhi FVenugopal S(2018)Anomaly Detection in Complex Real World Application SystemsIEEE Transactions on Network and Service Management10.1109/TNSM.2017.277140315:1(83-96)Online publication date: Mar-2018
https://doi.org/10.1109/TNSM.2017.2771403
Huang JMozafari BWenisch T(2017)Statistical Analysis of Latency Through Semantic ProfilingProceedings of the Twelfth European Conference on Computer Systems10.1145/3064176.3064179(64-79)Online publication date: 23-Apr-2017
https://dl.acm.org/doi/10.1145/3064176.3064179
Xu JWang YChen PWang P(2017)Lightweight and Adaptive Service API Performance Monitoring in Highly Dynamic Cloud Environment2017 IEEE International Conference on Services Computing (SCC)10.1109/SCC.2017.80(35-43)Online publication date: Jun-2017
https://doi.org/10.1109/SCC.2017.80
Chen AWu YHaeberlen AZhou WLoo BBarcellos MCrowcroft JVahdat AKatti S(2016)The Good, the Bad, and the DifferencesProceedings of the 2016 ACM SIGCOMM Conference10.1145/2934872.2934910(115-128)Online publication date: 22-Aug-2016
https://dl.acm.org/doi/10.1145/2934872.2934910
Yu LLan Z(2016)A Scalable, Non-Parametric Method for Detecting Performance Anomaly in Large Scale ComputingIEEE Transactions on Parallel and Distributed Systems10.1109/TPDS.2015.247574127:7(1902-1914)Online publication date: 1-Jul-2016
https://dl.acm.org/doi/10.1109/TPDS.2015.2475741
Show More Cited By

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents