Article

Mining precise performance-aware behavioral models from existing instrumentation

Authors:

Ivan Beschastnikh,

Yuriy BrunAuthors Info & Claims

ICSE Companion 2014: Companion Proceedings of the 36th International Conference on Software Engineering

Pages 484 - 487

https://doi.org/10.1145/2591062.2591107

Published: 31 May 2014 Publication History

Abstract

Software bugs often arise from differences between what developers envision their system does and what that system actually does. When faced with such conceptual inconsistencies, debugging can be very difficult. Inferring and presenting developers with accurate behavioral models of the system implementation can help developers reconcile their view of the system with reality and improve system quality.

We present Perfume, a model-inference algorithm that improves on the state of the art by using performance information to differentiate otherwise similar-appearing executions and to remove false positives from the inferred models. Perfume uses a system's runtime execution logs to infer a concise, precise, and predictive finite state machine model that describes both observed executions and executions that have not been observed but that the system can likely generate. Perfume guides the model inference process by mining temporal performance-constrained properties from the logs, ensuring precision of the model's predictions. We describe the model inference process and demonstrate how it improves precision over the state of the art.

References

[1]

M. K. Aguilera, J. C. Mogul, J. L. Wiener, P. Reynolds, and A. Muthitacharoen. Performance Debugging for Distributed Systems of Black Boxes. In SOSP, 2003.

Digital Library

[2]

P. Barham, A. Donnelly, R. Isaacs, and R. Mortier. Using Magpie for Request Extraction and Workload Modelling. In OSDI, 2004.

Digital Library

[3]

I. Beschastnikh, Y. Brun, S. Schneider, M. Sloan, and M. D. Ernst. Leveraging Existing Instrumentation to Automatically Infer Invariant-Constrained Models. In ESEC/FSE, 2011.

Digital Library

[4]

A. W. Biermann and J. A. Feldman. On the Synthesis of Finite-State Machines from Samples of Their Behavior. IEEE Transactions on Computers, 21(6):592–597, 1972.

Digital Library

[5]

T. Britton, L. Jeng, G. Carver, P. Cheak, and T. Katzenellenbogen. Reversible Debugging Software. Technical report, University of Cambridge, Judge Business School, 2013.

[6]

E. Clarke, O. Grumberg, S. Jha, Y. Lu, and H. Veith. Counterexampleguided Abstraction Refinement. In Computer Aided Verification, pages 154–169, 2000.

Digital Library

[7]

B. Dagenais, and M. P. Robillard. Creating and evolving developer documentation: Understanding the decisions of open source contributors. In FSE, 2010.

Digital Library

[8]

M. D. Ernst, J. Cockrell, W. G. Griswold, and D. Notkin. Dynamically discovering likely program invariants to support program evolution. IEEE TSE, 27(2):99–123, 2001.

Digital Library

[9]

R. Fonseca, G. Porter, R. H. Katz, S. Shenker, and I. Stoica. X-Trace: A Pervasive Network Tracing Framework. In NSDI, 2007.

Digital Library

[10]

M. Gabel and Z. Su. Javert: Fully Automatic Mining of General Temporal Properties from Dynamic Traces. In FSE, 2008.

Digital Library

[11]

P. Hooimeijer and W. Weimer. Modeling bug report quality. In ASE, 2007.

Digital Library

[12]

G. Jiang, H. Chen, and K. Yoshihira. Efficient and Scalable Algorithms for Inferring Likely Invariants in Distributed Systems. IEEE Transactions on Knowledge and Data Engineering, 19(11):1508– 1523, 2007.

Digital Library

[13]

M. Jovic, A. Adamoli, and M. Hauswirth. Catch Me if You Can: Performance Bug Detection in the Wild. In OOPSLA, 2011.

Digital Library

[14]

B. Liblit, A. Aiken, A. X. Zheng, and M. I. Jordan. Bug isolation via remote program sampling. In PLDI, 2003.

Digital Library

[15]

D. Lo and S. Maoz. Scenario-Based and Value-Based Specification Mining: Better Together. In ASE, 2010.

Digital Library

[16]

D. Lorenzoli, L. Mariani, and M. Pezzè. Automatic Generation of Software Behavioral Models. In ICSE, 2008.

Digital Library

[17]

R. R. Sambasivan, A. X. Zheng, M. De Rosa, E. Krevat, S. Whitman, M. Stroucken, W. Wang, L. Xu, and G. R. Ganger. Diagnosing Performance Changes by Comparing Request Flows. In NSDI, 2011.

Digital Library

[18]

M. Schur, A. Roth, and A. Zeller. Mining behavior models from enterprise web applications. In ESEC/FSE, 2013.

Digital Library

[19]

S. Sidiroglou-Douskos, S. Misailovic, H. Hoffmann, and M. Rinard. Managing Performance vs. Accuracy Trade-offs with Loop Perforation. In ESEC/FSE, 2011.

Digital Library

[20]

N. Walkinshaw and K. Bogdanov. Inferring Finite-State Models with Temporal Constraints. In ASE, 2008.

Digital Library

[21]

C. Weiß, R. Premraj, T. Zimmermann, and A. Zeller. How long will it take to fix this bug? In Mining Software Repositories, 2007.

Digital Library

[22]

W. Xu, L. Huang, A. Fox, D. Patterson, and M. Jordan. Experience Mining Google’s Production Console Logs. In SLAML, 2010.

Digital Library

[23]

J. Yang, D. Evans, D. Bhardwaj, T. Bhat, and M. Das. Perracotta: Mining Temporal API Rules from Imperfect Traces. In ICSE, 2006.

Digital Library

Cited By

Wei XWang JSun CTowey DZhang SZuo WYu YRuan RSong G(2024)Log‐based anomaly detection for distributed systems: State of the art, industry experience, and open issuesJournal of Software: Evolution and Process10.1002/smr.2650Online publication date: 7-Feb-2024
https://doi.org/10.1002/smr.2650
Aliabadi MHaghighi HAsl MMeybodi R(2020)Challenges of Specification Mining-Based Test Oracle for Cyber-Physical Systems2020 11th International Conference on Information and Knowledge Technology (IKT)10.1109/IKT51791.2020.9345615(1-7)Online publication date: 22-Dec-2020
https://doi.org/10.1109/IKT51791.2020.9345615
Inagaki TUeda YNakaike TOhara MApte VDi Marco ALitoiu MMerseguer J(2019)Profile-based Detection of Layered BottlenecksProceedings of the 2019 ACM/SPEC International Conference on Performance Engineering10.1145/3297663.3310296(197-208)Online publication date: 4-Apr-2019
https://dl.acm.org/doi/10.1145/3297663.3310296
Show More Cited By

Index Terms

Mining precise performance-aware behavioral models from existing instrumentation
1. General and reference
  1. Cross-computing tools and techniques
    1. Performance
2. Software and its engineering
  1. Software organization and properties
    1. Extra-functional properties
      1. Software performance

Recommendations

Behavioral resource-aware model inference
ASE '14: Proceedings of the 29th ACM/IEEE International Conference on Automated Software Engineering

Software bugs often arise because of differences between what developers think their system does and what the system actually does. These differences frustrate debugging and comprehension efforts. We describe Perfume, an automated approach for inferring ...
Inferring models of concurrent systems from logs of their behavior with CSight
ICSE 2014: Proceedings of the 36th International Conference on Software Engineering

Concurrent systems are notoriously difficult to debug and understand. A common way of gaining insight into system behavior is to inspect execution logs and documentation. Unfortunately, manual inspection of logs is an arduous process, and documentation ...
Leveraging existing instrumentation to automatically infer invariant-constrained models
ESEC/FSE '11: Proceedings of the 19th ACM SIGSOFT symposium and the 13th European conference on Foundations of software engineering

Computer systems are often difficult to debug and understand. A common way of gaining insight into system behavior is to inspect execution logs and documentation. Unfortunately, manual inspection of logs is an arduous process and documentation is often ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

ICSE Companion 2014: Companion Proceedings of the 36th International Conference on Software Engineering

May 2014

741 pages

ISBN:9781450327688

DOI:10.1145/2591062

General Chair:
Pankaj Jalote
IIIT-Delhi, India
,
Program Chairs:
Lionel Briand
University of Luxembourg, Luxembourg
,
André van der Hoek
University of California, Irvine, USA

Copyright © 2014 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

SIGSOFT: ACM Special Interest Group on Software Engineering

In-Cooperation

TCSE: IEEE Computer Society's Tech. Council on Software Engin.

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 31 May 2014

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Article

Conference

ICSE '14

Sponsor:

SIGSOFT

ICSE '14: 36th International Conference on Software Engineering

May 31 - June 7, 2014

Hyderabad, India

Acceptance Rates

Overall Acceptance Rate 276 of 1,856 submissions, 15%

Upcoming Conference

ICSE 2025

2025 IEEE/ACM 46th International Conference on Software Engineering

April 26 - May 3, 2025

Ottawa , ON , Canada

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

11
Total Citations
View Citations
130
Total Downloads

Downloads (Last 12 months)3
Downloads (Last 6 weeks)2

Reflects downloads up to 03 Feb 2025

Other Metrics

View Author Metrics

Citations

Cited By

Wei XWang JSun CTowey DZhang SZuo WYu YRuan RSong G(2024)Log‐based anomaly detection for distributed systems: State of the art, industry experience, and open issuesJournal of Software: Evolution and Process10.1002/smr.2650Online publication date: 7-Feb-2024
https://doi.org/10.1002/smr.2650
Aliabadi MHaghighi HAsl MMeybodi R(2020)Challenges of Specification Mining-Based Test Oracle for Cyber-Physical Systems2020 11th International Conference on Information and Knowledge Technology (IKT)10.1109/IKT51791.2020.9345615(1-7)Online publication date: 22-Dec-2020
https://doi.org/10.1109/IKT51791.2020.9345615
Inagaki TUeda YNakaike TOhara MApte VDi Marco ALitoiu MMerseguer J(2019)Profile-based Detection of Layered BottlenecksProceedings of the 2019 ACM/SPEC International Conference on Performance Engineering10.1145/3297663.3310296(197-208)Online publication date: 4-Apr-2019
https://dl.acm.org/doi/10.1145/3297663.3310296
Bouksiaa MTrahay FLescouet AVoron GDulong RGuermouche ABrunet EThomas G(2019)Using Differential Execution Analysis to Identify Thread InterferenceIEEE Transactions on Parallel and Distributed Systems10.1109/TPDS.2019.292748130:12(2866-2878)Online publication date: 1-Dec-2019
https://doi.org/10.1109/TPDS.2019.2927481
Goldstein MRaz DSegall I(2017)Experience Report: Log-Based Behavioral Differencing2017 IEEE 28th International Symposium on Software Reliability Engineering (ISSRE)10.1109/ISSRE.2017.14(282-293)Online publication date: Oct-2017
https://doi.org/10.1109/ISSRE.2017.14
Ohmann TStanley RBeschastnikh IBrun YDillon LVisser WWilliams L(2016)Visually reasoning about system and resource behaviorProceedings of the 38th International Conference on Software Engineering Companion10.1145/2889160.2889166(601-604)Online publication date: 14-May-2016
https://dl.acm.org/doi/10.1145/2889160.2889166
Grechanik MLuo QPoshyvanyk DPorter AAvritzer AIosup AZhu XBecker S(2016)Enhancing Rules For Cloud Resource Provisioning Via Learned Software Performance ModelsProceedings of the 7th ACM/SPEC on International Conference on Performance Engineering10.1145/2851553.2851568(209-214)Online publication date: 12-Mar-2016
https://dl.acm.org/doi/10.1145/2851553.2851568
Walls RBruii YLiberatore MLevine B(2015)Discovering specification violations in networked software systemsProceedings of the 2015 IEEE 26th International Symposium on Software Reliability Engineering (ISSRE)10.1109/ISSRE.2015.7381842(496-506)Online publication date: 2-Nov-2015
https://dl.acm.org/doi/10.1109/ISSRE.2015.7381842
Ohmann THerzberg MFiss SHalbert APalyart MBeschastnikh IBrun YCrnkovic IChechik MGrünbacher P(2014)Behavioral resource-aware model inferenceProceedings of the 29th ACM/IEEE International Conference on Automated Software Engineering10.1145/2642937.2642988(19-30)Online publication date: 15-Sep-2014
https://dl.acm.org/doi/10.1145/2642937.2642988
Krka IBrun YMedvidovic NCheung SOrso AStorey M(2014)Automatic mining of specifications from invocation traces and method invariantsProceedings of the 22nd ACM SIGSOFT International Symposium on Foundations of Software Engineering10.1145/2635868.2635890(178-189)Online publication date: 11-Nov-2014
https://dl.acm.org/doi/10.1145/2635868.2635890
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten