Article

MUX: algorithm selection for software model checkers

Authors:

Aditya V. NoriAuthors Info & Claims

MSR 2014: Proceedings of the 11th Working Conference on Mining Software Repositories

Pages 132 - 141

https://doi.org/10.1145/2597073.2597080

Published: 31 May 2014 Publication History

Abstract

With the growing complexity of modern day software, software model checking has become a critical technology for ensuring correctness of software. As is true with any promising technology, there are a number of tools for software model checking. However, their respective performance trade-offs are difficult to characterize accurately – making it difficult for practitioners to select a suitable tool for the task at hand. This paper proposes a technique called MUX that addresses the problem of selecting the most suitable software model checker for a given input instance. MUX performs machine learning on a repository of software verification instances. The algorithm selector, synthesized through machine learning, uses structural features from an input instance, comprising a program-property pair, at runtime and determines which tool to use.

We have implemented MUX for Windows device drivers and evaluated it on a number of drivers and model checkers. Our results are promising in that the algorithm selector not only avoids a significant number of timeouts but also improves the total runtime by a large margin, compared to any individual model checker. It also outperforms a portfolio-based algorithm selector being used in Microsoft at present. Besides, MUX identifies structural features of programs that are key factors in determining performance of model checkers.

References

[1]

A. Albarghouthi, A. Gurfinkel, Y. Li, S. Chaki, and M. Chechik. UFO: Verification with interpolants and abstract interpretation. In TACAS, pages 637–640, 2013.

Digital Library

[2]

S. Apel, D. Beyer, K. Friedberger, F. Raimondi, and A. Rhein. Domain types: Abstract-domain selection based on variable usage. In HVC, pages 262–278, 2013.

[3]

T. Ball, E. Bounimova, R. Kumar, and V. Levin. Slam2: static driver verification with under 4% false alarms. In FMCAD, pages 35–42, 2010.

Digital Library

[4]

T. Ball, B. Cook, V. Levin, and S. K. Rajamani. SLAM and Static Driver Verifier: Technology Transfer of Formal Methods inside Microsoft. In IFM, pages 1–20, 2004.

[5]

T. Ball, V. Levin, and S.K. Rajamani. A decade of software model checking with SLAM. CACM, pages 68–76, 2011.

Digital Library

[6]

T. Ball and S. K. Rajamani. SLIC: A specification language for interface checking (of C). Technical report, MSR-TR-2001-21, Microsoft Research, 2001.

[7]

D. Beyer. Second competition on software verification. In TACAS, pages 594–609, 2013.

Digital Library

[8]

D. Beyer, T. A. Henzinger, M. E. Keremoglu, and P. Wendler. Conditional model checking: a technique to pass information between verifiers. In FSE, page 57, 2012.

Digital Library

[9]

D. Beyer, T. A. Henzinger, and G. Théoduloz. Configurable software verification: Concretizing the convergence of model checking and program analysis. In CAV, pages 504–518, 2007.

Digital Library

[10]

B. Bischl, O. Mersmann, H. Trautmann, and M. Preuss. Algorithm selection based on exploratory landscape analysis and cost-sensitive learning. In GECCO, pages 313–320, 2012.

Digital Library

[11]

C. M. Bishop and N. M. Nasrabadi. Pattern recognition and machine learning. 2006.

Digital Library

[12]

G. Cabodi, S. Nocco, and S. Quer. Thread-based multi-engine model checking for multicore platforms. ACM Trans. Des. Autom. Electron. Syst., 18(3):36:1–36:28, 2013.

Digital Library

[13]

C. Chang and C. Lin. LIBSVM: a library for support vector machines. TIST, page 27, 2011.

Digital Library

[14]

B. Daniel and M. Boshernitsan. Predicting effectiveness of automatic testing tools. In ASE, pages 363–366, 2008.

Digital Library

[15]

R. Fan, K. Chang, C. Hsieh, X. Wang, and C. Lin. LIBLINEAR: A library for large linear classification. J. of ML Research, pages 1871–1874, 2008.

Digital Library

[16]

C. Gebruers, A. Guerri, B. Hnich, and M. Milano. Making choices using structure at the instance level within a case based reasoning framework. In Int. AI and OR Techn., pages 380– 386, 2004.

[17]

P. Godefroid. Software model checking: The verisoft approach. FMSD, pages 77–101, 2005.

Digital Library

[18]

Object Management Group. Unified Modeling Language Specification (Version 2.41). 2011.

[19]

A. Guerri and M. Milano. Learning techniques for automatic algorithm portfolio selection. In ECAI, page 475, 2004.

[20]

H. Guo. Algorithm selection for sorting and probabilistic inference: a machine learning-based approach. PhD thesis, Kansas State University, 2003.

Digital Library

[21]

M. Hall, E. Frank, G. Holmes, B. Pfahringer, P. Reutemann, and I. H. Witten. The WEKA data mining software: an update. SIGKDD Explorations, pages 10–18, 2009.

Digital Library

[22]

T. A. Henzinger, R. Jhala, R. Majumdar, and G. Sutre. Software verification with BLAST. In Model Checking Software, pages 235–239. 2003.

Digital Library

[23]

B. Huberman, R. Lukose, and T. Hogg. An economics approach to hard computational problems. Science, pages 51–54, 1997.

[24]

ITU-T. Specification and Description Language (SDL). 2010.

[25]

T. Johnson and R. Eigenmann. Context-sensitive domainindependent algorithm composition and selection. In PLDI, pages 181–192, 2006.

Digital Library

[26]

S. Kadioglu, Y. Malitsky, A. Sabharwal, H. Samulowitz, and M. Sellmann. Algorithm selection and scheduling. In CP, pages 454–469, 2011.

Digital Library

[27]

M. G. Kendall and J. D. Gibbons. Rank correlation methods. Oxford University Press, 1990.

[28]

P. Kulkarni, S. Hines, J. Hiser, D. Whalley, J. Davidson, and D. Jones. Fast searches for effective optimization phase sequences. In PLDI, pages 171–182, 2004.

Digital Library

[29]

M.G. Lagoudakis and M.L. Littman. Algorithm selection using reinforcement learning. In ICML, pages 511–518, 2000.

Digital Library

[30]

A. Lal, S. Qadeer, and S. K. Lahiri. A solver for reachability modulo theories. In CAV, pages 427–443, 2012.

Digital Library

[31]

T. B. Le and D. Lo. Will fault localization work for these failures? An automated approach to predict effectiveness of fault localization tools. In ICSM, pages 310–319, 2013.

Digital Library

[32]

G. T. Leavens, A. L. Baker, and C. Ruby. Preliminary design of JML: a behavioral interface specification language for Java. ACM SIGSOFT Soft. Engg Notes, pages 1–38, 2006.

Digital Library

[33]

T. Menzies, J. Greenwald, and A. Frank. Data mining static code attributes to learn defect predictors. IEEE Trans. Software Eng., 33(1):2–13, 2007.

Digital Library

[34]

M. Musuvathi, D. Y. W. Park, A. Chou, D. R. Engler, and D. L. Dill. CMC: a pragmatic approach to model checking real code. In OSDI, pages 75–88, 2002.

Digital Library

[35]

A. V. Nori and S. K. Rajamani. An empirical study of optimizations in YOGI. In ICSE, pages 355–364, 2010.

Digital Library

[36]

A. V. Nori, S. K. Rajamani, S. Tetali, and A. V. Thakur. The yogi project: Software property checking via static analysis and testing. In TACAS, pages 178–181. 2009.

Digital Library

[37]

J. R. Rice. The algorithm selection problem. Advances in Computers, 15:65–118, 1976.

[38]

H. Samulowitz and R. Memisevic. Learning to solve QBF. In AAAI, volume 7, pages 255–260, 2007.

Digital Library

[39]

K.A. Smith-Miles. Towards insightful algorithm selection for optimisation using meta-learning concepts. In IJCNN, pages 4118–4124, 2008.

[40]

N. Thomas, G. Tanase, O. Tkachyshyn, J. Perdue, N. M. Amato, and L. Rauchwerger. A framework for adaptive algorithm selection in STAPL. In PPoPP, pages 277–288, 2005.

Digital Library

[41]

A. M. Turing. On computable numbers, with an application to the entscheidungsproblem. Proc. of the London Math. Soc., pages 230–265, 1936.

[42]

K. Vaswani, M.J. Thazhuthaveetil, Y.N. Srikant, and P.J. Joseph. Microarchitecture sensitive empirical models for compiler optimizations. In CGO, pages 131–143, 2007.

Digital Library

[43]

W. Visser, K. Havelund, G. P. Brat, S. Park, and F. Lerda. Model checking programs. Autom. Softw. Eng., pages 203– 232, 2003.

Digital Library

[44]

S. Wang, D. Lo, L. Jiang, Lucia, and H. C. Lau. Search-based fault localization. In ASE, pages 556–559, 2011.

[45]

L. Xu, F. Hutter, H.H. Hoos, and K. Leyton-Brown. SATzilla: portfolio-based algorithm selection for SAT. Journal of AI Research, pages 565–606, 2008.

Digital Library

Cited By

Leeson WDwyer M(2024)Algorithm Selection for Software Verification Using Graph Neural NetworksACM Transactions on Software Engineering and Methodology10.1145/363722533:3(1-36)Online publication date: 14-Mar-2024
https://dl.acm.org/doi/10.1145/3637225
Darke PChimdyalwar BAlladawar MSulakhe SVenkatesh RChakraborty S(2024)Learning Strategies Using Boolean Program Metrics to Verify Industrial Code2024 IEEE International Conference on Software Maintenance and Evolution (ICSME)10.1109/ICSME58944.2024.00076(719-729)Online publication date: 6-Oct-2024
https://doi.org/10.1109/ICSME58944.2024.00076
Haltermann JJakobs MRichter CWehrheim H(2024)Parallel program analysis on path rangesScience of Computer Programming10.1016/j.scico.2024.103154238:COnline publication date: 1-Dec-2024
https://dl.acm.org/doi/10.1016/j.scico.2024.103154
Show More Cited By

Index Terms

MUX: algorithm selection for software model checkers

Recommendations

Algorithm selection for software validation based on graph kernels
Abstract
Algorithm selection is the task of choosing an algorithm from a given set of candidate algorithms when faced with a particular problem instance. Algorithm selection via machine learning (ML) has recently been successfully applied for various ...
A Comparative Study of Software Model Checkers as Unit Testing Tools: An Industrial Case Study

Conventional testing methods often fail to detect hidden flaws in complex embedded software such as device drivers or file systems. This deficiency incurs significant development and support/maintenance cost for the manufacturers. Model checking ...
Efficient modular glass box software model checking
OOPSLA '10: Proceedings of the ACM international conference on Object oriented programming systems languages and applications

Glass box software model checking incorporates novel techniques to identify similarities in the state space of a model checker and safely prune large numbers of redundant states without explicitly checking them. It is significantly more efficient than ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

MSR 2014: Proceedings of the 11th Working Conference on Mining Software Repositories

May 2014

427 pages

ISBN:9781450328630

DOI:10.1145/2597073

General Chair:
Premkumar Devanbu
University of California at Davis, USA
,
Program Chairs:
Sung Kim
Hong Kong University of Science and Technology, China
,
Martin Pinzger
University of Klagenfurt, Austria

Copyright © 2014 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGSOFT: ACM Special Interest Group on Software Engineering

In-Cooperation

TCSE: IEEE Computer Society's Tech. Council on Software Engin.

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 31 May 2014

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Article

Conference

ICSE '14

Sponsor:

SIGSOFT

ICSE '14: 36th International Conference on Software Engineering

May 31 - June 1, 2014

Hyderabad, India

Upcoming Conference

ICSE 2025

2025 IEEE/ACM 46th International Conference on Software Engineering

April 26 - May 3, 2025

Ottawa , ON , Canada

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

24
Total Citations
View Citations
249
Total Downloads

Downloads (Last 12 months)8
Downloads (Last 6 weeks)1

Reflects downloads up to 13 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Leeson WDwyer M(2024)Algorithm Selection for Software Verification Using Graph Neural NetworksACM Transactions on Software Engineering and Methodology10.1145/363722533:3(1-36)Online publication date: 14-Mar-2024
https://dl.acm.org/doi/10.1145/3637225
Darke PChimdyalwar BAlladawar MSulakhe SVenkatesh RChakraborty S(2024)Learning Strategies Using Boolean Program Metrics to Verify Industrial Code2024 IEEE International Conference on Software Maintenance and Evolution (ICSME)10.1109/ICSME58944.2024.00076(719-729)Online publication date: 6-Oct-2024
https://doi.org/10.1109/ICSME58944.2024.00076
Haltermann JJakobs MRichter CWehrheim H(2024)Parallel program analysis on path rangesScience of Computer Programming10.1016/j.scico.2024.103154238:COnline publication date: 1-Dec-2024
https://dl.acm.org/doi/10.1016/j.scico.2024.103154
Haltermann JWehrheim H(2024)Exchanging information in cooperative software validationSoftware and Systems Modeling (SoSyM)10.1007/s10270-024-01155-323:3(695-719)Online publication date: 1-Jun-2024
https://dl.acm.org/doi/10.1007/s10270-024-01155-3
Thoben NHaltermann JWehrheim H(2023)Timeout Prediction for Software AnalysesSoftware Engineering and Formal Methods10.1007/978-3-031-47115-5_19(340-358)Online publication date: 31-Oct-2023
https://doi.org/10.1007/978-3-031-47115-5_19
Haltermann JJakobs MRichter CWehrheim H(2023)Parallel Program Analysis via Range SplittingFundamental Approaches to Software Engineering10.1007/978-3-031-30826-0_11(195-219)Online publication date: 22-Apr-2023
https://dl.acm.org/doi/10.1007/978-3-031-30826-0_11
Cao WWu YWang QZhang JZhang XQiu M(2022)A Novel RVFL-Based Algorithm Selection Approach for Software Model CheckingKnowledge Science, Engineering and Management10.1007/978-3-031-10989-8_33(414-425)Online publication date: 19-Jul-2022
https://doi.org/10.1007/978-3-031-10989-8_33
Jakobs M(2022)Reusing Predicate Precision in Value AnalysisIntegrated Formal Methods10.1007/978-3-031-07727-2_5(63-85)Online publication date: 1-Jun-2022
https://doi.org/10.1007/978-3-031-07727-2_5
Wang QJiang JZhao YCao WWang CLi S(2021)Algorithm selection for software verification based on adversarial LSTM2021 7th IEEE Intl Conference on Big Data Security on Cloud (BigDataSecurity), IEEE Intl Conference on High Performance and Smart Computing, (HPSC) and IEEE Intl Conference on Intelligent Data and Security (IDS)10.1109/BigDataSecurityHPSCIDS52275.2021.00026(87-92)Online publication date: May-2021
https://doi.org/10.1109/BigDataSecurityHPSCIDS52275.2021.00026
Koc UMordahl AWei SFoster JPorter A(2021)SATune: A Study-Driven Auto-Tuning Approach for Configurable Software Verification Tools2021 36th IEEE/ACM International Conference on Automated Software Engineering (ASE)10.1109/ASE51524.2021.9678761(330-342)Online publication date: Nov-2021
https://doi.org/10.1109/ASE51524.2021.9678761
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents