Article

Controlling factors in evaluating path-sensitive error detection techniques

Authors:

Matthew B. Dwyer,

Suzette Person,

Sebastian ElbaumAuthors Info & Claims

SIGSOFT '06/FSE-14: Proceedings of the 14th ACM SIGSOFT international symposium on Foundations of software engineering

Pages 92 - 104

https://doi.org/10.1145/1181775.1181787

Published: 05 November 2006 Publication History

Abstract

Recent advances in static program analysis have made it possible to detect errors in applications that have been thoroughly tested and are in wide-spread use. The ability to find errors that have eluded traditional validation methods is due to the development and combination of sophisticated algorithmic techniques that are embedded in the implementations of analysis tools. Evaluating new analysis techniques is typically performed by running an analysis tool on a collection of subject programs, perhaps enabling and disabling a given technique in different runs. While seemingly sensible, this approach runs the risk of attributing improvements in the cost-effectiveness of the analysis to the technique under consideration, when those improvements may actually be due to details of analysis tool implementations that are uncontrolled during evaluation.In this paper, we focus on the specific class of path-sensitive error detection techniques and identify several factors that can significantly influence the cost of analysis. We show, through careful empirical studies, that the influence of these factors is sufficiently large that, if left uncontrolled, they may lead researchers to improperly attribute improvements in analysis cost and effectiveness. We make several recommendations as to how the influence of these factors can be mitigated when evaluating techniques.

References

[1]

J. C. Corbett. Evaluating Deadlock Detection Methods for Concurrent Software. IEEE Transactions on Software Engineering, 22(3), Mar. 1996.

Digital Library

[2]

J. C. Corbett, M. B. Dwyer, J. Hatcliff, and Robby. Expressing checkable properties of dynamic systems: The Bandera Specification Language. International Journal on Software Tools for Technology Transfer, 2002.

[3]

D. L. Dill, A. J. Drexler, A. J. Hu, and C. H. Yang. Protocol verification as a hardware design aid. In IEEE International Conference on Computer Design, October 1992.

Digital Library

[4]

H. Do, S. G. Elbaum, and G. Rothermel. Subject infrastructure repository. http://esquared.unl.edu/sir.

[5]

Y. Dong, X. Du, G. J. Holzmenn, and S. A. Smolka. Fighting livelock in the gnu i-protocol: a case study in explicit-state model checking. Int'l. Journal on Software Tools for Tech. Transfer, 4(4):505--528, 2003.

Digital Library

[6]

M. B. Dwyer, J. Hatcliff, M. Hoosier, V. Ranganath, Robby, and T. Wallentine. Evaluating the effectiveness of program slicing for model reduction of concurrent object-oriented programs. In Proc. of the Twelfth Int'l. Conf. Tools and Algorithms for the Construction and Analysis of Systems, 2006. LNCS 3920.

Digital Library

[7]

M. B. Dwyer, J. Hatcliff, V. R. Prasad, and Robby. Exploiting object escape and locking information in partial order reductions for concurrent object-oriented programs. Formal Methods in System Designs, 25(2-3):199--240, September-November 2004.

Digital Library

[8]

S. Edelkamp, S. Leue, and A. Lluch-Lafuente. Partial-order reduction and trail improvement in directed model checking. International Journal on Software Tools for Technology Transfer, 6(4), 2004.

Digital Library

[9]

http://home.att.net/~ddavies/NewSmulator.html.

[10]

Y. Eytani, K. Havelund, S. D. Stoller, and S. Ur. Toward a framework and benchmark for testing tools for multi-threaded programs. Concurrency and Computation: Practice and Experience, to appear.

Digital Library

[11]

Y. Eytani and S. Ur. Compiling a benchmark of documented multi-threaded bugs. In Proc. of the Workshop on Parallel and Distributed Systems: Testing and Debugging, 2004.

[12]

E. Farchi, Y. Nir, and S. Ur. Concurrent bug patterns and how to test them. In Proc. of the 17th Int'l. Symp. on Parallel and Distributed Processing, 2003.

Digital Library

[13]

S. Graf and H. Saidi. Construction of abstract state graphs with PVS. In Proc. 9th International Conference on Computer Aided Verification, June 1997.

Digital Library

[14]

A. Groce and W. Visser. Heuristics for model checking java programs. Int'l. Journal on Software Tools for Tech. Transfer, 6(4):260--276, 2004.

Digital Library

[15]

D. Hamlet and J. Voas. Faults on its sleeve: amplifying software reliability testing. In Proc. of the 1993 ACM SIGSOFT international symposium on Software testing and analysis, pages 89--98, 1993.

Digital Library

[16]

K. Havelund, S. D. Stoller, and S. Ur. Benchmark and framework for encouraging research on multi-threaded testing. In Proc. of the Workshop on Parallel and Distributed Systems: Testing and Debugging, 2003.

Digital Library

[17]

G. J. Holzmann. The model checker SPIN. IEEE Transactions on Software Engineering, 23(5): 279--294, May 1997.

Digital Library

[18]

http://www.kclee.de/clemens/java/javancss.

[19]

M. Musuvathi and D. R. Engler. Model Checking Large Network Protocol Implementations. In Proc. of the First Symp. on Networked Systems Design and Implementation, Mar. 2004.

Digital Library

[20]

M. Musuvathi, D. Park, A. Chou, D. R. Engler, and D. L. Dill. CMC: A Pragmatic Approach to Model Checking Real Code. In Proc. of the Fifth Symp. on Operating Systems Design and Implementation, Dec. 2002.

Digital Library

[21]

C. Păsăreanu, M. B. Dwyer, and W. Visser. Finding feasible abstract counter-examples. Int'l. Journal on Software Tools for Tech. Transfer, 5(1):34--48, 2003.

Digital Library

[22]

http://research.microsoft.com/qadeer/cav-issta.htm.

[23]

Robby, M. B. Dwyer, and J. Hatcliff. Bogor: An extensible and highly-modular model checking framework. In Proceedings of the 9th European Software Engineering Conference held jointly with the 11th ACM SIGSOFT Symposium on the Foundations of Software Engineering, 2003.

Digital Library

[24]

Robby, M. B. Dwyer, J. Hatcliff, and R. Iosif. Space-reduction strategies for model checking dynamic systems. In Proceedings of the 2003 Workshop on Software Model Checking, July 2003.

[25]

S. D. Stoller. Testing concurrent java programs using randomized scheduling. In Proc. Workshop on Runtime Verification, 2002.

[26]

J. Tan, G. S. Avrunin, L. A. Clarke, S. Zilberstein, and S. Leue. Heuristic-guided counterexample search in flavers. In Proc. of the 12th ACM SIGSOFT Twelfth Int'l. Symp. on Foundations of Software Engineering, pages 201--210, 2004.

Digital Library

[27]

W. Visser, K. Havelund, G. Brat, and S. Park. Model Checking Programs. In Proceedings of the 15th IEEE Conference on Automated Software Engineering, Sept. 2000.

Digital Library

[28]

J. Yang, P. Twohey, D. R. Engler, and M. Musuvathi. Using Model Checking to Find Serious File System Errors. In Proc. of the Seventh Symp. on Operating Systems Design and Implementation, Dec. 2004.

Digital Library

Cited By

Böhme MSzekeres LMetzman JDwyer MDamian DZeller A(2022)On the reliability of coverage-based fuzzer benchmarkingProceedings of the 44th International Conference on Software Engineering10.1145/3510003.3510230(1621-1633)Online publication date: 21-May-2022
https://dl.acm.org/doi/10.1145/3510003.3510230
Gerrard MBorges MDwyer MFilieri A(2022)Conditional Quantitative Program AnalysisIEEE Transactions on Software Engineering10.1109/TSE.2020.301677848:4(1212-1227)Online publication date: 1-Apr-2022
https://doi.org/10.1109/TSE.2020.3016778
Santos IFurlanetti AMelo Sde Souza PDelamaro MSouza SCavalcante EDantas FBatista T(2020)Contributions to improve the combined selection of concurrent software testing techniquesProceedings of the 5th Brazilian Symposium on Systematic and Automated Software Testing10.1145/3425174.3425214(69-78)Online publication date: 20-Oct-2020
https://dl.acm.org/doi/10.1145/3425174.3425214
Show More Cited By

Index Terms

Controlling factors in evaluating path-sensitive error detection techniques

Recommendations

Tracking pointers with path and context sensitivity for bug detection in C programs

This paper proposes a pointer alias analysis for automatic error detection. State-of-the-art pointer alias analyses are either too slow or too imprecise for finding errors in real-life programs. We propose a hybrid pointer analysis that tracks actively ...
Tracking pointers with path and context sensitivity for bug detection in C programs
ESEC/FSE-11: Proceedings of the 9th European software engineering conference held jointly with 11th ACM SIGSOFT international symposium on Foundations of software engineering

This paper proposes a pointer alias analysis for automatic error detection. State-of-the-art pointer alias analyses are either too slow or too imprecise for finding errors in real-life programs. We propose a hybrid pointer analysis that tracks actively ...
Thresher: precise refutations for heap reachability
PLDI '13

We present a precise, path-sensitive static analysis for reasoning about heap reachability, that is, whether an object can be reached from another variable or object via pointer dereferences. Precise reachability information is useful for a number of ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

SIGSOFT '06/FSE-14: Proceedings of the 14th ACM SIGSOFT international symposium on Foundations of software engineering

November 2006

298 pages

ISBN:1595934685

DOI:10.1145/1181775

General Chair:
Michal Young
University of Oregon
,
Program Chair:
Premkumar Devanbu
University of California, Davis

Copyright © 2006 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 05 November 2006

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Article

Conference

SIGSOFT06/FSE-14

Sponsor:

SIGSOFT06/FSE-14: SIGSOFT 2006 -14th International Symposium on the Foundations of Software Engineering

November 5 - 11, 2006

Oregon, Portland, USA

Acceptance Rates

Overall Acceptance Rate 17 of 128 submissions, 13%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

37
Total Citations
View Citations
574
Total Downloads

Downloads (Last 12 months)4
Downloads (Last 6 weeks)0

Reflects downloads up to 16 Feb 2025

Other Metrics

View Author Metrics

Citations

Cited By

Böhme MSzekeres LMetzman JDwyer MDamian DZeller A(2022)On the reliability of coverage-based fuzzer benchmarkingProceedings of the 44th International Conference on Software Engineering10.1145/3510003.3510230(1621-1633)Online publication date: 21-May-2022
https://dl.acm.org/doi/10.1145/3510003.3510230
Gerrard MBorges MDwyer MFilieri A(2022)Conditional Quantitative Program AnalysisIEEE Transactions on Software Engineering10.1109/TSE.2020.301677848:4(1212-1227)Online publication date: 1-Apr-2022
https://doi.org/10.1109/TSE.2020.3016778
Santos IFurlanetti AMelo Sde Souza PDelamaro MSouza SCavalcante EDantas FBatista T(2020)Contributions to improve the combined selection of concurrent software testing techniquesProceedings of the 5th Brazilian Symposium on Systematic and Automated Software Testing10.1145/3425174.3425214(69-78)Online publication date: 20-Oct-2020
https://dl.acm.org/doi/10.1145/3425174.3425214
Kalhauge CPalsberg J(2018)Sound deadlock predictionProceedings of the ACM on Programming Languages10.1145/32765162:OOPSLA(1-29)Online publication date: 24-Oct-2018
https://dl.acm.org/doi/10.1145/3276516
Gyori ALambeth BKhurshid SMarinov D(2017)Exploring Underdetermined Specifications using Java PathFinderACM SIGSOFT Software Engineering Notes10.1145/3011286.301129541:6(1-5)Online publication date: 5-Jan-2017
https://dl.acm.org/doi/10.1145/3011286.3011295
Lin ZMarinov DZhong HChen YZhao JCohen MGrunske LWhalen M(2015)JaConTeBeProceedings of the 30th IEEE/ACM International Conference on Automated Software Engineering10.1109/ASE.2015.87(178-189)Online publication date: 9-Nov-2015
https://dl.acm.org/doi/10.1109/ASE.2015.87
Hong SStaats MAhn JKim MRothermel G(2015)Are concurrency coverage metrics effective for testingSoftware Testing, Verification & Reliability10.1002/stvr.153925:4(334-370)Online publication date: 1-Jun-2015
https://dl.acm.org/doi/10.1002/stvr.1539
Groce AAlipour MGopinath RBlack AKrishnamurthi SBruegge BRuskiewicz J(2014)Coverage and Its DiscontentsProceedings of the 2014 ACM International Symposium on New Ideas, New Paradigms, and Reflections on Programming & Software10.1145/2661136.2661157(255-268)Online publication date: 20-Oct-2014
https://dl.acm.org/doi/10.1145/2661136.2661157
Francesco NLettieri GSantone AVaglini G(2014)GreASEACM Transactions on Software Engineering and Methodology10.1145/256056323:3(1-26)Online publication date: 2-Jun-2014
https://dl.acm.org/doi/10.1145/2560563
Hong SStaats MAhn JKim MRothermel G(2013)The Impact of Concurrent Coverage Metrics on Testing EffectivenessProceedings of the 2013 IEEE Sixth International Conference on Software Testing, Verification and Validation10.1109/ICST.2013.32(232-241)Online publication date: 18-Mar-2013
https://dl.acm.org/doi/10.1109/ICST.2013.32
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten