Article

Leveraging test generation and specification mining for automated bug detection without false positives

Authors:

Michael Pradel,

Thomas R. GrossAuthors Info & Claims

ICSE '12: Proceedings of the 34th International Conference on Software Engineering

Pages 288 - 298

Published: 02 June 2012 Publication History

Abstract

Mining specifications and using them for bug detection is a promising way to reveal bugs in programs. Existing approaches suffer from two problems. First, dynamic specification miners require input that drives a program to generate common usage patterns. Second, existing approaches report false positives, that is, spurious warnings that mislead developers and reduce the practicability of the approach. We present a novel technique for dynamically mining and checking specifications without relying on existing input to drive a program and without reporting false positives. Our technique leverages automatically generated tests in two ways: Passing tests drive the program during specification mining, and failing test executions are checked against the mined specifications. The output are warnings that show with concrete test cases how the program violates commonly accepted specifications. Our implementation reports no false positives and 54 true positives in ten well-tested Java programs.

References

[1]

C. Csallner and Y. Smaragdakis, "JCrasher: an automatic robustness tester for Java," Software Pract Exper, vol. 34, no. 11, pp. 1025-1050, Sep. 2004.

Digital Library

[2]

C. Pacheco, S. K. Lahiri, M. D. Ernst, and T. Ball, "Feedback-directed random test generation," in ICSE, 2007, pp. 75-84.

Digital Library

[3]

I. Ciupa, A. Leitner, M. Oriol, and B. Meyer, "ARTOO: adaptive random testing for object-oriented software," in ICSE, 2008, pp. 71-80.

Digital Library

[4]

G. Ammons, R. Bodík, and J. R. Larus, "Mining specifications," in POPL, 2002, pp. 4-16.

Digital Library

[5]

M. Pradel and T. R. Gross, "Automatic generation of object usage specifications from large method traces," in ASE, 2009, pp. 371-382.

Digital Library

[6]

M. Pradel, P. Bichsel, and T. R. Gross, "A framework for the evaluation of specification miners based on finite state machines," in ICSM, 2010, pp. 1-10.

Digital Library

[7]

C. Allan, P. Avgustinov, A. S. Christensen, L. Hendren, S. Kuzins, O. Lhoták, O. de Moor, D. Sereni, G. Sittampalam, and J. Tibble, "Adding trace matching with free variables to AspectJ," in OOPSLA, 2005, pp. 345-364.

Digital Library

[8]

M. C. Martin, V. B. Livshits, and M. S. Lam, "Finding application errors and security flaws using PQL: A program query language," in OOPSLA, 2005, pp. 365-383.

Digital Library

[9]

F. Chen and G. Rosu, "MOP: An efficient and generic runtime verification framework," in OOPSLA, 2007, pp. 569-588.

Digital Library

[10]

S. M. Blackburn et al., "The DaCapo benchmarks: Java benchmarking development and analysis," in OOPSLA, 2006, pp. 169-190.

Digital Library

[11]

T. A. Henzinger, R. Jhala, and R. Majumdar, "Permissive interfaces," in ESEC/FSE, 2005, pp. 31-40.

Digital Library

[12]

M. Gabel and Z. Su, "Online inference and enforcement of temporal properties," in ICSE, 2010, pp. 15-24.

Digital Library

[13]

H. Zhong, L. Zhang, T. Xie, and H. Mei, "Inferring resource specifications from natural language API documentation," in ASE, 2009, pp. 307-318.

Digital Library

[14]

J. Whaley, M. C. Martin, and M. S. Lam, "Automatic extraction of object-oriented component interfaces," in ISSTA, 2002, pp. 218-228.

Digital Library

[15]

S. Shoham, E. Yahav, S. Fink, and M. Pistoia, "Static specification mining using automata-based abstractions," in ISSTA, 2007, pp. 174-184.

Digital Library

[16]

M. Gabel and Z. Su, "Javert: Fully automatic mining of general temporal properties from dynamic traces," in FSE, 2008, pp. 339-349.

Digital Library

[17]

D. Lorenzoli, L. Mariani, and M. Pezzè, "Automatic generation of software behavioral models," in ICSE, 2008, pp. 501-510.

Digital Library

[18]

C. Lee, F. Chen, and G. Rosu, "Mining parametric specifications," in ICSE, 2011, pp. 591-600.

Digital Library

[19]

I. Ciupa, A. Pretschner, A. Leitner, M. Oriol, and B. Meyer, "On the predictability of random tests for object-oriented software," in ICST, 2008, pp. 72-81.

Digital Library

[20]

A. Bessey, K. Block, B. Chelf, A. Chou, B. Fulton, S. Hallem, C. Henri-Gros, A. Kamsky, S. McPeak, and D. R. Engler, "A few billion lines of code later: Using static analysis to find bugs in the real world," Commun ACM, vol. 53, no. 2, pp. 66-75, 2010.

Digital Library

[21]

C. Pacheco, S. K. Lahiri, and T. Ball, "Finding errors in .NET with feedback-directed random testing," in ISSTA, 2008, pp. 87-96.

Digital Library

[22]

R. Vallée-Rai, P. Co, E. Gagnon, L. J. Hendren, P. Lam, and V. Sundaresan, "Soot - a Java bytecode optimization framework," in CASCON, 1999, pp. 125-135.

Digital Library

[23]

M. D. Ernst, J. Cockrell, W. G. Griswold, and D. Notkin, "Dynamically discovering likely program invariants to support program evolution," IEEE T Software Eng, vol. 27, no. 2, pp. 213-224, 2001.

Digital Library

[24]

S. Hangal and M. S. Lam, "Tracking down software bugs using automatic anomaly detection," in ICSE, 2002, pp. 291- 301.

Digital Library

[25]

A. Wasylkowski and A. Zeller, "Mining temporal specifications from object usage," in ASE, 2009, pp. 295-306.

Digital Library

[26]

S. Thummalapenta and T. Xie, "Mining exception-handling rules as sequence association rules," in ICSE, 2009, pp. 496- 506.

Digital Library

[27]

T. T. Nguyen, H. A. Nguyen, N. H. Pham, J. M. Al-Kofahi, and T. N. Nguyen, "Graph-based mining of multiple object usage patterns," in ESEC/FSE, 2009, pp. 383-392.

Digital Library

[28]

M. Monperrus, M. Bruch, and M. Mezini, "Detecting missing method calls in object-oriented software," in ECOOP, 2010, pp. 2-25.

Digital Library

[29]

M. Pradel, C. Jaspan, J. Aldrich, and T. R. Gross, "Statically checking API protocol conformance with mined multiobject specifications," in International Conference on Software Engineering (ICSE), 2012.

Digital Library

[30]

H. Jaygarl, K.-S. Lu, and C. K. Chang, "GenRed: A tool for generating and reducing object-oriented test cases," in COMPSAC, 2010, pp. 127-136.

Digital Library

[31]

W. Zheng, Q. Zhang, M. R. Lyu, and T. Xie, "Random unittest generation with MUT-aware sequence recommendation," in ASE, 2010, pp. 293-296.

Digital Library

[32]

W. Jin, A. Orso, and T. Xie, "Automated behavioral regression testing," in ICST, 2010, pp. 137-146.

Digital Library

[33]

S. Thummalapenta, T. Xie, N. Tillmann, J. de Halleux, and W. Schulte, "MSeqGen: Object-oriented unit-test generation via mining source code," in ESEC/FSE, 2009, pp. 193-202.

Digital Library

[34]

T. Xie and D. Notkin, "Automatic extraction of object-oriented observer abstractions from unit-test executions," in ICFEM, 2004, pp. 290-305.

[35]

S. Thummalapenta, J. de Halleux, N. Tillmann, and S. Wadsworth, "DyGen: Automatic generation of high-coverage tests via mining gigabytes of dynamic traces," in TAP, 2010, pp. 77-93.

Digital Library

[36]

S. Zhang, D. Saff, Y. Bu, and M. D. Ernst, "Combined static and dynamic automated test generation," in ISSTA, 2011, pp. 353-363.

Digital Library

[37]

T. Xie and D. Notkin, "Mutually enhancing test generation and specification inference," in FATES, 2003, pp. 60-69.

[38]

V. Dallmeier, N. Knopp, C. Mallon, S. Hack, and A. Zeller, "Generating test cases for specification mining," in ISSTA, 2010, pp. 85-96.

Digital Library

[39]

T. Xie and D. Notkin, "Tool-assisted unit test selection based on operational violations," in ASE, 2003, pp. 40-48.

Digital Library

[40]

M. Renieris and S. P. Reiss, "Fault localization with nearest neighbor queries," in ASE, 2003, pp. 30-39.

[41]

V. Dallmeier, C. Lindig, and A. Zeller, "Lightweight defect localization for Java," in ECOOP, 2005, pp. 528-550.

Digital Library

[42]

L. Mariani, F. Pastore, and M. Pezzè, "Dynamic analysis for diagnosing integration faults," IEEE Trans Sw Eng, vol. 37, no. 4, pp. 486-508, 2011.

Digital Library

[43]

W. Weimer, T. Nguyen, C. Le Goues, and S. Forrest, "Automatically finding patches using genetic programming," in ICSE, 2009, pp. 363-374.

Digital Library

[44]

Y. Wei, Y. Pei, C. A. Furia, L. S. Silva, S. Buchholz, B. Meyer, and A. Zeller, "Automated fixing of programs with contracts," in ISSTA, 2010, pp. 61-72.

Digital Library

Cited By

Ye HRao JShi YLi Z(2021)ProExtor: Mining API Protocols for Program Vulnerability DetectionProceedings of the 3rd International Conference on Advanced Information Science and System10.1145/3503047.3503100(1-7)Online publication date: 26-Nov-2021
https://dl.acm.org/doi/10.1145/3503047.3503100
Amann SNguyen HNadi SNguyen TMezini MStorey MAdams BHaiduc S(2019)Investigating next steps in static API-misuse detectionProceedings of the 16th International Conference on Mining Software Repositories10.1109/MSR.2019.00053(265-275)Online publication date: 26-May-2019
https://dl.acm.org/doi/10.1109/MSR.2019.00053
He ZChen YHuang EWang QPei YYuan HAtlee JBultan TWhittle J(2019)A system identification based Oracle for control-CPS software fault localizationProceedings of the 41st International Conference on Software Engineering10.1109/ICSE.2019.00029(116-127)Online publication date: 25-May-2019
https://dl.acm.org/doi/10.1109/ICSE.2019.00029
Show More Cited By

Recommendations

Specification Mining with Few False Positives
TACAS '09: Proceedings of the 15th International Conference on Tools and Algorithms for the Construction and Analysis of Systems: Held as Part of the Joint European Conferences on Theory and Practice of Software, ETAPS 2009,

Formal specifications can help with program testing, optimization, refactoring, documentation, and, most importantly, debugging and repair. Unfortunately, formal specifications are difficult to write manually, while techniques that infer specifications ...
Controlling false positives in association rule mining

Association rule mining is an important problem in the data mining area. It enumerates and tests a large number of rules on a dataset and outputs rules that satisfy user-specified constraints. Due to the large number of rules being tested, rules that do ...
False Positives vs. False Negatives: The Effects of Recovery Time and Cognitive Costs on Input Error Preference
UIST '21: The 34th Annual ACM Symposium on User Interface Software and Technology

Existing approaches to trading off false positive versus false negative errors in input recognition are based on imprecise ideas of how these errors affect user experience that are unlikely to hold for all situations. To inform dynamic approaches to ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

ICSE '12: Proceedings of the 34th International Conference on Software Engineering

June 2012

1657 pages

ISBN:9781467310673

General Chair:
Martin Glinz,
Program Chairs:
Gail Murphy,
Mauro Pezzè

Sponsors

SIGSOFT: ACM Special Interest Group on Software Engineering

Publisher

IEEE Press

Publication History

Published: 02 June 2012

Check for updates

Qualifiers

Article

Conference

ICSE '12

Sponsor:

SIGSOFT

ICSE '12: 34th International Conference on Software Engineering

June 2 - 9, 2012

Zurich, Switzerland

Acceptance Rates

Overall Acceptance Rate 276 of 1,856 submissions, 15%

Upcoming Conference

ICSE 2025

2025 IEEE/ACM 46th International Conference on Software Engineering

April 26 - May 3, 2025

Ottawa , ON , Canada

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

22
Total Citations
View Citations
285
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 11 Aug 2024

Other Metrics

View Author Metrics

Citations

Cited By

Ye HRao JShi YLi Z(2021)ProExtor: Mining API Protocols for Program Vulnerability DetectionProceedings of the 3rd International Conference on Advanced Information Science and System10.1145/3503047.3503100(1-7)Online publication date: 26-Nov-2021
https://dl.acm.org/doi/10.1145/3503047.3503100
Amann SNguyen HNadi SNguyen TMezini MStorey MAdams BHaiduc S(2019)Investigating next steps in static API-misuse detectionProceedings of the 16th International Conference on Mining Software Repositories10.1109/MSR.2019.00053(265-275)Online publication date: 26-May-2019
https://dl.acm.org/doi/10.1109/MSR.2019.00053
He ZChen YHuang EWang QPei YYuan HAtlee JBultan TWhittle J(2019)A system identification based Oracle for control-CPS software fault localizationProceedings of the 41st International Conference on Software Engineering10.1109/ICSE.2019.00029(116-127)Online publication date: 25-May-2019
https://dl.acm.org/doi/10.1109/ICSE.2019.00029
(2018)Mining balanced API protocolsInternational Journal of Computational Science and Engineering10.1504/IJCSE.2018.09176416:3(289-302)Online publication date: 1-Jan-2018
https://dl.acm.org/doi/10.1504/IJCSE.2018.091764
Kim MCheung SKim SLeavens GGarcia APăsăreanu C(2018)Which generated test failures are fault revealing? prioritizing failures based on inferred precondition violations using PAFProceedings of the 2018 26th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering10.1145/3236024.3236058(679-690)Online publication date: 26-Oct-2018
https://dl.acm.org/doi/10.1145/3236024.3236058
Legunsen OHassan WXu XRoşu GMarinov DLo DApel SKhurshid S(2016)How good are the specs? a study of the bug-finding effectiveness of existing Java API specificationsProceedings of the 31st IEEE/ACM International Conference on Automated Software Engineering10.1145/2970276.2970356(602-613)Online publication date: 25-Aug-2016
https://dl.acm.org/doi/10.1145/2970276.2970356
Ermuth MPradel MZeller ARoychoudhury A(2016)Monkey see, monkey do: effective generation of GUI tests with inferred macro eventsProceedings of the 25th International Symposium on Software Testing and Analysis10.1145/2931037.2931053(82-93)Online publication date: 18-Jul-2016
https://dl.acm.org/doi/10.1145/2931037.2931053
Wu RXiao XCheung SZhang HZhang C(2016)Casper: an efficient approach to call trace collectionACM SIGPLAN Notices10.1145/2914770.283761951:1(678-690)Online publication date: 11-Jan-2016
https://dl.acm.org/doi/10.1145/2914770.2837619
Pereira OBrandão WSong MOssowski S(2016)Automatic formal specification generation of APIs by mining unit testsProceedings of the 31st Annual ACM Symposium on Applied Computing10.1145/2851613.2851969(1542-1545)Online publication date: 4-Apr-2016
https://dl.acm.org/doi/10.1145/2851613.2851969
Wu RXiao XCheung SZhang HZhang CBodik RMajumdar R(2016)Casper: an efficient approach to call trace collectionProceedings of the 43rd Annual ACM SIGPLAN-SIGACT Symposium on Principles of Programming Languages10.1145/2837614.2837619(678-690)Online publication date: 11-Jan-2016
https://dl.acm.org/doi/10.1145/2837614.2837619
Show More Cited By

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents