research-article

Nighthawk: a two-level genetic-random unit test data generator

Authors:

James H. Andrews,

Felix C. H. Li,

Tim MenziesAuthors Info & Claims

ASE '07: Proceedings of the 22nd IEEE/ACM International Conference on Automated Software Engineering

Pages 144 - 153

https://doi.org/10.1145/1321631.1321654

Published: 05 November 2007 Publication History

Abstract

Randomized testing has been shown to be an effective method fortesting software units. However, the thoroughness of randomized unit testing varies widely according to the settings of certain parameters, such as the relative frequencies with which methods are called. In this paper, we describe a system which uses agenetic algorithm to find parameters for randomized unit testing that optimize test coverage. We compare our coverage results to previous work, and report on case studies and experiments on system options

References

[1]

J. H. Andrews, S. Haldar, Y. Lei, and C. H. F. Li. Tool support for randomized unit testing. In Proceedings of the First International Workshop on Randomized Testing (RT'06), pages 36--45, Portland, Maine, July 2006.

Digital Library

[2]

S. Antoy and R. G. Hamlet. Automatically checking an implementation against its formal specification. IEEE Transactions on Software Engineering, 2 (1):55--69, January 2000.

Digital Library

[3]

T. Ball. A theory of predicate-complete test coverage and generation. In Third International Symposium on Formal Methods for Components and Objects (FMCO 2004), pages 1--22, Leiden, The Netherlands, November 2004.

Digital Library

[4]

R. R. Bouckaert. Choosing between two learning algorithms based on calibrated tests. In Proceedings of the Twentieth International Conference on Machine Learning (ICML 2003), pages 51--58, Washington, DC, USA, August 2003.

[5]

L. Burdy, Y. Cheon, D. R. Cok, M. D. Ernst, J. R. Kiniry, and G. T. Leavens. An overview of JML tools and applications. International Journal on Software Tools for Technology Transfer, 7(3):212--232, June 2005.

Digital Library

[6]

K. Claessen and J. Hughes. QuickCheck: A lightweight tool for random testing of Haskell programs. In Proceedings of the Fifth ACM SIGPLAN International Conference on Functional Programming (ICFP '00), pages 268--279, Montreal, Canada, September 2000.

Digital Library

[7]

L. A. Clarke. A system to generate test data and symbolically execute programs. IEEE Transactions on Software Engineering, SE--2(3):215--222, September 1976.

Digital Library

[8]

Cobertura Development Team. Cobertura web site. cobertura.sourceforge.net, accessed February 2007.

[9]

S. Cornett. Minimum acceptable code coverage. http://www.bullseye.com/minimum.html, 2006.

[10]

J. Costa, P. Silva, and N. Lopes. JDEAL Java Distributed Evolutionary Algorithms Library version 1.0: Getting started. Technical report, LaSEEB Instituto Superior Técnico, Universidade Técnica de Lisboa, Portugal, 2005.

[11]

C. Csallner and Y. Smaragdakis. J. Crasher: an automatic robustness tester for Java. Software Practice and Experience, 34(11):1025--1050, 2004.

Digital Library

[12]

K. A. DeJong and W. M. Spears. An analysis of the interacting roles of population size and crossover in genetic algorithms. In First Workshop on Parallel Problem Solving from Nature, pages 38--47. Springer, 1990.

Digital Library

[13]

J. Demsar. Statistical comparisons of classifiers over multiple data sets. Journal of Machine Learning Research, 7:1--30, 2006.

Digital Library

[14]

R.-K. Doong and P. G. Frankl. The ASTOOT approach to testing object-oriented programs. ACM Transactions on Software Engineering and Methodology, 3(2):101--130, April 1994.

Digital Library

[15]

M. D. Ernst, J. Cockrell, W. G. Griswold, and D. Notkin. Dynamically discovering likely program invariants to support program evolution. IEEE Transactions on Software Engineering, 27(2):99--123, February 2001.

Digital Library

[16]

P. Godefroid, N. Klarlund, and K. Sen. DART: Directed automated random testing. In Proceedings of the ACM SIGPLAN 2005 Conference on Programming Language Design and Implementation (PLDI), pages 213--223, Chicago, June 2005.

Digital Library

[17]

D. E. Goldberg. Genetic Algorithm in Search, Optimization, and Machine Learning. Addison-Wesley, 1989.

Digital Library

[18]

A. Groce, G. J. Holzmann, and R. Joshi. Randomized differential testing as a prelude to formal verification. In Proceedings of the 29th International Conference on Software Engineering (ICSE 2007), pages 621--631, Minneapolis, MN, May 2007.

Digital Library

[19]

Q. Guo, R. M. Hierons, M. Harman, and K. Derderian. Computing unique input/output sequences using genetic algorithms. In 3rd International Workshop on Formal Approaches to Testing of Software (FATES 2003), volume 2931 of LNCS, pages 164--177. Springer, 2004.

[20]

N. Gupta, A. P. Mathur, and M. L. Soffa. Automated test data generation using an iterative relaxation method. In Sixth International Symposium on the Foundations of Software Engineering (FSE 98), pages 224--232, November 1998.

Digital Library

[21]

K. Havelund and T. Pressburger. Model checking Java programs using Java PathFinder. International Journal on Software Tools for Technology Transfer, 2(4):366--381, 2000.

[22]

W. C. Hetzel, editor. Program Test Methods. Automatic Computation. Prentice-Hall, Englewood Cliffs, N.J., 1973.

[23]

M. Kelly. Beyond the black art. EvoWeb News and Features, July 2001. evonet.lri.fr/evoweb/.

[24]

W. K. Leow, S. C. Khoo, and Y. Sun. Automated generation of test programs from closed specifications of classes and test cases. In Proceedings of the 26th International Conference on Software Engineering (ICSE 2004), pages 96--105, Edinburgh, UK, May 2004.

Digital Library

[25]

F. C. H. Li. Applications of genetic algorithms to randomized unit testing. Master's thesis, Department of Computer Science, University of Western Ontario, December 2006.

[26]

C. C. Michael, G. McGraw, and M. A. Schatz. Generating software test data by evolution. IEEE Transactions on Software Engineering, 27(12), December 2001.

Digital Library

[27]

B. P. Miller, L. Fredriksen, and B. So. An empirical study of the reliability of UNIX utilities. Commun. ACM, 33(12):32--44, December 1990.

Digital Library

[28]

D. Owen and T. Menzies. Lurch: a lightweight alternative to model checking. In Proceedings of the Fifteenth International Conference on Software Engineering and Knowledge Engineering (SEKE'2003), pages 158--165, San Francisco, July 2003.

[29]

C. Pacheco, S. K. Lahiri, M. D. Ernst, and T. Ball. Feedback-directed random test generation. In Proceedings of the 29th International Conference on Software Engineering (ICSE 2007), pages 75--84, Minneapolis, MN, May 2007.

Digital Library

[30]

L. Rela. Evolutionary computing in search-based software engineering. Master's thesis, Lappeenranta University of Technology, 2004.

[31]

K. Sen, D. Marinov, and G. Agha. CUTE: a concolic unit testing engine for C. In Proceedings of the 13th ACM SIGSOFT International Symposium on Foundations of Software Engineering (ESEC/FSE), pages 263--272, Lisbon, September 2005.

Digital Library

[32]

P. Tonella. Evolutionary testing of classes. In Proceedings of the ACM/SIGSOFT International Symposium on Software Testing and Analysis (ISSTA 2004), pages 119--128, Boston, Massachusetts, USA, July 2004.

Digital Library

[33]

W. Visser, C. S. Pasareanu, and S. Khurshid. Test input generation with Java PathFinder. In Proceedings of the ACM/SIGSOFT International Symposium on Software Testing and Analysis (ISSTA 2004), pages 97--107, Boston, MA, July 2004.

Digital Library

[34]

W. Visser, C. S. Pasareanu, and R. Pelánek. Test input generation for Java containers using state matching. In Proceedings of the International Symposium on Software Testing and Analysis (ISSTA 2006), pages 37--48, Portland, Maine, July 2006.

Digital Library

Cited By

Holmes JAhmed IBrindescu CGopinath RZhang HGroce A(2020)Using Relative Lines of Code to Guide Automated Test Generation for PythonACM Transactions on Software Engineering and Methodology10.1145/340889629:4(1-38)Online publication date: 26-Sep-2020
https://dl.acm.org/doi/10.1145/3408896
Dass SNamin AHung CCerny TShin DBechini A(2020)Vulnerability coverage for adequacy security testingProceedings of the 35th Annual ACM Symposium on Applied Computing10.1145/3341105.3374099(540-543)Online publication date: 30-Mar-2020
https://dl.acm.org/doi/10.1145/3341105.3374099
Krishna RMenzies T(2020)Learning actionable analytics from multiple software projectsEmpirical Software Engineering10.1007/s10664-020-09843-625:5(3468-3500)Online publication date: 24-Jul-2020
https://dl.acm.org/doi/10.1007/s10664-020-09843-6
Show More Cited By

Index Terms

Nighthawk: a two-level genetic-random unit test data generator

Recommendations

Genetic Algorithms for Randomized Unit Testing

Randomized testing is an effective method for testing software units. The thoroughness of randomized unit testing varies widely according to the settings of certain parameters, such as the relative frequencies with which methods are called. In this ...
Checked Coverage and Object Branch Coverage: New Alternatives for Assessing Student-Written Tests
SIGCSE '15: Proceedings of the 46th ACM Technical Symposium on Computer Science Education

Many educators currently use code coverage metrics to assess student-written software tests. While test adequacy criteria such as statement or branch coverage can also be used to measure the thoroughness of a test suite, they have limitations. Coverage ...
Towards a deeper understanding of test coverage

Test coverage is traditionally considered as how much of the code is covered by the test suite in whole. However, test suites typically contain different types of tests with different roles, such as unit tests, integration tests and functional tests. As ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

ASE '07: Proceedings of the 22nd IEEE/ACM International Conference on Automated Software Engineering

November 2007

590 pages

ISBN:9781595938824

DOI:10.1145/1321631

General Chair:
Kurt Stirewalt
Michigan State University, USA
,
Program Chairs:
Alexander Egyed
Teknowledge Corporation, USA
,
Bernd Fischer
University of Southampton, UK

Copyright © 2007 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 05 November 2007

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

ASE07

Sponsor:

ASE07: International Conference on Automated Software Engineering 2007

November 5 - 9, 2007

Georgia, Atlanta, USA

Acceptance Rates

Overall Acceptance Rate 82 of 337 submissions, 24%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

33
Total Citations
View Citations
470
Total Downloads

Downloads (Last 12 months)3
Downloads (Last 6 weeks)0

Reflects downloads up to 24 Dec 2024

Other Metrics

View Author Metrics

Citations

Cited By

Holmes JAhmed IBrindescu CGopinath RZhang HGroce A(2020)Using Relative Lines of Code to Guide Automated Test Generation for PythonACM Transactions on Software Engineering and Methodology10.1145/340889629:4(1-38)Online publication date: 26-Sep-2020
https://dl.acm.org/doi/10.1145/3408896
Dass SNamin AHung CCerny TShin DBechini A(2020)Vulnerability coverage for adequacy security testingProceedings of the 35th Annual ACM Symposium on Applied Computing10.1145/3341105.3374099(540-543)Online publication date: 30-Mar-2020
https://dl.acm.org/doi/10.1145/3341105.3374099
Krishna RMenzies T(2020)Learning actionable analytics from multiple software projectsEmpirical Software Engineering10.1007/s10664-020-09843-625:5(3468-3500)Online publication date: 24-Jul-2020
https://dl.acm.org/doi/10.1007/s10664-020-09843-6
Shamshiri SRojas JGaleotti JWalkinshaw NFraser G(2018)How Do Automatically Generated Unit Tests Influence Software Maintenance?2018 IEEE 11th International Conference on Software Testing, Verification and Validation (ICST)10.1109/ICST.2018.00033(250-261)Online publication date: Apr-2018
https://doi.org/10.1109/ICST.2018.00033
Shamshiri SRojas JGazzola LFraser GMcMinn PMariani LArcuri A(2018)Random or evolutionary search for object‐oriented test suite generation?Software Testing, Verification and Reliability10.1002/stvr.166028:4Online publication date: 30-Mar-2018
https://doi.org/10.1002/stvr.1660
Galhotra SBrun YMeliou ABodden ESchäfer WDeursen AZisman A(2017)Fairness testing: testing software for discriminationProceedings of the 2017 11th Joint Meeting on Foundations of Software Engineering10.1145/3106237.3106277(498-510)Online publication date: 21-Aug-2017
https://dl.acm.org/doi/10.1145/3106237.3106277
Daka ERojas JFraser GBultan TSen K(2017)Generating unit tests with descriptive names or: would you name your children thing1 and thing2?Proceedings of the 26th ACM SIGSOFT International Symposium on Software Testing and Analysis10.1145/3092703.3092727(57-67)Online publication date: 10-Jul-2017
https://dl.acm.org/doi/10.1145/3092703.3092727
Alipour MGroce AGopinath RChristi AZeller ARoychoudhury A(2016)Generating focused random tests using directed swarm testingProceedings of the 25th International Symposium on Software Testing and Analysis10.1145/2931037.2931056(70-81)Online publication date: 18-Jul-2016
https://dl.acm.org/doi/10.1145/2931037.2931056
Rojas JFraser GArcuri AYoung MXie T(2015)Automated unit test generation during software development: a controlled experiment and think-aloud observationsProceedings of the 2015 International Symposium on Software Testing and Analysis10.1145/2771783.2771801(338-349)Online publication date: 13-Jul-2015
https://dl.acm.org/doi/10.1145/2771783.2771801
Shamshiri SRojas JFraser GMcMinn PEsparcia-Alcázar A(2015)Random or Genetic Algorithm Search for Object-Oriented Test Suite Generation?Proceedings of the 2015 Annual Conference on Genetic and Evolutionary Computation10.1145/2739480.2754696(1367-1374)Online publication date: 11-Jul-2015
https://dl.acm.org/doi/10.1145/2739480.2754696
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents