Towards automatic scenario generation from coverage information
Abstract
Nowadays, the design of software systems is pushed towards agile development practices. One of its most fundamental approaches is Test Driven Development (TDD). This procedure is based on test cases which are incrementally written prior to the implementation. Recently, Behavior Driven Development (BDD) has been introduced as an extension of TDD, in which natural language scenarios are the starting point for the test cases. This description offers a ubiquitous communication mean for both the software developers and stakeholders.
Following the BDD methodology thoroughly, one would expect 100% code coverage, since code is only written to make the test cases pass. However, as we show in an empirical study this expectation is not valid in practice. It becomes even worse in the process of development, i.e. the coverage decreases over time. To close the coverage gap, we sketch an algorithm that generates BDD-style scenarios based on uncovered code.
References
[1]
K. Beck, Test Driven Development. By Example. Amsterdam: Addison-Wesley Longman, Nov. 2003.
[2]
T. Bhat and N. Naghappan, "Evaluating the efficacy of test-driven development: industrial case studies," in Empirical Software Engineering, 2006, pp. 356--363.
[3]
B. George and L. A. Williams, "A structured experiment of test-driven development," Information & Software Technology, vol. 46, no. 5, pp. 337--342, 2004.
[4]
D. North, "Behavior Modification: The evolution of behavior-driven development," Better Software, vol. 8, no. 3, 2006.
[5]
R. Drechsler, M. Diepenbeck, D. Große, U. Kühne, H. M. Le, J. Seiter, M. Soeken, and R. Wille, "Completeness-driven development," in International Conference on Graph Transformations, 2012, pp. 38--50.
[6]
J. Offutt and A. Abdurazik, "Generating tests from uml specifications," in International Conference on the Unified Modeling Language, 1999, pp. 416--429.
[7]
S. Kansomkeat and W. Rivepiboon, "Automated-generating test case using uml statechart diagrams," in Annual Research Conference of the South African Institute of Computer Scientists and Information Technologists on Enablement through Technology, 2003, pp. 296--300.
[8]
C. Doungsa-ard, K. Dahal, A. Hossain, and T. Suwannasart, "Test Data Generation from UML state machine diagrams using GAs," in International Conference on Software Engineering Advances, 2007, pp. 47--52.
[9]
P. Samuel, R. Mall, and P. Kanth, "Automatic test case generation from UML communication diagrams," Information and Software Technology, vol. 49, no. 2, pp. 158--171, 2007.
[10]
S. Horwitz, "Tool support for improving test coverage," in European Symposium on Programming, 2002.
[11]
C. Pacheco, S. K. Lahiri, M. D. Ernst, and T. Ball, "Feedback-directed random test generation," in International Conference on Software Engineering, 2007, pp. 75--84.
[12]
P. Ammann, P. Black, and W. Majurski, "Using model checking to generate tests from specifications," in International Conference on Formal Engineering Methods, 1998, pp. 46--54.
[13]
A. Gargantini and C. Heitmeyer, "Using model checking to generate tests from requirements specifications," in European Software Engineering Conference held jointly with the 7th ACM SIGSOFT International Symposium on Foundations of Software Engineering, 1999, pp. 146--162.
[14]
K. Havelund, "Java PathFinder, a translator from Java to Promela," in 5th and 6th International SPIN Workshops on Theoretical and Practical Aspects of SPIN Model Checking, 1999, p. 152.
[15]
D. Flanagan and Y. Matsumoto, The Ruby Programming Language. O'Reilly Media, 2008.
[16]
M. Wynne and A. Hellesøy, The Cucumber Book: Behaviour-Driven Development for Testers and Developers. The Pragmatic Bookshelf, 2012.
[17]
D. Chelimsky, D. Astels, B. Helmkamp, D. North, Z. Dennis, and A. Hellesøy, The RSpec Book: Behaviour Driven Development with Rspec, Cucumber, and Friends. The Pragmatic Bookshelf, 2010.
[18]
M. Siniaalto and P. Abrahamsson, "A comparative case study on the impact of test-driven development on program design and test coverage," in International Symposium on Empirical Software Engineering and Measurement, 2007.
[19]
M. Diepenbeck, M. Soeken, D. Grose, and R. Drechsler, "Behavior driven development for circuit design and verification," in High Level Design Validation and Test Workshop, 2012, pp. 9--16.
- Towards automatic scenario generation from coverage information
Recommendations
System state coverage through automatic test case generation
Achieving reasonable coverage of system states during testing is a non-trivial problem, since the number of system states for practical systems is usually very large. To further complicate the problem, system developers often do not construct a system ...
Towards a deeper understanding of test coverage
Test coverage is traditionally considered as how much of the code is covered by the test suite in whole. However, test suites typically contain different types of tests with different roles, such as unit tests, integration tests and functional tests. As ...
Automated coverage calculation and test case generation
SAICSIT '12: Proceedings of the South African Institute for Computer Scientists and Information Technologists ConferenceThis article describes the use of symbolic execution, a formal method of static analysis, to calculate code coverage of a program's existing JUnit test suites. Code coverage is measured with respect to a number of test adequacy criteria, including ...
Comments
Information & Contributors
Information
Published In
Sponsors
Publisher
IEEE Press
Publication History
Published: 18 May 2013
Check for updates
Qualifiers
- Research-article
Conference
ICSE '13
Sponsor:
ICSE '13: 35th International Conference on Software Engineering
May 18 - 19, 2013
California, San Francisco
Upcoming Conference
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 87Total Downloads
- Downloads (Last 12 months)1
- Downloads (Last 6 weeks)0
Reflects downloads up to 22 Dec 2024
Other Metrics
Citations
View Options
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in