Appears in Proceedings of the 3rd International Workshop on Formal Approaches to Testing of Software (FATES 2003)
Mutually Enhancing Test Generation and Specification
Inference
Tao Xie and David Notkin
Department of Computer Science and Engineering
University of Washington at Seattle
WA 98195, USA
{taoxie, notkin}@cs.washington.edu
Abstract. Generating effective tests and inferring likely program specifications
are both difficult and costly problems. We propose an approach in which we
can mutually enhance the tests and specifications that are generated by iteratively applying each in a feedback loop. In particular, we infer likely specifications from the executions of existing tests and use these specifications to guide
automatic test generation. Then the existing tests, as well as the new tests, are
used to infer new specifications in the subsequent iteration. The iterative process continues until there is no new test that violates specifications inferred in
the previous iteration. Inferred specifications can guide test generation to focus
on particular program behavior, reducing the scope of analysis; and newly generated tests can improve the inferred specifications. During each iteration, the
generated tests that violate inferred specifications are collected to be inspected.
These violating tests are likely to have a high probability of exposing faults or
exercising new program behavior. Our hypothesis is that such a feedback loop
can mutually enhance test generation and specification inference.
1 Introduction
There are a variety of software quality assurance (SQA) methods being adopted in
practice. Since there are particular dependences or correlations among some SQA
methods, these methods could be integrated synergistically to provide value considerably beyond what the separate methods can provide alone [28, 32, 35, 11]. Two
such exemplary methods are specification-based test generation and dynamic specification inference. Specification-based test generation requires specifications a priori
[13, 25, 5]. In practice, however, formal specifications are often not written for programs. On the other hand, dynamic specification inference relies on good tests to infer
high quality specifications [10, 31, 21]. There is a circular dependency between tests
in specification-based test generation and specifications in dynamic specification
inference.
In addition, when formal specifications are not available, automatic test generation,
such as white-box test generation or random test generation, does not sufficiently
address output checking. Without specifications, output checking is limited to detect-
60
ing a program crash, or an exception is thrown but not caught. In other words, there is
a lack of test oracles in automatic test generation without specifications a priori.
In this research, without a priori specifications, we want to mutually enhance test
generation and specification inference. At the same time, from a large number of
generated tests, we want to have a way to identify valuable tests for inspection. Valuable tests can be fault-revealing tests or tests that exercise new program behavior. The
solution we propose is a method and tools for constructing a feedback loop between
test generation and specification inference, using and adapting existing specificationbased test generation and dynamic specification inference techniques. We implement
the method for three types of inferred specifications: axiomatic specifications [10],
protocol specifications [31], and algebraic specifications [21]. We demonstrate the
usefulness of the method by initially focusing on the unit test generation and the
specification inference for object-oriented components, such as Java classes.
2 Background
2.1 Formal Specifications
A formal specification expresses the desired behavior of a program. We model the
specification in a style of requires/ensures. Requires describe the constraints of using
APIs provided by a class. When requires are satisfied during execution, ensures describe the desired behavior of the class. Requires can be used to guard against illegal
inputs, and ensures can be used as test oracles for correctness checking.
Axiomatic specifications [22] are defined in the granularity of a method in a class
interface. Preconditions for a method are requires for the method, whereas postconditions for a method are ensures for the method. Object invariants in axiomatic
specifications can be viewed as the pre/post-conditions for each method in the class
interface. The basic elements in requires/ensures consist of method arguments, returns, and class fields.
Protocol specifications [6] are defined in the granularity of a class. Requires are the
sequencing constraints in the form of finite state machines. Although extensions to
protocol specifications can describe ensures behavior, there are no ensures in basic
protocol specifications. The basic elements in requires consist of method calls, including method signatures, but usually no method arguments or returns.
Algebraic specifications [18] are also defined in the granularity of a class. Ensures
are the AND combination of all axioms in algebraic specifications. In the AND combination, each axiom, in the form of LHS=RHS, is interpreted as “if a current call
sequence window instantiates LHS, then its result is equal to RHS”. The basic elements in ensures consist of method calls, including method signature, method arguments and returns, but no class fields. Therefore, algebraic specifications are in a
higher-level abstraction than axiomatic specifications are. Usually there are no explicit requires in algebraic specifications. Indeed, sequencing constraints, which are
requires, can be derived from the axiom whose RHS is an error or exception [4].
61
2.2 Dynamic Specification Inference
Dynamic specification inference discovers operational abstractions from the executions of tests [20]. An operational abstraction is syntactically identical to a formal
specification. The discovered operational abstractions consist of those properties that
hold for all the observed executions. These abstractions can be used to approximate
specifications or indicate the deficiency of tests.
Ernst et al. [10] develop a dynamic invariant detection tool, called Daikon, to infer
likely axiomatic specifications from executions of test suites. It examines the variable
values that a program computes, generalizes over them, and reports the generalizations in the form of pre/post-conditions and class invariants.
Whaley et al. [31] develop a tool to infer likely protocol specifications from
method call traces collected while a Java class interface is being used. These specifications are in the form of multiple finite state machines, each of which contains
methods accessing the same class field. Ammons et al. [1] develop a tool to infer
likely protocol specifications from C method call traces by using an off-the-shelf
probabilistic finite state automaton learner. Hagerer et al. [19] present the regular
extrapolation technique to discover protocol specifications from execution traces of
reactive systems.
Henkel and Diwan [21] develop a tool to derive a large number of terms for a Java
class and generate tests to evaluate them. The observational equivalence technique [3,
9] is used to evaluate the equality among these terms. Based on the evaluation results,
equations among these terms are proposed, and are further generalized to infer axioms
in algebraic specifications.
2.3 Specification-Based Test Generation
We categorize specification-based test generation into test generation for functionality
and test generation for robustness. Test generation for functionality generates tests
that satisfy requires, and checks whether ensures are satisfied during test executions.
Test generation for robustness generates tests that may not satisfy requires, and
checks whether a program can handle these test executions gracefully, such as throwing appropriate exceptions.
We divide the test generation problem into three sub-problems: object state setup,
method parameter generation, and method sequence generation. Object state setup
puts the class under test into particular states before invoking methods on it. Method
parameter generation produces particular arguments for methods to be invoked.
Method sequence generation creates particular method call sequences to exercise the
class on certain object states. Axiomatic specifications provide more guidance on
both method parameter generation and object state setup, whereas algebraic specifications provide more guidance on method sequence generation. Protocol specifications
provide more guidance on both object state setup and method sequence generation.
Dick and Faivre develop a tool to reduce axiomatic specifications to a disjunctive
normal form and generate tests based on them [8]. Boyapati et al. develop a tool to
generate tests effectively by filtering the test input space based on preconditions in
62
axiomatic specifications [5]. Gupta develop a structural testing technique to generate
test inputs to exercise some particular post-conditions or assertions [16]. A commercial Java unit testing tool, ParaSoft Jtest [24], can automatically generate test inputs to
perform white box testing when no axiomatic specifications are provided, and perform black box testing when axiomatic specifications are equipped.
There is a variety of test generation techniques based on protocol specifications,
which is in the form of finite state machines [25]. There are several test generation
tools based on algebraic specifications. They generate tests to execute the LHS and
RHS of an axiom. The DAISTS tool developed by Gannon et al. [13] and the Daistish
tool developed by Hughes and Stotts [23] use an implementation-supplied equality
method to compare the results of LHS and RHS. A tool developed by Bernot et al. [3]
and the ASTOOT tool developed by Doong and Frankl [9] uses observational equivalence to determine whether LHS and RHS are equal.
3 Feedback Loop Framework
T he existing
test suite
Instrum ented
program
Autom atically
generated
test inputs
Selected
tests
Test
G eneration
Test
Selection
Autom atically
generated
test inputs
Trace
Likely
specs
D ata trace
Spec
Collection
Inference
Fig. 1. An overview of the feedback loop framework
Our approach can be viewed as a black box into which a developer feeds a program
and its existing tests, and from which the developer gets a set of valuable tests, inferred specifications, and reasons why these tests are valuable. Then the developer
can inspect the valuable tests and inferred specifications for problems. Our feedback
loop framework consists of multiple iterations. Each iteration is given the program, a
set of tests, and specifications inferred from the previous iteration (except for the first
iteration). After each iteration, a complete set of new tests, a valuable subset of new
tests, reasons for being valuable, and new inferred specifications are produced. The
subsequent iteration is given the original tests augmented by the complete set of new
tests or the valuable subset of new tests, as well as the new inferred specifications.
Optionally the developer can specify some iteration-terminating conditions, such as a
stack size being equal to the maximum capacity, or the number of iterations reaching
the specified number. The iterations continue until user-specified conditions are satis-
63
fied and there is no new test that violates specifications inferred in the previous iteration.
Figure 1 shows an overview of the feedback loop framework. The framework defines four stages for each iteration: trace collection, specification inference, test generation, and test selection. Human intervention is only needed for inspecting selected
tests and inferred specifications in the end of the feedback loop. But human intervention may be incorporated in the end of each iteration and should improve results.
In the trace collection stage, the given tests are run on the instrumented Java program and traces are collected from the executions. Object states are defined by some
particular relevant class field values. The values of method arguments, returns, and
object states are recorded at the entry and exit of a method execution. To collect object states, we instrument invocations of this.equals(this) at the entry and exit of each
public method in the Java class file. Then we monitor the class field values accessed
by the execution of this.equals(this). These values are collected as object states. The
collected method arguments, returns, and object states are used in the specification
inference stage and the test generation stage.
In the specification inference stage, the collected traces are used to infer specifications. The axiomatic and protocol specification inference techniques in Section 2.2
are used in this stage. Instead of using the algebraic specification inference technique
based on observational equivalence [21], we develop a tool prototype to infer algebraic specifications based on an implementation-supplied equality method. Since it is
expensive to execute the equality method to compare object states among all method
executions, we use object states collected in the trace collection stage to compare the
object states offline. Based on a set of pre-defined axiom-pattern templates, the tool
looks for equality patterns among collected object states, method arguments, and
returns of methods. We infer algebraic specifications by using these equality patterns
as axioms.
In the test generation stage, inferred specifications are used to guide test generation.
Jtest [24] is used to automatically generate tests based on axiomatic specifications. In
protocol and algebraic specification-based test generation, we grow new object states
and method parameters based on the collected traces in the present iteration. In addition, we generate the method sequences based on inferred protocol and algebraic
specifications.
Because inferred preconditions in axiomatic specifications may be overconstrained,
only generating test inputs that satisfy them would leave some interesting legal test
inputs out of scope. One solution is to remove all the inferred preconditions before
the specifications are used to guide test generation. Then both legal and illegal test
inputs can be generated. Allowing some illegal inputs can still be useful in testing
program robustness. However, removing inferred preconditions makes test generation
based on preconditions unguided. In future work, we plan to investigate techniques to
remove or relax parts of inferred preconditions. There are similar overconstrained
problems with protocol specifications. To address these problems, we can deliberately
generate some method sequences that do not follow the transitions in the inferred
finite state machines. For example, we can generate test sequences to exercise the
complement of the inferred finite state machines. The test generation based on inferred algebraic specifications also needs some adaptations. If not all the combina-
64
tions of method pairs are exercised, we need to generate tests to exercise those uncovered method pairs besides those method pairs in the inferred axioms.
In the test selection stage, the generated tests are executed, and checked against the
inferred specifications. Two types of tests are selected for inspection. The first type of
test is the test whose execution causes an uncaught runtime exception or a program
crash. If the test is a legal input, it may expose a program fault. The second type of
test is the test whose execution violates ensures in axiomatic specifications or algebraic specifications. If the ensures violated by the test are overconstrained ones, this
may indicate the insufficiency of the existing tests. If the violated ensures are actual
ones and the test is a legal input, it may expose a program fault. These selected tests
are collected as the candidates of valuable tests. In the end of the feedback loop, the
developer can inspect these selected tests and their violated specifications for problems. If a selected test input is an illegal one, the developer can either add preconditions to guard against this test input in the subsequent iteration, or adopt defensive
programming to throw appropriate exceptions for this test input. If a selected test
input is a legal one and it exposes a program fault, the developer can fix the bug that
causes the fault, and augment the regression test suite with this test input after adding
an oracle for it. If a selected test input is a legal one and it does not expose a fault but
exercise certain new program behavior, the developer can add it to the regression test
suite together with its oracle. Besides selecting these two types of tests, the developer
can also select those tests that exercise at least one new structural entity, such as
statement or branch.
In our experiments with the feedback loop for axiomatic specifications, the number
of selected tests is not large, which makes the human inspection effort affordable [34].
In addition, the selected tests have a high probability of exposing faults or exercising
new program behavior. We observed the similar phenomena in our preliminary experiment with the feedback loop for algebraic specifications.
The selected tests, or all the newly generated tests from the present iteration are
used to augment the existing tests in the subsequent iteration. The inferred specifications from the present iteration are also used in the specification inference stage of the
subsequent iteration. In this specification inference stage, conditional specifications
might be inferred to refine some of those specifications that are violated by the generated tests in the present iteration.
4 Related Work
There have been several lines of work that use feedback loops in static analyses. Ball
and Rajamani construct a feedback loop between program abstraction and model
checking to validate user-specified temporal safety properties of interfaces [2].
Flanagan and Leino use a feedback loop between annotation guessing and theorem
proving to infer specifications statically [12]. Wild guesses of annotations are automatically generated based on heuristics before the first iteration. Human interventions
are needed to insert manual annotations in subsequent iterations. Giannakopoulou et
al. construct a feedback loop between assumption generation and model checking to
infer assumptions for a user-specified property in compositional verification [14, 7].
65
Given crude program abstractions or properties, these feedback loops in static analyses use model checkers or theorem provers to find counterexamples or refutations.
Then these counterexamples or refutations are used to refine the abstractions or properties iteratively. Our work is to construct a feedback loop in dynamic analyses, corresponding to the ones in static analyses. Our work does not require users to specify
properties, which are inferred from test executions instead.
Naumovich and Frankl propose to construct a feedback loop between finite state
verification and testing to dynamically confirm the statically detected faults [26].
When a finite state verifier detects a property violation, a testing tool uses the violation to guide test data selection, execution, and checking. The tool hopes to find test
data that shows the violation to be real. Based on the test information, human intervention is used to refine the model and restart the verifier. This is an example of a
feedback loop between static analysis and dynamic analysis. Another example of a
feedback loop between static analysis and dynamic analysis is profile-guided optimization [30]. Our work focuses on the feedback loop in dynamic analyses.
Peled et al. present the black box checking [29] and the adaptive model checking
approach [15]. Black box checking tests whether an implementation with unknown
structure or model satisfies certain given properties. Adaptive model checking performs model checking in the presence of an inaccurate model. In these approaches, a
feedback loop is constructed between model learning and model checking, which is
similar to the preceding feedback loops in static analyses. Model checking is performed on the learned model against some given properties. When a counterexample
is found for a given property, the counterexample is compared with the actual system.
If the counterexample is confirmed, a fault is reported. If the counterexample is refuted, it is fed to the model learning algorithm to improve the learned model. Another
feedback loop is constructed between model learning and conformance testing. If no
counterexample is found for the given property, conformance testing is conducted to
test whether the learned model and the system conform. If they do not conform, the
discrepancy-exposing test sequence is fed to the model learning algorithm, in order to
improve the approximate model. Then the improved model is used to perform model
checking in the subsequent iteration. The dynamic specification inference in our
feedback loop is corresponding to the model learning in their feedback loop, and the
specification-based test generation in our feedback loop is corresponding to the conformance testing in their feedback loop. Our feedback loop does not require some
given properties, but their feedback loop requires user-specified properties in order to
perform model checking.
Gupta et al. use a feedback loop between test data generation and branch predicate
constraint solving to generate test data for a given path [17]. An arbitrarily chosen
input from a given domain is executed to exercise the program statements relevant to
the evaluation of each branch predicate on the given path. Then a set of linear constraints is derived. These constraints can be solved to produce the increments for the
input. These increments are added to the current input in the subsequent iteration. The
specification inference in our work is corresponding to the branch predicate constraints in their approach. Our work does not require users to specify a property,
whereas the work of Gupta et al. requires users to specify the path to be covered.
66
5 Conclusion
We have proposed a feedback loop between specification-based test generation and
dynamic specification inference. This feedback loop can mutually enhance both test
generation and specification inference. The feedback loop provides aids in test generation by improving the underlying specifications, and aids in specification inference
by improving the underlying test suites. We have implemented a feedback loop for
axiomatic specifications, and demonstrated its usefulness [33, 34]. We have developed an initial implementation of feedback loop for algebraic specifications, and plan
to do more experiments and refine the implementation. In future work, we plan to
implement and experiment the feedback loop for protocol specifications. At the same
time, the following research questions are to be further investigated. In the first iteration, the inferred specifications can be used to generate a relatively large number of
new tests. In the subsequent iterations, the marginal improvements on tests and specifications come from the specification refinement and object state growth. We need to
explore effective ways to maximize these marginal improvements. We also plan to
investigate other SQA methods, such as static verification techniques, in evaluating
the quality of the inferred specifications in iterations [27].
6 Acknowledgment
We thank Michael Ernst, Andrew Petersen, and Darko Marinov for fruitful discussion.
This work was supported in part by the National Science Foundation under grant ITR
0086003. We acknowledge support through the High Dependability Computing Program from NASA Ames cooperative agreement NCC-2-1298.
References
1.
2.
3.
4.
5.
6.
Ammons, G., Bodik, R., Larus, J. R.: Mining specification. In Proceedings of Principles of
Programming Languages, Portland, Oregon, (2002) 4-16
Ball, T., Rajamani, S. K.: Automatically Validating Temporal Safety Properties of Interfaces. In Proceedings of SPIN Workshop on Model Checking of Software, (2001) 103-122
Bernot, G., Gaudel, M. C., Marre, B.: Software testing based on formal specifications: a
theory and a tool. Software Engineering Journal, Nov. (1991) 387-405
Bieman, J.M., Olender, K.M.: Using algebraic specifications to find sequencing defects. In
Proceedings of 4th International Symposium on Software Reliability Engineering, (1993)
226–232
Boyapati, C., Khurshid, S., Marinov, D.: Korat: Automated testing based on Java predicates. In Proceedings of the 2002 International Symposium on Software Testing and
Analysis, Rome, Italy, (2002) 123-133
Campbell, R. H., Habermann, A. N.: The Specification of Process Synchronization by
Path Expressions. vol. 16, Lecture Notes in Computer Science, (1974) 89-102
67
7.
8.
9.
10.
11.
12.
13.
14.
15.
16.
17.
18.
19.
20.
21.
22.
23.
24.
25.
26.
Cobleigh, J. M., Giannakopoulou, D., Pasareanu, C. S.: Learning Assumptions for Compositional Verification. In Proceedings of the 9th International Conference on Tools and
Algorithms for the Construction and Analysis of Systems, (2003) 331-346
Dick, J., Faivre, A.: Automating the Generation and Sequencing of Test Cases from
Model-Based Specifications. In Proceedings of International Symposium of Formal Methods Europe, (1993) 268-284
Doong, R., Frankl, P. G.: The ASTOOT approach to testing object-oriented programs.
ACM Transactions on Software Engineering, 3(2), Apr. (1994) 101-130
Ernst, M. D., Cockrell, J., Griswold, W. G., Notkin, D.: Dynamically discovering likely
program invariants to support program evolution. IEEE Transactions on Software Engineering, vol. 27, no. 2, Feb. (2001) 1-25
Ernst, M. D.: Static and dynamic analysis: Synergy and duality. In Proceedings of ICSE
Workshop on Dynamic Analysis. Portland, Oregon, May (2003) 24-27
Flanagan, C., Leino, K. R. M.: Houdini, an annotation assistant for ESC/Java. In Proceedings of International Symposium of Formal Methods Europe, March (2001) 500-517
Gannon, J., McMullin, P., Hamlet, R.: Data-Abstraction Implementation, Specification,
and Testing. ACM Transactions on Programming Languages and Systems, Vol.31, No. 3,
July (1981) 211-223
Giannakopoulou, D., Pasareanu, C. S., Barringer, H.: Assumption Generation for Software
Component Verification. In Proceedings of the 17th IEEE International Conference on
Automated Software Engineering, (2002) 3-12
Groce, A., Peled, D., Yannakakis, M.: Adaptive model checking. In Proceedings of the 8th
International Conference on Tools and Algorithms for the Construction and Analysis of
Systems, Apr. (2002) 357-370
Gupta, N.: Generating Test Data for Dynamically Discovering Likely Program Invariants.
In Proceedings of ICSE 2003 Workshop on Dynamic Analysis, May (2003) 21-24
Gupta, N., Mathur, A. P., Soffa, M. L.: Automated test data generation using an iterative
relaxation method. In Proceedings of International Symposium on Foundations of Software Engineering, November (1998) 231-244
Guttag, J. V., Horning, J. J.: The algebraic specification of abstract data types. Acta Information, 10, (1978) 27-52
Hagerer, A., Hungar, H., Niese, O., Steffen, B.: Model Generation by Moderated Regular
Extrapolation. In Proceedings of International Conference on Fundamental Approaches to
Software Engineering, Heidelberg, Germany, April (2002) 80-95
Harder, M., Mellen, J., Ernst, M. D.: Improving test suites via operational abstraction. In
Proceedings of the International Conference on Software Engineering, Portland, Oregon,
May (2003) 60-71
Henkel, J., Diwan, A.: Discovering algebraic specifications from Java classes. In Proceedings of European Conference on Object-Oriented Programming, July (2003) 431-456
Hoare, C. A. R.: An axiomatic basis for computer programming. Communications of the
ACM, 12(10), (1969) 576–580
Hughes, M., Stotts, D.: Daistish: Systematic algebraic testing for OO programs in the
presence of side-effects. In Proceedings of the International Symposium on Software Testing and Analysis, San Diego, California, (1996) 53-61
Parasoft Corporation, Jtest manuals version 4.5, http://www.parasoft.com/, October 23,
2002
Lee, D., Yannakakis, M.: Principles and methods of testing finite state machines - a survey. In Proceedings of the IEEE, vol. 84, 8, (1996) 1090-1123
Naumovich, G., Frankl, P. G.: Toward Synergy of Finite State Verification and Testing. In
Proceedings of the First International Workshop on Automated Program Analysis, Testing
and Verification, June (2000) 89-94
68
27. Nimmer, J. W., Ernst, M. D.: Static verification of dynamically detected program invariants: Integrating Daikon and ESC/Java. In Proceedings of First Workshop on Runtime
Verification, Paris, France, ENTCS 55 (2), (2001)
28. Osterweil, L. J., et al.: Strategic directions in software quality. ACM Computing Surveys,
(4), Dec. (1996) 738-750
29. Peled, D., Vardi, M. Y., Yannakakis, M.: Black Box Checking. In Proceedings of
FORTE/PSTV, Beijing, China, Kluwer, (1999) 225-240
30. Pettis, K., Hansen, R. C.: Profile guided code positioning. In Proceedings of the ACM
SIGPLAN Conference on Programming Language Design and Implementation, White
Plains, NY, USA, (1990) 16-27
31. Whaley, J., Martin, M. C., Lam, M. S.: Automatic Extraction of Object-Oriented Component Interfaces. In Proceedings of the International Symposium on Software Testing and
Analysis, July (2002) 218-228
32. Xie, T., Notkin, D.: Macro and Micro Perspectives on Strategic Software Quality Assurance in Resource Constrained Environments. In Proceedings of 4th International Workshop on Economics-Driven Software Engineering Research, May (2002) 66-70
33. Xie, T., Notkin, D.: Exploiting Synergy between Testing and Inferred Partial Specifications. In Proceedings of ICSE 2003 Workshop on Dynamic Analysis, Portland, Oregon,
(2003) 17-20
34. Xie, T., Notkin, D.: Tool-Assisted Unit Test Selection Based on Operational Violations. In
Proceedings of 18th IEEE International Conference on Automated Software Engineering,
Montreal, Canada, Oct. (2003) 40-48
35. Young, M.: Symbiosis of Static Analysis and Program Testing. In Proceedings of 6th
International Conference on Fundamental Approaches to Software Engineering, Warsaw,
Poland, April (2003) 1-5
69