Article

Incrementally discovering testable specifications from program executions

Authors:

Neil Walkinshaw,

John DerrickAuthors Info & Claims

FMCO'09: Proceedings of the 8th international conference on Formal methods for components and objects

Pages 272 - 289

Published: 04 November 2009 Publication History

Abstract

The ProTest project is an EU FP7 project to develop techniques that improve the testing and verification of concurrent and distributed software systems. One of the four main work packages is concerned with the automated identification of specifications that could serve as a suitable basis for testing; this is currently a tedious and error-prone manual task that tends to be neglected in practice. This paper describes how this problem has been addressed in the ProTest project. It describes a technique that uses test executions to refine the specification from which they are generated. It shows how the technique has been implemented and applied to real Erlang systems. It also describes in detail the major challenges that remain to be addressed in future work.

References

[1]

Derrick, J., Walkinshaw, N.: Property-based testing: The protest project. In: FMCO (2009).

Digital Library

[2]

Armstrong, J.: Programming Erlang: Software for a Concurrent World. Pragmatic Bookshelf (July 2007).

Digital Library

[3]

Walkinshaw, N., Derrick, J., Guo, Q.: Iterative refinement of reverse-engineered models by model-based testing. In: Cavalcanti, A., Dams, D.R. (eds.) FM 2009. LNCS, vol. 5850, pp. 305-320. Springer, Heidelberg (2009).

Digital Library

[4]

Walkinshaw, N., Bogdanov, K., Holcombe, M., Salahuddin, S.: Improving Dynamic Software Analysis by Applying Grammar Inference Principles. Journal of Software Maintenance and Evolution: Research and Practice (2008).

Digital Library

[5]

Claessen, K., Hughes, J.: Quickcheck: A Lightweight Tool for Random Testing of Haskell Programs. In: Proceedings of the International Conference on Functional Programming (ICFP), pp. 268-279 (2000).

Digital Library

[6]

Walkinshaw, N., Bogdanov, K., Ali, S., Holcombe, M.: Automated discovery of state transitions and their functions in source code. Software Testing, Verification and Reliability 18(2) (2008).

Digital Library

[7]

Moore, E.F.: Gedanken-experiments on sequential machines. In: Shannon, C.E., McCarthy, J. (eds.) Annals of Mathematics Studies (34), Automata Studies, pp. 129-153. Princeton University Press, Princeton (1956).

[8]

Nerode, A.: Linear automata transformations. Proceedings of the American Mathematical Society 9, 541-544 (1958).

[9]

Gold, E.: Language Identification in the Limit. Information and Control 10, 447- 474 (1967).

[10]

Biermann, A., Feldman, J.: On the Synthesis of Finite-State Machines from Samples of their Behavior. IEEE Transactions on Computers 21, 592-597 (1972).

Digital Library

[11]

Dupont, P., Lambeau, B., Damas, C., van Lamsweerde, A.: The QSM Algorithm and its Application to Software Behavior Model Induction. Applied Artificial Intelligence 22, 77-115 (2008).

Digital Library

[12]

Walkinshaw, N., Bogdanov, K., Holcombe, M., Salahuddin, S.: Reverse Engineering State Machines by Interactive Grammar Inference. In: 14th IEEE International Working Conference on Reverse Engineering, WCRE (2007).

Digital Library

[13]

Cook, J., Wolf, A.: Discovering Models of Software Processes from Event-Based Data. ACM Transactions on Software Engineering and Methodology 7(3), 215-249 (1998).

Digital Library

[14]

Ammons, G., Bodík, R., Larus, J.: Mining Specifications. In: 29th SIGPLAN-SIGACT Symposium on Principles of Programming Languages (POPL), Portland, Oregon, pp. 4-16 (2002).

Digital Library

[15]

Angluin, D.: Learning Regular Sets from Queries and Counterexamples. Information and Computation 75, 87-106 (1987).

Digital Library

[16]

Shahbaz, M., Groz, R.: Inferring mealy machines. In: Cavalcanti, A., Dams, D.R. (eds.) FM 2009. LNCS, vol. 5850, pp. 207-222. Springer, Heidelberg (2009).

Digital Library

[17]

Lang, K., Pearlmutter, B., Price, R.: Results of the Abbadingo One DFA Learning Competition and a New Evidence-Driven State Merging Algorithm. In: Honavar, V.G., Slutzki, G. (eds.) ICGI 1998. LNCS (LNAI), vol. 1433, pp. 1-12. Springer, Heidelberg (1998).

Digital Library

[18]

Biermann, A.W., Krishnaswamy, R.: Constructing programs from example computations. IEEE Trans. on Software Engineering SE 2, 141-153 (1976).

Digital Library

[19]

Cicchello, O., Kremer, S.: Inducing grammars from sparse data sets: A survey of algorithms and results. Journal of Machine Learning Research 4, 603-632 (2003).

Digital Library

[20]

Cheng, K., Krishnakumar, A.: Automatic functional test generation using the extended finite state machine model. In: 30th ACM/IEEE Design Automation Conference, pp. 86-91 (1993).

Digital Library

[21]

Walkinshaw, N., Bogdanov, K.: Inferring Finite-State Models with Temporal Constraints. In: Proceedings of the 23rd International Conference on Automated Software Engineering, ASE (2008).

Digital Library

[22]

Paris, J., Arts, T.: Automatic testing of tcp/ip implementations using quickcheck. In: Proceedings of the 8th ACM SIGPLAN workshop on Erlang, Erlang 2009, pp. 83-92. ACM, New York (2009).

Digital Library

[23]

Postel, J.: Transmission control protocol. Technical Report 793, DDN Network Information Center, SRI International, RFC (September 1981).

Digital Library

[24]

Ostrand, T., Balcer, M.: The category-partition method for specifying and generating functional tests. Communications of the ACM 31(6), 676-686 (1988).

Digital Library

[25]

Damas, C., Lambeau, B., Dupont, P., van Lamsweerde, A.: Generating Annotated Behavior Models from End-User Scenarios. IEEE Transactions on Software Engineering 31(12), 1056-1073 (2005).

Digital Library

[26]

Lorenzoli, D., Mariani, L., Pezzè, M.: Automatic generation of software behavioral models. In: Proceedings of the 30th international conference on Software engineering, ICSE 2008, pp. 501-510. ACM, New York (2008).

Digital Library

[27]

Ernst, M., Cockrell, J., Griswold, W., Notkin, D.: Dynamically Discovering Likely Program Invariants to Support Program Evolution. Transactions on Software Engineering 27(2), 1-25 (2001).

Digital Library

[28]

Bongard, J.C., Lipson, H.: Nonlinear system identification using coevolution of models and tests. IEEE Trans. Evolutionary Computation 9(4), 361-384 (2005).

Digital Library

[29]

Cornelissen, B., Zaidman, A., Holten, D., Moonen, L., van Deursen, A., van Wijk, J.: Execution trace analysis through massive sequence and circular bundle views. Journal of Systems and Software 81(12), 2252-2268 (2008).

Digital Library

[30]

Afshan, S., McMinn, P., Walkinshaw, N.: Using dictionary compression algorithms to identify phases in program traces. Technical Report CS-10-01, Department of Computer Science, The University of Sheffield (2010).

[31]

Walkinshaw, N., Afshan, S., McMinn, P.: Using compression algorithms to support the comprehension of program traces. In: Proceedings of the Eighth International Workshop on Dynamic Analysis (WODA 2010). ACM, New York (2010).

Digital Library

[32]

Nevill-Manning, C., Witten, I.: Compression and explanation using hierarchical grammars. Computer Journal 40(2/3), 103-116 (1997).

Cited By

Derrick JWalkinshaw NArts TEarle CCesarini FFredlund LGulias VHughes JThompson S(2009)Property-based testingProceedings of the 8th international conference on Formal methods for components and objects10.5555/1939101.1939123(250-271)Online publication date: 4-Nov-2009
https://dl.acm.org/doi/10.5555/1939101.1939123

Incrementally discovering testable specifications from program executions
1. Software and its engineering
  1. Software creation and management
    1. Software verification and validation
      1. Software defect analysis

Recommendations

Guiding dynamic symbolic execution toward unverified program executions
ICSE '16: Proceedings of the 38th International Conference on Software Engineering

Most techniques to detect program errors, such as testing, code reviews, and static program analysis, do not fully verify all possible executions of a program. They leave executions unverified when they do not check certain properties, fail to verify ...
Discovering bugs, or ensuring success?

The Communications Web site, http://cacm.acm.org, features more than a dozen bloggers in the BLOG@CACM community. In each issue of Communications, we'll publish selected posts or excerpts.
twitter
Follow us on Twitter at http://twitter.com/blogCACM
http://...
More testable properties

Testing remains a widely used validation technique for software systems. However, recent needs in software development (e.g., in terms of security concerns) may require to extend this technique to address a larger set of properties. In this article, we ...

Comments

Information & Contributors

Information

Published In

cover image Guide Proceedings

FMCO'09: Proceedings of the 8th international conference on Formal methods for components and objects

November 2009

338 pages

ISBN:3642170706

Editors:
Frank S. de Boer
Centre for Mathematics and Computer Science, Amsterdam, The Netherlands
,
Marcello M. Bonsangue
Leiden University, Leiden Institute of Advanced Computer Science, Leiden, The Netherlands
,
Stefan Hallerstede
Heinrich-Heine University of Dusseldorf, Department of Computer Science, Dusseldorf, Germany
,
Michael Leuschel
Heinrich-Heine University of Dusseldorf, Department of Computer Science, Dusseldorf, Germany

Sponsors

NWO: Dutch Orgartisation for Scientific Research

Publisher

Springer-Verlag

Berlin, Heidelberg

Publication History

Published: 04 November 2009

Qualifiers

Article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

1
Total Citations
View Citations
0
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 26 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Derrick JWalkinshaw NArts TEarle CCesarini FFredlund LGulias VHughes JThompson S(2009)Property-based testingProceedings of the 8th international conference on Formal methods for components and objects10.5555/1939101.1939123(250-271)Online publication date: 4-Nov-2009
https://dl.acm.org/doi/10.5555/1939101.1939123

View Options

View options

Figures

Tables

Media

View Table of Conten