research-article

A Framework for Specification-Based Testing

Authors:

David CarringtonAuthors Info & Claims

IEEE Transactions on Software Engineering, Volume 22, Issue 11

Pages 777 - 793

https://doi.org/10.1109/32.553698

Published: 01 November 1996 Publication History

Abstract

Test templates and a test template framework are introduced as useful concepts in specification-based testing. The framework can be defined using any model-based specification notation and used to derive tests from model-based specifications in this paper, it is demonstrated using the Z notation. The framework formally defines test data sets and their relation to the operations in a specification and to other test data sets, providing structure to the testing process. Flexibility is preserved, so that many testing strategies can be used. Important application areas of the framework are discussed, including refinement of test data, regression testing, and test oracles.

References

[1]

O.-J. Dahl E.W. Dijkstra and C.A.R. Hoare, Structured Programming, APIC Studies in Data Processing, no. 8. Academic Press, 1972.

Digital Library

[2]

A.S. Tanenbaum, "In Defense of Program Testing or Correctness Proofs Considered Harmful," ACM SIGPLAN Notices, vol. 11, no. 5, pp. 64-68, May 1976.

Digital Library

[3]

D. Carrington and P. Stocks, "A Tale of Two Paradigms: Formal Methods and Software Testing," Proc. Eighth Z User Meeting, Springer-Verlag, pp. 51-68, 1994.

[4]

J.B. Goodenough and S.L. Gerhart, "Toward a Theory of Test Data Selection," IEEE Trans. Software Eng., vol. 1, no. 2, pp. 156-173, June 1975.

Digital Library

[5]

D.J. Richardson S.L. Aha and T.O. O'Malley, "Specification-Based Test Oracles for Reactive Systems," Proc. 14th Int'l Conf. Software Eng., pp. 105-118, May 1992.

Digital Library

[6]

B. Hetzel, The Complete Guide to Software Testing, QED Information Sciences, Wellesley, Mass., second edition, 1988.

Digital Library

[7]

C.B. Jones, Systematic Software Development Using VDM, Series in Computer Science. Prentice Hall Int'l, second edition, 1990.

Digital Library

[8]

Specification Case Studies, Series in Computer Science. I. Hayes, ed., Prentice Hall Int'l, second edition, 1993.

[9]

J.M. Spivey, The Z Notation: A Reference Manual, Series in Computer Science. Prentice Hall Int'l, second edition, 1992.

Digital Library

[10]

P.A.V. Hall, "Towards Testing with Respect to Formal Specifications," Second IEE/BCS Conf. Software Eng. 88. pp. 159-163, July 1988.

[11]

P.A.V. Hall, "Relationship Between Specifications and Testing," Information and Software Technology, vol. 33, no. 1, pp. 47-52, 1991.

Digital Library

[12]

J. Dick and A. Faivre, "Automating the Generating and Sequencing of Test Cases from Model-Based Specification," FME '93: Industrial Strength Formal Methods, Fifth Int'l Symp. Formal Methods Europe, J.C.P. Woodcock and P.G. Larsen, eds., Odense, Denmark, vol. 670 of Lecture Notes in Computer Science, pp. 268-284. Springer-Verlag, Apr. 1992.

Digital Library

[13]

H.-M. Hörcher, "Improving Software Tests Using Z Specifications," Bowen and Hinchey {41}, pp. 152-166, Proc. Ninth Int'l Conf. Z Users.

Digital Library

[14]

H.-M. Hörcher and J. Peleska, "Using Formal Specifications to Support Software Testing," Software Quality J., vol. 4, no. 4, pp. 309-327, Dec. 1995.

[15]

S. Stepney, "Testing as Abstraction," Bowen and Hinchey {41}, pp. 137-151, Proc. Ninth Int'l Conf. Z Users.

Digital Library

[16]

P. Stocks and D.A. Carrington, "Test Templates: A Specification-Based Testing Framework," Proc. 15th Int'l Conf. Software Eng., pp. 405-414, May 1993.

Digital Library

[17]

P. Stocks, "Applying Formal Methods to Software Testing," PhD thesis, the Univ. of Queensland, Australia, 1993.

[18]

T.J. Ostrand and M.J. Balcer, "The Category-Partition Method for Specifying and Generating Functional Tests," Comm. ACM, vol. 31, no. 6, pp. 676-686, June 1988.

Digital Library

[19]

M.J. Balcer W.M. Hasling and T.J. Ostrand, "Automatic Generation of Test Scripts from Formal Test Specifications," Software Eng. Notes, vol. 14, no. 8, pp. 210-218, Proc. ACM SIGSOFT '89 Third Symp. Software Testing, Analysis, and Verification, Dec. 1989.

Digital Library

[20]

J. Pachl, "A Notation for Specifying Test Selection Criteria," Protocol Specification, Testing, and Verification, X, L. Logrippo, R.L. Probert, and H. Ural, eds., pp. 71-84. North-Holland, 1990.

Digital Library

[21]

T.J. Ostrand R. Sigal and E.J. Weyuker, "Design for a Tool to Manage Specification-Based Testing," Workshop on Software Testing, Banff, Canada, pp. 41-50, IEEE CS Press, July 1986.

[22]

R.A. Kemmerer, "Testing Formal Specifications to Detect Design Errors," IEEE Trans. Software Eng., vol. 11, no. 1, pp. 32-43, Jan. 1985.

Digital Library

[23]

R. Kneuper, "Symbolic Execution as a Tool for Validation of Specifications," PhD dissertation, Technical Report UMCS-89-7-1, Dept. of Computer Science, Univ. of Manchester, 1989.

[24]

E.J. Weyuker and T.J. Ostrand, "Theories of Program Testing and the Application of Revealing Subdomains," IEEE Trans. Software Eng., vol. 6, no. 3, pp. 236-246, May 1980.

Digital Library

[25]

D.J. Richardson and L.A. Clarke, "Partition Analysis: A Method Combining Testing and Verification," IEEE Trans. Software Eng., vol. 11, no. 12, pp. 1,477-1,490, Dec. 1985.

[26]

J.W. Duran and S.C. Ntafos, "An Evaluation of Random Testing," IEEE Trans. Software Eng., vol. 10, no. 4, pp. 438-444, July 1984.

Digital Library

[27]

P. Stocks and D.A. Carrington, "Test Template Framework: A Specification-Based Testing Case Study," Proc. Int'l Symp. Software Testing and Analysis, ISSTA '93, pp. 11-18, June 1993.

Digital Library

[28]

I. MacColl D. Carrington and P. Stocks, "An Experiment in Specification-Based Testing," Proc. 19th Australasian Computer Science Conf., ACSC'96, K. Ramamohanarao, ed., vol. 18, pp. 159-168, Feb. 1996.

[29]

The RAISE Language Group, The RAISE Specification Language, 1992.

Digital Library

[30]

E.J. Weyuker and B. Jeng, "Analyzing Partition Testing Strategies," IEEE Trans. Software Eng., vol. 17, no. 7, pp. 703-711, July 1991.

Digital Library

[31]

A. Parrish and S.H. Zweben, "Analysis and Refinement of Software Test Data Adequacy Properties," IEEE Trans. Software Eng., vol. 17, no. 6, pp. 565-581, June 1991.

Digital Library

[32]

E. Weyuker T. Goradia and A. Singh, "Automatically Generating Test Data from a Boolean Specification," IEEE Trans. Software Eng., vol. 20, no. 5, pp. 353-363, May 1994.

Digital Library

[33]

P. Stocks and D.A. Carrington, "Deriving Software Test Cases from Formal Specifications," Sixth Australian Software Eng. Conf., pp. 327-340, July 1991.

[34]

R. Hamlet, "Theoretical Comparison of Testing Methods," Software Eng. Notes, vol. 14, no. 8, pp. 28-37, Proc. ACM SIGSOFT '89 Third Symp. Software Testing, Analysis, and Verification, Dec. 1989.

Digital Library

[35]

R. Hamlet and J. Voas, "Faults on Its Sleeve: Amplifying Software Reliability Testing," Proc. Int'l Symp. Software Testing and Analysis, ISSTA '93, pp. 89-98, June 1993.

Digital Library

[36]

I.J. Hayes, "Specification Directed Module Testing," IEEE Trans. Software Eng., vol. 12, no. 1, pp. 124-133, Jan. 1986.

Digital Library

[37]

W.E. Howden, "A Functional Approach to Program Testing and Analysis," IEEE Trans. Software Eng., vol. 12, no. 10, pp. 997-1,005, Oct. 1986.

Digital Library

[38]

W.E. Howden, Functional Program Testing and Analysis. McGraw-Hill, 1986.

Digital Library

[39]

D. Carrington D. Duke I. Hayes and J. Welsh, "Deriving Modular Designs from Formal Specifications," Software Eng. Notes, vol. 18, no. 5, pp. 89-98, Proc. First ACM SIGSOFT Symp. Foundations of Software Eng., Dec. 1993.

Digital Library

[40]

J. Gannon P. McMullin and R. Hamlet, "Data-Abstraction Implementation, Specification, and Testing," ACM Trans. Programming Languages and Systems, vol. 3, no. 2, pp. 211-223, July 1981.

Digital Library

[41]

"ZUM '95: The Z Formal Specification Notation," J.P. Bowen and M.G. Hinchey, eds., vol. 967, Lecture Notes in Computer Science, Limerick, Ireland, Proc. Ninth Int'l Conf. of Z Users, Springer-Verlag, June 1995.

Cited By

Hossain SDwyer MElbaum SNguyen-Tuong AGrundy JPollock LPenta M(2023)Measuring and Mitigating Gaps in Structural TestingProceedings of the 45th International Conference on Software Engineering10.1109/ICSE48619.2023.00147(1712-1723)Online publication date: 14-May-2023
https://dl.acm.org/doi/10.1109/ICSE48619.2023.00147
Cristiá MRossi G(2021)An Automatically Verified Prototype of the Tokeneer ID Station SpecificationJournal of Automated Reasoning10.1007/s10817-021-09602-265:8(1125-1151)Online publication date: 1-Dec-2021
https://dl.acm.org/doi/10.1007/s10817-021-09602-2
Lam PDietrich JPearce DKell SVerna D(2020)Putting the semantics into semantic versioningProceedings of the 2020 ACM SIGPLAN International Symposium on New Ideas, New Paradigms, and Reflections on Programming and Software10.1145/3426428.3426922(157-179)Online publication date: 18-Nov-2020
https://dl.acm.org/doi/10.1145/3426428.3426922
Show More Cited By

Index Terms

A Framework for Specification-Based Testing
1. Software and its engineering
  1. Software creation and management
    1. Designing software
      1. Requirements analysis
    2. Software verification and validation
      1. Software defect analysis
        Software testing and debugging

Recommendations

Fault-based test suite prioritization for specification-based testing

Context: Existing test suite prioritization techniques usually rely on code coverage information or historical execution data that serve as indicators for estimating the fault-detecting ability of test cases. Such indicators are primarily empirical in ...
Evaluation of Three Specification-Based Testing Criteria
ICECCS '00: Proceedings of the 6th IEEE International Conference on Complex Computer Systems

Abstract: This paper compares three specification-based testing criteria using Mathur and Wong's PROBSUBSUMES measure. The three criteria are specification-mutation coverage, full predicate coverage, and transition-pair coverage. A novel aspect of the ...
Automatic Specification-Based Testing: Challenges and Possibilities
TASE '11: Proceedings of the 2011 Fifth International Conference on Theoretical Aspects of Software Engineering

Automatic specification-based testing has long been a goal of software engineering, but due to various challenges and difficulties, the goal still remains far from realized. If the specification is informal, full automation would be impossible in ...

Comments

Information & Contributors

Information

Published In

cover image IEEE Transactions on Software Engineering

IEEE Transactions on Software Engineering Volume 22, Issue 11

November 1996

62 pages

ISSN:0098-5589

Editor:
Richard A. Kemmerer
Univ. of California, Santa Barbara, CA

Issue’s Table of Contents

Copyright © Copyright © 1996 IEEE. All Rights Reserved.

Publisher

IEEE Press

Publication History

Published: 01 November 1996

Author Tags

Qualifiers

Research-article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

69
Total Citations
View Citations
0
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 09 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Hossain SDwyer MElbaum SNguyen-Tuong AGrundy JPollock LPenta M(2023)Measuring and Mitigating Gaps in Structural TestingProceedings of the 45th International Conference on Software Engineering10.1109/ICSE48619.2023.00147(1712-1723)Online publication date: 14-May-2023
https://dl.acm.org/doi/10.1109/ICSE48619.2023.00147
Cristiá MRossi G(2021)An Automatically Verified Prototype of the Tokeneer ID Station SpecificationJournal of Automated Reasoning10.1007/s10817-021-09602-265:8(1125-1151)Online publication date: 1-Dec-2021
https://dl.acm.org/doi/10.1007/s10817-021-09602-2
Lam PDietrich JPearce DKell SVerna D(2020)Putting the semantics into semantic versioningProceedings of the 2020 ACM SIGPLAN International Symposium on New Ideas, New Paradigms, and Reflections on Programming and Software10.1145/3426428.3426922(157-179)Online publication date: 18-Nov-2020
https://dl.acm.org/doi/10.1145/3426428.3426922
(2016)An incremental method for extracting tests from object-oriented specificationInformation and Software Technology10.1016/j.infsof.2016.05.00578:C(1-26)Online publication date: 1-Oct-2016
https://dl.acm.org/doi/10.1016/j.infsof.2016.05.005
Sabor KThiel SHao DSubramanyan RMariani L(2015)Adaptive random testing by static partitioningProceedings of the 10th International Workshop on Automation of Software Test10.5555/2819261.2819271(28-32)Online publication date: 16-May-2015
https://dl.acm.org/doi/10.5555/2819261.2819271
Tonella PTiella RNguyen CJalote PBriand LHoek A(2014)Interpolated n-grams for model based testingProceedings of the 36th International Conference on Software Engineering10.1145/2568225.2568242(562-572)Online publication date: 31-May-2014
https://dl.acm.org/doi/10.1145/2568225.2568242
Cristiá MMesuro JFrydman C(2014)Integration Testing in the Test Template FrameworkProceedings of the 17th International Conference on Fundamental Approaches to Software Engineering - Volume 841110.1007/978-3-642-54804-8_28(400-414)Online publication date: 5-Apr-2014
https://dl.acm.org/doi/10.1007/978-3-642-54804-8_28
Bowen JReeves S(2013)UI-design driven model-based testingInnovations in Systems and Software Engineering10.1007/s11334-013-0199-69:3(201-215)Online publication date: 1-Sep-2013
https://dl.acm.org/doi/10.1007/s11334-013-0199-6
Cristiá MRossi GFrydman C(2013){log} as a Test Case Generator for the Test Template FrameworkProceedings of the 11th International Conference on Software Engineering and Formal Methods - Volume 813710.1007/978-3-642-40561-7_16(229-243)Online publication date: 25-Sep-2013
https://dl.acm.org/doi/10.1007/978-3-642-40561-7_16
Tonella PTiella RNguyen C(2013)N-Gram Based Test Sequence Generation from Finite State ModelsFuture Internet Testing10.1007/978-3-319-07785-7_4(59-74)Online publication date: 12-Nov-2013
https://dl.acm.org/doi/10.1007/978-3-319-07785-7_4
Show More Cited By

View Options

View options

Media

Figures

Other

Tables

View Issue’s Table of Contents