research-article

Open access

Information gain of black-box testing

Authors:

Thomas R. FischerAuthors Info & Claims

Formal Aspects of Computing, Volume 23, Issue 4

Pages 513 - 539

https://doi.org/10.1007/s00165-011-0175-6

Published: 01 July 2011 Publication History

Abstract

For model-based black-box testing, test cases are often selected from the syntactic appearance of the specification of the system under test, according to a pre-given test data adequacy criterion. We introduce a novel approach that is semantics-based, independent of the syntactic appearance of the system specification. Basically, we model the system under test as a random variable, whose sample space consists of all possible behavior sets (with respect to the specification) over the known interface of the black-box. The entropy of the system is measured as the (Shannon) entropy of the random variable. In our criterion, the coverage of a test set is measured as the expected amount of entropy decrease (i.e. the expected amount of information gained) once the test set is run. Since our criterion is syntactic independent, we study the notion of information-optimal software testing where, within a given constraint, a test set is selected to gain the most information.

References

References

[1]

Ammann P and Offutt J Introduction to Software Testing 2008 Cambridge Cambridge University Press

Digital Library

[2]

Ammann P, Offutt J, Huang H (2003) Coverage criteria for logical expressions. In: ISSRE ’03: Proceedings of the 14th International Symposium on Software Reliability Engineering. IEEE Computer Society, USA, pp 99–107

[3]

Andrés C, Llana L, and Rodríguez I Formally transforming user-model testing problems into implementer-model testing problems and viceversa J Log Algebraic Program 2009 78 6 425-453

[4]

Beizer B Black-box testing: techniques for functional testing of software and systems 1995 New York Wiley

Digital Library

[5]

Broy M, Jonsson B, Katoen J-P, Leucker M, Pretschner A (2005) Model-based testing of reactive systems: advanced lectures. In: Lecture notes in computer science, vol 3472. Springer, Berlin

[6]

Clarke EM, Grumberg O, and Peled DA Model checking 1999 Cambridge The MIT Press

Digital Library

[7]

Cover TM and Thomas JA Elements of information theory 2006 2 New York Wiley

Digital Library

[8]

Duran JW and Ntafos S An evaluation of random testing IEEE Trans Softw Eng 1984 10 4 438-444

Digital Library

[9]

Gaudel MC (1995) Testing can be formal, too. In: TAPSOFT ’95: Proceedings of the 6th international joint conference CAAP/FASE on theory and practice of software development, 1995. Lecture notes in computer science, vol 915. Springer, Berlin, pp 82–96

[10]

Goodenough JB, Gerhart SL (1975) Toward a theory of test data selection. In: Proceedings of the international conference on reliable software. ACM, New York, pp 493–510

[11]

Harel D Statecharts: a visual formalism for complex systems Sci Comput Program 1987 8 3 231-274

Digital Library

[12]

Harel D and Pnueli A Apt KR On the development of reactive systems Logics and models of concurrent systems, vol 13. NATO ASI Series 1985 New York Springer 477-498

[13]

Heitmeyer C Software cost reduction. Encyclopedia of software engineering 2002 2 New York Wiley

[14]

Holzmann GJ The model checker SPIN IEEE Trans Softw Eng 1997 23 5 279-295

Digital Library

[15]

Hopcroft JE, Motwani R, and Ullman JD Introduction to automata theory, languages, and computation 2007 3 Reading Addison-Wesley

Digital Library

[16]

Ladyman J (2007) Physics and computation: the status of landauer’s principle. In: CiE ’07: Proceedings of the 3rd conference on computability in Europe, 2007. Lecture notes in computer science, vol 4497. Springer, Berlin, pp 446–454

[17]

Landauer R Irreversibility and heat generation in the computing process IBM J Res Dev 1961 5 183-191

Digital Library

[18]

Last M, Friedman M, Kandel A (2003) The data mining approach to automated software testing. In: KDD ’03: Proceedings of the 9th ACM SIGKDD international conference on knowledge discovery and data mining. ACM, New York, pp 388–396

[19]

Lynch NA and Tuttle MR An introduction to input/output automata CWI-Q 1989 2 3 219-246

[20]

Myers GJ and Sandler C The art of software testing 2004 New York Wiley

Digital Library

[21]

Nachmanson L, Veanes M, Schulte W, Tillmann N, and Grieskamp W Optimal strategies for testing nondeterministic systems SIGSOFT Softw Eng Notes 2004 29 4 55-64

Digital Library

[22]

Peled D, Vardi MY, and Yannakakis M Black box checking J Autom Lang Comb 2001 7 2 225-246

Digital Library

[23]

Petrenko A, Yevtushenko N, Huo J (2003) Testing transition systems with input and output testers. In: TestCom ’03, 2003. Lecture notes in computer science, vol 2644. Springer, Berlin, pp 129–145

[24]

Pnueli A (1977) The temporal logic of programs. In: SFCS ’77: Proceedings of the 18th annual symposium on foundations of computer science. IEEE Computer Society, USA, pp 46–57

[25]

Shannon CE A mathematical theory of communication Bell Syst Tech J 1948 27 379-423, 623656

[26]

Tan L, Sokolsky O, Lee I (2004) Specification-based testing with linear temporal logic. In: IRI’04: proceedings of IEEE internation conference on information reuse and integration. IEEE Computer Society, USA, pp 483–498

[27]

Tretmans J (2008) Model based testing with labelled transition systems. In: Formal methods and testing, 2008. Lecture notes in computer science, vol 4949. Springer, Berlin, pp 1–38

[28]

Tretmans J, Brinksma E (2003) Torx: automated model-based testing. In: First European conference on model-driven software engineering. N/A, pp 31–43

[29]

van der Bijl M, Peureux F (2005) I/O-automata based testing. In: Model-based testing of reactive systems, Chapt. 7. Lecture notes in computer science, vol 3472, Springer, Berlin, pp 173–200

[30]

Weyuker EJ and Jeng B Analyzing partition testing strategies IEEE Trans Softw Eng 1991 17 7 703-711

Digital Library

[31]

Xie G (2005) Fundamental studies on automatic verification of component-based systems—a decompositional and hybrid approach. PhD thesis, Washington State University

[32]

Xie G, Dang Z (2006) Testing systems of concurrent black-boxes—an automata-theoretic and decompositional approach. In: FATES’05: Proceedings of the 5th international workshop on formal approaches to software testing, 2006. Lecture notes in computer science, vol 3997. Springer, Berlin, pp 170–186

[33]

Yang L, Dang Z, Fischer TR (2009) Optimal software testing—a cooling down process. In: FCS’09: Proceedings of the international conference on foundations of computer science. CSREA Press, pp 162–168

[34]

Zhu H and Hall PAV Test data adequacy measurement Softw Eng J 1992 8 1 21-29

Digital Library

[35]

Zhu H, Hall PAV, and May JHR Software unit test coverage and adequacy ACM Comput Surv 1997 29 4 366-427

Digital Library

Cited By

Cui CDang ZFischer TIbarra O(2017)Information rate of some classes of non-regular languagesInformation and Computation10.1016/j.ic.2017.04.008256:C(45-61)Online publication date: 1-Oct-2017
https://dl.acm.org/doi/10.1016/j.ic.2017.04.008
Cui CDang ZFischer TIbarra O(2016)Execution information rate for some classes of automataInformation and Computation10.1016/j.ic.2015.11.006246:C(20-29)Online publication date: 1-Feb-2016
https://dl.acm.org/doi/10.1016/j.ic.2015.11.006

Recommendations

Selection and Prioritization of Test Cases by Combining White-Box and Black-Box Testing Methods
ECBS-EERC '13: Proceedings of the 2013 3rd Eastern European Regional Conference on the Engineering of Computer Based Systems

In this paper, we present a methodology that combines both white-box and black-box testing, in order to improve testing quality for a given class of embedded systems. The goal of this methodology is generation of test cases for the new functional ...
Multi-objective black-box test case selection for system testing
GECCO '17: Proceedings of the Genetic and Evolutionary Computation Conference

Testing is a fundamental task to ensure software quality. Regression testing aims to ensure that changes to software do not introduce new failures. As resources are often limited and testing comprises a vast amount of test cases, different regression ...
RESTest: automated black-box testing of RESTful web APIs
ISSTA 2021: Proceedings of the 30th ACM SIGSOFT International Symposium on Software Testing and Analysis

Testing RESTful APIs thoroughly is critical due to their key role in software integration. Existing tools for the automated generation of test cases in this domain have shown great promise, but their applicability is limited as they mostly rely on ...

Comments

Information & Contributors

Information

Published In

cover image Formal Aspects of Computing

Formal Aspects of Computing Volume 23, Issue 4

Jul 2011

189 pages

ISSN:0934-5043

EISSN:1433-299X

Issue’s Table of Contents

© British Computer Society 2011.

Publisher

Springer-Verlag

Berlin, Heidelberg

Publication History

Published: 01 July 2011

Accepted: 19 February 2011

Revision received: 31 July 2010

Received: 09 February 2010

Published in FAC Volume 23, Issue 4

Author Tags

Qualifiers

Research-article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

2
Total Citations
View Citations
114
Total Downloads

Downloads (Last 12 months)41
Downloads (Last 6 weeks)9

Reflects downloads up to 26 Sep 2024

Other Metrics

View Author Metrics

Citations

Cited By

Cui CDang ZFischer TIbarra O(2017)Information rate of some classes of non-regular languagesInformation and Computation10.1016/j.ic.2017.04.008256:C(45-61)Online publication date: 1-Oct-2017
https://dl.acm.org/doi/10.1016/j.ic.2017.04.008
Cui CDang ZFischer TIbarra O(2016)Execution information rate for some classes of automataInformation and Computation10.1016/j.ic.2015.11.006246:C(20-29)Online publication date: 1-Feb-2016
https://dl.acm.org/doi/10.1016/j.ic.2015.11.006

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

Media

Figures

Other

Tables

View Issue’s Table of Contents