research-article

Testing Relative to Usage Scope: Revisiting Software Coverage Criteria

Authors:

Antonia BertolinoAuthors Info & Claims

ACM Transactions on Software Engineering and Methodology (TOSEM), Volume 29, Issue 3

Article No.: 18, Pages 1 - 24

https://doi.org/10.1145/3389126

Published: 01 June 2020 Publication History

Abstract

Coverage criteria provide a useful and widely used means to guide software testing; however, indiscriminately pursuing full coverage may not always be convenient or meaningful, as not all entities are of interest in any usage context. We aim at introducing a more meaningful notion of coverage that takes into account how the software is going to be used. Entities that are not going to be exercised by the user should not contribute to the coverage ratio. We revisit the definition of coverage measures, introducing a notion of relative coverage. According to this notion, we provide a definition and a theoretical framework of relative coverage, within which we discuss implications on testing theory and practice. Through the evaluation of three different instances of relative coverage, we could observe that relative coverage measures provide a more effective strategy than traditional ones: we could reach higher coverage measures, and test cases selected by relative coverage could achieve higher reliability. We hint at several other useful implications of relative coverage notion on different aspects of software testing.

References

[1]

Andrea Arcuri and Gordon Fraser. 2011. On parameter tuning in search based software engineering. In Proceedings of the International Symposium on Search Based Software Engineering. 33--47.

[2]

Matthew Arnold and Barbara G. Ryder. 2001. A framework for reducing the cost of instrumented code. ACM SIGPLAN Notices 36, 5 (2001), 168--179.

Digital Library

[3]

Earl T. Barr, Mark Harman, Phil McMinn, Muzammil Shahbaz, and Shin Yoo. 2015. The oracle problem in software testing: A survey. IEEE Transactions on Software Engineering 41, 5 (2015), 507--525.

Digital Library

[4]

Cesare Bartolini, Antonia Bertolino, Sebastian Elbaum, and Eda Marchetti. 2011. Bringing white-box testing to service oriented architectures through a service oriented approach. Journal of Systems and Software 84, 4 (2011), 655--668.

Digital Library

[5]

Antonia Bertolino, Breno Miranda, Roberto Pietrantuono, and Stefano Russo. 2017. Adaptive coverage and operational profile-based testing for reliability improvement. In Proceedings of the 39th International Conference on Software Engineering (ICSE’17). IEEE, Los Alamitos, CA, 541--551.

Digital Library

[6]

A. Bertolino, B. Miranda, R. Pietrantuono, and S. Russo. 2019. Adaptive test case allocation, selection and generation using coverage spectrum and operational profile. IEEE Transactions on Software Engineering. Early access.

[7]

Antonia Bertolino and Lorenzo Strigini. 1996. On the use of testability measures for dependability assessment. IEEE Transactions on Software Engineering 22, 2 (1996), 97--108.

Digital Library

[8]

Lionel C. Briand, Yvan Labiche, and Michal M. Sówka. 2006. Automated, contract-based user testing of commercial-off-the-shelf components. In Proceedings of the 28th International Conference on Software Engineering (ICSE’06). ACM, New York, NY, 92--101.

[9]

Laura Brandán Briones, Ed Brinksma, and Mariëlle Stoelinga. 2006. A semantic framework for test coverage. In Proceedings of the International Symposium on Automated Technology for Verification and Analysis. 399--414.

Digital Library

[10]

Cristian Cadar and Koushik Sen. 2013. Symbolic execution for software testing: Three decades later. Communications of the ACM 56, 2 (Feb. 2013), 82--90.

Digital Library

[11]

Gerardo Canfora, Aniello Cimitile, and Andrea De Lucia. 1998. Conditioned program slicing. Information and Software Technology 40, 11 (1998), 595--607.

[12]

Boyuan Chen, Jian Song, Peng Xu, Xing Hu, and Zhen Ming Jack Jiang. 2018. An automated approach to estimating code coverage measures via execution logs. In Proceedings of the 33rd ACM/IEEE International Conference on Automated Software Engineering. ACM, New York, NY, 305--316.

Digital Library

[13]

Myra B. Cohen, Matthew B. Dwyer, and Jiangfan Shi. 2006. Coverage and adequacy in software product line testing. In Proceedings of the ISSTA 2006 Workshop on Role of Software Architecture for Testing and Analysis (ROSATEA’06). ACM, New York, NY, 53--63.

Digital Library

[14]

Domenico Cotroneo, Roberto Natella, and Stefano Russo. 2009. Assessment and improvement of hang detection in the Linux operating system. In Proceedings of the 28th IEEE International Symposium on Reliable Distributed Systems (SRDS’09). IEEE, Los Alamitos, CA, 288--294.

Digital Library

[15]

Andrea De Lucia. 2001. Program slicing: Methods and applications. In Proceedings of the 1st IEEE International Workshop on Source Code Analysis and Manipulation. IEEE, Los Alamitos, CA, 142--149.

[16]

Arilo C. Dias Neto, Rajesh Subramanyan, Marlon Vieira, and Guilherme H. Travassos. 2007. A survey on model-based testing approaches: A systematic review. In Proceedings of the 1st ACM International Workshop on Empirical Assessment of Software Engineering Languages, and Technologies: Held in Conjunction with the 22nd IEEE/ACM International Conference on Automated Software Engineering (ASE’07). ACM, New York, NY, 31--36.

[17]

Gencer Erdogan, Yan Li, Ragnhild Kobro Runde, Fredrik Seehusen, and Ketil Stølen. 2014. Approaches for the combined use of risk analysis and testing: A systematic literature review. International Journal on Software Tools for Technology Transfer 16, 5 (Oct. 2014), 627--642.

Digital Library

[18]

Michael Felderer and Rudolf Ramler. 2014. Integrating risk-based testing in industrial test processes. Software Quality Journal 22, 3 (Sept. 2014), 543--575.

[19]

Michael Felderer and Ina Schieferdecker. 2014. A taxonomy of risk-based testing. International Journal on Software Tools for Technology Transfer 16, 5 (Oct. 2014), 559--568.

[20]

P. G. Frankl, R. G. Hamlet, Bev Littlewood, and L. Strigini. 1998. Evaluating testing methods by delivered reliability. IEEE Transactions on Software Engineering 24, 8 (Aug. 1998), 586--601.

Digital Library

[21]

P. G. Frankl and E. J. Weyuker. 1988. An applicable family of data flow testing criteria. IEEE Transactions on Software Engineering 14, 10 (Oct. 1988), 1483--1498.

Digital Library

[22]

Phyllis G. Frankl and Elaine J. Weyuker. 1993. A formal analysis of the fault-detecting ability of testing methods. IEEE Transactions on Software Engineering 19, 3 (1993), 202--213.

Digital Library

[23]

Gregory Gay. 2018. To call, or not to call: Contrasting direct and indirect branch coverage in test generation. In Proceedings of the 11th International Workshop on Search-Based Software Testing. ACM, New York, NY, 43--50.

Digital Library

[24]

Markus Geimer, Sameer S. Shende, Allen D. Malony, and Felix Wolf. 2009. A generic and configurable source-code instrumentation component. In Proceedings of the International Conference on Computational Science. 696--705.

Digital Library

[25]

Patrice Godefroid, Nils Klarlund, and Koushik Sen. 2005. DART: Directed automated random testing. In Proceedings of the 2005 ACM SIGPLAN Conference on Programming Language Design and Implementation (PLDI’05). ACM, New York, NY, 213--223.

Digital Library

[26]

J. B. Goodenough and S. L. Gerhart. 1975. Toward a theory of test data selection. IEEE Transactions on Software Engineering SE-1, 2 (1975), 156--173.

Digital Library

[27]

J. B. Goodenough and S. L. Gerhart. 1977. Toward a theory of testing: Data selection criteria. Current Trends in Programming Methodology 2, 2 (1977), 44--79.

[28]

John S. Gourlay. 1983. A mathematical framework for the investigation of testing. IEEE Transactions on Software Engineering6 (1983), 686--709.

Digital Library

[29]

Robert J. Hall. 1995. Automatic extraction of executable program subsets by simultaneous dynamic program slicing. Automated Software Engineering 2, 1 (1995), 33--53.

[30]

Mary Jean Harrold, Gregg Rothermel, Rui Wu, and Liu Yi. 1998. An empirical investigation of program spectra. In Proceedings of the SIGPLAN/SIGSOFT Workshop on Program Analysis for Software Tools and Engineering (PASTE’98). 83--90.

Digital Library

[31]

W. E. Howden. 1976. Reliability of the path analysis testing strategy. IEEE Transactions on Software Engineering 2, 3 (May 1976), 208--215.

[32]

Gunel Jahangirova, David Clark, Mark Harman, and Paolo Tonella. 2016. Test oracle assessment and improvement. In Proceedings of the 25th International Symposium on Software Testing and Analysis (ISSTA’16). ACM, New York, NY, 247--258.

Digital Library

[33]

Baris Kasikci, Thomas Ball, George Candea, John Erickson, and Madanlal Musuvathi. 2014. Efficient tracing of cold code via bias-free sampling. In Proceedings of the USENIX Annual Technical Conference. 243--254.

[34]

Bogdan Korel and Janusz Laski. 1988. Dynamic program slicing. Information Processing Letters 29, 3 (1988), 155--163.

Digital Library

[35]

Rupak Majumdar and Koushik Sen. 2007. Hybrid concolic testing. In Proceedings of the 29th International Conference on Software Engineering (ICSE’07). IEEE, Los Alamitos, CA, 416--426.

Digital Library

[36]

Matthias Meitner and Francesca Saglietti. 2014. Target-specific adaptations of coupling-based software reliability testing. In Measurement, Modelling, and Evaluation of Computing Systems and Dependability and Fault Tolerance. Springer, 192--206.

[37]

Breno Miranda. 2014. A proposal for revisiting coverage testing metrics. In Proceedings of the ACM/IEEE International Conference on Automated Software Engineering (ASE’14). 899--902.

Digital Library

[38]

Breno Miranda. 2016. Redefining and Evaluating Coverage Criteria Based on the Testing Scope. Ph.D. Dissertation. University of Pisa, Italy.

[39]

Breno Miranda and Antonia Bertolino. 2014. Social coverage for customized test adequacy and selection criteria. In Proceedings of the 9th International Workshop on Automation of Software Test (AST’14). 22--28.

Digital Library

[40]

Breno Miranda and Antonia Bertolino. 2015. Improving test coverage measurement for reused software. In Proceedings of the 41st Euromicro Conference on Software Engineering and Advanced Applications (EUROMICRO-SEAA’15). 27--34.

Digital Library

[41]

Breno Miranda and Antonia Bertolino. 2016. Does code coverage provide a good stopping rule for operational profile based testing? In Proceedings of the 11th International Workshop on Automation of Software Test (AST’16). ACM, New York, NY, 22--28.

Digital Library

[42]

Breno Miranda and Antonia Bertolino. 2017. Scope-aided test prioritization, selection and minimization for software reuse. Journal of Systems and Software 131 (2017), 528--549.

Digital Library

[43]

Breno Miranda and Antonia Bertolino. 2018. An assessment of operational coverage as both an adequacy and a selection criterion for operational profile based testing. Software Quality Journal 26, 4 (Dec. 2018), 1571--1594.

Digital Library

[44]

John D. Musa. 1993. Operational profiles in software-reliability engineering. IEEE Software 10, 2 (1993), 14--32.

Digital Library

[45]

John D. Musa. 1996. Software-reliability-engineered testing. IEEE Computer 29, 11 (1996), 61--68.

Digital Library

[46]

Alessandro Orso, Donglin Liang, Mary Jean Harrold, and Richard Lipton. 2002. Gamma system: Continuous evolution of software after deployment. In Proceedings of the International Symposium on Software Testing and Analysis (ISSTA’02). ACM, New York, NY, 65--69.

Digital Library

[47]

Pablo Ponzio, Nazareno Aguirre, Marcelo F. Frias, and Willem Visser. 2016. Field-exhaustive testing. In Proceedings of the 2016 24th ACM SIGSOFT International Symposium on Foundations of Software Engineering. ACM, New York, NY, 908--919.

Digital Library

[48]

Rudolf Ramler, Theodorich Kopetzky, and Wolfgang Platz. 2012. Value-based coverage measurement in requirements-based testing: Lessons learned from an approach implemented in the TOSCA testsuite. In Proceedings of the 2012 38th Euromicro Conference on Software Engineering and Advanced Applications. IEEE, Los Alamitos, CA, 363--366.

Digital Library

[49]

T. Ravichandran and Marcus A. Rothenberger. 2003. Software reuse strategies and component markets. Communications of the ACM 46, 8 (Aug. 2003), 109--114.

Digital Library

[50]

Sanjai Rayadurgam and Mats Per Erik Heimdahl. 2001. Coverage based test-case generation using model checkers. In Proceedings of the 8th Annual IEEE International Conference and Workshop on the Engineering of Computer-Based Systems (ICBS’01). IEEE, Los Alamitos, CA, 83--91.

[51]

Björn Regnell, Per Runeson, and Claes Wohlin. 2000. Towards integration of use case modelling and usage-based testing. Journal of Systems and Software 50, 2 (2000), 117--130.

Digital Library

[52]

Thomas Reps. 1998. Program analysis via graph reachability. Information and Software Technology 40, 11–12 (1998), 701--726.

[53]

José Miguel Rojas, José Campos, Mattia Vivanti, Gordon Fraser, and Andrea Arcuri. 2015. Combining multiple coverage criteria in search-based unit test generation. In Proceedings of the International Symposium on Search Based Software Engineering. 93--108.

[54]

D. S. Rosenblum. 1997. Adequate Testing of Component-Based Software. Technical Report UCI-ICS-97-34. University of California, Irvine.

[55]

Gregg Rothermel and Mary Jean Harrold. 1997. A safe, efficient regression test selection technique. ACM Transactions on Software Engineering and Methodology 6, 2 (1997), 173--210.

Digital Library

[56]

Per Runeson and Claes Wohlin. 1995. Statistical usage testing for software reliability control. Informatica 19, 2 (1995), 195--207.

[57]

Koushik Sen, Darko Marinov, and Gul Agha. 2005. CUTE: A concolic unit testing engine for C. SIGSOFT Software Engineering Notes 30, 5 (Sept. 2005), 263--272.

Digital Library

[58]

Carol Smidts, Chetan Mutha, Manuel Rodríguez, and Matthew J. Gerber. 2014. Software testing with operational profile—OP definition. ACM Computing Surveys 46, 3 (Feb. 2014), Article 39, 39 pages.

Digital Library

[59]

Matt Staats, Gregory Gay, Michael Whalen, and Mats Heimdahl. 2012. On the danger of coverage directed test case generation. In Fundamental Approaches to Software Engineering. Lecture Notes in Computer Science, Vol. 7212. Springer, 409--424.

[60]

Matt Staats, Michael W. Whalen, and Mats P. E. Heimdahl. 2011. Programs, tests, and oracles: The foundations of testing revisited. In Proceedings of the 33rd International Conference on Software Engineering. ACM, New York, NY, 391--400.

[61]

Tomohiko Takagi, Kazuya Nishimachi, Masayuki Muragishi, Takashi Mitsuhashi, and Zengo Furukawa. 2009. Usage distribution coverage: What percentage of expected use has been executed in software testing? In Software Engineering, Artificial Intelligence, Networking and Parallel/Distributed Computing. Springer, 57--67.

[62]

Mustafa M. Tikir and Jeffrey K. Hollingsworth. 2002. Efficient instrumentation for code coverage testing. ACM SIGSOFT Software Engineering Notes 27, 86--96.

Digital Library

[63]

R. Torkar and S. Mankefors. 2003. A survey on testing and reuse. In Proceedings of the IEEE International Conference onSoftware Science, Technology, and Engineering (SwSTE’03). 164--173.

[64]

Guda A. Venkatesh. 1991. The semantic approach to program slicing. In Proceedings of the ACM SIGPLAN 1991 Conference on Programming Language Design and Implementation. 107--119.

Digital Library

[65]

J. Voas, L. Morell, and K. Miller. 1991. Predicting where faults can hide from testing. IEEE Software 8, 2 (March 1991), 41--48.

Digital Library

[66]

Jeffrey M. Voas. 1992. PIE: A dynamic failure-based technique. IEEE Transactions on Software Engineering 18, 8 (1992), 717--727.

Digital Library

[67]

Mark David Weiser. 1979. Program Slices: Formal, Psychological, and Practical Investigations of an Automatic Program Abstraction Method. Ph.D. Dissertation, University of Michigan.

[68]

E. J. Weyuker. 1998. Testing component-based software: A cautionary tale. IEEE Software 15, 5 (Sept. 1998), 54--59.

Digital Library

[69]

E. J. Weyuker and T. J. Ostrand. 1980. Theories of program testing and the application of revealing subdomains. IEEE Transactions on Software Engineering SE-6, 3 (1980), 236--246.

Digital Library

[70]

Yucheng Zhang and Ali Mesbah. 2015. Assertions are strongly correlated with test suite effectiveness. In Proceedings of the 2015 10th Joint Meeting on Foundations of Software Engineering (ESEC/FSE’15). 214--224.

Digital Library

[71]

Hong Zhu, Patrick A. V. Hall, and John H. R. May. 1997. Software unit test coverage and adequacy. ACM Computing Surveys 29, 4 (1997), 366--427.

Digital Library

Cited By

Kolchyn OPotiyenko S(2024)Increasing The Thoroughness Of Data Flow Testing With The Required k-Use ChainsProceedings of the 2024 10th International Conference on Computer Technology Applications10.1145/3674558.3674563(27-32)Online publication date: 15-May-2024
https://dl.acm.org/doi/10.1145/3674558.3674563
Walkinshaw NFoster MRojas JHierons R(2024)Bounding Random Test Set Size with Computational Learning TheoryProceedings of the ACM on Software Engineering10.1145/36608191:FSE(2538-2560)Online publication date: 12-Jul-2024
https://dl.acm.org/doi/10.1145/3660819
Dhulipala HYadavally ANguyen TLo DPenta MXia XHu X(2024)Planning to Guide LLM for Code Coverage PredictionProceedings of the 2024 IEEE/ACM First International Conference on AI Foundation Models and Software Engineering10.1145/3650105.3652292(24-34)Online publication date: 14-Apr-2024
https://dl.acm.org/doi/10.1145/3650105.3652292
Show More Cited By

Index Terms

Testing Relative to Usage Scope: Revisiting Software Coverage Criteria
1. General and reference
  1. Cross-computing tools and techniques
    1. Metrics
    2. Verification
2. Software and its engineering
  1. Software creation and management
    1. Software verification and validation
      1. Software defect analysis
        Software testing and debugging

Recommendations

An assessment of operational coverage as both an adequacy and a selection criterion for operational profile based testing

While the relation between code coverage measures and fault detection is actively studied, only few works have investigated the correlation between measures of coverage and of reliability. In this work, we introduce a novel approach to measuring code ...
A proposal for revisiting coverage testing metrics
ASE '14: Proceedings of the 29th ACM/IEEE International Conference on Automated Software Engineering

Test coverage information can be very useful for guiding testers in enhancing their test suites to exercise possible uncovered entities and in deciding when to stop testing. Since the concept of test criterion was born, several contributions have been ...
Does code coverage provide a good stopping rule for operational profile based testing?
AST '16: Proceedings of the 11th International Workshop on Automation of Software Test

We introduce a new coverage measure, called the operational coverage, which is customized to the usage profile (count spectrum) of the entities to be covered. Operational coverage is proposed as an adequacy criterion for operational profile based ...

Comments

Information & Contributors

Information

Published In

cover image ACM Transactions on Software Engineering and Methodology

ACM Transactions on Software Engineering and Methodology Volume 29, Issue 3

July 2020

292 pages

ISSN:1049-331X

EISSN:1557-7392

DOI:10.1145/3403667

Editor:
Mauro Pezzè
Università della Svizzera italiana and Università di Milano-Bicocca, Switzerland

Issue’s Table of Contents

Copyright © 2020 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 01 June 2020

Online AM: 07 May 2020

Accepted: 01 March 2020

Revised: 01 January 2020

Received: 01 December 2018

Published in TOSEM Volume 29, Issue 3

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

9
Total Citations
View Citations
502
Total Downloads

Downloads (Last 12 months)57
Downloads (Last 6 weeks)6

Reflects downloads up to 03 Oct 2024

Other Metrics

View Author Metrics

Citations

Cited By

Kolchyn OPotiyenko S(2024)Increasing The Thoroughness Of Data Flow Testing With The Required k-Use ChainsProceedings of the 2024 10th International Conference on Computer Technology Applications10.1145/3674558.3674563(27-32)Online publication date: 15-May-2024
https://dl.acm.org/doi/10.1145/3674558.3674563
Walkinshaw NFoster MRojas JHierons R(2024)Bounding Random Test Set Size with Computational Learning TheoryProceedings of the ACM on Software Engineering10.1145/36608191:FSE(2538-2560)Online publication date: 12-Jul-2024
https://dl.acm.org/doi/10.1145/3660819
Dhulipala HYadavally ANguyen TLo DPenta MXia XHu X(2024)Planning to Guide LLM for Code Coverage PredictionProceedings of the 2024 IEEE/ACM First International Conference on AI Foundation Models and Software Engineering10.1145/3650105.3652292(24-34)Online publication date: 14-Apr-2024
https://dl.acm.org/doi/10.1145/3650105.3652292
Bertolino ADe Angelis GDi Giandomenico FLonetti F(2023)Cross-coverage testing of functionally equivalent programs2023 IEEE/ACM International Conference on Automation of Software Test (AST)10.1109/AST58925.2023.00014(101-111)Online publication date: May-2023
https://doi.org/10.1109/AST58925.2023.00014
Hacks SPersson LHersén N(2022)Measuring and achieving test coverage of attack simulations extended versionSoftware and Systems Modeling10.1007/s10270-022-01042-922:1(31-46)Online publication date: 17-Sep-2022
https://doi.org/10.1007/s10270-022-01042-9
Kolchin APotiyenko S(2022)Extending Data Flow Coverage to Test Constraint RefinementsIntegrated Formal Methods10.1007/978-3-031-07727-2_17(313-321)Online publication date: 1-Jun-2022
https://doi.org/10.1007/978-3-031-07727-2_17
Al-Sharif ZAbdalrahman WJeffery C(2021)Encoding Test Cases using Execution Traces2021 12th International Conference on Information and Communication Systems (ICICS)10.1109/ICICS52457.2021.9464569(239-244)Online publication date: 24-May-2021
https://doi.org/10.1109/ICICS52457.2021.9464569
Amrita Gupta PSingh P(2021)Priority-wise Test Case Allocation using Fuzzy LogicInternational Journal of System Assurance Engineering and Management10.1007/s13198-021-01247-zOnline publication date: 24-Aug-2021
https://doi.org/10.1007/s13198-021-01247-z
Hersén NHacks SFögen K(2021)Towards Measuring Test Coverage of Attack SimulationsEnterprise, Business-Process and Information Systems Modeling10.1007/978-3-030-79186-5_20(303-317)Online publication date: 17-Jun-2021
https://doi.org/10.1007/978-3-030-79186-5_20
Bertolino ALonetti Fde Oliveira Neves V(2020)Standing on the Shoulders of Software Product Line Research for Testing Systems of Systems2020 IEEE International Symposium on Software Reliability Engineering Workshops (ISSREW)10.1109/ISSREW51248.2020.00074(209-214)Online publication date: Oct-2020
https://doi.org/10.1109/ISSREW51248.2020.00074

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

HTML Format

View this article in HTML Format.

Media

Figures

Other

Tables

View Issue’s Table of Contents