research-article

Accounting for defect characteristics in evaluations of testing techniques

Authors:

Jaymie Strecker,

Atif M. MemonAuthors Info & Claims

ACM Transactions on Software Engineering and Methodology (TOSEM), Volume 21, Issue 3

Article No.: 17, Pages 1 - 43

https://doi.org/10.1145/2211616.2211620

Published: 03 July 2012 Publication History

Abstract

As new software-testing techniques are developed, before they can achieve widespread acceptance, their effectiveness at detecting defects must be evaluated. The most common way of evaluating testing techniques is with empirical studies, in which one or more techniques are tried out on software with known defects. However, the defects used can affect the performance of the techniques. To complicate matters, it is not even clear how to effectively describe or characterize defects. To address these problems, this article describes an experiment architecture for empirically evaluating testing techniques which takes both defect and test-suite characteristics into account. As proof of concept, an experiment on GUI-testing techniques is conducted. It provides evidence that the defect characteristics proposed do help explain defect detection, at least for GUI testing, and it explores the relationship between the coverage of defective code and the detection of defects.

References

[1]

Agresti, A. 2007. An Introduction to Categorical Data Analysis, 2nd Ed. John Wiley & Sons, New York, NY.

[2]

Ammann, P. and Offutt, J. 2008. Introduction to Software Testing. Cambridge University Press, New York, NY.

Digital Library

[3]

Andrews, J. H., Briand, L. C., Labiche, Y., and Namin, A. S. 2006. Using mutation analysis for assessing and comparing testing coverage criteria. IEEE Trans. Softw. Eng. 32, 8, 608--624.

Digital Library

[4]

Basili, V. R. and Selby, R. W. 1987. Comparing the effectiveness of software testing strategies. IEEE Trans. Softw. Eng. 13, 12, 1278--1296.

Digital Library

[5]

Basili, V. R., Shull, F., and Lanubile, F. 1999. Building knowledge through families of experiments. IEEE Trans. Softw. Eng. 25, 4, 456--473.

Digital Library

[6]

Briand, L. C., Melo, W. L., and Wüst, J. 2002. Assessing the applicability of fault-proneness models across object-oriented software projects. IEEE Trans. Softw. Eng. 28, 7, 706--720.

Digital Library

[7]

Do, H. and Rothermel, G. 2006. On the use of mutation faults in empirical assessments of test case prioritization techniques. IEEE Trans. Softw. Eng. 32, 9, 733--752.

Digital Library

[8]

Elbaum, S., Gable, D., and Rothermel, G. 2001. Understanding and measuring the sources of variation in the prioritization of regression test suites. In Proceedings of the 7th International Symposium on Software Metrics (METRICS'01). IEEE Computer Society, Los Alamitos, CA, 169--179.

Digital Library

[9]

Fenton, N. E. and Neil, M. 1999. A critique of software defect prediction models. IEEE Trans. Softw. Eng. 25, 5, 675--689.

Digital Library

[10]

Frankl, P. G., Hamlet, R. G., Littlewood, B., and Strigini, L. 1998. Evaluating testing methods by delivered reliability. IEEE Trans. Softw. Eng. 24, 8, 586--601.

Digital Library

[11]

Frankl, P. G. and Weiss, S. N. 1991. An experimental comparison of the effectiveness of the all-uses and all-edges adequacy criteria. In Proceedings of the Symposium on Testing Analysis and Verification (TAV4). ACM, New York, NY, USA, 154--164.

Digital Library

[12]

Frankl, P. G. and Weyuker, E. J. 1993. A formal analysis of the fault-detecting ability of testing methods. IEEE Trans. Softw. Eng. 19, 3, 202--213.

Digital Library

[13]

Garson, G. D. 2006. Statnotes: Topics in multivariate analysis. http://www2.chass.ncsu.edu/garson/PA765/statnote.htm.

[14]

Graves, T. L., Harrold, M. J., Kim, J.-M., Porter, A., and Rothermel, G. 2001. An empirical study of regression test selection techniques. ACM Trans. Softw. Eng. Methodol. 10, 2, 184--208.

Digital Library

[15]

Harrold, M. J., Offutt, A. J., and Tewary, K. 1997. An approach to fault modeling and fault seeding using the program dependence graph. J. Syst. Softw. 36, 3, 273--295.

Digital Library

[16]

Hutchins, M., Foster, H., Goradia, T., and Ostrand, T. 1994. Experiments on the effectiveness of dataflow- and controlflow-based test adequacy criteria. In Proceedings of the International Conference on Software Engineering (ICSE'94). IEEE Computer Society, Los Alamitos, CA, USA, 191--200.

Digital Library

[17]

Juristo, N., Moreno, A. M., and Vegas, S. 2004. Reviewing 25 years of testing technique experiments. Empirical Softw. Eng. 9, 1--2, 7--44.

Digital Library

[18]

Lenth, R. V. 2000. Two sample-size practices that i don't recommend. In Proceedings of the Section on Physical and Engineering Sciences. American Statistical Association.

[19]

McMaster, S. and Memon, A. 2008. Call-stack coverage for GUI test suite reduction. IEEE Trans. Softw. Eng. 34, 1, 99--115.

Digital Library

[20]

Memon, A. M. 2007. An event-flow model of GUI-based applications for testing. Softw. Test. Verification Reliability 17, 3, 137--157.

Digital Library

[21]

Memon, A. M. 2009. Using reverse engineering for automated usability evaluation of gui-based applications. In Software Engineering Models, Patterns and Architectures for HCI. Springer-Verlag London Ltd, London.

[22]

Morgan, J. A., Knafl, G. J., and Wong, W. E. 1997. Predicting fault detection effectiveness. In Proceedings of International Symposium on Software Metrics (METRICS'97). IEEE Computer Society, Los Alamitos, CA, 82--89.

Digital Library

[23]

Myers, G. J. 1978. A controlled experiment in program testing and code walkthroughs/inspections. Commun. ACM 21, 9, 760--768.

Digital Library

[24]

Offutt, A. J. and Hayes, J. H. 1996. A semantic model of program faults. In Proceedings of the International Symposium on Software Testing and Analysis (ISSTA'96). ACM Press, New York, NY, 195--200.

Digital Library

[25]

Offutt, A. J., Lee, A., Rothermel, G., Untch, R. H., and Zapf, C. 1996. An experimental determination of sufficient mutant operators. ACM Trans. Softw. Eng. Methodol. 5, 2, 99--118.

Digital Library

[26]

Richardson, D. J. and Thompson, M. C. 1993. An analysis of test data selection criteria using the RELAY model of fault detection. IEEE Trans. Softw. Eng. 19, 6, 533--553.

Digital Library

[27]

Ripley, B. D. 2001. The R project in statistical computing. MSOR Connections: Newsletter of the LTSN Maths, Stats and OR Network 1, 1, 23--25.

[28]

Rosner, B. 2000. Fundamentals of Biostatistics, 5th Ed. Duxbury, Pacific Grove, CA.

[29]

Rothermel, G., Elbaum, S., Malishevsky, A. G., Kallakuri, P., and Qiu, X. 2004. On test suite composition and cost-effective regression testing. ACM Trans. Softw. Eng. Methodol. 13, 3, 277--331.

Digital Library

[30]

Slud, E. V. 2008. Personal communication with Prof. Eric V. Slud, Professor, Statistics Program, Department of Mathematics, University of Maryland College Park, MD.

[31]

Staiger, S. 2007. Static analysis of programs with graphical user interface. In Proceedings of the European Conference on Software Reengineering (CSMR'07). IEEE Computer Society, Los Alamitos, CA, 252--264.

Digital Library

[32]

Strecker, J. 2009. Accounting for defect characteristics in empirical studies of software testing. Ph.D. dissertation, University of Maryland, College Park, MD.

Digital Library

[33]

Strecker, J. and Memon, A. M. 2007. Faults' context matters. In Proceedings of the International Workshop on Software Quality Assurance (SOQUA'07). ACM Press, New York, NY, 112--115.

Digital Library

[34]

Strecker, J. and Memon, A. M. 2008. Relationships between test suites, faults, and fault detection in GUI testing. In Proceedings of the International Conference on Software Testing, Verification, and Validation (ICST'08). IEEE Computer Society, Los Alamitos, CA, 12--21.

Digital Library

[35]

Strecker, J. and Memon, A. M. 2009. Testing graphical user interfaces. In Encyclopedia of Information Science and Technology, 2nd Ed. IGI Global, Hershey, PA.

[36]

Testbeds 2009. In Proceedings of the 1st International Workshop on Testing Techniques and Experimentation Benchmarks for Event-Driven Software.

[37]

Voas, J. M. 1992. PIE: A dynamic failure-based technique. IEEE Trans. Softw. Eng. 18, 8, 717--727.

Digital Library

[38]

Xie, Q. and Memon, A. 2006. Studying the characteristics of a “good” GUI test suite. In Proceedings of the IEEE International Symposium on Software Reliability Engineering (ISSRE'06). IEEE Computer Society, Los Alamitos, CA, 159--168.

Digital Library

[39]

Xie, Q. and Memon, A. M. 2007. Designing and comparing automated test oracles for gui-based software applications. ACM Trans. Softw. Eng. Methodol. 16, 1, 4.

Digital Library

[40]

Yuan, X., Cohen, M. B., and Memon, A. M. 2011. Gui interaction testing: Incorporating event context. IEEE Trans. Softw. Eng. 37, 4, 559--574.

Digital Library

[41]

Yuan, X. and Memon, A. M. 2007. Using GUI run-time state as feedback to generate test cases. In Proceedings of the International Conference on Software Engineering (ICSE'07). IEEE Computer Society, Los Alamitos, CA, 396--405.

Digital Library

[42]

Zelkowitz, M. V. and Wallace, D. 1997. Experimental validation in software engineering. Inform. Softw. Technol. 39, 11, 735--743.

Digital Library

Cited By

Yang YXia XLo DBi TGrundy JYang X(2022)Predictive Models in Software Engineering: Challenges and OpportunitiesACM Transactions on Software Engineering and Methodology10.1145/350350931:3(1-72)Online publication date: 9-Apr-2022
https://dl.acm.org/doi/10.1145/3503509
Coppola RAlégroth E(2022)A taxonomy of metrics for GUI-based testing research: A systematic literature reviewInformation and Software Technology10.1016/j.infsof.2022.107062152(107062)Online publication date: Dec-2022
https://doi.org/10.1016/j.infsof.2022.107062
Pecorelli FPalomba FDe Lucia A(2021)The Relation of Test-Related Factors to Software Quality: A Case Study on Apache SystemsEmpirical Software Engineering10.1007/s10664-020-09891-y26:2Online publication date: 1-Mar-2021
https://dl.acm.org/doi/10.1007/s10664-020-09891-y
Show More Cited By

Index Terms

Accounting for defect characteristics in evaluations of testing techniques
1. Software and its engineering
  1. Software creation and management
    1. Software development techniques
      1. Reusability
        Software product lines
    2. Software verification and validation
      1. Software defect analysis
        Software testing and debugging

Recommendations

Accounting for defect characteristics in empirical studies of software testing
Comparing model-based and dynamic event-extraction based GUI testing techniques

HighlightsWe compare model-based and dynamic event-extraction based GUI testing techniques.We present a exible GUI testing framework that enables comparison of techniques.We show that nonexecutable events weaken effectiveness and increase costs of ...
Defect classification and defect types revisited
DEFECTS '08: Proceedings of the 2008 workshop on Defects in large software systems

There have been various attempts to establish common defect classification systems in standards, academia and industry. The resulted classifications are partly similar but also contain variations and none of which is accepted as a basic tool in software ...

Comments

Information & Contributors

Information

Published In

cover image ACM Transactions on Software Engineering and Methodology

ACM Transactions on Software Engineering and Methodology Volume 21, Issue 3

June 2012

239 pages

ISSN:1049-331X

EISSN:1557-7392

DOI:10.1145/2211616

Issue’s Table of Contents

Copyright © 2012 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 03 July 2012

Accepted: 01 January 2011

Revised: 01 July 2010

Received: 01 July 2009

Published in TOSEM Volume 21, Issue 3

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed

Funding Sources

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

14
Total Citations
View Citations
747
Total Downloads

Downloads (Last 12 months)5
Downloads (Last 6 weeks)2

Reflects downloads up to 09 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Yang YXia XLo DBi TGrundy JYang X(2022)Predictive Models in Software Engineering: Challenges and OpportunitiesACM Transactions on Software Engineering and Methodology10.1145/350350931:3(1-72)Online publication date: 9-Apr-2022
https://dl.acm.org/doi/10.1145/3503509
Coppola RAlégroth E(2022)A taxonomy of metrics for GUI-based testing research: A systematic literature reviewInformation and Software Technology10.1016/j.infsof.2022.107062152(107062)Online publication date: Dec-2022
https://doi.org/10.1016/j.infsof.2022.107062
Pecorelli FPalomba FDe Lucia A(2021)The Relation of Test-Related Factors to Software Quality: A Case Study on Apache SystemsEmpirical Software Engineering10.1007/s10664-020-09891-y26:2Online publication date: 1-Mar-2021
https://dl.acm.org/doi/10.1007/s10664-020-09891-y
Pecorelli FDumas MPfahl DApel SRusso A(2019)Test-related factors and post-release defects: an empirical studyProceedings of the 2019 27th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering10.1145/3338906.3342500(1235-1237)Online publication date: 12-Aug-2019
https://dl.acm.org/doi/10.1145/3338906.3342500
Aho PSuarez MMemon AKanstrén T(2015)Making GUI Testing PracticalProceedings of the 2015 12th International Conference on Information Technology - New Generations10.1109/ITNG.2015.77(439-444)Online publication date: 13-Apr-2015
https://dl.acm.org/doi/10.1109/ITNG.2015.77
Yang WChen ZGao ZZou YXu X(2014)GUI testing assisted by human knowledgeJournal of Systems and Software10.5555/2747476.274752889:C(76-86)Online publication date: 1-Mar-2014
https://dl.acm.org/doi/10.5555/2747476.2747528
Herzig K(2014)Using Pre-Release Test Failures to Build Early Post-Release Defect Prediction ModelsProceedings of the 2014 IEEE 25th International Symposium on Software Reliability Engineering10.1109/ISSRE.2014.21(300-311)Online publication date: 3-Nov-2014
https://dl.acm.org/doi/10.1109/ISSRE.2014.21
Aho PSuarez MKanstrén TMemon A(2014)Murphy ToolsProceedings of the 2014 IEEE International Conference on Software Testing, Verification, and Validation Workshops10.1109/ICSTW.2014.39(343-348)Online publication date: 31-Mar-2014
https://dl.acm.org/doi/10.1109/ICSTW.2014.39
Darvish AChang C(2014)GUIDiVaProceedings of the 2014 IEEE 38th Annual Computer Software and Applications Conference10.1109/COMPSAC.2014.93(65-74)Online publication date: 21-Jul-2014
https://dl.acm.org/doi/10.1109/COMPSAC.2014.93
Bae GRothermel GBae D(2014)Comparing model-based and dynamic event-extraction based GUI testing techniquesJournal of Systems and Software10.1016/j.jss.2014.06.03997:C(15-46)Online publication date: 1-Oct-2014
https://dl.acm.org/doi/10.1016/j.jss.2014.06.039
Show More Cited By

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Issue’s Table of Contents