article

Empirical studies of software testing techniques: challenges, practical strategies, and future research

Authors:

Lionel Briand and

Yvan LabicheAuthors Info & Claims

ACM SIGSOFT Software Engineering Notes, Volume 29, Issue 5

Pages 1 - 3

https://doi.org/10.1145/1022494.1022541

Published: 01 September 2004 Publication History

Get Access

Abstract

This position paper aims at discussing a number of issues that typically arise when performing empirical studies with software testing techniques. Though some problems are general to all empirical disciplines, software testing studies face a number of specific challenges. Some of the main ones are discussed in sequence below.

References

[1]

G. Antoniol, L. C. Briand, M. Di Penta and Y. Labiche, "A Case Study Using the Round-Trip Strategy for State-Based Class Testing," Proc. 13th IEEE International Symposium on Software Reliability Engineering (ISSRE'02), Annapolis (MD, USA), IEEE Computer Society, pp. 269--279, November 12-15, 2002.

Digital Library

Google Scholar

[2]

L. Briand, Y. Labiche and Y. Wang, "Using Simulation to Empirically Investigate Test Coverage Criteria," Proc. (to appear) IEEE/ACM International Conference on Software Engineering, Edinburgh, May, 2004.

Digital Library

Google Scholar

[3]

L. C. Briand, Y. Labiche and H. Sun, "Investigating the Use of Analysis Contracts to Improve the Testability of Object-Oriented Code," Software - Practice and Experience, vol. 33 (7), pp. 637--672, 2003.

Digital Library

Google Scholar

[4]

M. E. Delamaro, J. C. Maldonado and A. P. Mathur, "Interface Mutation: An Approach for Integration Testing," IEEE Transactions of Software Engineering, vol. 27 (3), pp. 228--247, 2001.

Digital Library

Google Scholar

[5]

R. A. DeMillo, "Hints on Test Data Selection: Help for the Practicing Programmer," in IEEE Computer, vol. 11, 1978, pp. 34--41.

Digital Library

Google Scholar

[6]

T. L. Graves, M. J. Harrold, J.-M. Kim, A. Porter and G. Rothermel, "An Empirical Study of Regression Test Selection Techniques," ACM Transactions on Software Engineering and Methodology, vol. 10 (2), pp. 184--208, 2001.

Digital Library

Google Scholar

[7]

M. Hutchins, H. Froster, T. Goradia and T. Ostrand, "Experiments on the Effectiveness of Dataflow- and Controlflow-Based Test Adequacy Criteria," Proc. 16th IEEE International Conference on Software Engineering, Sorrento (Italy), pp. 191--200, May 16-21, 1994.

Digital Library

Google Scholar

[8]

N. Juristo and A. M. Moreno, Basics of Software Engineering Experimentation, Kluwer, 2001.

Digital Library

Google Scholar

[9]

S. Kim, J. A. Clark and J. A. McDermid, "Class Mutation: Mutation Testing for Object-Oriented Programs," Proc. Net. ObjectDays, Erfurt, Germany, October 9-12, 2000.

Google Scholar

[10]

K. N. King and A. J. Offutt, "A Fortran Language System for Mutation-Based Software Testing," Software - Practice and Experience, vol. 21 (7), pp. 686--718, 1991.

Digital Library

Google Scholar

[11]

A. J. Offutt, "Investigations of the Software Testing Coupling Effect," ACM Transactions on Software Engineering and Methodology, vol. 1 (1), pp. 3--18, 1992.

Digital Library

Google Scholar

[12]

E. Weyuker, "More Experience with Data Flow Testing," IEEE Transactions on Software Engineering, vol. 19 (9), pp. 912--919, 1993.

Digital Library

Google Scholar

[13]

C. Wohlin, P. Runeson, M. Host, M. C. Ohlsson, B. Regnell and A. Wesslen, Experimentation in Software Engineering - An Introduction, Kluwer, 2000.

Digital Library

Google Scholar

Cited By

View all

Zhou WWang YCheng XWang X(2020)Predicate Testing Generation for Safety-critical Systems2020 6th International Symposium on System and Software Reliability (ISSSR)10.1109/ISSSR51244.2020.00032(143-150)Online publication date: Oct-2020
https://doi.org/10.1109/ISSSR51244.2020.00032
Melo SCarver JSouza PSouza S(2019)Empirical research on concurrent software testing: A systematic mapping studyInformation and Software Technology10.1016/j.infsof.2018.08.017105(226-251)Online publication date: Jan-2019
https://doi.org/10.1016/j.infsof.2018.08.017
Lazzarini Lemos OFagundes Silveira FCutigi Ferrari FGarcia A(2018)The impact of Software Testing education on code reliability: An empirical assessmentJournal of Systems and Software10.1016/j.jss.2017.02.042137(497-511)Online publication date: Mar-2018
https://doi.org/10.1016/j.jss.2017.02.042
Show More Cited By

Index Terms

Empirical studies of software testing techniques: challenges, practical strategies, and future research
1. Software and its engineering
  1. Software creation and management
    1. Software verification and validation
      1. Software defect analysis
        Software testing and debugging

Index terms have been assigned to the content through auto-classification.

Recommendations

Ethical Issues in Empirical Studies of Software Engineering

The popularity of empirical methods in software engineering research is on the rise. Surveys, experiments, metrics, case studies, and field studies are examples of empirical methods used to investigate both software engineering processes and products. ...
Read More
A Critical Analysis of Empirical Research in Software Testing
ESEM '07: Proceedings of the First International Symposium on Empirical Software Engineering and Measurement

In the foreseeable future, software testing will remain one of the best tools we have at our disposal to ensure software dependability. Empirical studies are crucial to software testing research in order to compare and improve software testing ...
Read More
Empirical Studies of Test Case Prioritization in a JUnit Testing Environment
ISSRE '04: Proceedings of the 15th International Symposium on Software Reliability Engineering

Test case prioritization provides a way to run test cases with the highest priority earliest. Numerous empirical studies have shown that prioritization can improve a test suite's rate of fault detection, but the extent to which these results generalize ...
Read More

Comments

Information & Contributors

Information

Published In

cover image ACM SIGSOFT Software Engineering Notes

ACM SIGSOFT Software Engineering Notes Volume 29, Issue 5

September 2004

321 pages

ISSN:0163-5948

DOI:10.1145/1022494

Issue’s Table of Contents

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 01 September 2004

Published in SIGSOFT Volume 29, Issue 5

Check for updates

Qualifiers

Article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

32
Total Citations
View Citations
1,305
Total Downloads

Downloads (Last 12 months)18
Downloads (Last 6 weeks)0

Other Metrics

View Author Metrics

Citations

Cited By

View all

Zhou WWang YCheng XWang X(2020)Predicate Testing Generation for Safety-critical Systems2020 6th International Symposium on System and Software Reliability (ISSSR)10.1109/ISSSR51244.2020.00032(143-150)Online publication date: Oct-2020
https://doi.org/10.1109/ISSSR51244.2020.00032
Melo SCarver JSouza PSouza S(2019)Empirical research on concurrent software testing: A systematic mapping studyInformation and Software Technology10.1016/j.infsof.2018.08.017105(226-251)Online publication date: Jan-2019
https://doi.org/10.1016/j.infsof.2018.08.017
Lazzarini Lemos OFagundes Silveira FCutigi Ferrari FGarcia A(2018)The impact of Software Testing education on code reliability: An empirical assessmentJournal of Systems and Software10.1016/j.jss.2017.02.042137(497-511)Online publication date: Mar-2018
https://doi.org/10.1016/j.jss.2017.02.042
Rei SAbreu R(2017)A Database of Existing Vulnerabilities to Enable Controlled Testing StudiesInternational Journal of Secure Software Engineering10.4018/IJSSE.20170701018:3(1-23)Online publication date: 1-Jul-2017
https://dl.acm.org/doi/10.4018/IJSSE.2017070101
da Costa Araújo Ida Silva Wde Sousa Nunes JNeto F(2016)ARRESTTProceedings of the 1st Brazilian Symposium on Systematic and Automated Software Testing10.1145/2993288.2993303(1-10)Online publication date: 19-Sep-2016
https://dl.acm.org/doi/10.1145/2993288.2993303
Deak AStålhane TSindre G(2016)Challenges and strategies for motivating software testing personnelInformation and Software Technology10.1016/j.infsof.2016.01.00273:C(1-15)Online publication date: 1-May-2016
https://dl.acm.org/doi/10.1016/j.infsof.2016.01.002
de Oliveira Neto FTorkar RMachado PBertolino ACanfora GElbaum S(2015)An initiative to improve reproducibility and empirical evaluation of software testing techniquesProceedings of the 37th International Conference on Software Engineering - Volume 210.5555/2819009.2819106(575-578)Online publication date: 16-May-2015
https://dl.acm.org/doi/10.5555/2819009.2819106
Lemos OFerrari FSilveira FGarcia A(2015)Experience reportProceedings of the 2015 IEEE 26th International Symposium on Software Reliability Engineering (ISSRE)10.1109/ISSRE.2015.7381829(359-369)Online publication date: 2-Nov-2015
https://dl.acm.org/doi/10.1109/ISSRE.2015.7381829
Oliveira Neto FTorkar RMachado P(2015)An Initiative to Improve Reproducibility and Empirical Evaluation of Software Testing Techniques2015 IEEE/ACM 37th IEEE International Conference on Software Engineering10.1109/ICSE.2015.197(575-578)Online publication date: May-2015
https://doi.org/10.1109/ICSE.2015.197
Alrmuny D(2014)Open Problems in Software Test CoverageLecture Notes on Software Engineering10.7763/LNSE.2014.V2.107(121-125)Online publication date: 2014
https://doi.org/10.7763/LNSE.2014.V2.107
Show More Cited By

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Cited By

Index Terms

Recommendations

Ethical Issues in Empirical Studies of Software Engineering

A Critical Analysis of Empirical Research in Software Testing

Empirical Studies of Test Case Prioritization in a JUnit Testing Environment

Comments

Published In

Publisher

Publication History

Check for updates

Qualifiers

Other Metrics

Article Metrics

Other Metrics

Cited By

Login options

Full Access

PDF

eReader

Abstract

References

Cited By

Index Terms

Recommendations

Ethical Issues in Empirical Studies of Software Engineering

A Critical Analysis of Empirical Research in Software Testing

Empirical Studies of Test Case Prioritization in a JUnit Testing Environment

Comments

Information

Published In

Publisher

Publication History

Check for updates

Qualifiers

Contributors

Other Metrics

Bibliometrics

Article Metrics

Other Metrics

Citations

Cited By

Get Access

Login options

Full Access

View options

PDF

eReader

Figures

Other

Share

Share this Publication link

Share on social media

Affiliations