Article

Economic perspectives in test automation: balancing automated and manual testing with opportunity cost

Authors:

Klaus WolfmaierAuthors Info & Claims

AST '06: Proceedings of the 2006 international workshop on Automation of software test

Pages 85 - 91

https://doi.org/10.1145/1138929.1138946

Published: 23 May 2006 Publication History

Abstract

Testing is a major cost factor in software development. Test automation has been proposed as one solution to reduce these costs. Test automation tools promise to increase the number of tests they run and the frequency at which they run them. So why not automate every test? In this paper we discuss the question "When should a test be automated?" and the trade-off between automated and manual testing. We reveal problems in the overly simplistic cost models commonly used to make decisions about automating testing. We introduce an alternative model based on opportunity cost and present influencing factors on the decision of whether or not to invest in test automation. Our aim is to stimulate discussion about these factors as well as their influence on the benefits and costs of automated testing in order to support researchers and practitioners reflecting on proposed automation approaches.

References

[1]

Amland, S., Risk Based Testing and Metrics. 5th Int. Software Testing Analysis & Review Conference (EuroSTAR'99), Barcelona, Spain, Nov. 1999.

[2]

Bach, J., Test Automation Snake Oil, 14th International Conference and Exposition on Testing Computer Software, Washington, DC, 1999.

[3]

Boehm B.: Software Risk Management: Principles and Practices. IEEE Software, pp. 32--41, Jan. 1991.

Digital Library

[4]

Boehm, B., Value-Based Software Engineering: Overview and Agenda. In: Biffl S. et al.: Value-Based Software Engineering. Springer, 2005.

[5]

Boehm, B., Basili, V.R., Software Defect Reduction Top 10 List, IEEE Computer, pp. 135--137, January 2001.

Digital Library

[6]

Compuware QADirector, Advanced risk-based test management. Compuware2006 (www.compuware.com/products/qacenter/qadirector.htm).

[7]

Dustin, E. et. al., Automated Software Testing, Addison-Wesley, 1999.

[8]

Fewster, M., Graham, D., Software Test Automation: Effective Use of Text Execution Tool, Addison-Wesley, 1999.

Digital Library

[9]

Harrold, M.J.: Testing: A Roadmap. In: The Future of Software Engineering, ed by Finkelstein, A., 22th International Conference on Software Engineering, Limerick, Ireland, June 2000.

Digital Library

[10]

Highsmith, J., Agile Software Development Ecosystems, Addison-Wesley, 2002.

Digital Library

[11]

Hoffman, D., Cost Benefits Analysis of Test Automation, Software Testing Analysis & Review Conference (STAR East). Orlando, FL, May 1999.

[12]

Kaner, C., Bach, J., and Pettichord, B., Lessons Learned in Software Testing, Wiley, 2002.

Digital Library

[13]

Kaner, C., What is a Good Test Case, Software Testing Analysis & Review Conference (STAR East). May 2003.

[14]

Kruchten, P., The Rational Unified Process: An Introduction, 3rd Ed., Addison-Wesley, 2003.

Digital Library

[15]

Link, J., Unit Testing in Java: How Tests Drive the Code, Morgan Kaufmann, 2003.

Digital Library

[16]

Linz, T., Daigl, M., GUI Testing Made Painless. Implementation and results of the ESSI Project Number 24306, 1998. In: Dustin, et. al., Automated Software Testing, Addison-Wesley, 1999, pp. 52.

[17]

Mankiw G. W., Principles of Economics, 2nd Ed. Hartcourt College Publishers, 1998.

[18]

Marick, B., When Should a Test Be Automated, Software Testing Analysis & Review Conference (STAR East). Orlando, FL, May 1999.

[19]

Persson, C., Yilmaztürk, N., Establishment of Automated Regression Testing at ABB: Industiral Experience Report on Avoiding the Pitfalls, 19th IEEE International Conference on Automated Software Engineering (ASE'04), Linz, Austria, 2004.

Digital Library

[20]

Ramler R., Weippl E., Winterer M., Schwinger W., and Altmann J., A Quality-Driven Approach to Web-Testing. 2nd Conf. on Web Engineering (ICWE'02), Santa Fe, Argentina, Sept. 2002.

[21]

Ramler R., Biffl S., Grünbacher P., Value-based Management of Software Testing. In: Biffl S. et al.: Value-Based Software Engineering. Springer, 2005.

[22]

Redmill, F., Exploring risk-based testing and its implications. Software Testing, Verification and Reliability. 14, 3--15 (2004).

Digital Library

[23]

Rothermel, G., Elbaum, S., Putting Your Best Tests Forward. IEEE Software, 20(5), pp. 74--77, Sept/Oct 2003.

Digital Library

[24]

Schwaber, C., Gilpin, M., Evaluating Automated Functional Testing Tools, Forrester Research, February 2005.

[25]

Voas, J. M., Morell, L. J., and Miller, K. W., Predicting where faults can hide from testing. IEEE Software, 8(2), pp. 41--48, March 1991.

Digital Library

Cited By

Yin HMohammed HBoyapati S(2024)Leveraging Pre-Trained Large Language Models (LLMs) for On-Premises Comprehensive Automated Test Case Generation: An Empirical Study2024 9th International Conference on Intelligent Informatics and Biomedical Sciences (ICIIBMS)10.1109/ICIIBMS62405.2024.10792720(597-607)Online publication date: 21-Nov-2024
https://doi.org/10.1109/ICIIBMS62405.2024.10792720
Florea RStray VSjøberg D(2023)On the roles of software testers: An exploratory studyJournal of Systems and Software10.1016/j.jss.2023.111742204(111742)Online publication date: Oct-2023
https://doi.org/10.1016/j.jss.2023.111742
Abo-eleneen APalliyali ACatal C(2023)The role of Reinforcement Learning in software testingInformation and Software Technology10.1016/j.infsof.2023.107325164(107325)Online publication date: Dec-2023
https://doi.org/10.1016/j.infsof.2023.107325
Show More Cited By

Index Terms

Economic perspectives in test automation: balancing automated and manual testing with opportunity cost
1. Social and professional topics
  1. Professional topics
    1. Management of computing and information systems
      1. System management
        Quality assurance
2. Software and its engineering
  1. Software creation and management
    1. Software verification and validation
      1. Software defect analysis
        Software testing and debugging

Recommendations

The complementary aspect of automatically and manually generated test case sets
A-TEST 2016: Proceedings of the 7th International Workshop on Automating Test Case Design, Selection, and Evaluation

The test is a mandatory activity for software quality assurance. The knowledge about the software under testing is necessary to generate high-quality test cases, but to execute more than 80% of its source code is not an easy task, and demands an in-...
When to automate software testing? A decision-support approach based on process simulation

Software test processes are complex and costly. To reduce testing effort without compromising effectiveness and product quality, automation of test activities has been adopted as a popular approach in software industry. However, because test automation ...
When to automate software testing? decision support based on system dynamics: an industrial case study
ICSSP '14: Proceedings of the 2014 International Conference on Software and System Process

Software test processes are complex and costly. To reduce testing effort without compromising effectiveness and product quality, automation of test activities has been adopted as a popular approach in software industry. However, since test automation ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

AST '06: Proceedings of the 2006 international workshop on Automation of software test

May 2006

128 pages

ISBN:1595934081

DOI:10.1145/1138929

Conference Chair:
Hong Zhu
Oxford Brookes University, UK
,
Program Chairs:
Joseph R. Horgan
Telcordia Technologies, USA
,
S. C. Cheung
The Hong Kong University of Science and Technology, Hong Kong
,
J. Jenny Li
Avaya Labs, USA

Copyright © 2006 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 23 May 2006

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Article

Conference

ICSE06

Sponsor:

ICSE06: International Conference on Software Engineering

May 23, 2006

Shanghai, China

Upcoming Conference

ICSE 2025

2025 IEEE/ACM 46th International Conference on Software Engineering

April 26 - May 3, 2025

Ottawa , ON , Canada

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

65
Total Citations
View Citations
3,357
Total Downloads

Downloads (Last 12 months)73
Downloads (Last 6 weeks)10

Reflects downloads up to 23 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Yin HMohammed HBoyapati S(2024)Leveraging Pre-Trained Large Language Models (LLMs) for On-Premises Comprehensive Automated Test Case Generation: An Empirical Study2024 9th International Conference on Intelligent Informatics and Biomedical Sciences (ICIIBMS)10.1109/ICIIBMS62405.2024.10792720(597-607)Online publication date: 21-Nov-2024
https://doi.org/10.1109/ICIIBMS62405.2024.10792720
Florea RStray VSjøberg D(2023)On the roles of software testers: An exploratory studyJournal of Systems and Software10.1016/j.jss.2023.111742204(111742)Online publication date: Oct-2023
https://doi.org/10.1016/j.jss.2023.111742
Abo-eleneen APalliyali ACatal C(2023)The role of Reinforcement Learning in software testingInformation and Software Technology10.1016/j.infsof.2023.107325164(107325)Online publication date: Dec-2023
https://doi.org/10.1016/j.infsof.2023.107325
Giray G(2022)A software engineering perspective on engineering machine learning systemsJournal of Systems and Software10.1016/j.jss.2021.111031180:COnline publication date: 22-Apr-2022
https://dl.acm.org/doi/10.1016/j.jss.2021.111031
Junior NCosta HKarita LMachado ISoares L(2021)Experiences and Practices in GUI Functional Testing: A Software Practitioners’ ViewProceedings of the XXXV Brazilian Symposium on Software Engineering10.1145/3474624.3474640(195-204)Online publication date: 27-Sep-2021
https://dl.acm.org/doi/10.1145/3474624.3474640
Halani KKavita Saxena R(2021)Critical Analysis of Manual Versus Automation Testing2021 International Conference on Computational Performance Evaluation (ComPE)10.1109/ComPE53109.2021.9752388(132-135)Online publication date: 1-Dec-2021
https://doi.org/10.1109/ComPE53109.2021.9752388
Clark AWalkinshaw NHierons R(2021)Test case generation for agent-based models: A systematic literature reviewInformation and Software Technology10.1016/j.infsof.2021.106567135(106567)Online publication date: Jul-2021
https://doi.org/10.1016/j.infsof.2021.106567
García BMunoz-Organero MAlario-Hoyos CDelgado Kloos C(2021)Automated driver management for selenium WebDriverEmpirical Software Engineering10.1007/s10664-021-09975-326:5Online publication date: 23-Jul-2021
https://doi.org/10.1007/s10664-021-09975-3
Ricca FStocco A(2021)Web Test Automation: Insights from the Grey LiteratureSOFSEM 2021: Theory and Practice of Computer Science10.1007/978-3-030-67731-2_35(472-485)Online publication date: 11-Jan-2021
https://doi.org/10.1007/978-3-030-67731-2_35
Angara JPrasad SSridevi G(2020)DevOps Project Management Tools for Sprint Planning, Estimation and Execution MaturityCybernetics and Information Technologies10.2478/cait-2020-001820:2(79-92)Online publication date: 12-Jun-2020
https://doi.org/10.2478/cait-2020-0018
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents