research-article

An Empirical Study of Testing and Integration Strategies Using Artificial Software Systems

Authors:

J. H. RowlandAuthors Info & Claims

IEEE Transactions on Software Engineering, Volume 19, Issue 10

Pages 941 - 949

https://doi.org/10.1109/32.245736

Published: 01 October 1993 Publication History

Abstract

There has been much discussion about the merits of various testing and integration strategies. Top-down, bottom-up, big-bang, and sandwich integration strategies are advocated by various authors. Also, some authors insist that modules be unit tested, while others believe that unit testing diverts resources from more effective verification processes. This article addresses the ability of the aforementioned integration strategies to detect defects, and produce reliable systems. It also explores the efficacy of spot unit testing, and compares phased and incremental versions of top-down and bottom-up integration strategies. Relatively large artificial software systems were constructed using a code generator with ten basic module templates. These systems were seeded with known defects and tested using the above testing and integration strategies. A number of experiments were then conducted using a simulator whose validity was established by comparing results against these artificial systems. The defect detection ability and resulting system reliability were measured for each strategy. Results indicated that top-down integration strategies are generally most effective in terms of defect correction. Top-down and big-bang strategies produced the most reliable systems. Results favored neither those strategies that incorporate spot unit testing nor those that do not; also, results favored neither phased nor incremental strategies.

References

[1]

{1} V. R. Basili and R. W. Selby, "Comparing the effectiveness of software testing strategies," IEEE Trans. Software Eng., vol. SE-13, no. 12, pp. 1278-1296, Dec. 1987.

Digital Library

[2]

{2} B. Beizer, Software System Testing and Quality Assurance. New York: Van Nostrand Reinhold, 1984, pp. 4, 40, 143, 161-162.

[3]

{3} G. Collins and G. Blay, Structured Systems Development Techniques: Strategic Planning to System Testing. New York: Wiley, 1983, p. 206.

[4]

{4} M. S. Deutsch, Software Verification and Validation: Realistic Project Approaches. Englewood Cliffs, NJ: Prentice-Hall, 1982, pp. 95-101.

[5]

{5} J. W. Duran and S. C. Ntafos, "An evaluation of random testing," IEEE Trans. Software Eng., vol. SE-10, no. 4, pp. 438-444, July 1984.

Digital Library

[6]

{6} M. R. Girgis and M. R. Woodward, "An experimental comparison of the error exposing ability of program testing criteria," in Proc. Workshop Software Testing. Washington, DC: IEEE Computer Society Press, 1986, pp. 64-73.

[7]

{7} B. Hetzel, The Complete Guide to Software Testing. Wellesley, MA: QED Information Sciences, 1988, pp. 117-118.

[8]

{8} W. E. Howden, "Functional program testing," IEEE Trans. Software Eng., vol. SE-6, pp. 162-169, 1980.

Digital Library

[9]

{9} H. D. Mills, "Structured programming: Retrospect and prospect," IEEE Software, vol. 3, no. 6, pp. 58-66, Nov. 1986.

Digital Library

[10]

{10} H. D. Mills, M. Dyer, and R. C. Linger, "Cleanroom software engineering," IEEE Software, vol. 4, no. 5, pp. 19-25, Sept. 1987.

Digital Library

[11]

{11} G. J. Myers, The Art of Software Testing. New York: Wiley, 1979, pp. 36, 89.

[12]

{12} G. J. Myers, Software Reliability: Principles and Practices. New York: Wiley, 1976, pp. 187, 189.

[13]

{13} S. L. Pfleeger, Software Engineering. New York: Macmillan, 1991.

[14]

{14} R. M. Poston and M. W. Bruen, "Counting down to zero software failures," IEEE Software, vol. 4, no. 5, pp. 54-61, Sept. 1987.

Digital Library

[15]

{15} J. H. Rowland, "Artificial systems for software engineering studies," Proc. 2nd Workshop Software Testing, Verification, and Analysis. Washington, DC: IEEE Computer Society Press, 1988, pp. 80-88.

[16]

{16} J. H. Rowland and Y. Zuyuan, "Experimental comparison of three system test strategies: Preliminary report," Proc. ACM SIGSOFT '89 3rd Symp. Software Testing, Analysis, and Verification. New York: ACM Press, 1989, pp. 141-150.

Digital Library

[17]

{17} H. F. Smith, Data Structures: Form and Function. San Diego: Harcourt Brace Jovanovich, 1987, pp. 204-206.

[18]

{18} J. A. Solheim, "Testing and integration strategies of large software systems", Ph.D, dissertation, Univ. Wyoming, Laramie, 1992 (microfiche available through UMI Dissertation Services, Ann Arbor, MI).

[19]

{19} I. Sommerville, Software Engineering. Reading, MA: Addison-Wesley, 1992.

[20]

{20} P. Thévenod-Fosse and H. Waeselynck, "An investigation of statistical software testing," J. Software Testing, Verification and Reliability, vol. 1, no. 2, pp. 5-25, July-Sept. 1991.

Digital Library

[21]

{21} R. Walpole and R. Myers, Probability and Statistics for Engineers and Scientists. New York: Macmillan, 1978, pp. 382-383.

[22]

{22} E. Yourdon, Managing the Structured Techniques. Englewood Cliffs, NJ: Prentice-Hall, 1989, p. 79.

Cited By

Perry DPorter AVotta LFinkelstein A(2000)Empirical studies of software engineeringProceedings of the Conference on The Future of Software Engineering10.1145/336512.336586(345-355)Online publication date: 1-May-2000
https://dl.acm.org/doi/10.1145/336512.336586
Voas J(1998)Certifying Off-the-Shelf Software ComponentsComputer10.1109/2.68300831:6(53-59)Online publication date: 1-Jun-1998
https://dl.acm.org/doi/10.1109/2.683008
Zhu HHall PMay J(1997)Software unit test coverage and adequacyACM Computing Surveys10.1145/267580.26759029:4(366-427)Online publication date: 1-Dec-1997
https://dl.acm.org/doi/10.1145/267580.267590
Show More Cited By

Recommendations

Testing and integration strategies of large software systems
On some reliability estimation problems in random and partition testing

Studies have shown that random testing can be an effective testing strategy. One of the goals of testing is to estimate the reliability of the program from the test outcomes. The authors extend the Thayer-Lipow-Nelson reliability model (R. Thayer et al.,...
A demand-driven analyzer for data flow testing at the integration level
ICSE '96: Proceedings of the 18th international conference on Software engineering

Data-flow testing relies on static analysis for computing the definition-use pairs that serve as the test case requirements for a program. When testing large programs, the individual procedures are first tested in isolation during unit testing. ...

Comments

Information & Contributors

Information

Published In

cover image IEEE Transactions on Software Engineering

IEEE Transactions on Software Engineering Volume 19, Issue 10

October 1993

71 pages

ISSN:0098-5589

Issue’s Table of Contents

Copyright © Copyright © 1993 IEEE. All Rights Reserved.

Publisher

IEEE Press

Publication History

Published: 01 October 1993

Author Tags

Qualifiers

Research-article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

4
Total Citations
View Citations
0
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 11 Aug 2024

Other Metrics

View Author Metrics

Citations

Cited By

Perry DPorter AVotta LFinkelstein A(2000)Empirical studies of software engineeringProceedings of the Conference on The Future of Software Engineering10.1145/336512.336586(345-355)Online publication date: 1-May-2000
https://dl.acm.org/doi/10.1145/336512.336586
Voas J(1998)Certifying Off-the-Shelf Software ComponentsComputer10.1109/2.68300831:6(53-59)Online publication date: 1-Jun-1998
https://dl.acm.org/doi/10.1109/2.683008
Zhu HHall PMay J(1997)Software unit test coverage and adequacyACM Computing Surveys10.1145/267580.26759029:4(366-427)Online publication date: 1-Dec-1997
https://dl.acm.org/doi/10.1145/267580.267590
Voas JMcGraw GKassab LVoas L(1997)A 'Crystal Ball' for Software LiabilityComputer10.1109/2.58754530:6(29-36)Online publication date: 1-Jun-1997
https://dl.acm.org/doi/10.1109/2.587545

View Options

View options

Media

Figures

Other

Tables

View Issue’s Table of Contents