research-article

Public Access

Impact of Using Tools in an Undergraduate Software Testing Course Supported by WReSTT

Authors:

Peter J. Clarke,

Debra L. Davis,

Raymond Chang-Lau,

Tariq M. KingAuthors Info & Claims

ACM Transactions on Computing Education (TOCE), Volume 17, Issue 4

Article No.: 18, Pages 1 - 28

https://doi.org/10.1145/3068324

Published: 26 August 2017 Publication History

Abstract

Software continues to affect a major part of our daily lives, including the way we use our phones, home appliances, medical devices, and cars. The pervasiveness of software has led to a growing demand for software developers over the next decade. To ensure the high quality of software developed in industry, students being trained in software engineering also need to be trained on how to use testing techniques and supporting tools effectively at all levels of development.

In this article, we investigate how testing tools are used in the software project of an undergraduate testing course. We also investigate how a cyberlearning environment—the Web-Based Repository of Software Testing Tutorials (WReSTT)—is used to supplement the learning materials presented in class, particularly the tutorials on different software testing tools. The results of a study spanning three semesters of the undergraduate course suggest that (1) the use of code coverage tools motivates students to improve their test suites; (2) the number of bugs found when using coverage tools slightly increased, which is similar to the results found in the research literature; and (3) students find WReSTT to be a useful resource for learning about software testing techniques and the use of code coverage tools.

References

[1]

AAAS and NSF. 2013. Transforming Undergraduate Education in STEM: Making and Measuring Impacts. In Proceedings of the 2013 TUES Principal Investigators (Pls) Conference. http://ccliconference.org/files/2013/01/2013-Tues-Conference-Program.pdf

[2]

Kalle Aaltonen, Petri Ihantola, and Otto Seppälä. 2010. Mutation analysis vs. code coverage in automated assessment of students’ testing skills. In Proceedings of the ACM International Conference Companion on Object Oriented Programming Systems Languages and Applications Companion (OOPSLA’10). ACM, New York, NY, 153--160.

Digital Library

[3]

ACM/IEEE-CS Interim Review Task Force. 2008. Computer Science Curriculum 2008: An Interim Revision of CS 2001. Retrieved June 12, 2017, from http://www.acm.org/education/curricula/ComputerScience 2008.pdf.

[4]

Paul Ammann and Jeff Offutt. 2008. Introduction to Software Testing. Cambridge University Press, New York, NY.

[5]

Robert V. Binder. 1999. Testing Object-Oriented Systems: Models, Patterns, and Tools. Addison Wesley Longman, Boston, MA.

Digital Library

[6]

P. Bourque and R. Dupuis. 2004. Guide to the Software Engineering Body of Knowledge 2004 Version. IEEE, Los Alamitos, California.

[7]

Tom Britton, Lisa Jeng, Graham Carver, and Paul Cheak. 2013. Reversible Debugging Software “Quantify the Time and Cost Saved Using Reversible Debuggers.” Available at http://citeseerx.ist.psu.edu/viewdoc/summary? doi=10.1.1.370.9611

[8]

Bernd Bruegge and Allen H. Dutoit. 2009. Object-Oriented Software Engineering Using UML, Patterns, and Java (3rd ed.). Prentice Hall, Upper Saddle River, NJ.

[9]

J. C. Carver and N. A. Kraft. 2011. Evaluating the testing ability of senior-level computer science students. In Proceedings of the 24th IEEE-CS Conference on CSEET. IEEE, Los Alamitos, CA, 169--178.

Digital Library

[10]

Peter J. Clarke, Debra Davis, Raymond Chang-Lau, and Tariq King. 2014a. Observations on student use of tools in an undergraduate testing class. In 121st American Society for Engineering Education (ASEE)—Software Engineering Constituent Committee Division Track (SWECC). Paper No. 10123. ASEE, Washington DC.

[11]

Peter J. Clarke, Debra Davis, Tariq M. King, Jairo Pava, and Edward L. Jones. 2014b. Integrating testing into software engineering courses supported by a collaborative learning environment. Transactions on Computing Education 14, 3, Article No. 18.

Digital Library

[12]

Peter J. Clarke, Jairo Pava, Debra Davis, and Tariq M. King. 2012. Using WReSTT in SE courses: An empirical study. In Proceedings of the 43rd SIGCSE Conference. ACM, New York, NY, 307--312.

Digital Library

[13]

CodeCover Team. 2011. CodeCover Home Page. Retrieved June 12, 2017, from http://codecover.org/

[14]

Jacob Cohen. 1988. Statistical Power Analysis for the Behavioral Sciences (2nd ed.). Lawrence Erlbaum, NJ.

[15]

John W. Creswell. 2004. Educational Research: Planning, Conducting, and Evaluating Quantitative and Qualitative Research (2nd ed.). Prentice Hall, Upper Saddle River, NJ.

[16]

Sebastian Deterding, Dan Dixon, Rilla Khaled, and Lennart Nacke. 2011. From game design elements to gamefulness: Defining “gamification.” In Proceedings of the 15th International Academic MindTrek Conference: Envisioning Future Media Environments (MindTrek’11). ACM, New York, NY, 9--15.

Digital Library

[17]

E. W. Dijkstra. 1970. Software Engineering Techniques. Report on a conference sponsored by the NATO Science Committee, Rome, Italy, October 1969.

[18]

T. Dvornik, D. S. Janzen, J. Clements, and O. Dekhtyar. 2011. Supporting introductory test-driven labs with WebIDE. In Proceedings of the 24th IEEE-CS Software Engineering Education and Training Conference (CSEET’11). IEEE, Los Alamitos, CA, 51--60.

Digital Library

[19]

Stephen Edwards. 2015. Web-CAT: The Web-Based Center for Automated Testing. http://web-cat.org/.

[20]

Stephen H. Edwards. 2004. Using software testing to move students from trial-and-error to reflection-in-action. In Proceedings of the 35th SIGCSE Technical Symposium on Computer Science Education (SIGCSE’04). ACM, New York, NY, 26--30.

Digital Library

[21]

Stephen H. Edwards and Zalia Shams. 2014. Comparing test quality measures for assessing student-written tests. In Companion Proceedings of the 36th International Conference on Software Engineering (ICSE Companion’14). ACM, New York, NY, 354--363.

Digital Library

[22]

FitNesse Team. 2015. FitNesse Home Page. Retrieved June 12, 2017, from http://www.fitnesse.org/.

[23]

P. G. Frankl and E. J. Weyuker. 1993. A formal analysis of the fault-detecting ability of testing methods. IEEE Transactions on Software Engineering 19, 3, 202--213. http://dx.doi.org/10.1109/32.221133

Digital Library

[24]

S. Frezza. 2002. Integrating testing and design methods for undergraduates: Teaching software testing in the context of software design. In Proceedings of the 32nd Annual Conference on Frontiers in Education (FIE’02). Vol. 3. IEEE, Los Alamitos, CA.

[25]

E. Gamma and K. Beck. 2015. JUnit Home Page. Retrieved June 12, 2017, from http://www.junit.org/.

[26]

V. Garousi. 2011. Incorporating real-world industrial testing projects in software testing courses: Opportunities, challenges, and lessons learned. In Proceedings of the 24th IEEE-CS Conference on CSEET. IEEE, Los Alamitos, CA, 396--400.

Digital Library

[27]

Michael Hackett. 2013. The changing landscape of software testing. LogiGear Magazine 7, 1, 7--15. http://www.logigear.com/magazine/wp-content/uploads/2013/02/LogiGear-Magazine-The-Rapidly-Changing-Testing-Landscape1.pdf.

[28]

Joachim Hofer. 2010. eCobertura Home Page. (August2010). Retrieved June 12, 2017, from http://ecobertura.johoop.de/.

[29]

Marc R. Hoffmann, Brock Janiczak, and Evgeny Mandrikov. 2015. EclEmma Home Page. Retrieved June 12, 2017, from http://www.eclemma.org/.

[30]

IBM. 2015a. Rational Functional Tester. Retrieved June 12, 2017, from http://www-01.ibm.com/software/awdtools/tester/functional/.

[31]

IBM. 2015b. Rational Test RealTime. Retrieved June 12, 2017, from http://www-03.ibm.com/software/products/en/realtime.

[32]

Laura Inozemtseva and Reid Holmes. 2014. Coverage is not strongly correlated with test suite effectiveness. In Proceedings of the 36th International Conference on Software Engineering (ICSE’14). ACM, New York, NY, 435--445.

Digital Library

[33]

David S. Janzen and Hossein Saiedian. 2006. Test-driven learning: Intrinsic integration of testing into the CS/SE curriculum. ACM SIGCSE Bulletin 38, 1, 254--258.

Digital Library

[34]

Edward L. Jones. 2000. Software testing in the computer science curriculum—a holistic approach. In Proceedings of the Australasian Conference on Computing Education (ACSE’00). ACM, New York, NY, 153--157.

Digital Library

[35]

C. Kaner and S. Padmanabhan. 2007. Practice and transfer of learning in the teaching of software testing. In Proceedings of the 20th Conference on Software Engineering Education Training (CSEET’07). IEEE, Los Alamitos, CA, 157--166.

Digital Library

[36]

Alex Koohang. 2004. Creating learning objects in collaborative e-learning settings. Issues in Information Systems 4, 2, 584--590. http://www.iacis.org/iis/2004/Koohang.pdf

[37]

Ilaria Liccardi, Asma Ounnas, Reena Pau, Elizabeth Massey, Päivi Kinnunen, Sarah Lewthwaite, Marie-Anne Midy, and Chandan Sarkar. 2007. The role of social networks in students’ learning experiences. ACM SIGCSE Bulletin 39, 4, 224--237. http://doi.acm.org/10.1145/1345375.1345442

Digital Library

[38]

Thomas W. Malone. 1980. What makes things fun to learn? Heuristics for designing instructional computer games. In Proceedings of the 3rd ACM SIGSMALL Symposium and the 1st SIGPC Symposium on Small Systems (SIGSMALL’80). ACM, New York, NY, 162--169.

Digital Library

[39]

Aditya P. Mathur. 2008. Foundations of Software Testing. Pearson Education, Delhi, India.

[40]

John D. McGregor and David A. Sykes. 2001. A Practical Guide to Testing Object-Oriented Software. Addison Wesley Longman, Boston, MA.

[41]

NUnit Team. 2015. NUnit Home Page. Retrieved June 12, 2017, from http://www.nunit.org/.

[42]

Selenium Development Team. 2014. SeleniumHQ Home Page. Retrieved June 12, 2017, from http://www.seleniumhq.org/.

[43]

Charlie Y. Shim, Mina Choi, and Jung Yeop Kim. 2009. Promoting collaborative learning in software engineering by adapting the PBL strategy. In Proceedings of the WASET International Conference on Computer and Information Technology (ICCIT’09). IEEE, Los Alamitos, CA, 1167--1170.

[44]

Barbara Leigh Smith and Jean T. MacGregor. 1992. What is collaborative learning? In Collaborative Learning: A Sourcebook for Higher Education, A. S. Goodsell, M. R. Maher, and V. Tinto (Eds.). National Center on Postsecondary Teaching, Learning, and Assessment, University Park, PA.

[45]

Rachel S. Smith. 2004. Guidelines for Authors of Learning Objects. New Media Consortium. Available at https://www.nmc.org/publication/guidelines-for-authors-of-learning-objects.

[46]

Jean Tesser. 2010. Dependency Finder. Retrieved June 12, 2017, from http://depfind.sourceforge.net.

[47]

Wikipedia. 2013. Category:Software testing tools. Retrieved June 12, 2017, from http://en.wikipedia.org/wiki/Category:Software_testing_tools.

[48]

WReSTT Team. 2015. WReSTT: Web-Based Repository for Software Testing Tutorials. Retrieved June 12, 2017, from http://wrestt.cis.fiu.edu/.

[49]

Qian Yang, J. Jenny Li, and David Weiss. 2007. A survey of coverage-based testing tools. Computer Journal 52, 5, 589--597.

Digital Library

[50]

Hong Zhu. 1996. A formal analysis of the subsume relation between software test adequacy criteria. IEEE Transactions on Software Engineering 22, 4, 248--255.

Digital Library

[51]

Hong Zhu, Patrick A. V. Hall, and John H. R. May. 1997. Software unit test coverage and adequacy. ACM Computing Surveys 29, 366--427.

Digital Library

Cited By

Shin AKazerouni A(2023)A Model of How Students Engineer Test Cases With FeedbackACM Transactions on Computing Education10.1145/362860424:1(1-31)Online publication date: 20-Oct-2023
https://dl.acm.org/doi/10.1145/3628604
Ardic BZaidman A(2023)Hey Teachers, Teach Those Kids Some Software Testing2023 IEEE/ACM 5th International Workshop on Software Engineering Education for the Next Generation (SEENG)10.1109/SEENG59157.2023.00007(9-16)Online publication date: May-2023
https://doi.org/10.1109/SEENG59157.2023.00007
Blanco RTrinidad MSuárez-Cabal MCalderón ARuiz MTuya J(2023)Can gamification help in software testing education? Findings from an empirical studyJournal of Systems and Software10.1016/j.jss.2023.111647200:COnline publication date: 1-Jun-2023
https://dl.acm.org/doi/10.1016/j.jss.2023.111647
Show More Cited By

Index Terms

Impact of Using Tools in an Undergraduate Software Testing Course Supported by WReSTT

Recommendations

Survey of Learning Environments for Software Testing Education
ECBS 2021: 7th Conference on the Engineering of Computer Based Systems

Software development is one of the fastest growing industries today. The defects in software can be very costly, either in terms of losing money, reputation, or even lives in case of some critical applications. Consequently, there is a high and always ...
Integrating Testing into Software Engineering Courses Supported by a Collaborative Learning Environment

As software becomes more ubiquitous and complex, the cost of software bugs continues to grow at a staggering rate. To remedy this situation, there needs to be major improvement in the knowledge and application of software validation techniques. Although ...
A survey of coverage based testing tools
AST '06: Proceedings of the 2006 international workshop on Automation of software test

Test coverage is sometimes used as a way to measure how thoroughly software is tested. Coverage is used by software developers and sometimes by vendors to indicate their confidence in the readiness of their software. This survey studies and compares 17 ...

Comments

Information & Contributors

Information

Published In

cover image ACM Transactions on Computing Education

ACM Transactions on Computing Education Volume 17, Issue 4

December 2017

123 pages

EISSN:1946-6226

DOI:10.1145/3134765

Editor:
Chris Hundhausen
Washington State University, USA

Issue’s Table of Contents

Copyright © 2017 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 26 August 2017

Accepted: 01 December 2016

Revised: 01 August 2016

Received: 01 April 2015

Published in TOCE Volume 17, Issue 4

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed

Funding Sources

National Science Foundation

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

11
Total Citations
View Citations
586
Total Downloads

Downloads (Last 12 months)138
Downloads (Last 6 weeks)15

Reflects downloads up to 30 Aug 2024

Other Metrics

View Author Metrics

Citations

Cited By

Shin AKazerouni A(2023)A Model of How Students Engineer Test Cases With FeedbackACM Transactions on Computing Education10.1145/362860424:1(1-31)Online publication date: 20-Oct-2023
https://dl.acm.org/doi/10.1145/3628604
Ardic BZaidman A(2023)Hey Teachers, Teach Those Kids Some Software Testing2023 IEEE/ACM 5th International Workshop on Software Engineering Education for the Next Generation (SEENG)10.1109/SEENG59157.2023.00007(9-16)Online publication date: May-2023
https://doi.org/10.1109/SEENG59157.2023.00007
Blanco RTrinidad MSuárez-Cabal MCalderón ARuiz MTuya J(2023)Can gamification help in software testing education? Findings from an empirical studyJournal of Systems and Software10.1016/j.jss.2023.111647200:COnline publication date: 1-Jun-2023
https://dl.acm.org/doi/10.1016/j.jss.2023.111647
Muramatsu MNoguchi YKogure SYamashita KKonishi TItoh Y(2023)Classroom Practice with Learning Support System for Program Design Using Mock Technique Based on TestabilitySN Computer Science10.1007/s42979-023-02096-24:5Online publication date: 17-Aug-2023
https://dl.acm.org/doi/10.1007/s42979-023-02096-2
De Lira CWong RAdesope O(2022)A systematic review on the effectiveness of programming camps on middle school students' programming knowledge and attitudes of computingJournal of Computing Sciences in Colleges10.5555/3575618.357562738:1(89-98)Online publication date: 1-Nov-2022
https://dl.acm.org/doi/10.5555/3575618.3575627
Ramasamy VNarasareddygari MWalia GAllen ADuke DKiper JDavis D(2022)A Multi-institutional Analysis of CS1 Students’ Common Misconceptions of Key Programming ConceptsHandbook on Artificial Intelligence-Empowered Applied Software Engineering10.1007/978-3-031-07650-3_8(127-144)Online publication date: 18-Oct-2022
https://doi.org/10.1007/978-3-031-07650-3_8
Clarke PDavis DBuckley IPotvin GThirunarayanan MJones E(2021)Combining Learning and Engagement Strategies in a Software Testing Learning EnvironmentACM Transactions on Computing Education10.1145/346913122:2(1-25)Online publication date: 1-Nov-2021
https://dl.acm.org/doi/10.1145/3469131
Clarke PThirunarayanan MAllala SSotomayor JRoss M(2020)Experiences of Integrating Learning and Engagement Strategies (LESs) into Software Engineering Courses2020 ASEE Virtual Annual Conference Content Access Proceedings10.18260/1-2--34630Online publication date: Jun-2020
https://doi.org/10.18260/1-2--34630
Alomari HRamasamy VKiper JPotvin G(2020)A User Interface (UI) and User eXperience (UX) evaluation framework for cyberlearning environments in computer science and software engineering educationHeliyon10.1016/j.heliyon.2020.e039176:5(e03917)Online publication date: May-2020
https://doi.org/10.1016/j.heliyon.2020.e03917
Paschoal Lde Souza SMalucelli AReinehr S(2018)A Survey on Software Testing Education in BrazilProceedings of the XVII Brazilian Symposium on Software Quality10.1145/3275245.3275289(334-343)Online publication date: 17-Oct-2018
https://dl.acm.org/doi/10.1145/3275245.3275289
Show More Cited By

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

Media

Figures

Other

Tables

View Issue’s Table of Contents