article

Test-Driven Development in scientific software: a survey

Authors:

Aziz Nanthaamornphong,

Jeffrey C. CarverAuthors Info & Claims

Software Quality Journal, Volume 25, Issue 2

Pages 343 - 372

https://doi.org/10.1007/s11219-015-9292-4

Published: 01 June 2017 Publication History

Abstract

Scientific software developers are increasingly employing various software engineering practices. Specifically, scientists are beginning to use Test-Driven Development (TDD). Even with this increasing use of TDD, the effect of TDD on scientific software development is not fully understood. To help scientific developers determine whether TDD is appropriate for their scientific projects, we surveyed scientific developers who use TDD to understand: (1) TDDs effectiveness, (2) the benefits and challenges of using TDD, and (3) the use of refactoring practices (an important part of the TDD process). Some key positive results include: (1) TDD helps scientific developers increase software quality, in particular functionality and reliability; and (2) TDD helps scientific developers reduce the number of problems in the early phase of projects. Conversely, some key challenges include: (1) TDD may not be effective for all types of scientific projects; and (2) Writing a good test is the most difficult task in TDD, particularly in a parallel computing environment. To summarize, TDD generally has a positive effect on the quality of scientific software, but it often requires a large effort investment. The results of this survey indicate the need for additional empirical evaluation of the use of TDD for the development of scientific software to help organizations make better decisions.

References

[1]

Abdel-Hamid, A. (2013). Refactoring as a lifeline: Lessons learned from refactoring. In Agile Conference (AGILE), 2013, pp. 129-136

Digital Library

[2]

Anselm, L. S., & Juliet, M. C. (1990). Basics of qualitative research: Rounded theory procedures and techniques. Newbury Park, CA: Sage Publications.

[3]

Beck, K. (2002). Test driven development: By example. Boston, MA: Addison-Wesley Longman Publishing Co. Inc.

[4]

Beck, K., & Andres, C. (2004). Extreme programming explained: Embrace change (2nd ed.). Boston, MA: Addison-Wesley Professional.

[5]

Carver, J. (2011). Development of a mesh generation code with a graphical front-end: A case study. Journal of End User Computing, 23(4), 1-16.

Digital Library

[6]

Carver, J. C., Kendall, R. P., Squires, S. E., & Post, D. E. (2007). Software development environments for scientific and engineering software: A series of case studies. In The 29th international conference on software engineering (pp. 550-559). MN: Minneapolis.

[7]

Causevic, A., Sundmark, D., & Punnekkat, S. (2011). Factors limiting industrial adoption of test driven development: A systematic review. In The 4th international conference on software testing (pp. 337-346). Berlin: Verification and Validation.

[8]

Desai, C., Janzen, D., & Savage, K. (2008). A survey of evidence for Test-Driven Development in academia. SIGCSE Bulletin, 40(2), 97-101.

Digital Library

[9]

Eclipse. (2013). Photran--An integrated development environment and refactoring tool for fortran. http://www.eclipse.org/photran/. Accessed December 2013

[10]

Erdogmus, H., Morisio, M., & Torchiano, M. (2005). On the effectiveness of the test-first approach to programming. IEEE Transactions on Software Engineering, 31(3), 226-237.

Digital Library

[11]

Fowler, M. (1999). Refactoring: Improving the design of existing code. Boston, MA: Addison-Wesley Longman Publishing Co. Inc.

[12]

Gamma, E., Helm, R., Johnson, R., & Vlissides, J. (1995). Design patterns: Elements of reusable object-oriented software. Boston, MA: Addison-Wesley Longman Publishing Co. Inc.

[13]

ISO IEC. (2011). Systems and software engineering: System and software quality requirements and evaluation (SQuaRE)--System and software quality models. ISO/IEC, 25010, 2011.

[14]

Janzen, D., & Saiedian, H. (2005). Test-Driven Development concepts, taxonomy, and future direction. Computer, 38(9), 43-50.

Digital Library

[15]

Kollanus, S. (2010). Test-Driven Development--Still a promising approach? In Proceedings of the 7th international conference on the quality of information and communications technology (pp. 403-408). Portugal: Porto.

[16]

Koskela, L. (2007). Test driven: Practical TDD and acceptance TDD for java developers. Greenwich, CO: Manning Publications Co.

[17]

Mens, T., & Tourwé, T. (2004). A survey of software refactoring. IEEE Transactions on Software Engineering, 30(2), 126-139.

Digital Library

[18]

Nanthaamornphong, A., Morris, K., Rouson, D., & Michelsen, H. (2013). A case study: Agile development in the community laser-induced incandescence modeling environment (CLiiME). In The 5th international workshop on software engineering for computational science and engineering (pp. 9-18).

[19]

Nanthaamornphong, A., Carver, J., Morris, K., Michelsen, H., & Rouson, D. (2014). Building cliime via Test-Driven Development: A case study. Computing in Science Engineering, 16(3), 36-46.

[20]

Opdyke, W. F. (1992). Refactoring object-oriented frameworks. PhD thesis, University of Illinois at Urbana-Champaign, Champaign, Illinois, USA.

[21]

Orchard, D., & Rice, A. (2013). Upgrading fortran source code using automatic refactoring. Proceedings of the international workshop on refactoring tools (pp. 29-32). Indiana: Indianapolis.

[22]

Overbey, J., Xanthos, S., Johnson, R., & Foote, B. (2005). Refactorings for fortran and high-performance computing. In Proceedings of the 2nd international workshop on software engineering for high performance computing system applications (pp. 37-39). Missouri: St. Louis.

[23]

Overbey, J. L., Negara, S., & Johnson, R. E. (2009). Refactoring and the evolution of fortran. In Proceedings of the international workshop on software engineering for computational science and engineering (pp. 28-34). British Columbia: Vancouver.

[24]

Rafique, Y., & Misic, V. (2013). The effects of Test-Driven Development on external quality and productivity: A meta-analysis. IEEE Transactions on Software Engineering, 39(6), 835-856.

Digital Library

[25]

Ruparelia, N. B. (2010). Software development lifecycle models. SIGSOFT Software Engineering Notes, 35(3), 8-13.

Digital Library

[26]

Sanchez, J., Williams, L., Maximilien, E. (2007). On the sustained use of a Test-Driven Development practice at ibm. In Agile conference (AGILE), 2007 (pp 5-14)

Digital Library

[27]

Sanders, R., & Kelly, D. (2008). Dealing with risk in scientific software development. IEEE Software, 25(4), 21-28.

Digital Library

[28]

Sletholt, M., Hannay, J., Pfahl, D., & Langtangen, H. (2012). What do we know about scientific software development's agile practices? Computing in Science Engineering, 14(2), 24-37.

Digital Library

[29]

Sletholt, M. T., Hannay, J., Pfahl, D., Benestad, H. C., & Langtangen, H. P. (2011). A literature review of agile practices and their effects in scientific software development. In Proceedings of the 4th international workshop on software engineering for computational science and engineering (pp. 1-9). Hawaii: Honolulu.

Cited By

Zakeri-Nasrabadi MParsa SJafari S(2024)Measuring and improving software testability at the design levelInformation and Software Technology10.1016/j.infsof.2024.107511174:COnline publication date: 1-Oct-2024
https://dl.acm.org/doi/10.1016/j.infsof.2024.107511
Golmohammadi RParsa SZakeri-Nasrabadi M(2024)Dynamic domain testing with multi-agent Markov chain Monte Carlo methodSoft Computing - A Fusion of Foundations, Methodologies and Applications10.1007/s00500-024-09680-528:13-14(8293-8317)Online publication date: 1-Jul-2024
https://dl.acm.org/doi/10.1007/s00500-024-09680-5
Khoshniat NJamarani AAhmadzadeh AHaghi Kashani MMahdipour E(2024)Nature-inspired metaheuristic methods in software testingSoft Computing - A Fusion of Foundations, Methodologies and Applications10.1007/s00500-023-08382-828:2(1503-1544)Online publication date: 1-Jan-2024
https://dl.acm.org/doi/10.1007/s00500-023-08382-8
Show More Cited By

Test-Driven Development in scientific software: a survey
1. Software and its engineering
  1. Software creation and management
    1. Software verification and validation
      1. Software defect analysis

Recommendations

Test-Driven Development: Concepts, Taxonomy, and Future Direction

Test-driven development creates software in very short iterations with minimal upfront design. Poised for widespread adoption TDD has become the focus of an increasing number of researchers and developers.
A survey of scientific software development
ESEM '10: Proceedings of the 2010 ACM-IEEE International Symposium on Empirical Software Engineering and Measurement

Software for scientific research purposes has received increased attention in recent years. Case studies have noted development practices, limitations, and problems in the development of scientific software. However, applicability of the results of ...
Learning Test-Driven Development by Counting Lines

Test-driven development is an agile development practice that changes every minute of developers' daily lives. That's a big change! How can you best train developers in such a pervasive practice? One method is to write tests for counting lines of code. ...

Comments

Information & Contributors

Information

Published In

cover image Software Quality Journal

Software Quality Journal Volume 25, Issue 2

June 2017

252 pages

ISSN:0963-9314

Issue’s Table of Contents

Copyright © Copyright © 2017 Springer Science+Business Media New York.

Publisher

Kluwer Academic Publishers

United States

Publication History

Published: 01 June 2017

Author Tags

Qualifiers

Article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

4
Total Citations
View Citations
0
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 17 Oct 2024

Other Metrics

View Author Metrics

Citations

Cited By

Zakeri-Nasrabadi MParsa SJafari S(2024)Measuring and improving software testability at the design levelInformation and Software Technology10.1016/j.infsof.2024.107511174:COnline publication date: 1-Oct-2024
https://dl.acm.org/doi/10.1016/j.infsof.2024.107511
Golmohammadi RParsa SZakeri-Nasrabadi M(2024)Dynamic domain testing with multi-agent Markov chain Monte Carlo methodSoft Computing - A Fusion of Foundations, Methodologies and Applications10.1007/s00500-024-09680-528:13-14(8293-8317)Online publication date: 1-Jul-2024
https://dl.acm.org/doi/10.1007/s00500-024-09680-5
Khoshniat NJamarani AAhmadzadeh AHaghi Kashani MMahdipour E(2024)Nature-inspired metaheuristic methods in software testingSoft Computing - A Fusion of Foundations, Methodologies and Applications10.1007/s00500-023-08382-828:2(1503-1544)Online publication date: 1-Jan-2024
https://dl.acm.org/doi/10.1007/s00500-023-08382-8
Vogel TDruskat SScheidgen MDraxl CGrunske L(2019)Challenges for verifying and validating scientific software in computational materials scienceProceedings of the 14th International Workshop on Software Engineering for Science10.1109/SE4Science.2019.00010(25-32)Online publication date: 28-May-2019
https://dl.acm.org/doi/10.1109/SE4Science.2019.00010
Paine DLee C(2017)"Who Has Plots?"Proceedings of the ACM on Human-Computer Interaction10.1145/31347201:CSCW(1-21)Online publication date: 6-Dec-2017
https://dl.acm.org/doi/10.1145/3134720

View Options

View options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Media

Figures

Other

Tables

View Issue’s Table of Contents