research-article

Why Research on Test-Driven Development is Inconclusive?

Authors:

Mohammad Ghafari,

Michael FeldererAuthors Info & Claims

ESEM '20: Proceedings of the 14th ACM / IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM)

Article No.: 25, Pages 1 - 10

https://doi.org/10.1145/3382494.3410687

Published: 23 October 2020 Publication History

Abstract

[Background] Recent investigations into the effects of Test-Driven Development (TDD) have been contradictory and inconclusive. This hinders development teams to use research results as the basis for deciding whether and how to apply TDD. [Aim] To support researchers when designing a new study and to increase the applicability of TDD research in the decision-making process in industrial context, we aim at identifying the reasons behind the inconclusive research results in TDD. [Method] We studied the state of the art in TDD research published in top venues in the past decade, and analyzed the way these studies were set up. [Results] We identified five categories of factors that directly impact the outcome of studies on TDD. [Conclusions] This work can help researchers to conduct more reliable studies, and inform practitioners of risks they need to consider when consulting research on TDD.

References

[1]

Steve Bannerman and Andrew Martin. 2011. A multiple comparative study of test-with development product changes and their effects on team speed and product quality. Empirical Software Engineering 16 (2011), 177--210. https://doi.org/10.1007/s10664-010-9137-5

Digital Library

[2]

Kent Beck. 1999. Extreme Programming Explained: Embrace Change. Addison-Wesley Longman Publishing Co., Inc., Boston, MA, USA.

[3]

Kent Beck. 2002. Test-Driven Development By Example. Addison-Wesley Longman, Amsterdam.

[4]

Moritz Beller, Georgios Gousios, Annibale Panichella, Sebastian Proksch, Sven Amann, and Andy Zaidman. 2019. Developer Testing in the IDE: Patterns, Beliefs, and Behavior. IEEE Transactions on Software Engineering 45, 3 (2019), 261--284. https://doi.org/10.1109/TSE.2017.2776152

Digital Library

[5]

Wilson Bissi, Adolfo Neto, and Maria Emer. 2016. The effects of test driven development on internal quality, external quality and productivity: A systematic review. Information and Software Technology 74 (2016), 45--54. https://doi.org/10.1016/j.infsof.2016.02.004

Digital Library

[6]

Neil Borle, Meysam Feghhi, Eleni Stroulia, Russ Greiner, and Abram Hindle. 2018. Analyzing the effects of test driven development in GitHub. Empirical Software Engineering 23, 4 (2018), 1931--1958.

Digital Library

[7]

Jim Buchan, Ling Li, and Stephen G Macdonell. 2011. Causal Factors, Benefits and Challenges of Test-Driven Development: Practitioner Perceptions. 2011 18th Asia-Pacific Software Engineering Conference (2011), 405--413. https://doi.org/10.1109/APSEC.2011.44

[8]

Adnan Causevic, Daniel Sundmark, and Sasikumar Punnekkat. 2011. Factors Limiting Industrial Adoption of Test Driven Development: A Systematic Review. Proceedings - 4th IEEE International Conference on Software Testing, Verification, and Validation, ICST 2011 (2011), 337--346. https://doi.org/10.1109/ICST.2011.19

Digital Library

[9]

Daniela S Cruzes, Tore Dybå, Per Runeson, and Martin Höst. 2015. Case studies synthesis: a thematic, cross-case, and narrative synthesis worked example. Empirical Software Engineering 20, 6 (2015), 1634--1665.

Digital Library

[10]

Tomaž Dogša and David Batic. 2011. The effectiveness of test-driven development: An industrial case study. Software Quality Journal 19 (2011), 643--661. https://doi.org/10.1007/s11219-011-9130-2

Digital Library

[11]

Davide Falessi, Natalia Juristo, Claes Wohlin, Burak Turhan, Jürgen Münch, Andreas Jedlitschka, and Markku Oivo. 2018. Empirical software engineering experts on the use of students and professionals in experiments. Empirical Software Engineering 23, 1 (2018), 452--489.

Digital Library

[12]

Robert Feldt, Thomas Zimmermann, Gunnar R. Bergersen, Davide Falessi, Andreas Jedlitschka, Natalia Juristo, Jürgen Münch, Markku Oivo, Per Runeson, Martin Shepperd, Dag I. K. Sjøberg, and Burak Turhan. 2018. Four commentaries on the use of students and professionals in empirical software engineering experiments. Empirical Software Engineering 23, 6 (2018), 3801--3820. https://doi.org/10.1007/s10664-018-9655-0

Digital Library

[13]

Uwe Flick. 2009. An Introduction to Qualitative Research. SAGE Publications.

[14]

Davide Fucci, Hakan Erdogmus, Burak Turhan, Markku Oivo, and Natalia Juristo. 2017. A Dissection of the Test-Driven Development Process: Does It Really Matter to Test-First or to Test-Last? IEEE Transactions on Software Engineering 43, 7 (2017), 597--614.

Digital Library

[15]

Davide Fucci, Simone Romano, Maria Teresa Baldassarre, Danilo Caivano, Giuseppe Scanniello, Burak Turhan, and Natalia Juristo. 2018. A Longitudinal Cohort Study on the Retainment of Test-Driven Development. Proceedings of the 12th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement, ESEM 2018 (2018). https://doi.org/10.1145/3239235.3240502

Digital Library

[16]

Davide Fucci and Burak Turhan. 2013. A Replicated Experiment on the Effectiveness of Test-First Development. In 2013 ACM/IEEE International Symposium on Empirical Software Engineering and Measurement, Baltimore, Maryland, USA, October 10-11, 2013. IEEE, 103--112. https://doi.org/10.1109/ESEM.2013.15

[17]

Mohammad Ghafari, Markus Eggiman, and Oscar Nierstrasz. 2019. Testability First!. In 2019 ACM/IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM). 1--6.

[18]

Christina Hansson, Yvonne Dittrich, Björn Gustafsson, and Stefan Zarnak. 2006. How agile are industrial software development practices? Journal of systems and software 79, 9 (2006), 1295--1311.

Digital Library

[19]

Theodore D Hellmann, Abhishek Sharma, Jennifer Ferreira, and Frank Maurer. 2012. Agile Testing: Past, Present, and Future-Charting a Systematic Map of Testing in Agile Software Development. In 2012 Agile Conference. IEEE, 55--63.

Digital Library

[20]

Martin Ivarsson and Tony Gorschek. 2011. A Method for Evaluating Rigor and Industrial Relevance of Technology Evaluations. Empirical Softw. Engg. 16, 3 (June 2011), 365--395. https://doi.org/10.1007/s10664-010-9146-4

Digital Library

[21]

Daniel Kahneman. 2015. Thinking, fast and slow. Farrar, Straus and Giroux.

[22]

Itir Karac and Burak Turhan. 2018. What Do We ( Really ) Know about Test-Driven Development ? IEEE Software 35 (2018), 81--85. https://doi.org/10.1109/MS.2018.2801554

[23]

Itir Karac, Burak Turhan, and Natalia Juristo. 2019. A Controlled Experiment with Novice Developers on the Impact of Task Description Granularity on Software Quality in Test-Driven Development. IEEE Transactions on Software Engineering (2019).

[24]

Ayaan M. Kazerouni, Clifford A. Shaffer, Stephen H. Edwards, and Francisco Servant. 2019. Assessing Incremental Testing Practices and Their Impact on Project Outcomes. In Proceedings of the 50th ACM Technical Symposium on Computer Science Education. ACM, New York, NY, USA, 407--413. https://doi.org/10.1145/3287324.3287366

Digital Library

[25]

Sami Kollanus. 2010. Test-Driven Development - Still a Promising Approach?. In 2010 Seventh International Conference on the Quality of Information and Communications Technology. IEEE, 403--408. https://doi.org/10.1109/QUATIC.2010.73

Digital Library

[26]

Roberto Latorre. 2014. Effects of Developer Experience on Learning and Applying Unit Test-Driven Development. IEEE Transactions on Software Engineering 40, 4 (2014), 381--395. https://doi.org/10.1109/TSE.2013.2295827

Digital Library

[27]

Lech Madeyski. 2010. Test-driven development: an empirical evaluation of agile practice. Springer-Verlag, Heidelberg New York.

[28]

Lech Madeyski. 2010. The impact of Test-First programming on branch coverage and mutation score indicator of unit tests: An experiment. Information and Software Technology 52, 2 (2010), 169--184. https://doi.org/10.1016/j.infsof.2009.08.007

Digital Library

[29]

Artem Marchenko, Pekka Abrahamsson, and Tuomas Ihme. 2009. Long-Term Effects of Test-Driven Development A Case Study. (2009), 13--22.

[30]

Hussan Munir, Misagh Moayyed, and Kai Petersen. 2014. Considering rigor and relevance when evaluating test driven development: A systematic review. Information and Software Technology 56, 4 (2014), 375--394. https://doi.org/10.1016/j.infsof.2014.01.002

Digital Library

[31]

Jeff Offutt. 2018. Why don't we publish more TDD research papers? Software Testing, Verification and Reliability 28, 4 (2018), e1670. https://doi.org/10.1002/stvr.1670 arXiv:https://www.onlinelibrary.wiley.com/doi/pdf/10.1002/sttvr.1670e1670 STVR-18-0033.

[32]

Matjaž Pančur and Mojca Ciglaric. 2011. Impact of test-driven development on productivity, code and tests: A controlled experiment. Information and Software Technology 53, 6 (2011), 557--573. https://doi.org/10.1016/j.infsof.2011.02.002

Digital Library

[33]

Yahya Rafique and Vojislav Misic. 2013. The Effects of Test-Driven Development on External Quality and Productivity: A Meta-Analysis. IEEE Transactions on Software Engineering 39, 6 (2013), 835--856. https://doi.org/10.1109/TSE.2012.28

Digital Library

[34]

Simone Romano, Davide Fucci, Maria Teresa Baldassarre, Danilo Caivano, and Giuseppe Scanniello. 2019. An Empirical Assessment on Affective Reactions of Novice Developers When Applying Test-Driven Development. In Product-Focused Software Process Improvement, Xavier Franch, Tomi Männistö, and Silverio Martínez-Fernández (Eds.). Springer International Publishing, Cham, 3--19.

[35]

Simone Romano, Davide Fucci, Giuseppe Scanniello, Burak Turhan, and Natalia Juristo. 2017. Findings from a multi-method study on test-driven development. Information and Software Technology 89 (2017), 64--77. https://doi.org/10.1016/j.infsof.2017.03.010

[36]

Adrian Santos, Janne Järvinen, Jari Partanen, Markku Oivo, and Natalia Juristo. 2018. Does the Performance of TDD Hold Across Software Companies and Premises? A Group of Industrial Experiments on TDD. In Product-Focused Software Process Improvement. Springer International Publishing, Cham, 227--242.

[37]

Adrian Santos, Jaroslav Spisak, Markku Oivo, and Natalia Juristo. 2018. Improving Development Practices through Experimentation: An Industrial TDD Case. In 25th Asia-Pacific Software Engineering Conference, APSEC 2018, Nara, Japan, December 4-7, 2018. 465--473. https://doi.org/10.1109/APSEC.2018.00061

[38]

Giuseppe Scanniello, Simone Romano, Davide Fucci, Burak Turhan, and Natalia Juristo. 2016. Students' and Professionals' Perceptions of Test-driven Development: A Focus Group Study. In Proceedings of the 31st Annual ACM Symposium on Applied Computing (SAC '16). ACM, New York, NY, USA, 1422--1427. https://doi.org/10.1145/2851613.2851778

Digital Library

[39]

Forrest Shull, Grigori Melnik, Burak Turhan, Lucas Layman, Madeline Diep, and Hakan Erdogmus. 2010. What Do We Know about Test-Driven Development? IEEE Softw. 27, 6 (Nov. 2010), 16--19. https://doi.org/10.1109/MS.2010.152

Digital Library

[40]

Maria Siniaalto. 2006. Test driven development: empirical body of evidence. Agile Software Development of Embedded Systems (2006).

[41]

Stripe.com. 2018. The Developer Coefficient. Technical Report. https://stripe.com/files/reports/the-developer-coefficient.pdf, last accessed on 15.12.2019.

[42]

Hussein Suleman, Stephan Jamieson, and Maria Keet. 2017. Testing Test-Driven Development. In ICT Education, Janet Liebenberg and Stefan Gruner (Eds.). Springer International Publishing, Cham, 241--248.

[43]

Anders Sundelin, Javier Gonzalez-Huerta, and Krzysztof Wnuk. 2018. Test-Driving FinTech Product Development: An Experience Report. In Product-Focused Software Process Improvement. Springer International Publishing, Cham, 219--226.

[44]

Christopher D Thomson, Mike Holcombe, and Anthony J H Simons. 2009. What Makes Testing Work: Nine Case Studies of Software Development Teams. In 2009 Testing: Academic and Industrial Conference - Practice and Research Techniques. 167--175. https://doi.org/10.1109/TAICPART.2009.12

[45]

Ayse Tosun, Muzamil Ahmed, Burak Turhan, and Natalia Juristo. 2018. On the Effectiveness of Unit Tests in Test-Driven Development. In Proceedings of the 2018 International Conference on Software and System Process (ICSSP '18). Association for Computing Machinery, New York, NY, USA, 113--122. https://doi.org/10.1145/3202710.3203153

Digital Library

[46]

Ayse Tosun, Oscar Dieste, Sira Vegas, Dietmar Pfahl, Kerli Rungi, and Natalia Juristo. 2019. Investigating the Impact of Development Task on External Quality in Test-Driven Development: An Industry Experiment. IEEE Transactions on Software Engineering (2019).

[47]

Burak Turhan, Lucas Layman, Madeline Diep, Hakan Erdogmus, and Forrest Shull. 2010. How Effective is Test-Driven Development? In Making Software. Number October. O'Reilly Media, 624.

Cited By

Karac IPanach JTurhan BJuristo N(2025)Does Treatment Adherence Impact Experiment Results in TDD?IEEE Transactions on Software Engineering10.1109/TSE.2024.349733251:1(135-152)Online publication date: 1-Jan-2025
https://dl.acm.org/doi/10.1109/TSE.2024.3497332
Parsa SZakeri-Nasrabadi MTurhan B(2025)Testability-driven developmentComputer Standards & Interfaces10.1016/j.csi.2024.10387791:COnline publication date: 1-Jan-2025
https://dl.acm.org/doi/10.1016/j.csi.2024.103877
Mock MMelegati JRusso B(2025)Generative AI for Test Driven Development: Preliminary ResultsAgile Processes in Software Engineering and Extreme Programming – Workshops10.1007/978-3-031-72781-8_3(24-32)Online publication date: 11-Jan-2025
https://doi.org/10.1007/978-3-031-72781-8_3
Show More Cited By

Index Terms

Why Research on Test-Driven Development is Inconclusive?
1. Software and its engineering
  1. Software creation and management
    1. Software development process management
      1. Software development methods
        Agile software development
    2. Software verification and validation
      1. Software defect analysis
        Software testing and debugging

Recommendations

An experimental evaluation of test driven development vs. test-last development with industry professionals
EASE '14: Proceedings of the 18th International Conference on Evaluation and Assessment in Software Engineering

Test-Driven Development (TDD) is a software development approach where test cases are written before actual development of the code in iterative cycles. Context: TDD has gained attention of many software practitioners during the last decade since it has ...
Test-Driven Development: Concepts, Taxonomy, and Future Direction

Test-driven development creates software in very short iterations with minimal upfront design. Poised for widespread adoption TDD has become the focus of an increasing number of researchers and developers.
Test-Driven Development of Relational Databases

Developers can use a test-driven development with database schema just as they use it with application code. Implementing test-driven database development (TDDD) involves three relatively simple steps: database refactoring, database regression testing, ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

ESEM '20: Proceedings of the 14th ACM / IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM)

October 2020

412 pages

ISBN:9781450375801

DOI:10.1145/3382494

Copyright © 2020 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 23 October 2020

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed limited

Conference

ESEM '20

Sponsor:

SIGSOFT

ESEM '20: ACM / IEEE International Symposium on Empirical Software Engineering and Measurement

October 5 - 9, 2020

Bari, Italy

Acceptance Rates

ESEM '20 Paper Acceptance Rate 26 of 123 submissions, 21%;

Overall Acceptance Rate 130 of 594 submissions, 22%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

11
Total Citations
View Citations
414
Total Downloads

Downloads (Last 12 months)86
Downloads (Last 6 weeks)7

Reflects downloads up to 08 Feb 2025

Other Metrics

View Author Metrics

Citations

Cited By

Karac IPanach JTurhan BJuristo N(2025)Does Treatment Adherence Impact Experiment Results in TDD?IEEE Transactions on Software Engineering10.1109/TSE.2024.349733251:1(135-152)Online publication date: 1-Jan-2025
https://dl.acm.org/doi/10.1109/TSE.2024.3497332
Parsa SZakeri-Nasrabadi MTurhan B(2025)Testability-driven developmentComputer Standards & Interfaces10.1016/j.csi.2024.10387791:COnline publication date: 1-Jan-2025
https://dl.acm.org/doi/10.1016/j.csi.2024.103877
Mock MMelegati JRusso B(2025)Generative AI for Test Driven Development: Preliminary ResultsAgile Processes in Software Engineering and Extreme Programming – Workshops10.1007/978-3-031-72781-8_3(24-32)Online publication date: 11-Jan-2025
https://doi.org/10.1007/978-3-031-72781-8_3
Marabesi MGarcía-Holgado AGarcía-Peñalvo F(2024)Exploring the Connection between the TDD Practice and Test Smells—A Systematic Literature ReviewComputers10.3390/computers1303007913:3(79)Online publication date: 18-Mar-2024
https://doi.org/10.3390/computers13030079
Lemos OSilveira FFerrari FSilva TGuerra EGarcia A(2024)Unraveling the code: an in-depth empirical study on the impact of development practices in auxiliary functions implementationSoftware Quality Journal10.1007/s11219-024-09682-432:3(1137-1174)Online publication date: 1-Sep-2024
https://dl.acm.org/doi/10.1007/s11219-024-09682-4
Moedt van Bolhuis WBernsteiner RHall MFruhling A(2023)Enhancing IoT Project Success through Agile Best PracticesACM Transactions on Internet of Things10.1145/35681704:1(1-31)Online publication date: 23-Feb-2023
https://dl.acm.org/doi/10.1145/3568170
Esposito MRomano SScanniello G(2023)Test-Driven Development and Embedded Systems: An Exploratory Investigation2023 49th Euromicro Conference on Software Engineering and Advanced Applications (SEAA)10.1109/SEAA60479.2023.00045(239-246)Online publication date: 6-Sep-2023
https://doi.org/10.1109/SEAA60479.2023.00045
Knight DTorri SBhowmik T(2023)A Preliminary Critical Review of the Impact of Three Popular Development Practices on Source Code Maintainability2023 IEEE 47th Annual Computers, Software, and Applications Conference (COMPSAC)10.1109/COMPSAC57700.2023.00252(1633-1637)Online publication date: Jun-2023
https://doi.org/10.1109/COMPSAC57700.2023.00252
Ernst NBaldassarre M(2023)Registered reports in software engineeringEmpirical Software Engineering10.1007/s10664-022-10277-528:2Online publication date: 11-Mar-2023
https://dl.acm.org/doi/10.1007/s10664-022-10277-5
Romano SZampetti FBaldassarre MDi Penta MScanniello G(2022)Do Static Analysis Tools Affect Software Quality when Using Test-driven Development?Proceedings of the 16th ACM / IEEE International Symposium on Empirical Software Engineering and Measurement10.1145/3544902.3546233(80-91)Online publication date: 19-Sep-2022
https://dl.acm.org/doi/10.1145/3544902.3546233
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten