article

Studying the co-evolution of production and test code in open source and industrial developer test processes through repository mining

Authors:

Serge DemeyerAuthors Info & Claims

Empirical Software Engineering, Volume 16, Issue 3

Pages 325 - 364

https://doi.org/10.1007/s10664-010-9143-7

Published: 01 June 2011 Publication History

Abstract

Many software production processes advocate rigorous development testing alongside functional code writing, which implies that both test code and production code should co-evolve. To gain insight in the nature of this co-evolution, this paper proposes three views (realized by a tool called TeMo) that combine information from a software project's versioning system, the size of the various artifacts and the test coverage reports. We validate these views against two open source and one industrial software project and evaluate our results both with the help of log messages, code inspections and the original developers of the software system. With these views we could recognize different co-evolution scenarios (i.e., synchronous and phased) and make relevant observations for both developers as well as test engineers.

References

[1]

Adams B, Jiang ZM, Hassan AE (2010) Identifying crosscutting concerns using historical code changes. In: Proceedings of the international conference on software engineering (ICSE). ACM, New York, pp 305-314.

[2]

Ant (2010) http://ant.apache.org. Website last visited: 4 August 2010.

[3]

Bachmann A, Bernstein A (2010) When process data quality affects the number of bugs: correlations in software engineering datasets. In: Proceedings of the 7th international working conference on mining software repositories (MSR). IEEE Computer Society, Washington, pp 62-71.

[4]

Ball T, Eick SG (1996) Software visualization in the large. IEEE Comput 29(4):33-43.

Digital Library

[5]

Ball T, Kim J, Porter A, Siy H (1997) If your version control system could talk. In: ICSE workshop on process modelling and empirical studies of software engineering. Boston, USA.

[6]

Beck K (1999) Extreme programming explained: embrace change. Addison Wesley, Reading, MA.

[7]

Beck K (2003) Test-driven development: by example. Addison-Wesley, Reading, MA.

[8]

Berner S, Weber R, Keller RK (2007) Enhancing software testing by judicious use of code coverage information. In: Proceedings of the international conference on software engineering (ICSE). IEEE Computer Society, Los Alamitos, pp 612-620.

[9]

Beyer D, Hassan AE (2006) Animated visualization of software history using evolution storyboards. In: Proceedings of the working conference on reverse engineering (WCRE). IEEE Computer Society, Washington, pp 199-210.

[10]

Beyer D, Noack A (2005) Clustering software artifacts based on frequent common changes. In: Proceedings of the international workshop on program comprehension (IWPC). IEEE Computer Society, Washington, pp 259-268.

[11]

Bhat T, Nagappan N (2006) Evaluating the efficacy of test-driven development: industrial case studies. In: Proceedings of the international symposium on empirical software engineering (ISESE). ACM, New York, pp 356-363.

[12]

Bible J, Rothermel G, Rosenblum D (2001) A comparative study of coarse- and fine-grained safe regression test selection. ACM Trans Softw Eng Methodol 10(2):149-183.

Digital Library

[13]

Binder R (2000) Testing object-oriented systems: models, patterns, and tools. Addison-Wesley, Reading, MA.

[14]

Bouktif S, Guéhéneuc YG, Antoniol G (2006) Extracting change-patterns from cvs repositories. In: Proceedings of the working conference on reverse engineering (WCRE). IEEE Computer Society, Washington, pp 221-230.

[15]

Breu S, Zimmermann T (2006) Mining aspects from version history. In: Proceedings of the international conference on automated software engineering (ASE). IEEE Computer Society, Washington, pp 221-230.

[16]

Bruntink M, van Deursen A (2006) An empirical study into class testability. J Syst Softw 79(9):1219- 1232.

Digital Library

[17]

Clover (2010) http://www.atlassian.com/software/clover/. Website last visited: 4 August 2010.

[18]

Cornelissen B, Zaidman A, van Deursen A, Van Rompaey B (2009) Trace visualization for program comprehension: a controlled experiment. In: Proceedings of the international conference on program comprehension (ICPC). IEEE Computer Society, Washington, pp 100-109.

[19]

D'Ambros M, Gall H, Lanza M, Pinzger M (2008) Analyzing software repositories to understand software evolution. In: Mens T, Demeyer S (eds) Software evolution. Springer, Berlin, pp 39-70.

[20]

D'Ambros M, Lanza M, Lungu M (2009) Visualizing co-change information with the evolution radar. IEEE Trans Softw Eng 35(5):720-735.

Digital Library

[21]

Demeyer S, Ducasse S, Nierstrasz O (2002) Object-oriented reengineering patterns. Morgan Kaufmann, San Francisco.

[22]

Elbaum S, Gable D, Rothermel G (2001) The impact of software evolution on code coverage information. In: Proceedings of the international conference on software maintenance (ICSM). IEEE Computer Society, Washington, pp 170-179.

[23]

Emma (2010) http://emma.sourceforge.net/. Website last visited: 4 August 2010.

[24]

Feathers M (2005) Working effectively with legacy code. Prentice Hall.

[25]

Fewster M, Graham D (1999) Software Test Automation: effective use of test execution tools. ACM Press/Addison-Wesley Publishing Co, New York.

[26]

Fluri B, Würsch M, Gall H (2007) Do code and comments co-evolve? On the relation between source code and comment changes. In: Proceedings of the working conference on reverse engineering (WCRE). IEEE Computer Society, Washington, pp 70-79.

[27]

Gaelli M, Lanza M, Nierstrasz O, Wuyts R (2004) Ordering broken unit tests for focused debugging. In: Proceedings of the international conference on software maintenance (ICSM). IEEE Computer Society, Washington, pp 114-123.

[28]

Gall H, Hajek K, Jazayeri M (1998) Detection of logical coupling based on product release history. In: Proceedings of the international conference on software maintenance (ICSM). IEEE Computer Society, Washington, pp 190-197.

[29]

Gamma E, Beck K (1998) Test infected: programmers love writing tests. Java Rep 3(7):37-50.

[30]

Gîrba T, Ducasse S (2006) Modeling history to analyze software evolution. J Softw Maint 18(3):207- 236.

Digital Library

[31]

Glover A (2006) In pursuit of code quality: don't be fooled by the coverage report. http://www. ibm.com/developerworks/java/library/j-cq01316/. Last visited: 4 August 2010.

[32]

gnuplot (2010) http://www.gnuplot.info. Website last visited: 4 August 2010.

[33]

Godfrey M, Tu Q (2000) Evolution in open source software: a case study. In: Proceedings of the international conference on software maintenance (ICSM). IEEE Computer Society, Washington, pp 131-142.

[34]

Hassan AE, Mockus A, Holt RC, Johnson PM (2005) Special issue on mining software repositories. IEEE Trans Softw Eng 31(6):426-428.

Digital Library

[35]

Hindle A, Godfrey M, Holt R (2007a) Release pattern discovery: a case study of database systems. In: Proceedings of the international conference on software maintenance (ICSM). IEEE Computer Society, Washington, pp 285-294.

[36]

Hindle A, Godfrey M, Holt R (2007b) Release pattern discovery via partitioning: methodology and case study. In: Proceedings of the international workshop on mining software repositories (MSR). IEEE Computer Society, Washington, pp 19-26.

[37]

JFreeChart (2010) http://www.jfree.org/jfreechart/. Website last visited: 4 August 2010.

[38]

Kagdi H, Collard ML, Maletic JI (2007) A survey and taxonomy of approaches for mining software repositories in the context of software evolution. J Softw Maint 19(2):77-131.

Digital Library

[39]

Kagdi H, Maletic J, Sharif B (2007) Mining software repositories for traceability links. In: Proceedings of the international conference on program comprehension (ICPC). IEEE Computer Society, Washington, pp 145-154.

[40]

Kanstrén T (2008) Towards a deeper understanding of test coverage. J Softw Maint 20(1):59-76.

Digital Library

[41]

Lanza M (2001) The evolution matrix: recovering software evolution using software visualization techniques. In: Proceedings of the international workshop on principles of software evolution (IWPSE). ACM, pp 37-42.

[42]

Lanza M (2010) The visual terminator. In: Proceedings of the 7th international working conference on mining software repositories (MSR). IEEE Computer Society, http://www.slideshare. net/michele.lanza/the-visual-terminator.

[43]

Lanza M, Ducasse S (2002) Understanding software evolution using a combination of software visualization and software metrics. In: Proceedings of LMO 2002 (Languages et Modèles à Objets). Hermes Publications, pp 135-149.

[44]

Lanza M, Ducasse S (2003) Polymetric views--a lightweight visual approach to reverse engineering. IEEE Trans Softw Eng 29(9):782-795.

Digital Library

[45]

Lehman M (1980) On understanding laws, evolution and conservation in the large program life cycle. J Syst Softw 1(3):213-221.

[46]

LOCC (2010) http://csdl.ics.hawaii.edu/Plone/research/locc/. Website last visited: 4 August 2010.

[47]

Lubsen Z, Zaidman A, Pinzger M (2009) Using association rules to study the co-evolution of production & test code. In: Proceedings of the 6th international working conference on mining software repositories (MSR). IEEE Computer Society, Washington, pp 151-154.

[48]

Luijten B, Visser J, Zaidman A (2010) Assessment of issue handling efficiency. In: Proceedings of the international working conference on mining software repositories (MSR). IEEE Computer Society, Washington, pp 94-97.

[49]

Mathur AP (2008) Foundations of software testing. Pearson Education.

[50]

Maven (2010) http://maven.apache.org. Website last visited: 4 August 2010.

[51]

Maximilien E, Williams L (2003) Assessing test-driven development at IBM. In: Proceedings of the international conference on software engineering (ICSE). IEEE Computer Society, Washington, pp 564-569.

[52]

Mens T, Wermelinger M, Ducasse S, Demeyer S, Hirschfeld R, Jazayeri M (2005) Challenges in software evolution. In: Proceedings of the international workshop on principles of software evolution (IWPSE). IEEE Computer Society, Washington, pp 13-22.

[53]

Meszaros G (2007) xUnit test patterns: refactoring test code. Addison-Wesley, Reading, MA.

[54]

Moonen L, van Deursen A, Zaidman A, Bruntink M (2008) The interplay between software testing and software evolution. In: Mens T, Demeyer S (eds) Software evolution. Springer, Berlin, pp 173-202.

[55]

Mulder F, Zaidman A (2010) Identifying cross-cutting concerns using software repository mining. In: Proceedings of the 4th international joint ERCIM/IWPSE symposium on software evolution (IWPSE-EVOL). ACM (To appear).

[56]

Prevalence and incidence of color blindness (2010) http://www.wrongdiagnosis.com/c/color_ blindness/prevalence.htm. Website last visited: 4 August 2010.

[57]

Ratzinger J, Fischer M, Gall H (2005) Evolens: lens-view visualizations of evolution data. In: Proceedings of the international workshop on principles of software evolution (IWPSE). IEEE Computer Society, Washington, pp 103-112.

Digital Library

[58]

Runeson P (2006) A survey of unit testing practices. IEEE Softw 25(4):22-29.

Digital Library

[59]

Runeson P, Höst M (2009) Guidelines for conducting and reporting case study research in software engineering. Empir Software Eng 14(2):131-164.

Digital Library

[60]

Sangwan R, Laplante P (2006) Test-driven development in large projects. IT Pro 8(5):25-29.

Digital Library

[61]

Schwaber K, Beedle M (2002) Agile software development with Scrum. Prentice Hall, Upper Saddle River.

[62]

Siniaalto M, Abrahamsson P (2007) A comparative case study on the impact of test-driven development on program design and test coverage. In: Proceedings of the first international symposium on empirical software engineering and measurement (ESEM). IEEE Computer Society, Washington, pp 275-284.

Digital Library

[63]

Slaughter SA, Harter DE, Krishnan MS (1998) Evaluating the cost of software quality. Commun ACM 41(8):67-73.

Digital Library

[64]

Storey MA, ¿ubranic D, German D (2005) On the use of visualization to support awareness of human activities in software development: a survey and a framework. In: Proceedings of the symposium on software visualization. ACM, New York, pp 193-202.

[65]

SVNKit (2010) http://svnkit.com/. Website last visited: 4 August 2010.

[66]

Tassey G (2002) Economic impacts of inadequate infrastructure for software testing. Planning Report 02-3, National Institute of Standards and Technology (NIST).

[67]

Telea A, Auber D (2008) Code flows: visualizing structural evolution of source code. Comput Graph Forum 27(3):831-838.

Digital Library

[68]

Van Rompaey B, Demeyer S (2008) Estimation of test code changes using historical release data. In: Proceedings of the working conference on reverse engineering (WCRE). IEEE Computer Society, Washington, pp 269-278.

[69]

Van Rompaey B, Du Bois B, Demeyer S, Rieger M (2007) On the detection of test smells: a metrics-based approach for general fixture and eager test. IEEE Trans Softw Eng 33(12): 800-817.

Digital Library

[70]

Van Rysselberghe F, Demeyer S (2004) Studying software evolution information by visualizing the change history. In: Proceedings of the international conference on software maintenance (ICSM). IEEE Computer Society, Washington, pp 328-337.

[71]

Voinea L, Telea A (2006) An open framework for cvs repository querying, analysis and visualization. In: Proceedings of the international workshop on mining software repositories (MSR). ACM, New York, pp 33-39.

[72]

Voinea L, Telea A (2007) Visual data mining and analysis of software repositories. Comput Graph 31(3):410-428.

Digital Library

[73]

Voinea L, Telea A, van Wijk JJ (2005) Cvsscan: visualization of code evolution. In: Proceedings of the ACM symposium on software visualization (SOFTVIS). ACM, pp 47-56.

[74]

Wu J, Holt RC, Hassan AE (2004) Exploring software evolution using spectographs. In: Proceedings of the working conference on reverse engineering (WCRE). IEEE Computer Society, Washington, pp 80-89.

[75]

XQuery (2010) http://www.w3.org/TR/xquery/. Website last visited: 4 August 2010.

[76]

Yamaura T (1998) How to design practical test cases. IEEE Softw 15(6):30-36.

Digital Library

[77]

Yin RK (2002) Case study research: design and methods, 3rd edn. Sage Publications.

[78]

Zaidman A, Van Rompaey B, Demeyer S, van Deursen A (2008) Mining software repositories to study co-evolution of production & test code. In: Proceedings of the international conference on software testing, verification, and validation (ICST). IEEE Computer Society, Washington, pp 220-229.

[79]

Zhu H, Hall PA, May J (1997) Software unit test coverage and adequacy. ACM Comput Surv 29(4):366-427.

Digital Library

[80]

Zimmermann T, Kim S, Whitehead J, Zeller A (2006) Mining version archives for co-changed lines. In: Proceedings of the international workshop on mining software repositories (MSR). ACM, New York, pp 72-75.

Cited By

Sterk AWessel MHooten EZaidman ASaadatmand MLonetti FBudnik CLi JGuerriero A(2024)Running a Red Light: An Investigation into Why Software Engineers (Occasionally) Ignore Coverage ChecksProceedings of the 5th ACM/IEEE International Conference on Automation of Software Test (AST 2024)10.1145/3644032.3644444(12-22)Online publication date: 15-Apr-2024
https://dl.acm.org/doi/10.1145/3644032.3644444
Sun WGuo ZYan MLiu ZLei YZhang H(2024)Method-Level Test-to-Code Traceability Link Construction by Semantic Correlation LearningIEEE Transactions on Software Engineering10.1109/TSE.2024.344991750:10(2656-2676)Online publication date: 1-Oct-2024
https://dl.acm.org/doi/10.1109/TSE.2024.3449917
van Dinten IDerakhshanfar PPanichella AZaidman A(2024)The slow and the furious? Performance antipattern detection in Cyber–Physical SystemsJournal of Systems and Software10.1016/j.jss.2023.111904210:COnline publication date: 1-Apr-2024
https://dl.acm.org/doi/10.1016/j.jss.2023.111904
Show More Cited By

Studying the co-evolution of production and test code in open source and industrial developer test processes through repository mining
1. Software and its engineering
  1. Software creation and management
    1. Software verification and validation
      1. Software defect analysis

Recommendations

Excluding code from test coverage: practices, motivations, and impact
Abstract
Test coverage measures the percentage of code that is covered (and uncovered) by tests. In practice, not all code is equally important for coverage analysis, like code that will not be executed during tests. Some coverage tools provide support for ...
Mining Software Repositories to Study Co-Evolution of Production & Test Code
ICST '08: Proceedings of the 2008 International Conference on Software Testing, Verification, and Validation

Engineering software systems is a multidisciplinary activity, whereby a number of artifacts must be created — and maintained — synchronously. In this paper we investigatewhether production code and the accompanying tests coevolveby exploring a project’s ...
Combined Source Code Approach for Test Case Prioritization
ICISS '18: Proceedings of the 1st International Conference on Information Science and Systems

Regression testing is an activity in the software testing process to ensure the software is validated and verified after modification occurred on software. It is costly process procedure which has been expected to reach half cost of the software ...

Comments

Information & Contributors

Information

Published In

cover image Empirical Software Engineering

Empirical Software Engineering Volume 16, Issue 3

June 2011

129 pages

ISSN:1382-3256

Issue’s Table of Contents

Copyright © Copyright © 2011 Springer Science+Business Media, LLC.

Publisher

Kluwer Academic Publishers

United States

Publication History

Published: 01 June 2011

Author Tags

Qualifiers

Article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

55
Total Citations
View Citations
0
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 17 Feb 2025

Other Metrics

View Author Metrics

Citations

Cited By

Sterk AWessel MHooten EZaidman ASaadatmand MLonetti FBudnik CLi JGuerriero A(2024)Running a Red Light: An Investigation into Why Software Engineers (Occasionally) Ignore Coverage ChecksProceedings of the 5th ACM/IEEE International Conference on Automation of Software Test (AST 2024)10.1145/3644032.3644444(12-22)Online publication date: 15-Apr-2024
https://dl.acm.org/doi/10.1145/3644032.3644444
Sun WGuo ZYan MLiu ZLei YZhang H(2024)Method-Level Test-to-Code Traceability Link Construction by Semantic Correlation LearningIEEE Transactions on Software Engineering10.1109/TSE.2024.344991750:10(2656-2676)Online publication date: 1-Oct-2024
https://dl.acm.org/doi/10.1109/TSE.2024.3449917
van Dinten IDerakhshanfar PPanichella AZaidman A(2024)The slow and the furious? Performance antipattern detection in Cyber–Physical SystemsJournal of Systems and Software10.1016/j.jss.2023.111904210:COnline publication date: 1-Apr-2024
https://dl.acm.org/doi/10.1016/j.jss.2023.111904
Chen ZJia CChen L(2024)Evaluating test quality of Python libraries for IoT applications at the network edgeWireless Networks10.1007/s11276-023-03479-230:7(6603-6618)Online publication date: 1-Oct-2024
https://dl.acm.org/doi/10.1007/s11276-023-03479-2
Winkler DUrbanke PRamler R(2024)Investigating the readability of test codeEmpirical Software Engineering10.1007/s10664-023-10390-z29:2Online publication date: 26-Feb-2024
https://dl.acm.org/doi/10.1007/s10664-023-10390-z
Asthana SSajnani HVoyloshnikova EAcharya BHerzig KChandra SBlincoe KTonella P(2023)A Case Study of Developer Bots: Motivations, Perceptions, and ChallengesProceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering10.1145/3611643.3616248(1268-1280)Online publication date: 30-Nov-2023
https://dl.acm.org/doi/10.1145/3611643.3616248
Liu LWang SLiu YDeng JLiu S(2023)Drift: Fine-Grained Prediction of the Co-Evolution of Production and Test Code via Machine LearningProceedings of the 14th Asia-Pacific Symposium on Internetware10.1145/3609437.3609449(227-237)Online publication date: 4-Aug-2023
https://dl.acm.org/doi/10.1145/3609437.3609449
Sun WYan MLiu ZXia XLei YLo D(2023)Revisiting the Identification of the Co-evolution of Production and Test CodeACM Transactions on Software Engineering and Methodology10.1145/360718332:6(1-37)Online publication date: 30-Sep-2023
https://dl.acm.org/doi/10.1145/3607183
Wang XXiao LYu TWoepse AWong S(2023)From Inheritance to Mockito: An Automatic Refactoring ApproachIEEE Transactions on Software Engineering10.1109/TSE.2022.323185049:4(2791-2814)Online publication date: 1-Apr-2023
https://dl.acm.org/doi/10.1109/TSE.2022.3231850
Chen AChen TWang S(2023)T-Evos: A Large-Scale Longitudinal Study on CI Test Execution and FailureIEEE Transactions on Software Engineering10.1109/TSE.2022.321826449:4(2352-2365)Online publication date: 1-Apr-2023
https://dl.acm.org/doi/10.1109/TSE.2022.3218264
Show More Cited By

View Options

View options

Figures

Tables

Media

View Issue’s Table of Contents