abstract

Finding and breaking test dependencies to speed up test execution

Author:

Sebastian KapplerAuthors Info & Claims

FSE 2016: Proceedings of the 2016 24th ACM SIGSOFT International Symposium on Foundations of Software Engineering

Pages 1136 - 1138

https://doi.org/10.1145/2950290.2983974

Published: 01 November 2016 Publication History

Abstract

Software testing takes up the major part of the build time, which hinders developers' ability to promptly identify and fix problems.

Test parallelization is an effective means to speed up test executions, hence improving software development. Effective and sound test parallelization requires that tests are independent or that test dependencies are known in advance. However, current techniques to detect test dependencies are either precise but slow, or fast but inaccurate. Further, available algorithms for test parallelization either over-constraint test executions, which reduces their level of parallelism, or re-execute the same tests multiple times, which increases the execution effort.

This research addresses both sides of the problem of speeding up test execution: it aims to devise a practical test detection technique that can suitably balance efficiency and accuracy, and develop a novel technique to break test dependencies which allows both sound and efficient test executions.

References

[1]

Github - cloudera/dist test. https://github.com/cloudera/dist test.

[2]

Github - grosser/parallel tests. https://github.com/grosser/parallel tests.

[3]

Jenkins and cloudbees. https: //www.cloudbees.com/jenkins/jenkins-cloudbees.

[4]

Junit 4 ordered testcases execution example - howtodoinjava. http://howtodoinjava.com/junit/ ordered-testcases-execution-in-junit-4/.

[5]

Maven surefire plugin. http: //maven.apache.org/surefire/maven-surefire-plugin/.

[6]

Testng dependent tests tutorial and examples. http://howtodoinjava.com/testng/ testng-dependent-tests-examples/.

[7]

S. Elbaum, A. G. Malishevsky, and G. Rothermel. Prioritizing test cases for regression testing. In Proceedings of the International Symposium on Software Testing and Analysis, ISSTA 00, pages 102–112, 2000.

Digital Library

[8]

S. Elbaum, G. Rothermel, and J. Penix. Techniques for improving regression testing in continuous integration development environments. In Proceedings of the International Symposium on Foundations of Software Engineering, FSE 14, pages 235–245, 2014.

Digital Library

[9]

M. Gligoric, L. Eloussi, and D. Marinov. Practical regression test selection with dynamic file dependencies. In Proceedings of the International Symposium on Software Testing and Analysis, ISSTA 15, pages 211–222, 2015.

Digital Library

[10]

W. Lam, S. Zhang, and M. D. Ernst. When tests collide: Evaluating and coping with the impact of test dependence. Technical Report UW-CSE-15-03-01, University of Washington Department of Computer Science and Engineering, Seattle, WA, USA, Mar. 2015.

[11]

D. Marijan, A. Gotlieb, and S. Sen. Test case prioritization for continuous regression testing: An industrial case study. In Proceedings of the International Conference on Software Maintenance, ICSM 13, pages 540–543, 2013.

Digital Library

[12]

A. Shi, A. Gyori, M. Gligoric, A. Zaytsev, and D. Marinov. Balancing trade-offs in test-suite reduction. In Proceedings of the International Symposium on Foundations of Software Engineering, FSE 14, pages 246–256, 2014.

Digital Library

[13]

A. Shi, T. Yung, A. Gyori, and D. Marinov. Comparing and combining test-suite reduction and regression test selection. In Proceedings Joint Meeting on Foundations of Software Engineering, ESEC/FSE 15, pages 237–247, 2015.

Digital Library

[14]

S. Yoo and M. Harman. Regression testing minimization, selection and prioritization: a survey. Software Testing, Verification & Reliability, 22(2):67–120, 2012.

Digital Library

[15]

S. Zhang, D. Jalali, J. Wuttke, K. Mu¸slu, W. Lam, M. D. Ernst, and D. Notkin. Empirically revisiting the test independence assumption. In Proceedings of the International Symposium on Software Testing and Analysis, ISSTA 14, pages 385–396, 2014.

Digital Library

[16]

Introduction and Motivation Background and Related Work Approach and Results Conclusions and Ongoing Work References

Cited By

Bagies TLe WSheaffer JJannesari A(2023)Reducing branch divergence to speed up parallel execution of unit testing on GPUsThe Journal of Supercomputing10.1007/s11227-023-05375-079:16(18340-18374)Online publication date: 13-May-2023
https://doi.org/10.1007/s11227-023-05375-0
Bluemke IMalanowska A(2021)Software Testing Effort Estimation and Related ProblemsACM Computing Surveys10.1145/344269454:3(1-38)Online publication date: 17-Apr-2021
https://dl.acm.org/doi/10.1145/3442694
Bagies TJannesari A(2021)An Empirical Study of Parallelizing Test Execution Using CUDA Unified Memory and OpenMP GPU Offloading2021 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW)10.1109/ICSTW52544.2021.00052(271-278)Online publication date: Apr-2021
https://doi.org/10.1109/ICSTW52544.2021.00052
Show More Cited By

Index Terms

Finding and breaking test dependencies to speed up test execution
1. Software and its engineering
  1. Software creation and management
    1. Software verification and validation
      1. Software defect analysis
        Software testing and debugging

Recommendations

Efficient dependency detection for safe Java test acceleration
ESEC/FSE 2015: Proceedings of the 2015 10th Joint Meeting on Foundations of Software Engineering

Slow builds remain a plague for software developers. The frequency with which code can be built (compiled, tested and packaged) directly impacts the productivity of developers: longer build times mean a longer wait before determining if a change to the ...
Empirically revisiting the test independence assumption
ISSTA 2014: Proceedings of the 2014 International Symposium on Software Testing and Analysis

In a test suite, all the test cases should be independent: no test should affect any other test’s result, and running the tests in any order should produce the same test results. Techniques such as test prioritization generally assume that the tests in ...
Repairing test dependence
FSE 2016: Proceedings of the 2016 24th ACM SIGSOFT International Symposium on Foundations of Software Engineering

In a test suite, all the tests should be independent: no test should affect another test's result, and running the tests in any order should yield the same test results. The assumption of such test independence is important so that tests behave ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

FSE 2016: Proceedings of the 2016 24th ACM SIGSOFT International Symposium on Foundations of Software Engineering

November 2016

1156 pages

ISBN:9781450342186

DOI:10.1145/2950290

General Chair:
Thomas Zimmermann
Microsoft Research, USA
,
Program Chairs:
Jane Cleland-Huang
University of Notre Dame, USA
,
Zhendong Su
University of California at Davis, USA

Copyright © 2016 Owner/Author.

Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for third-party components of this work must be honored. For all other uses, contact the Owner/Author.

Sponsors

SIGSOFT: ACM Special Interest Group on Software Engineering

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 01 November 2016

Check for updates

Author Tags

Qualifiers

Abstract

Conference

FSE'16

Sponsor:

SIGSOFT

FSE'16: 24nd ACM SIGSOFT International Symposium on the Foundations of Software Engineering

November 13 - 18, 2016

WA, Seattle, USA

Acceptance Rates

Overall Acceptance Rate 17 of 128 submissions, 13%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

9
Total Citations
View Citations
201
Total Downloads

Downloads (Last 12 months)3
Downloads (Last 6 weeks)0

Reflects downloads up to 16 Feb 2025

Other Metrics

View Author Metrics

Citations

Cited By

Bagies TLe WSheaffer JJannesari A(2023)Reducing branch divergence to speed up parallel execution of unit testing on GPUsThe Journal of Supercomputing10.1007/s11227-023-05375-079:16(18340-18374)Online publication date: 13-May-2023
https://doi.org/10.1007/s11227-023-05375-0
Bluemke IMalanowska A(2021)Software Testing Effort Estimation and Related ProblemsACM Computing Surveys10.1145/344269454:3(1-38)Online publication date: 17-Apr-2021
https://dl.acm.org/doi/10.1145/3442694
Bagies TJannesari A(2021)An Empirical Study of Parallelizing Test Execution Using CUDA Unified Memory and OpenMP GPU Offloading2021 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW)10.1109/ICSTW52544.2021.00052(271-278)Online publication date: Apr-2021
https://doi.org/10.1109/ICSTW52544.2021.00052
Biagiola MStocco AMesbah ARicca FTonella PDumas MPfahl DApel SRusso A(2019)Web test dependency detectionProceedings of the 2019 27th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering10.1145/3338906.3338948(154-164)Online publication date: 12-Aug-2019
https://dl.acm.org/doi/10.1145/3338906.3338948
Mondal SNasre R(2019) MahtabJournal of Systems and Software10.1016/j.jss.2019.110403158:COnline publication date: 1-Dec-2019
https://dl.acm.org/doi/10.1016/j.jss.2019.110403
Vahabzadeh AStocco AMesbah AChaudron MCrnkovic IChechik MHarman M(2018)Fine-grained test minimizationProceedings of the 40th International Conference on Software Engineering10.1145/3180155.3180203(210-221)Online publication date: 27-May-2018
https://dl.acm.org/doi/10.1145/3180155.3180203
Gambi ABell JZeller A(2018)Practical Test Dependency Detection2018 IEEE 11th International Conference on Software Testing, Verification and Validation (ICST)10.1109/ICST.2018.00011(1-11)Online publication date: Apr-2018
https://doi.org/10.1109/ICST.2018.00011
Griscom MIba T(2017)MetaAutomationProceedings of the 24th Conference on Pattern Languages of Programs10.5555/3290281.3290313(1-34)Online publication date: 23-Oct-2017
https://dl.acm.org/doi/10.5555/3290281.3290313
Gambi AKappler SLampel JZeller ABultan TSen K(2017)CUT: automatic unit testing in the cloudProceedings of the 26th ACM SIGSOFT International Symposium on Software Testing and Analysis10.1145/3092703.3098222(364-367)Online publication date: 10-Jul-2017
https://dl.acm.org/doi/10.1145/3092703.3098222

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten