research-article

Test execution checkpointing for web applications

Authors:

Marco Guarnieri,

Mohammad Torabi Dashti,

David BasinAuthors Info & Claims

ISSTA 2017: Proceedings of the 26th ACM SIGSOFT International Symposium on Software Testing and Analysis

Pages 203 - 214

https://doi.org/10.1145/3092703.3092710

Published: 10 July 2017 Publication History

Abstract

Test isolation is a prerequisite for the correct execution of test suites on web applications. We present Test Execution Checkpointing, a method for efficient test isolation. Our method instruments web applications to support checkpointing and exploits this support to isolate and optimize tests. We have implemented and evaluated this method on five popular PHP web applications. The results show that our method not only provides test isolation essentially for free, it also reduces testing time by 44% on average.

References

[1]

Derek Allard. Bamboo Invoice. http://github.com/derekallard/BambooInvoice (Retrieved: January 2016).

[2]

Anneliese A. Andrews, Jeff Offutt, and Roger T. Alexander. 2005. Testing web applications by modeling with FSMs. Software and Systems Modeling 4, 3 (2005), 326–345.

Digital Library

[3]

Jonathan Bell. 2014. Detecting, isolating, and enforcing dependencies among and within test cases. In Proceedings of the 22nd ACM SIGSOFT International Symposium on Foundations of Software Engineering. ACM, 799–802.

Digital Library

[4]

Jonathan Bell and Gail Kaiser. 2014. Unit test virtualization with VMVM. In Proceedings of the 36th International Conference on Software Engineering. ACM, 550–561.

Digital Library

[5]

Jonathan Bell, Gail Kaiser, Eric Melski, and Mohan Dattatreya. 2015. Efficient dependency detection for safe Java test acceleration. In Proceedings of the 2015 10th Joint Meeting on Foundations of Software Engineering. ACM, 770–781.

Digital Library

[6]

Francesco Bolis, Angelo Gargantini, Marco Guarnieri, and Eros Magri. 2012. Evolutionary testing of PHP web applications with WETT. In International Symposium on Search Based Software Engineering. Springer, 285–291.

Digital Library

[7]

Francesco Bolis, Angelo Gargantini, Marco Guarnieri, Eros Magri, and Lorenzo Musto. 2012. Model-driven testing for web applications using abstract state machines. In Current Trends in Web Engineering: ICWE 2012 International Workshops. Springer, 71–78.

Digital Library

[8]

Greg Bronevetsky, Daniel Marques, Keshav Pingali, and Paul Stodghill. 2003. Automated Application-level Checkpointing of MPI Programs. In Proceedings of the 9th ACM SIGPLAN Symposium on Principles and Practice of Parallel Programming (PPoPP ’03). ACM, New York, NY, USA, 84–94. https://

Digital Library

[9]

781513

[10]

Greg Bronevetsky, Daniel Marques, Keshav Pingali, Peter Szwed, and Martin Schulz. 2004. Application-level Checkpointing for Shared Memory Programs. In Proceedings of the 11th International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS XI). ACM, New York, NY, USA, 235–247. https://

Digital Library

[11]

Tristan Buchs. 2015. Checkpointing-based testing. Master’s thesis. ETH Zurich, Switzerland.

[12]

Tsong Yueh Chen and Man Fai Lau. 1998. A new heuristic for test suite reduction. Information and Software Technology 40, 5-6 (1998), 347–354.

[13]

Tsong Yueh Chen and Man Fai Lau. 1998. A simulation study on some heuristics for test suite reduction. Information and Software Technology 40, 13 (1998), 777–787.

[14]

Jong-Deok Choi and Harini Srinivasan. 1998. Deterministic Replay of Java Multithreaded Applications. In Proceedings of the SIGMETRICS Symposium on Parallel and Distributed Tools (SPDT ’98). ACM, New York, NY, USA, 48–59. https://

Digital Library

[15]

CRIU. Checkpoint/Restore In Userspace. http://criu.org (Retrieved: January 2016).

[16]

Pranavadatta Devaki, Suresh Thummalapenta, Nimit Singhania, and Saurabh Sinha. 2013. Efficient and Flexible GUI Test Execution via Test Merging. In Proceedings of the 2013 International Symposium on Software Testing and Analysis (ISSTA 2013). ACM, New York, NY, USA, 34–44. https://

Digital Library

[17]

2483781

[18]

E. N. Elnozahy, D. B. Johnson, and W. Zwaenepoel. 1992. The performance of consistent checkpointing. In Proceedings 11th Symposium on Reliable Distributed Systems. 39–47. https://

[19]

Elmootazbellah N. Elnozahy and James S. Plank. 2004. Checkpointing for Peta-Scale Systems: A Look into the Future of Practical Rollback-Recovery. IEEE Transactions on Dependable and Secure Computing 1, 2 (2004), 97–108.

Digital Library

[20]

Sequel: The Database Toolkit for Ruby. Testing with Sequel. http://sequel. jeremyevans.net/rdoc/files/doc/testing_rdoc.html (Retrieved: January 2016).

[21]

Gordon Fraser and Franz Wotawa. 2007. Redundancy based test-suite reduction. In Proceedings of the 10th International Conference on Fundamental Approaches to Software Engineering (FASE 2007). 291–305.

Digital Library

[22]

Gallery. Gallery - Your photos on your website. http://galleryproject.org/ (Retrieved: January 2016).

[23]

Alex Gyori, August Shi, Farah Hariri, and Darko Marinov. 2015. Reliable Testing: Detecting State-polluting Tests to Prevent Test Dependency. In Proceedings of the 2015 International Symposium on Software Testing and Analysis (ISSTA 2015). ACM, New York, NY, USA, 223–233. https://

Digital Library

[24]

Dan Hao, Lu Zhang, Xingxia Wu, Hong Mei, and Gregg Rothermel. 2012. Ondemand Test Suite Reduction. In Proceedings of the 34th International Conference on Software Engineering (ICSE ’12). IEEE Press, Piscataway, NJ, USA, 738–748.

Digital Library

[25]

M Jean Harrold, Rajiv Gupta, and Mary Lou Soffa. 1993. A methodology for controlling the size of a test suite. ACM Transactions on Software Engineering and Methodology (TOSEM) 2, 3 (1993), 270–285.

Digital Library

[26]

Mats PE Heimdahl and Devaraj George. 2004. Test-suite reduction for model based tests: Effects on test quality and implications for testing. In Proceedings of the 19th IEEE International Conference on Automated Software Engineering. IEEE, 176–185.

Digital Library

[27]

Dennis Jeffrey and Neelam Gupta. 2007. Improving fault detection capability by selectively retaining test cases during test suite reduction. IEEE Transactions on software Engineering 33, 2 (2007).

Digital Library

[28]

James A Jones and Mary Jean Harrold. 2003. Test-suite reduction and prioritization for modified condition/decision coverage. IEEE Transactions on software Engineering 29, 3 (2003), 195–209.

Digital Library

[29]

S. A. Khalek and S. Khurshid. 2011. Efficiently Running Test Suites Using Abstract Undo Operations. In Proceedings of the 22nd IEEE International Symposium on Software Reliability Engineering. 110–119. https://

Digital Library

[30]

Donald E. Knuth. 1998. The Art of Computer Programming (2nd ed.). Addison-Wesley Longman Publishing Co., Inc., Boston, MA, USA.

Digital Library

[31]

Berkeley Lab. Berkeley Lab Checkpoint/Restart (BLCR) for LINUX. http://crd.lbl. gov/departments/computer-science/CLaSS/research/BLCR/ (Retrieved: January 2016).

[32]

Wing Lam, Sai Zhang, and Michael D. Ernst. 2015. When tests collide: Evaluating and coping with the impact of test dependence. Technical Report UW-CSE-15-03- 01. University of Washington Department of Computer Science and Engineering, Seattle, WA, USA.

[33]

J. L. Lawall and G. Muller. 2000. Efficient incremental checkpointing of Java programs. In Proceedings of the 2000 International Conference on Dependable Systems and Networks. 61–70. https://

Digital Library

[34]

LibCkpt. A Portable Checkpointer for Unix. http://web.eecs.utk.edu/plank/plank/ www/libckpt.html (Retrieved: January 2016).

[35]

Giuseppe Antonio Di Lucca, Anna Rita Fasolino, Francesco Faralli, and Ugo De Carlini. 2002. Testing web applications. In ICSM.

[36]

Qingzhou Luo, Farah Hariri, Lamyaa Eloussi, and Darko Marinov. 2014. An Empirical Analysis of Flaky Tests. In Proceedings of the 22nd ACM SIGSOFT International Symposium on Foundations of Software Engineering (FSE 2014). ACM, New York, NY, USA, 643–653. https://

Digital Library

[37]

Gerard Meszaros. 2007. xUnit test patterns: Refactoring test code. Pearson Education.

Digital Library

[38]

James Mickens, Jeremy Elson, and Jon Howell. 2010. Mugshot: Deterministic Capture and Replay for JavaScript Applications. In 7th USENIX Symposium on Networked Systems Design and Implementation (NSDI 10). USENIX Association, San Jose, CA. https://www.usenix.org/conference/nsdi10-0/ mugshot-deterministic-capture-and-replay-javascript-applications

Digital Library

[39]

Microsoft MSDN. How to: Write a SQL Server Unit Test that Runs within the Scope of a Single Transaction. http://msdn.microsoft.com/en-US/library/ jj851217(v=vs.103).aspx (Retrieved: January 2016).

[40]

Kivanç Muşlu, Bilge Soran, and Jochen Wuttke. 2011. Finding Bugs by Isolating Unit Tests. In Proceedings of the 19th ACM SIGSOFT Symposium and the 13th European Conference on Foundations of Software Engineering (ESEC/FSE ’11). ACM, New York, NY, USA, 496–499. https://

Digital Library

[41]

OSCommerce. OSCommerce - Creating Online Stores Worldwide. http://www. oscommerce.com/ (Retrieved: January 2016).

[42]

phpBB. phpBB - Free and Open Source Forum Software. http://www.phpbb.com/ (Retrieved: January 2016).

[43]

Feng Qin, Joseph Tucek, Jagadeesan Sundaresan, and Yuanyuan Zhou. 2005. Rx: Treating Bugs As Allergies—a Safe Method to Survive Software Failures. In Proceedings of the 20th ACM Symposium on Operating Systems Principles (SOSP ’05). ACM, New York, NY, USA, 235–248. https://

Digital Library

[44]

Filippo Ricca and Paolo Tonella. 2001. Analysis and Testing of Web Applications. In Proceedings of the 23rd International Conference on Software Engineering (ICSE ’01). IEEE Computer Society, Washington, DC, USA, 25–34.

[45]

Gregg Rothermel, Mary Jean Harrold, Jeffery Ostrin, and Christie Hong. 1998. An empirical study of the effects of minimization on the fault detection capabilities of test suites. In Proceedings of the 14th International Conference on Software Maintenance. IEEE, 34–43.

Digital Library

[46]

Sreedevi Sampath, Valentin Mihaylov, Amie Souter, and Lori Pollock. 2004. Composing a Framework to Automate Testing of Operational Web-Based Software. In Proceedings of the 20th IEEE International Conference on Software Maintenance (ICSM ’04). IEEE Computer Society, Washington, DC, USA, 104–113.

Digital Library

[47]

Sara Sprenkle, Emily Gibson, Sreedevi Sampath, and Lori Pollock. 2005. Automated Replay and Failure Detection for Web Applications. In Proceedings of the 20th IEEE/ACM International Conference on Automated Software Engineering (ASE ’05). ACM, New York, NY, USA, 253–262. https://

Digital Library

[48]

Sriraman Tallam and Neelam Gupta. 2005. A Concept Analysis Inspired Greedy Algorithm for Test Suite Minimization. In Proceedings of the 6th ACM SIGPLANSIGSOFT Workshop on Program Analysis for Software Tools and Engineering (PASTE ’05). ACM, New York, NY, USA, 35–42. https://

Digital Library

[49]

tSQLt. tSQLt – Database Unit Testing for SQL Server. http://tsqlt.org/ (Retrieved: January 2016).

[50]

A. Vahabzadeh, A. M. Fard, and A. Mesbah. 2015. An empirical study of bugs in test code. In 2015 IEEE International Conference on Software Maintenance and Evolution (ICSME). 101–110. https://

Digital Library

[51]

w3af. Open Source Web Application Security Scanner. http://w3af.org/ (Retrieved: January 2016). ISSTA’17, July 2017, Santa Barbara, CA, USA M. Guarnieri, P. Tsankov, T. Buchs, M. Torabi Dashti, and D. Basin

[52]

W. Eric Wong, Joseph R. Horgan, Saul London, and Hira Agrawal Bellcore. 1997. A Study of Effective Regression Testing in Practice. In Proceedings of the 8th International Symposium on Software Reliability Engineering (ISSRE ’97). IEEE Computer Society, Washington, DC, USA, 264–274.

Digital Library

[53]

Wordpress. Wordpress - Blog, Tool, Publishing Platform, and CMS. http://wordpress.org/ (Retrieved: January 2016).

[54]

Guoqing Xu, Atanas Rountev, Yan Tang, and Feng Qin. 2007. Efficient Checkpointing of Java Software Using Context-sensitive Capture and Replay. In Proceedings of the the 6th Joint Meeting of the European Software Engineering Conference and the ACM SIGSOFT Symposium on The Foundations of Software Engineering (ESEC-FSE ’07). ACM, New York, NY, USA, 85–94. https: //

Digital Library

[55]

Sai Zhang, Darioush Jalali, Jochen Wuttke, Kç Muşlu, Wing Lam, Michael D. Ernst, and David Notkin. 2014. Empirically Revisiting the Test Independence Assumption. In Proceedings of the 2014 International Symposium on Software Testing and Analysis (ISSTA 2014). ACM, New York, NY, USA, 385–396. https: //

Digital Library

[56]

Hao Zhong, Lu Zhang, and Hong Mei. 2006. An Experimental Comparison of Four Test Suite Reduction Techniques. In Proceedings of the 28th International Conference on Software Engineering (ICSE ’06). ACM, New York, NY, USA, 636–640. https://

Digital Library

[57]

Qian Zhongsheng. 2010. Test case generation and optimization for user sessionbased web application testing. Journal of Computers 5, 11 (2010), 1655–1662.

Cited By

An KTilevich E(2024)EdgStr: Automating Client-Cloud to Client-Edge-Cloud Transformation2024 IEEE 44th International Conference on Distributed Computing Systems (ICDCS)10.1109/ICDCS60910.2024.00061(589-600)Online publication date: 23-Jul-2024
https://doi.org/10.1109/ICDCS60910.2024.00061
Chen WCao HBlanc X(2021)An Improving Approach for DOM-Based Web Test Suite RepairWeb Engineering10.1007/978-3-030-74296-6_29(372-387)Online publication date: 11-May-2021
https://doi.org/10.1007/978-3-030-74296-6_29
An KTilevich E(2020)Client Insourcing: Bringing Ops In-House for Seamless Re-engineering of Full-Stack JavaScript ApplicationsProceedings of The Web Conference 202010.1145/3366423.3380105(179-189)Online publication date: 20-Apr-2020
https://dl.acm.org/doi/10.1145/3366423.3380105
Show More Cited By

Index Terms

Test execution checkpointing for web applications
1. Software and its engineering
  1. Software creation and management
    1. Software verification and validation
      1. Software defect analysis
        Software testing and debugging

Recommendations

DOM-based test adequacy criteria for web applications
ISSTA 2014: Proceedings of the 2014 International Symposium on Software Testing and Analysis

To assess the quality of web application test cases, web developers currently measure code coverage. Although code coverage has traditionally been a popular test adequacy criterion, we believe it alone is not adequate for assessing the quality of web ...
Parallel execution of prioritized test cases for regression testing of web applications
ACSC '13: Proceedings of the Thirty-Sixth Australasian Computer Science Conference - Volume 135

We present a new approach for automatically prioritizing and distributing test cases on multiple machines. Our approach is based on a functional dependency graph (FDG) of a web application. We partition the test suite into test sets according to the ...
Carving and Replaying Differential Unit Test Cases from System Test Cases

Unit test cases are focused and efficient. System tests are effective at exercising complex usage patterns. Differential unit tests (DUT) are a hybrid of unit and system tests that exploits their strengths. They are generated by carving the system ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

ISSTA 2017: Proceedings of the 26th ACM SIGSOFT International Symposium on Software Testing and Analysis

July 2017

447 pages

ISBN:9781450350761

DOI:10.1145/3092703

General Chair:
Tevfik Bultan
University of California at Santa Barbara, USA
,
Program Chair:
Koushik Sen
University of California at Berkeley, USA

Copyright © 2017 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

SIGSOFT: ACM Special Interest Group on Software Engineering

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 10 July 2017

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

ISSTA '17

Sponsor:

SIGSOFT

ISSTA '17: International Symposium on Software Testing and Analysis

July 10 - 14, 2017

CA, Santa Barbara, USA

Acceptance Rates

Overall Acceptance Rate 58 of 213 submissions, 27%

Upcoming Conference

ISSTA '25

Sponsor:
sigsoft

34th ACM SIGSOFT International Symposium on Software Testing and Analysis

June 25 - 28, 2025

Trondheim , Norway

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

4
Total Citations
View Citations
283
Total Downloads

Downloads (Last 12 months)8
Downloads (Last 6 weeks)1

Reflects downloads up to 13 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

An KTilevich E(2024)EdgStr: Automating Client-Cloud to Client-Edge-Cloud Transformation2024 IEEE 44th International Conference on Distributed Computing Systems (ICDCS)10.1109/ICDCS60910.2024.00061(589-600)Online publication date: 23-Jul-2024
https://doi.org/10.1109/ICDCS60910.2024.00061
Chen WCao HBlanc X(2021)An Improving Approach for DOM-Based Web Test Suite RepairWeb Engineering10.1007/978-3-030-74296-6_29(372-387)Online publication date: 11-May-2021
https://doi.org/10.1007/978-3-030-74296-6_29
An KTilevich E(2020)Client Insourcing: Bringing Ops In-House for Seamless Re-engineering of Full-Stack JavaScript ApplicationsProceedings of The Web Conference 202010.1145/3366423.3380105(179-189)Online publication date: 20-Apr-2020
https://dl.acm.org/doi/10.1145/3366423.3380105
Mondal SNasre R(2019) MahtabJournal of Systems and Software10.1016/j.jss.2019.110403158:COnline publication date: 1-Dec-2019
https://dl.acm.org/doi/10.1016/j.jss.2019.110403
Bell JLegunsen OHilton MEloussi LYung TMarinov DChaudron MCrnkovic IChechik MHarman M(2018)DeFlakerProceedings of the 40th International Conference on Software Engineering10.1145/3180155.3180164(433-444)Online publication date: 27-May-2018
https://dl.acm.org/doi/10.1145/3180155.3180164

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents