Article

Automated replay and failure detection for web applications

Authors:

Sreedevi Sampath,

Lori PollockAuthors Info & Claims

ASE '05: Proceedings of the 20th IEEE/ACM International Conference on Automated Software Engineering

Pages 253 - 262

https://doi.org/10.1145/1101908.1101947

Published: 07 November 2005 Publication History

Abstract

User-session-based testing of web applications gathers user sessions to create and continually update test suites based on real user input in the field. To support this approach during maintenance and beta testing phases, we have built an automated framework for testing web-based software that focuses on scalability and evolving the test suite automatically as the application's operational profile changes. This paper reports on the automation of the replay and oracle components for web applications, which pose issues beyond those in the equivalent testing steps for traditional, stand-alone applications. Concurrency, nondeterminism, dependence on persistent state and previous user sessions, a complex application infrastructure, and a large number of output formats necessitate developing different replay and oracle comparator operators, which have tradeoffs in fault detection effectiveness, precision of analysis, and efficiency. We have designed, implemented, and evaluated a set of automated replay techniques and oracle comparators for user-session-based testing of web applications. This paper describes the issues, algorithms, heuristics, and an experimental case study with user sessions for two web applications. From our results, we conclude that testers performing user-session-based testing should consider their expectations for program coverage and fault detection when choosing a replay and oracle technique.

References

[1]

A. Andrews, J. Offutt, and R. Alexander. Testing web applications by modeling with FSMs. Software Systems and Modeling, 4(2), April 2005.

[2]

J. H. Andrews, L. C. Briand, and Y. Labiche. Is mutation an appropriate tool for testing experiments? In ICSE, 2005.

Digital Library

[3]

L. Baresi and M. Young. Test oracles. Technical Report CIS-TR01-02, University of Oregon, 2001.

[4]

R. Binder. Testing Object-Oriented Systems. Addison Wesley, 2000.

Digital Library

[5]

J. Callahan, F. Schneider, and S. Easterbrook. Automated software testing using model-checking. In SPIN Wksp, 1996.

[6]

D. Chays, Y. Deng, P. Frankl, S. Dan, F. Vokolos, and E. Weyuker. An agenda for testing relational database applications. Soft Testing, Ver and Rel, 14:17--44, Mar. 2004.

Digital Library

[7]

Clover: Code coverage tool for Java. <http://www.cenqua.com/clover/>, 2005.

[8]

L. K. Dillon and Q. Yu. Oracles for checking temporal properties of concurrent systems. In SIGSOFT FSE, 1994.

Digital Library

[9]

S. Elbaum, S. Karre, and G. Rothermel. Improving web application testing with user session data. In ICSE, 2003.

Digital Library

[10]

S. Elbaum, G. Rothermel, S. Karre, and M. F. II. Leveraging user session data to support web application testing. IEEE Trans on Soft Eng, 2005.

Digital Library

[11]

Open source web applications with source code. <http://www.gotocode.com>, 2003.

[12]

M. J. Harrold, R. Gupta, and M. L. Soffa. A methodology for controlling the size of a test suite. ACM Trans on Soft Eng Meth, 2(3):270--285, 1993.

Digital Library

[13]

C. Hunter and P. Strooper. Systematically deriving partial oracles for testing concurrent programs. In Proceedings of the 24th Australasian conference on Computer science, 2001.

Digital Library

[14]

I. Jacobson. The use-case construct in object-oriented software engineering. In J. M. Carroll, editor, Scenario-based Design: Envisioning Work and Technology in System Development, 1995.

Digital Library

[15]

E. Kirda, M. Jazayeri, C. Kerer, and M. Schranz. Experiences in engineering flexible web service. IEEE MultiMedia, 8(1):58--65, 2001.

Digital Library

[16]

B. Long, D. Hoffman, and P. Strooper. Tool support for testing concurrent Java components. Trans on Software Engineering, 29(6):555--566, June 2003.

Digital Library

[17]

G. D. Lucca, A. Fasolino, F. Faralli, and U. D. Carlini. Testing web applications. In Int Conf on Software Maintenance, 2002.

Digital Library

[18]

A. M. Memon, M. E. Pollack, and M. L. Soffa. Automated test oracles for guis. In SIGSOFT/FSE-8, 2000.

Digital Library

[19]

A. Orso and B. Kennedy. Selective capture and replay of program executions. In Int Work on Dyn Anal (WODA), 2005.

Digital Library

[20]

Parasoft WebKing. <http://www.parsoft.com>, 2004.

[21]

Rational Robot. <http://www-306.ibm.com/software/ awdtools/tester/robot/>, 2005.

[22]

Caucho resin. <http://www.caucho.com/resin/>, 2005.

[23]

D. Richardson. TAOS: testing with analysis and oracle support. In Int Sym on Software Testing and Analysis, 1994.

Digital Library

[24]

D. J. Richardson, S. L. Aha, and T. O. O'Malley. Specification- based test oracles for reactive systems. In ICSE, 1992.

Digital Library

[25]

S. Sampath, V. Mihaylov, A. Souter, and L. Pollock. Composing a framework to automate testing of operational web-based software. In Int Conf on Soft Main, 2004.

Digital Library

[26]

S. Sampath, V. Mihaylov, A. Souter, and L. Pollock. A scalable approach to user-session based testing of web applications through concept analysis. In Automated Soft Eng Conf, 2004.

Digital Library

[27]

J. Sant, A. Souter, and L. Greenwald. An exploration of statistical models of automated test case generation. In Int Work on Dyn Anal (WODA), 2005.

Digital Library

[28]

S. Sprenkle, S. Sampath, E. Gibson, L. Pollock, and A. Souter. An empirical comparison of test suite reduction techniques for user-session-based testing of web applications. In Int Conf on Software Maintenance, 2005.

Digital Library

[29]

S. D. Stoller. Model-checking multi-threaded distributed Java programs. Int Journal on Soft Tools for Technology Transfer, 4(1):71--91, Oct. 2002.

[30]

E. Weyuker. On testing non-testable programs. The Computer Journal, 25(4):465--70, 1982.

[31]

GNU wget. <http://www.gnu.org/software/wget/>, 2005.

Cited By

Wang DGalster MMorales-Trujillo M(2024)Application Monitoring for bug reproduction in web-based applicationsJournal of Systems and Software10.1016/j.jss.2023.111834207:COnline publication date: 1-Jan-2024
https://dl.acm.org/doi/10.1016/j.jss.2023.111834
Zhang SWu LLi YZhang ZLei HLi DGuo YChen X(2023)ReSPlay: Improving Cross-Platform Record-and-Replay with GUI Sequence Matching2023 IEEE 34th International Symposium on Software Reliability Engineering (ISSRE)10.1109/ISSRE59848.2023.00056(439-450)Online publication date: 9-Oct-2023
https://doi.org/10.1109/ISSRE59848.2023.00056
Li CJiang YXu CRoychoudhury ACadar CKim M(2022)Cross-device record and replay for Android appsProceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering10.1145/3540250.3549083(395-407)Online publication date: 7-Nov-2022
https://dl.acm.org/doi/10.1145/3540250.3549083
Show More Cited By

Index Terms

Automated replay and failure detection for web applications
1. Software and its engineering
  1. Software creation and management
    1. Software verification and validation
      1. Software defect analysis
        Software testing and debugging

Recommendations

Leveraging existing tests in automated test generation for web applications
ASE '14: Proceedings of the 29th ACM/IEEE International Conference on Automated Software Engineering

To test web applications, developers currently write test cases in frameworks such as Selenium. On the other hand, most web test generation techniques rely on a crawler to explore the dynamic states of the application. The first approach requires much ...
Applying Concept Analysis to User-Session-Based Testing of Web Applications

The continuous use of the web for daily operations by businesses, consumers, and the government has created a great demand for reliable web applications. One promising approach to testing the functionality of web applications leverages user-session data ...
Mining Executable Specifications of Web Applications from Selenium IDE Tests
SERE '12: Proceedings of the 2012 IEEE Sixth International Conference on Software Security and Reliability

A common practice for system testing of web-based applications is to perform the test cases through a web browser. These tests are often recorded and managed by a record and replay tool, such as Selenium IDE. Mining specifications from such tests can be ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

ASE '05: Proceedings of the 20th IEEE/ACM International Conference on Automated Software Engineering

November 2005

482 pages

ISBN:1581139934

DOI:10.1145/1101908

General Chair:
David Redmiles
University of California, Irvine, CA
,
Program Chairs:
Tom Ellman
Vassar College
,
Andrea Zisman
City University, UK

Copyright © 2005 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 07 November 2005

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Article

Conference

ASE05

Sponsor:

ASE05: International Conference on Automated Software Engineering 2005

November 7 - 11, 2005

CA, Long Beach, USA

Acceptance Rates

Overall Acceptance Rate 82 of 337 submissions, 24%

Upcoming Conference

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

65
Total Citations
View Citations
1,258
Total Downloads

Downloads (Last 12 months)11
Downloads (Last 6 weeks)0

Reflects downloads up to 27 Jul 2024

Other Metrics

View Author Metrics

Citations

Cited By

Wang DGalster MMorales-Trujillo M(2024)Application Monitoring for bug reproduction in web-based applicationsJournal of Systems and Software10.1016/j.jss.2023.111834207:COnline publication date: 1-Jan-2024
https://dl.acm.org/doi/10.1016/j.jss.2023.111834
Zhang SWu LLi YZhang ZLei HLi DGuo YChen X(2023)ReSPlay: Improving Cross-Platform Record-and-Replay with GUI Sequence Matching2023 IEEE 34th International Symposium on Software Reliability Engineering (ISSRE)10.1109/ISSRE59848.2023.00056(439-450)Online publication date: 9-Oct-2023
https://doi.org/10.1109/ISSRE59848.2023.00056
Li CJiang YXu CRoychoudhury ACadar CKim M(2022)Cross-device record and replay for Android appsProceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering10.1145/3540250.3549083(395-407)Online publication date: 7-Nov-2022
https://dl.acm.org/doi/10.1145/3540250.3549083
Jambigi NBach TSchabernack FFelderer M(2022)Automatic Error Classification and Root Cause Determination while Replaying Recorded Workload Data at SAP HANA2022 IEEE Conference on Software Testing, Verification and Validation (ICST)10.1109/ICST53961.2022.00041(323-333)Online publication date: Apr-2022
https://doi.org/10.1109/ICST53961.2022.00041
Bach TAndrzejak ASeo CBierstedt CLemke CRitter DHwang DSheshi ESchabernack FRenkes FGaumnitz GMartens JHoemke LFelderer MRudolf MJambigi NMay NJoy RScheja RSchwedes SSeibel SSeifert SHaas SKraft SKroll TScheuer TLehner W(2022)Testing Very Large Database Management Systems: The Case of SAP HANADatenbank-Spektrum10.1007/s13222-022-00426-x22:3(195-215)Online publication date: 24-Nov-2022
https://doi.org/10.1007/s13222-022-00426-x
Althomali IKapfhammer GMcMinn P(2021)Automated visual classification of DOM‐based presentation failure reports for responsive web pagesSoftware Testing, Verification and Reliability10.1002/stvr.175631:4Online publication date: 14-Feb-2021
https://doi.org/10.1002/stvr.1756
Corrêa CDelamaro MChaim MNunes F(2020)Software Testing Automation of VR-Based Systems With Haptic InterfacesThe Computer Journal10.1093/comjnl/bxaa05464:5(826-841)Online publication date: 11-Jun-2020
https://doi.org/10.1093/comjnl/bxaa054
Walsh TKapfhammer GMcMinn P(2020)Automatically identifying potential regressions in the layout of responsive web pagesSoftware Testing, Verification and Reliability10.1002/stvr.174830:6Online publication date: 3-Aug-2020
https://doi.org/10.1002/stvr.1748
Kıraç MAktemur BSözer HGebizli C(2019)Automatically learning usage behavior and generating event sequences for black-box testing of reactive systemsSoftware Quality Journal10.1007/s11219-018-9439-127:2(861-883)Online publication date: 1-Jun-2019
https://dl.acm.org/doi/10.1007/s11219-018-9439-1
Lee SChen YMa SLee W(2018)Test Command Auto-Wait Mechanisms for Record and Playback-Style Web Application Testing2018 IEEE 42nd Annual Computer Software and Applications Conference (COMPSAC)10.1109/COMPSAC.2018.10206(75-80)Online publication date: Jul-2018
https://doi.org/10.1109/COMPSAC.2018.10206
Show More Cited By

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents