research-article

GUI testing using computer vision

Authors:

Tsung-Hsiang Chang,

Robert C. MillerAuthors Info & Claims

CHI '10: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems

Pages 1535 - 1544

https://doi.org/10.1145/1753326.1753555

Published: 10 April 2010 Publication History

Abstract

Testing a GUI's visual behavior typically requires human testers to interact with the GUI and to observe whether the expected results of interaction are presented. This paper presents a new approach to GUI testing using computer vision for testers to automate their tasks. Testers can write a visual test script that uses images to specify which GUI components to interact with and what visual feedback to be observed. Testers can also generate visual test scripts by demonstration. By recording both input events and screen images, it is possible to extract the images of components interacted with and the visual feedback seen by the demonstrator, and generate a visual test script automatically. We show that a variety of GUI behavior can be tested using this approach. Also, we show how this approach can facilitate good testing practices such as unit testing, regression testing, and test-driven development.

References

[1]

Autoit. http://www.autoitscript.com/autoit3/, 1999.

[2]

M. Boshernitsan, S.L. Graham, and M.A. Hearst. Aligning development tools with the way programmers think about code changes. In CHI '07, pages 567--576, New York, NY, USA, 2007. ACM.

Digital Library

[3]

HP Mercury Interactive. Winrunner. http://www.winrunner.com.

[4]

D.J. Kasik and H.G. George. Toward automatic generation of novice user test scripts. In CHI '96, pages 244--251, New York, NY, USA, 1996. ACM.

Digital Library

[5]

A.J. Koand, B.A. Myers. Designing the why line: a debugging interface for asking questions about program behavior. In CHI '04, pages 151--158, New York, NY, USA, 2004. ACM.

Digital Library

[6]

A.J. Koand, B.A. Myers. Finding causes of program output with the java why line. In CHI '09, pages 1569--1578, New York, NY, USA, 2009. ACM.

Digital Library

[7]

G. Leshed, E.M. Haber, T. Matthews, and T. Lau. Coscripter: automating & sharing how-to knowledge in the enterprise. In CHI '08, pages 1719--1728, New York, NY, USA, 2008. ACM.

Digital Library

[8]

G. Little, T.A. Lau, A. Cypher, J. Lin, E.M. Haber, and E. Kandogan. Koala: capture, share, automate, personalize business processes on the web. In CHI '07, pages 943--946, New York, NY, USA, 2007. ACM.

Digital Library

[9]

A. Memon. GUI testing: pitfalls and process. Computer, 35(8):87--88, Aug 2002.

Digital Library

[10]

A. Memon, I. Banerjee, and A. Nagarajan. GUI ripping: reverse engineering of graphical user interfaces for testing. In Reverse Engineering, 2003. WCRE 2003. Proceedings. 10th Working Conference on, pages 260--269, Nov. 2003.

Digital Library

[11]

T. Ostrand, A. Anodide, H. Foster, and T. Goradia. A visual test development environment for GUI systems. SIGSOFT Softw. Eng. Notes, 23(2):82--92, 1998.

Digital Library

[12]

G. Singh and Z. Cuie. Sage: creating reusable, modularized interactive behaviors by demonstration. In CHI '94, pages 297--298, New York, NY, USA, 1994. ACM.

Digital Library

[13]

SOFTWARERESEARCHINC.Capbak.http://soft.com, 1999.

[14]

R. St. Amant, H. Lieberman, R. Potter, and L. Zettlemoyer. Programming by example: visual generalization in programming by example. Commun. ACM, 43(3):107--114, 2000.

Digital Library

[15]

N. Subrahmaniyan, L. Beckwith, V. Grigoreanu, M. Burnett, S. Wiedenbeck, V. Narayanan, K. Bucht, R. Drummond, and X. Fern. Testing vs. code inspection vs. what else?: male and female end users' debugging strategies. In CHI '08, pages 617--626, New York, NY, USA, 2008. ACM.

Digital Library

[16]

E.M. Wilcox, J.W. Atwood, M.M. Burnett, J.J. Cadiz, and C.R. Cook. Does continuous visual feedback aid debugging in direct-manipulation programming systems? In CHI '97, pages 258--265, New York, NY, USA, 1997. ACM.

Digital Library

[17]

Q. Xie and A.M. Memon. Designing and comparing automated test oracles for GUI-based software applications. ACM Trans. Softw. Eng. Methodol., 16(1):4, 2007.

Digital Library

[18]

T. Yeh, T.-H. Chang, and R.C. Miller. Sikuli: Using GUI screenshots for search and automation. In UIST '09, pages 183--192. ACM, 2009.

Digital Library

[19]

L.S. Zettlemoyer and R. St. Amant. A visual medium for programmatic control of interactive applications. In CHI '99, pages 199--206, New York, NY, USA, 1999. ACM.

Digital Library

Cited By

Liu SWang SSun K(2024)Having Difficulty Understanding Manuals? Automatically Converting User Manuals into Instructional VideosProceedings of the ACM on Human-Computer Interaction10.1145/36602458:EICS(1-19)Online publication date: 17-Jun-2024
https://dl.acm.org/doi/10.1145/3660245
Wang YZhao YWang XTang WZhang JWang SWang PHu J(2024)Robotic Process Automation Efficiency for Mobile App Testing: An Empirical InvestigationInternational Journal of Software Engineering and Knowledge Engineering10.1142/S021819402450011634:07(1025-1046)Online publication date: 14-May-2024
https://doi.org/10.1142/S0218194024500116
Yu SFang CDu MDing ZChen ZSu Z(2024)Practical, Automated Scenario-Based Mobile App TestingIEEE Transactions on Software Engineering10.1109/TSE.2024.341467250:7(1949-1966)Online publication date: 1-Jul-2024
https://dl.acm.org/doi/10.1109/TSE.2024.3414672
Show More Cited By

Index Terms

GUI testing using computer vision
1. Human-centered computing
  1. Human computer interaction (HCI)
    1. Interaction paradigms
      1. Graphical user interfaces
2. Software and its engineering
  1. Software creation and management
    1. Software verification and validation
      1. Software defect analysis
        Software testing and debugging

Recommendations

Coverage criteria for GUI testing
ESEC/FSE-9: Proceedings of the 8th European software engineering conference held jointly with 9th ACM SIGSOFT international symposium on Foundations of software engineering

A widespread recognition of the usefulness of graphical user interfaces (GUIs) has established their importance as critical components of today's software. GUIs have characteristics different from traditional software, and conventional testing ...
Coverage criteria for GUI testing

A widespread recognition of the usefulness of graphical user interfaces (GUIs) has established their importance as critical components of today's software. GUIs have characteristics different from traditional software, and conventional testing ...
GUI Interaction Testing: Incorporating Event Context

Graphical user interfaces (GUIs), due to their event-driven nature, present an enormous and potentially unbounded way for users to interact with software. During testing, it is important to “adequately cover” this interaction space. In this paper, we ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

CHI '10: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems

April 2010

2690 pages

ISBN:9781605589299

DOI:10.1145/1753326

General Chair:
Elizabeth Mynatt
Georgia Institute of Technology
,
Program Chairs:
Geraldine Fitzpatrick
Vienna University of Technology
,
Scott Hudson
Carnegie Mellon University
,
Keith Edwards
Georgia Tech
,
Tom Rodden
University of Nottingham

Copyright © 2010 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGCHI: ACM Special Interest Group on Computer-Human Interaction

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 10 April 2010

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

CHI '10

Sponsor:

SIGCHI

CHI '10: CHI Conference on Human Factors in Computing Systems

April 10 - 15, 2010

Georgia, Atlanta, USA

Acceptance Rates

Overall Acceptance Rate 6,199 of 26,314 submissions, 24%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

145
Total Citations
View Citations
2,190
Total Downloads

Downloads (Last 12 months)133
Downloads (Last 6 weeks)7

Reflects downloads up to 30 Aug 2024

Other Metrics

View Author Metrics

Citations

Cited By

Liu SWang SSun K(2024)Having Difficulty Understanding Manuals? Automatically Converting User Manuals into Instructional VideosProceedings of the ACM on Human-Computer Interaction10.1145/36602458:EICS(1-19)Online publication date: 17-Jun-2024
https://dl.acm.org/doi/10.1145/3660245
Wang YZhao YWang XTang WZhang JWang SWang PHu J(2024)Robotic Process Automation Efficiency for Mobile App Testing: An Empirical InvestigationInternational Journal of Software Engineering and Knowledge Engineering10.1142/S021819402450011634:07(1025-1046)Online publication date: 14-May-2024
https://doi.org/10.1142/S0218194024500116
Yu SFang CDu MDing ZChen ZSu Z(2024)Practical, Automated Scenario-Based Mobile App TestingIEEE Transactions on Software Engineering10.1109/TSE.2024.341467250:7(1949-1966)Online publication date: 1-Jul-2024
https://dl.acm.org/doi/10.1109/TSE.2024.3414672
Sandnes F(2024)“Consent notices are obstructing my view”: Viewing sticky elements on responsive websites under the magnifying glassDisplays10.1016/j.displa.2023.10257981(102579)Online publication date: Jan-2024
https://doi.org/10.1016/j.displa.2023.102579
Wang ZXi KGao CJin XChen YLu C(2024)A real-time object detection method for electronic screen GUI test systemsThe Journal of Supercomputing10.1007/s11227-024-06319-y80:15(22803-22835)Online publication date: 1-Jul-2024
https://doi.org/10.1007/s11227-024-06319-y
Khalili FMariani LMohebbi APezzè MTerragni V(2024)Semantic matching in GUI test reuseEmpirical Software Engineering10.1007/s10664-023-10406-829:3Online publication date: 9-May-2024
https://doi.org/10.1007/s10664-023-10406-8
Sandnes F(2024)How Order and Omission of Web Content Can Vary Unintentionally Across User Cohorts: A ReviewUniversal Access in Human-Computer Interaction10.1007/978-3-031-60881-0_6(80-99)Online publication date: 1-Jun-2024
https://doi.org/10.1007/978-3-031-60881-0_6
Zhang YGuo ZSun T(2023)A Non-Intrusive Automated Testing System for Internet of Vehicles App Based on Deep LearningElectronics10.3390/electronics1213287312:13(2873)Online publication date: 29-Jun-2023
https://doi.org/10.3390/electronics12132873
Duan PWarner JHartmann B(2023)Towards Generating UI Design Feedback with LLMsAdjunct Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology10.1145/3586182.3615810(1-3)Online publication date: 29-Oct-2023
https://dl.acm.org/doi/10.1145/3586182.3615810
Zhang SWu LLi YZhang ZLei HLi DGuo YChen X(2023)ReSPlay: Improving Cross-Platform Record-and-Replay with GUI Sequence Matching2023 IEEE 34th International Symposium on Software Reliability Engineering (ISSRE)10.1109/ISSRE59848.2023.00056(439-450)Online publication date: 9-Oct-2023
https://doi.org/10.1109/ISSRE59848.2023.00056
Show More Cited By

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents