research-article

Seeking the user interface

Author:

Steven P. ReissAuthors Info & Claims

ASE '14: Proceedings of the 29th ACM/IEEE International Conference on Automated Software Engineering

Pages 103 - 114

https://doi.org/10.1145/2642937.2642976

Published: 15 September 2014 Publication History

Abstract

User interface design and coding can be complex and messy. We describe a system that uses code search to simplify and automate the generation of such code. We start with a simple sketch of the desired interface along with a set of keywords describing the application context. We then use existing code search engines to find results based on the keywords. We look for potential Java-based user interface solutions within those results and apply a series of code transformations to the solutions to generate derivative solutions, aiming to get solutions that constitute only the user interface and that will compile and run. We run the resultant solutions and compare the generated interfaces to the user's sketches. Finally, we let programmers interact with the matched solutions and return the running code for the solutions they choose. The system can be used not only for generating initial user interface code for an application, but also for exploring alternative interfaces and for looking at the user interfaces in a code repository.

References

[1]

Marat Akhin, Nikolai Tillmann, Manual Fahndrich, Jonathan de Halleux, and Michal Moskal, "Search by example in touch develop: code search made easy," Proceedings SUITE 2013, pp. 5--8 (June 2012).

[2]

Sushil Bajracharya, Trung Ngo, Erik Linstead, Yimeng Dou, Paul Rigor, Pierre Baldi, and Cristina Lopes, "Sourcerer: a search engine for open source code supporting structure based search," Proceedings ACM SIGPLAN Conference on Object-Oriented Programming, Systems, Languages, and Applications 2006, pp. 682--682 (October 2006).

Digital Library

[3]

Andrew Begel, "Codifier: a programmer-centric search user interface," Workshop on Human-Coputer Interaction and Information Retrieval, (October 2007).

[4]

Bojana Bislimovska, Alessandro Bozzon, Marco Brambilla, and Piero Fraternali, "Search upon UML repositories with text matching techniques," Proceedings SUITE 2013, pp. 9--12 (June 2012).

[5]

Wing-Kwan Chan, Hong Cheng, and David Lo, "Searching connected API subgraph via text phrases," pp. 1--11 in Proceedings of the ACM SIGSOFT 20th International Symposium on the Foundations of Software Engineering, (2012).

Digital Library

[6]

Shih-Chien Chou, Jen-Yen Chen, and Chyan-Goei Chung, "A behavior-based classification and retrieval technique for object-oriented specification reuse," Software Practice and Experience 26(7) pp. 815--832 (July 1996).

Digital Library

[7]

Shih-Chien Chou and Yuan-Chien Chen, "Retrieving reusable components with variation points from software product lines," Information Processing Letters 99 pp. 106--110 (2006).

Digital Library

[8]

Christopher G. Drummond, Dan Ionescu, and Robert C. Holte, "A learning agent that assists the browsing of software libraries," IEEE Transactions on Software Engineering 26(12) pp. 1179--1196 (December 2000).

Digital Library

[9]

Michael D. Ernst, Raimondas Lencevisius, and Jeff H. Perkins, "Detection of web service substitutability and composability," WS-MaTe 2006: International Workshop on Web Services -- Modeling and Testing, pp. 123--135 (June 2006).

[10]

William B. Frakes and Thomas P. Pole, "An empiracal study of representation methods for reusable software components," IEEE Transactions on Software Engineering 20(8) pp. 617--630 (August 1994).

Digital Library

[11]

Krzysztof Z. Gajos, Daniel S. Weld, and Jacob O. Wobbrock, "Decision-theoretic user interface generation," In Proceedings of the 22nd AAAI Conf. on Artificial Intelligence (AAAI-08), (2008).

Digital Library

[12]

Ben Goodger, Ian Hickson, David Hyatt, and Chris Waterson, "XML user interface language (XUL) 1.0 Specficiation," http://www.mozilla.org/projects/xul/xul.html (2003).

[13]

Mark Grechanik, Chen Fu, Qing Xie, Collin McMillan, Denys Poshyvanyk, and Chad Cumby, "A search engine for finding highly relevant applications," Proceedings of the 32nd ACM/IEEE International Conference on Software Engineering, (May 2010).

Digital Library

[14]

Saul Greenberg, "Toolkits and interface creativity," Journal on Multimedia Tools and Applications 32 pp. 139--159 (2007).

Digital Library

[15]

Robert J. Hall, "Generalized behavior-based retrieval," Proceedings International Conference on Software Engineering'93, pp. 371--380 (May 1993).

Digital Library

[16]

Bjorn Hartmann, Leith Abdulla, Manas Mittal, and Scott R. Klemmer, "Authoring sensor based interactions through direct manipulation and pattern matching," Proceedings of chi 2007: ACM Conference on Human Factors in Computing Systems, pp. 145--154 (2007).

Digital Library

[17]

David Hemer and Peter Lindsay, "Supporting component based reuse in CARE," Australian Computer Science Communications 24(1) pp. 95--104 (2002).

Digital Library

[18]

Raphael Hoffmann and James Fogarty, "Assieme: finding and leveraging implicit references in a web search interface for programmers," Proceedings UIST 2007, pp. 13--22 (October 2007).

Digital Library

[19]

Werner Janjic, Dietmar Stoll, Philipp Bostan, and Colin Atkinson, "Lowering the barrier to reuse through test- driven search," SUITE'09, pp. 21--24 (May 2009).

Digital Library

[20]

Werner Janjic and Colin Atkinson, "Leveraging software search and reuse with automated software adaptation," Proceedings SUITE 2013, pp. 23--26 (June 2012).

[21]

Jun-Jang Jeng and Betty H. C. Cheng, "Specification matching for software reuse: a foundation," Proceedings ACM Symposium on Software Reuse, pp. 97--105 (April 1995).

Digital Library

[22]

Otavio Lemos, Sushil Bajracharya, Joel Ossher, Ricardo Morla, Paulo Masiero, Pierre Baldi, and Cristina Lopes, "CodeGenie: using test-cases to search and reuse source code," ASE '07, pp. 525--526 (November 2007).

Digital Library

[23]

Otavio Augusto Lazzarini Lemos, Sushil Bajracharya, Joel Ossher, Paulo Cesar Masiero, and Cristina Lopes, "A test driven approach to code search and its application to the reuse of auxiliary functionality," Information and Software Technology 53(4) pp. 294--306 (April 2011).

Digital Library

[24]

Daniel Lucredio, Antonio Franciso do Prado, and Eduardo Santana de Almeida, "A survey of software components search and retrieval," Proceedings EUROMICRO'04, pp. 152--159 (2004).

Digital Library

[25]

Yoelle S. Maarek, Daniel M. Berry, and Gail E. Kaiser, "An information retrieval approach for automatically constructing software libraries," IEEE Transactions on Software Engineering 17(8) pp. 800--813 (August 1991).

Digital Library

[26]

Collin McMillan, Mark Grechanik, Denys Poshyvanyk, Qing Xie, and Chen Fu, "Portfolio: finding relevant functions and their usage," Proceeding of the 33rd International Conference on Software engineering, (May 2011).

Digital Library

[27]

Gerrit Meixner, Fabio Patern, and Jean Vanderdonckt, "Past, present, and future of model-based user interface development," i-com 10(3) pp. 2--11 (2011).

[28]

Rym Mili, Ali Mili, and Roland T. Mittermeir, "Storing and retrieving software components: a refinement based system," IEEE Transactions on Software Engineering 23(7)(July 1997).

Digital Library

[29]

Brad Myers, Scott E. Hudson, and Randy Pausch, "Past, present and future of user interface software tools," ACM Transactions on Computer-Human Interaction 7(1) pp. 3--28 (March 2000).

Digital Library

[30]

Jeffrey Nichols and Andrew Faulring, "Automatic interface generation and future user interface tools," ACM CHI 2005 Workshop on The Future of User Interface Design Tools, (2005).

[31]

Stina Nylander, "Semi-automatic generation of device adapted user interfaces," UIST conference companion, (October 2005).

[32]

Andy Podgurski and Lynn Pierce, "Retrieving reusable software by sampling behavior," ACM Transactions on Software Engineering and Methodology 2(3) pp. 286--303 (July 1993).

Digital Library

[33]

David Raneburger, Roman Popp, and Jean Vanderdonckt, "An automated layout approach for model-driven WIMP-UI generation," Proceedings of the 4th ACM SIGCHI symposium on Engineering interactive computing systems (EICS '12), pp. 91--100 (2012).

Digital Library

[34]

Steven P. Reiss, "A component model for Internet-scale applications," Proceedings ASE 2005, pp. 34--43 (November 2005).

Digital Library

[35]

Steven P. Reiss, "Automatic code stylizing," Proceedings ASE '07, pp. 74--83 (November 2007).

Digital Library

[36]

Steven P. Reiss, "Semantics-based code search," International Conference on Software Engineering 2009, pp. 243--253 (May 2009).

Digital Library

[37]

Steven P. Reiss, "Specifying what to search for," Proceedings SUITE 2009, (May 2009).

Digital Library

[38]

Steven P. Reiss, "Browsing software repositories," Unpublished manuscript submitted for publication, (2014).

[39]

Eugene J. Rollins and Jeannette M. Wing, "Specifications as search keys for software libraries," Proceedings 8th International Conference on Logic Programming, pp. 173--187 (1991).

[40]

Colin Runciman and Ian Toyn, "Retrieving re-usable software components by polymorphic type," Proceedings 4th International Conference on Functional Programming Languages and Computer Architecture, pp. 166--173 (1989).

Digital Library

[41]

Paulo Pinheiro da Silva, "User interface declarative models and development environments: a survey," Proceeding of the 7th International Conference on Design, Specficiation, and Verification of Interactive Systems, pp. 207--226 Springer-Verlag, (2000).

Digital Library

[42]

Jamie Starke, Chris Luce, and Jonathan Sillito, "Working with search results," SUITE'09, pp. 53--56 (May 2009).

Digital Library

[43]

Vijayan Sugumaran and Veda C. Storey, "A semantic-based approach to component retrieval," Advances in Information Systems 34(3) pp. 8--24 (2003).

Digital Library

[44]

Suresh Thummalapenta and Tao Xie, "PARSEWeb: a programmer assistant for reusing open source code on the web," Proceedings ASE'07, pp. 204--213 (November 2007).

Digital Library

[45]

Thung, Ferdian, Wang, Shaowei, Lo, David, and Lawall, Julia, "Automatic recommendation of api methods from feature requests," Proceedings of Automated Software Engineering (ASE) 2013, pp. 290--300 (2013).

[46]

Taciana A. Vanderlei, Frederico A. Durao, Alexandre C. Martins, Vinicius C. Garcia, Eduardo S. Almeida, and Silvio R. de L. Meira, "A cooperative classification mechanism for search and retrieval software components," Proceedings SAC'07, pp. 866--871 (March 2007).

Digital Library

[47]

Haining Yao and Letha Etzkorn, "Towards a semantic- based approach for software reusable component classification and retrieval," ACMSE'04, pp. 110--115 (April 2004).

Digital Library

[48]

Yunwen Ye and Gerhard Fischer, "Supporting reuse by delivering task relevant and personalized information," Proceedings International Conference on Software Engineering'02, pp. 513--523 (May 2002).

Digital Library

[49]

Yunwen Ye, "Programming with an intelligent agent," IEEE Intelligent Systems 18(3) pp. 43--47 (May 2003).

Digital Library

Cited By

Pan XHuan ZLi YCao Y(2024)Enhancement of GUI Display Error Detection Using Improved Faster R-CNN and Multi-Scale Attention MechanismApplied Sciences10.3390/app1403114414:3(1144)Online publication date: 30-Jan-2024
https://doi.org/10.3390/app14031144
Raptis GKatsini C(2023)Transforming Ideas into Code: A Novel Approach to Design-to-Development Workflow2023 International Conference on Computer and Applications (ICCA)10.1109/ICCA59364.2023.10401412(1-6)Online publication date: 28-Nov-2023
https://doi.org/10.1109/ICCA59364.2023.10401412
Li HXie RKong XWang LLi B(2020)An Analysis of Utility for API Recommendation: Do the Matched Results Have the Same Efforts?2020 IEEE 20th International Conference on Software Quality, Reliability and Security (QRS)10.1109/QRS51102.2020.00067(479-488)Online publication date: Dec-2020
https://doi.org/10.1109/QRS51102.2020.00067
Show More Cited By

Index Terms

Seeking the user interface
1. Software and its engineering
  1. Software notations and tools
    1. Development frameworks and environments
      1. Integrated and visual development environments

Recommendations

Seeking the user interface

User interface design and coding can be complex and messy. We describe a system that uses code search to simplify and automate the exploration of such code. We start with a simple sketch of the desired interface along with a set of keywords describing ...
User-Interface Tools: Introduction and Survey

An overview is given of user-interface development systems (UIDS). Systems are classified by how they let the programmer specify the interfaces, and examples of each type are given. The three types are language-based, graphical, and automatic creation ...
Interface-Walkthroughs: Efficient Collaborative Testing

A walkthrough facility for testing the usability of a user interface design before the software is written or a prototype is built is described. The use of this method to evaluate screen panels, which may include pull-down menus, dialogue boxes, ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

ASE '14: Proceedings of the 29th ACM/IEEE International Conference on Automated Software Engineering

September 2014

934 pages

ISBN:9781450330138

DOI:10.1145/2642937

General Chair:
Ivica Crnkovic
Mälardalen University, Sweden
,
Program Chairs:
Marsha Chechik
University of Toronto, Canada
,
Paul Grünbacher
Johannes Kepler Universität Linz, Austria

Copyright © 2014 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

SIGAI: ACM Special Interest Group on Artificial Intelligence
SIGSOFT: ACM Special Interest Group on Software Engineering
Mälardalen University: Mälardalen University
IEEE CS

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 15 September 2014

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

Division of Computing and Communication Foundations

Conference

ASE '14

Sponsor:

SIGAI
SIGSOFT
Mälardalen University

ASE '14: ACM/IEEE International Conference on Automated Software Engineering

September 15 - 19, 2014

Vasteras, Sweden

Acceptance Rates

ASE '14 Paper Acceptance Rate 82 of 337 submissions, 24%;

Overall Acceptance Rate 82 of 337 submissions, 24%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

10
Total Citations
View Citations
541
Total Downloads

Downloads (Last 12 months)20
Downloads (Last 6 weeks)0

Reflects downloads up to 24 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Pan XHuan ZLi YCao Y(2024)Enhancement of GUI Display Error Detection Using Improved Faster R-CNN and Multi-Scale Attention MechanismApplied Sciences10.3390/app1403114414:3(1144)Online publication date: 30-Jan-2024
https://doi.org/10.3390/app14031144
Raptis GKatsini C(2023)Transforming Ideas into Code: A Novel Approach to Design-to-Development Workflow2023 International Conference on Computer and Applications (ICCA)10.1109/ICCA59364.2023.10401412(1-6)Online publication date: 28-Nov-2023
https://doi.org/10.1109/ICCA59364.2023.10401412
Li HXie RKong XWang LLi B(2020)An Analysis of Utility for API Recommendation: Do the Matched Results Have the Same Efforts?2020 IEEE 20th International Conference on Software Quality, Reliability and Security (QRS)10.1109/QRS51102.2020.00067(479-488)Online publication date: Dec-2020
https://doi.org/10.1109/QRS51102.2020.00067
Ge XMussbacher GAtlee JBultan T(2019)Android GUI search using hand-drawn sketchesProceedings of the 41st International Conference on Software Engineering: Companion Proceedings10.1109/ICSE-Companion.2019.00060(141-143)Online publication date: 25-May-2019
https://dl.acm.org/doi/10.1109/ICSE-Companion.2019.00060
Bielik PFischer MVechev M(2018)Robust relational layout synthesis from examples for AndroidProceedings of the ACM on Programming Languages10.1145/32765262:OOPSLA(1-29)Online publication date: 24-Oct-2018
https://dl.acm.org/doi/10.1145/3276526
Kim BPark SWon THeo JKim BHung CSaid L(2018)Deep-learning based web UI automatic programmingProceedings of the 2018 Conference on Research in Adaptive and Convergent Systems10.1145/3264746.3264807(64-65)Online publication date: 9-Oct-2018
https://dl.acm.org/doi/10.1145/3264746.3264807
Behrang FReiss SOrso AJulien CLewis GSegall I(2018)GUIfetchProceedings of the 5th International Conference on Mobile Software Engineering and Systems10.1145/3197231.3197244(236-246)Online publication date: 27-May-2018
https://dl.acm.org/doi/10.1145/3197231.3197244
Chen CSu TMeng GXing ZLiu YChaudron MCrnkovic IChechik MHarman M(2018)From UI design image to GUI skeletonProceedings of the 40th International Conference on Software Engineering10.1145/3180155.3180240(665-676)Online publication date: 27-May-2018
https://dl.acm.org/doi/10.1145/3180155.3180240
Reiss SMiao YXin Q(2018)Seeking the user interfaceAutomated Software Engineering10.1007/s10515-017-0216-325:1(157-193)Online publication date: 1-Mar-2018
https://dl.acm.org/doi/10.1007/s10515-017-0216-3
Nguyen TCsallner CCohen MGrunske LWhalen M(2015)Reverse engineering mobile application user interfaces with REMAUIProceedings of the 30th IEEE/ACM International Conference on Automated Software Engineering10.1109/ASE.2015.32(248-259)Online publication date: 9-Nov-2015
https://dl.acm.org/doi/10.1109/ASE.2015.32

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten