research-article

Brassau: automatic generation of graphical user interfaces for virtual assistants

Authors:

Michael Fischer,

Giovanni Campagna,

Monica S. LamAuthors Info & Claims

MobileHCI '18: Proceedings of the 20th International Conference on Human-Computer Interaction with Mobile Devices and Services

September 2018

Article No.: 33, Pages 1 - 12

https://doi.org/10.1145/3229434.3229481

Published: 03 September 2018 Publication History

Abstract

This paper presents Brassau, a graphical virtual assistant that converts natural language commands into GUIs. A virtual assistant with a GUI has the following benefits compared to text or speech based virtual assistants: users can monitor multiple queries simultaneously, it is easy to re-run complex commands, and user can adjust settings using multiple modes of interaction. Brassau introduces a novel template-based approach that leverages a large corpus of images to make GUIs visually diverse and interesting. Brassau matches a command from the user to an image to create a GUI. This approach decouples the commands from GUIs and allows for reuse of GUIs across multiple commands. In our evaluation, users prefer the widgets produced by Brassau over plain GUIs.

References

[1]

Amazon Alexa 2017. (2017). https://developer.amazon.com/alexa

[2]

Scott Bateman, Carl Gutwin, and Miguel Nacenta. 2008. Seeing Things in the Clouds: The Effect of Visual Features on Tag Cloud Selections. In Proceedings of the Nineteenth ACM Conference on Hypertext and Hypermedia (HT '08). ACM, New York, NY, USA, 193--202.

Digital Library

[3]

Scott Bateman, Regan L. Mandryk, Carl Gutwin, Aaron Genest, David McDine, and Christopher Brooks. 2010. Useful Junk: The Effects of Visual Embellishment on Comprehension and Memorability of Charts. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '10). ACM, New York, NY, USA, 2573--2582.

Digital Library

[4]

Dan Bloomberg. 2008. Color quantization using modified median cut. Leptonica. http://www.leptonica.com/papers/mediancut.pdf

[5]

Giovanni Campagna, Rakesh Ramesh, Silei Xu, Michael Fischer, and Monica S. Lam. 2017. Almond: The Architecture of an Open, Crowdsourced, Privacy-Preserving, Programmable Virtual Assistant. In Proceedings of the 26th International Conference on World Wide Web (WWW '17). ACM Press, New York, New York, USA, 341--350.

Digital Library

[6]

Dennis JMJ De Baar, James D Foley, and Kevin E Mullet. 1992. Coupling Application Design and User Interface Design. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '92). ACM, 259--266.

Digital Library

[7]

Lokesh Dhakar. 2017. Color Thief. (2017). http://lokeshdhakar.com/projects/color-thief/

[8]

James Fogarty and Scott E Hudson. 2003. GADGET: A Toolkit for Optimization-Based Approaches to Interface and Display Generation. In Proceedings of the 16th Annual ACM Symposium on User Interface Software and Technology (UIST '03). ACM, 125--134.

Digital Library

[9]

Krzysztof Gajos and Daniel S. Weld. 2004. SUPPLE: Automatically Generating User Interfaces. In Proceedings of the 9th International Conference on Intelligent User Interfaces (IUI '04). ACM, New York, NY, USA, 93--100.

Digital Library

[10]

Krzysztof Z. Gajos, Daniel S. Weld, and Jacob O. Wobbrock. 2010. Automatically Generating Personalized User Interfaces with Supple. Artificial Intelligence 174, 12--13 (Aug. 2010), 910--950.

Digital Library

[11]

Tong Gao, Mira Dontcheva, Eytan Adar, Zhicheng Liu, and Karrie G. Karahalios. 2015. DataTone: Managing Ambiguity in Natural Language Interfaces for Data Visualization. In Proceedings of the 28th Annual ACM Symposium on User Interface Software Technology (UIST '15). ACM, New York, NY, USA, 489--500.

Digital Library

[12]

Yusef Hassan-Montero and Victor Herrero-Solana. 2006. Improving Tag-Clouds as Visual Information Retrieval Interfaces. In International Conference on Multidisciplinary Information Sciences and Technologies. 25--28.

[13]

Michael Johnston, John Chen, Patrick Ehlen, Hyuckchul Jung, Jay Lieske, Aarthi M Reddy, Ethan Selfridge, Svetlana Stoyanchev, Brant Vasilieff, and Jay G Wilpon. 2014. MVA: The Multimodal Virtual Assistant. In Proceedings of the 15th Annual Meeting of the Special Interest Group on Discourse and Dialogue (SIGDIAL '14). 257--259.

[14]

Donald E. Knuth and Michael F. Plass. 1981. Breaking paragraphs into lines. Software: Practice and Experience 11, 11 (1981), 1119--1184.

[15]

Ranjitha Kumar, Arvind Satyanarayan, Cesar Torres, Maxine Lim, Salman Ahmad, Scott R Klemmer, and Jerry O Talton. 2013. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM, 3083--3092.

Digital Library

[16]

Gitte Lindgaard, Cathy Dudek, Devjani Sen, Livia Sumegi, and Patrick Noonan. 2011. An Exploration of Relations Between Visual Appeal, Trustworthiness and Perceived Usability of Homepages. ACM Transactions on Computer-Human Interaction 18, 1 (2011), 1:1--1:30.

Digital Library

[17]

Mark Otto, Jacob Thornton, Chris Rebert, Julian Thilo, and others. 2011. Twitter Bootstrap. (2011). http://getbootstrap.com/

[18]

Shankar R. Ponnekanti, Brian Lee, Armando Fox, Pat Hanrahan, and Terry Winograd. 2001. ICrafter: A Service Framework for Ubiquitous Computing Environments. In International Conference on Ubiquitous Computing. 56--75.

Digital Library

[19]

Óscar Sánchez Ramón, Jesús Sánchez Cuadrado, Jesús García Molina, and Jean Vanderdonckt. 2016. A Layout Inference Algorithm for Graphical User Interfaces. Information and Software Technology 70 (2016), 155--175.

Digital Library

[20]

David Reitter, Erin Marie Panttaja, and Fred Cummins. 2004. UI on the Fly: Generating a Multimodal User Interface. In Proceedings of HLT-NAACL 2004: Short Papers (HLT-NAACL-Short '04). Association for Computational Linguistics, Stroudsburg, PA, USA, 45--48. http://dl.acm.org/citation.cfm?id=1613984.1613996

Digital Library

[21]

Arvind Satyanarayan, Dominik Moritz, Kanit Wongsuphasawat, and Jeffrey Heer. 2017. Vega-Lite: A Grammar of Interactive Graphics. IEEE Transactions on Visualization and Computer Graphics 23, 1 (Jan. 2017), 341--350.

Digital Library

[22]

Andrew Sears. 1993. Layout Appropriateness: A Metric for Evaluating User Interface Widget Layout. IEEE Transactions on Software Engineering 19, 7 (1993), 707--719.

Digital Library

[23]

Ethan Selfridge and Michael Johnston. 2015. Interact: Tightly-coupling Multimodal Dialog with an Interactive Virtual Assistant. In Proceedings of the 2015 ACM on International Conference on Multimodal Interaction (ICMI '15). ACM, New York, NY, USA, 381--382.

Digital Library

[24]

Vidya Setlur, Sarah E. Battersby, Melanie Tory, Rich Gossweiler, and Angel X. Chang. 2016. Eviza: A Natural Language Interface for Visual Analysis. In Proceedings of the 29th Annual Symposium on User Interface Software and Technology (UIST '16). ACM, New York, NY, USA, 365--377.

Digital Library

[25]

Yiwen Sun, Jason Leigh, Andrew Johnson, and Sangyoon Lee. 2010. Articulate: A Semi-automated Model for Translating Natural Language Queries into Meaningful Visualizations. In International Symposium on Smart Graphics. Springer Berlin Heidelberg, 184--195.

Digital Library

[26]

Ilya Sutskever, Oriol Vinyals, and Quoc V Le. 2014. Sequence to Sequence Learning with Neural Networks. In Advances in Neural Information Processing Systems 27, Z. Ghahramani, M. Welling, C. Cortes, N. D. Lawrence, and K. Q. Weinberger (Eds.). Curran Associates, Inc., 3104--3112. http://papers.nips.cc/paper/5346-sequence-to-sequence-learning-with-neural-networks.pdf

Digital Library

[27]

Pedro Szekely. 1990. Template-Based Mapping of Application Data Interactive Displays. In Proceedings of the 3rd annual ACM SIGGRAPH Symposium on User Interface Software and Technology (UST '90). ACM, 1--9.

Digital Library

[28]

Kashyap Todi, Daryl Weir, and Antti Oulasvirta. 2016. Sketchplore: Sketch and Explore with a Layout Optimiser. In Proceedings of the 2016 ACM Conference on Designing Interactive Systems. ACM, 543--555.

Digital Library

[29]

Tzutalin. 2015. LabelImg. Git code. (2015). https://github.com/tzutalin/labelImg

[30]

Wikipedia contributors. 2018. Pierre Brassau --- Wikipedia, The Free Encyclopedia. (2018). https://en.wikipedia.org/w/index.php?title=Pierre_Brassau

[31]

Brad Vander Zanden and Brad A Myers. 1990. Automatic, Look-and-Feel Independent Dialog Creation for Graphical User Interfaces. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '90). ACM, 27--34.

Digital Library

Cited By

Cho HYan YTodi KParent MSmith MJonker TBenko HLindlbauer D(2024)MineXR: Mining Personalized Extended Reality InterfacesProceedings of the CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642394(1-17)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3642394
Pan LYu CHe ZShi Y(2023)A Human-Computer Collaborative Editing Tool for Conceptual DiagramsProceedings of the 2023 CHI Conference on Human Factors in Computing Systems10.1145/3544548.3580676(1-29)Online publication date: 19-Apr-2023
https://dl.acm.org/doi/10.1145/3544548.3580676
Zhao MCui WRamakrishnan IZhai SBi X(2021)Voice and Touch Based Error-tolerant Multimodal Text Editing and Correction for SmartphonesThe 34th Annual ACM Symposium on User Interface Software and Technology10.1145/3472749.3474742(162-178)Online publication date: 10-Oct-2021
https://dl.acm.org/doi/10.1145/3472749.3474742
Show More Cited By

Recommendations

A Dialog-Oriented User Interface Generation Mechanism
APSEC '96: Proceedings of the Third Asia-Pacific Software Engineering Conference

Nowadays, for GUI application development, there are a number of interface builders, which make it possible for the user to create user interfaces easily, and UIMSs, which help to specify and design user interfaces. However, interface builders lack the ...
Read More
Completing the Job Interface Design

HyperNews, a user-interface management system that lets a user separate application and interface design to link a new interface to an old application with very little programming, is described. Much of a HyperNews interface can be designed without ...
Read More
Deep learning‐based prototyping of android GUI from hand‐drawn mockups

Recently, transforming graphical user interface (GUI) mockups into code becomes a common challenging practice for current software developers. However, this transformation usually takes time especially when GUI changes keep pace with evolutionary ...
Read More

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

MobileHCI '18: Proceedings of the 20th International Conference on Human-Computer Interaction with Mobile Devices and Services

September 2018

552 pages

ISBN:9781450358989

DOI:10.1145/3229434

General Chairs:
Lynne Bailie,
Nuria Oliver

Copyright © 2018 Owner/Author.

Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for third-party components of this work must be honored. For all other uses, contact the Owner/Author.

Sponsors

SIGCHI: ACM Special Interest Group on Computer-Human Interaction

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 03 September 2018

Check for updates

Author Tags

Qualifiers

Research-article

Conference

MobileHCI '18

Sponsor:

SIGCHI

MobileHCI '18: 20th International Conference on Human-Computer Interaction with Mobile Devices and Services

September 3 - 6, 2018

Barcelona, Spain

Acceptance Rates

Overall Acceptance Rate 202 of 906 submissions, 22%

Upcoming Conference

MOBILEHCI '24

Sponsor:
sigchi

26th International Conference on Mobile Human-Computer Interaction

September 30 - October 3, 2024

Melbourne , VIC , Australia

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

5
Total Citations
View Citations
237
Total Downloads

Downloads (Last 12 months)20
Downloads (Last 6 weeks)5

Other Metrics

View Author Metrics

Citations

Cited By

Cho HYan YTodi KParent MSmith MJonker TBenko HLindlbauer D(2024)MineXR: Mining Personalized Extended Reality InterfacesProceedings of the CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642394(1-17)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3642394
Pan LYu CHe ZShi Y(2023)A Human-Computer Collaborative Editing Tool for Conceptual DiagramsProceedings of the 2023 CHI Conference on Human Factors in Computing Systems10.1145/3544548.3580676(1-29)Online publication date: 19-Apr-2023
https://dl.acm.org/doi/10.1145/3544548.3580676
Zhao MCui WRamakrishnan IZhai SBi X(2021)Voice and Touch Based Error-tolerant Multimodal Text Editing and Correction for SmartphonesThe 34th Annual ACM Symposium on User Interface Software and Technology10.1145/3472749.3474742(162-178)Online publication date: 10-Oct-2021
https://dl.acm.org/doi/10.1145/3472749.3474742
Fischer MCampagna GChoi ELam MFreund SYahav E(2021)DIY assistant: a multi-modal end-user programmable virtual assistantProceedings of the 42nd ACM SIGPLAN International Conference on Programming Language Design and Implementation10.1145/3453483.3454046(312-327)Online publication date: 19-Jun-2021
https://dl.acm.org/doi/10.1145/3453483.3454046
Campagna GXu SRamesh RFischer MLam M(2018)Controlling Fine-Grain Sharing in Natural Language with a Virtual AssistantProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/32649052:3(1-28)Online publication date: 18-Sep-2018
https://dl.acm.org/doi/10.1145/3264905

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents