research-article

Umitation: Retargeting UI Behavior Examples for Website Design

Authors:

Tovi GrossmanAuthors Info & Claims

UIST '21: The 34th Annual ACM Symposium on User Interface Software and Technology

Pages 922 - 935

https://doi.org/10.1145/3472749.3474796

Published: 12 October 2021 Publication History

Abstract

Interface designers often refer to UI behavior examples found in the wild (e.g., commercial websites) for reference or design inspiration. While past research has looked at retargeting interface and webpage design, limited work has explored the challenges in retargeting interactive visual behaviors. We introduce Umitation, a system that helps designers extract, edit, and adapt example front-end UI behaviors to target websites. Umitation can also help designers specify the desired behaviors and reconcile their intended interaction details with their existing UI. In a qualitative evaluation, we found evidence that Umitation helps participants extract and retarget dynamic front-end UI behavior examples quickly and expressively.

References

[1]

2021. Microsoft Format Painter. https://tinyurl.com/afxc85n9 Accessed: April, 2021.

[2]

2021. Scrimba. https://www.scrimba.com/ Accessed: April, 2021.

[3]

Margaret A Boden 2004. The creative mind: Myths and mechanisms. Psychology Press.

[4]

Elizabeth Boling and Theodore W Frick. 1997. Holistic rapid prototyping for web design: Early usability testing is essential. Web-based instruction(1997), 319–328.

[5]

Brian Burg, Richard Bailey, Amy J Ko, and Michael D Ernst. 2013. Interactive record/replay for web application debugging. In Proceedings of the 26th annual ACM symposium on User interface software and technology. 473–484.

Digital Library

[6]

Bill Buxton. 2010. Sketching user experiences: getting the design right and the right design. Morgan kaufmann.

[7]

Caroline Chan, Shiry Ginosar, Tinghui Zhou, and Alexei A Efros. 2019. Everybody dance now. In Proceedings of the IEEE/CVF International Conference on Computer Vision. 5933–5942.

[8]

Kerry Shih-Ping Chang and Brad A Myers. 2012. WebCrystal: understanding and reusing examples in web authoring. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. 3205–3214.

Digital Library

[9]

Kerry Shih-Ping Chang and Brad A Myers. 2014. Creating interactive web data applications with spreadsheets. In Proceedings of the 27th annual ACM symposium on User interface software and technology. 87–96.

Digital Library

[10]

Sarah E Chasins, Maria Mueller, and Rastislav Bodik. 2018. Rousillon: Scraping Distributed Hierarchical Web Data. In Proceedings of the 31st Annual ACM Symposium on User Interface Software and Technology. 963–975.

Digital Library

[11]

Yan Chen, Sang Won Lee, and Steve Oney. 2021. CoCapture: Effectively Communicating UI Behaviors on Existing Websites by Demonstrating and Remixing. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems.

Digital Library

[12]

Yan Chen, Sang Won Lee, Yin Xie, YiWei Yang, Walter S Lasecki, and Steve Oney. 2017. Codeon: On-demand software development assistance. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems. 6220–6231.

Digital Library

[13]

Yan Chen, Steve Oney, and Walter S Lasecki. 2016. Towards providing on-demand expert support for software developers. In Proceedings of the 2016 CHI conference on human factors in computing systems. 3192–3203.

Digital Library

[14]

Pei-Yu Chi, Sen-Po Hu, and Yang Li. 2018. Doppio: Tracking ui flows and code changes for app development. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. 1–13.

Digital Library

[15]

Allen Cypher and Daniel Conrad Halbert. 1993. Watch what I do: programming by demonstration. MIT press.

Digital Library

[16]

Steven Dow, Julie Fortuna, Dan Schwartz, Beth Altringer, Daniel Schwartz, and Scott Klemmer. 2011. Prototyping dynamics: sharing multiple designs improves exploration, group rapport, and results. In Proceedings of the SIGCHI conference on human factors in computing systems. 2807–2816.

Digital Library

[17]

Mary L Gick and Keith J Holyoak. 1983. Schema induction and analogical transfer. Cognitive psychology 15, 1 (1983), 1–38.

[18]

Tovi Grossman, Justin Matejka, and George Fitzmaurice. 2010. Chronicle: capture, exploration, and playback of document workflow histories. In Proceedings of the 23nd annual ACM symposium on User interface software and technology. 143–152.

Digital Library

[19]

Björn Hartmann, Leslie Wu, Kevin Collins, and Scott R Klemmer. 2007. Programming by a sample: rapidly creating web applications with d. mix. In Proceedings of the 20th annual ACM symposium on User interface software and technology. 241–250.

Digital Library

[20]

Björn Hartmann, Loren Yu, Abel Allison, Yeonsoo Yang, and Scott R Klemmer. 2008. Design as exploration: creating interface alternatives through parallel authoring and runtime tuning. In Proceedings of the 21st annual ACM symposium on User interface software and technology. 91–100.

Digital Library

[21]

Yasunari Hashimoto and Takeo Igarashi. 2005. Retrieving Web Page Layouts using Sketches to Support Example-based Web Design. In SBM. Citeseer, 155–164.

[22]

Scarlett R Herring, Chia-Chen Chang, Jesse Krantzler, and Brian P Bailey. 2009. Getting inspired! Understanding how and why examples are used in creative design practice. In Proceedings of the SIGCHI conference on human factors in computing systems. 87–96.

Digital Library

[23]

Joshua Hibschman and Haoqi Zhang. 2015. Unravel: Rapid web application reverse engineering via interaction recording, source tracing, and library detection. In Proceedings of the 28th Annual ACM Symposium on User Interface Software & Technology. 270–279.

Digital Library

[24]

Joshua Hibschman and Haoqi Zhang. 2016. Telescope: Fine-tuned discovery of interactive web UI feature implementation. In Proceedings of the 29th Annual Symposium on User Interface Software and Technology. 233–245.

Digital Library

[25]

Yue Jiang, Wolfgang Stuerzlinger, and Christof Lutteroth. 2021. ReverseORC: Reverse Engineering of Resizable User Interface Layouts with OR-Constraints. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems. 1–18.

Digital Library

[26]

Rubaiat Habib Kazi, Fanny Chevalier, Tovi Grossman, and George Fitzmaurice. 2014. Kitty: sketching dynamic and interactive illustrations. In Proceedings of the 27th annual ACM symposium on User interface software and technology. 395–405.

Digital Library

[27]

Mary Beth Kery, Amber Horvath, and Brad A Myers. 2017. Variolite: Supporting Exploratory Programming by Data Scientists. In CHI, Vol. 10. 3025453–3025626.

Digital Library

[28]

Mary Beth Kery, Donghao Ren, Fred Hohman, Dominik Moritz, Kanit Wongsuphasawat, and Kayur Patel. 2020. mage: Fluid Moves Between Code and Graphical Work in Computational Notebooks. In Proceedings of the 33rd Annual ACM Symposium on User Interface Software and Technology. 140–151.

Digital Library

[29]

Donghwi Kim, Sooyoung Park, Jihoon Ko, Steven Y Ko, and Sung-Ju Lee. 2019. X-Droid: A Quick and Easy Android Prototyping Framework with a Single-App Illusion. In Proceedings of the 32nd Annual ACM Symposium on User Interface Software and Technology. 95–108.

Digital Library

[30]

Huhn Kim and Wan Chul Yoon. 2005. Supporting the cognitive process of user interface design with reusable design cases. International journal of human-computer studies 62, 4 (2005), 457–486.

Digital Library

[31]

Janet L Kolodner and Linda M Wills. 1993. Case-based creative design. AISB QUARTERLY 85(1993), 1–8.

[32]

Robert E Kraut, Susan R Fussell, and Jane Siegel. 2003. Visual information as a conversational resource in collaborative physical tasks. Human–computer interaction 18, 1-2 (2003), 13–49.

[33]

Ranjitha Kumar, Jerry O Talton, Salman Ahmad, and Scott R Klemmer. 2011. Bricolage: example-based retargeting for web design. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. 2197–2206.

Digital Library

[34]

James A Landay and Brad A Myers. 1995. Interactive sketching for the early stages of user interface design. In Proceedings of the SIGCHI conference on Human factors in computing systems. 43–50.

Digital Library

[35]

David Ledo, Steven Houben, Jo Vermeulen, Nicolai Marquardt, Lora Oehlberg, and Saul Greenberg. 2018. Evaluation strategies for HCI toolkit research. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. 1–17.

Digital Library

[36]

Brian Lee, Savil Srivastava, Ranjitha Kumar, Ronen Brafman, and Scott R Klemmer. 2010. Designing with interactive example galleries. In Proceedings of the SIGCHI conference on human factors in computing systems. 2257–2266.

Digital Library

[37]

Germán Leiva and Michel Beaudouin-Lafon. 2018. Montage: A Video Prototyping System to Reduce Re-Shooting and Increase Re-Usability. In Proceedings of the 31st Annual ACM Symposium on User Interface Software and Technology. 675–682.

Digital Library

[38]

Germán Leiva, Nolwenn Maudet, Wendy Mackay, and Michel Beaudouin-Lafon. 2019. Enact: Reducing designer–developer breakdowns when prototyping custom interactions. ACM Transactions on Computer-Human Interaction (TOCHI) 26, 3(2019), 1–48.

Digital Library

[39]

Germán Leiva, Cuong Nguyen, Rubaiat Habib Kazi, and Paul Asente. 2020. Pronto: Rapid Augmented Reality Video Prototyping Using Sketches and Enaction. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems (Honolulu, HI, USA) (CHI ’20). Association for Computing Machinery, New York, NY, USA, 1–13. https://doi.org/10.1145/3313831.3376160

Digital Library

[40]

Toby Jia-Jun Li, Amos Azaria, and Brad A Myers. 2017. SUGILITE: creating multimodal smartphone automation by demonstration. In Proceedings of the 2017 CHI conference on human factors in computing systems. 6038–6049.

[41]

Tom Lieber, Joel R Brandt, and Rob C Miller. 2014. Addressing misconceptions about code with always-on programming visualizations. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. 2481–2490.

Digital Library

[42]

Sarah Lim, Joshua Hibschman, Haoqi Zhang, and Eleanor O’Rourke. 2018. Ply: A visual web inspector for learning from professional webpages. In Proceedings of the 31st Annual ACM Symposium on User Interface Software and Technology. 991–1002.

Digital Library

[43]

James Lin, Mark W Newman, Jason I Hong, and James A Landay. 2000. DENIM: finding a tighter fit between tools and practice for Web site design. In Proceedings of the SIGCHI conference on Human Factors in Computing Systems. 510–517.

Digital Library

[44]

James Lin, Jeffrey Wong, Jeffrey Nichols, Allen Cypher, and Tessa A Lau. 2009. End-user programming of mashups with vegemite. In Proceedings of the 14th international conference on Intelligent user interfaces. 97–106.

Digital Library

[45]

Michael Xieyang Liu, Jane Hsieh, Nathan Hahn, Angelina Zhou, Emily Deng, Shaun Burley, Cynthia Taylor, Aniket Kittur, and Brad A Myers. 2019. Unakite: Scaffolding Developers’ Decision-Making Using the Web. In Proceedings of the 32nd Annual ACM Symposium on User Interface Software and Technology. 67–80.

Digital Library

[46]

Josip Maras, Maja Stula, Jan Carlson, and Ivica Crnkovic. 2013. Identifying code of individual features in client-side web applications. IEEE Transactions on Software Engineering 39, 12 (2013), 1680–1697.

Digital Library

[47]

Justin Matejka, Michael Glueck, Erin Bradner, Ali Hashemi, Tovi Grossman, and George Fitzmaurice. 2018. Dream lens: Exploration and visualization of large-scale generative design datasets. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. 1–12.

Digital Library

[48]

Brad Myers, Sun Young Park, Yoko Nakano, Greg Mueller, and Amy Ko. 2008. How designers design and program interactive behaviors. In 2008 IEEE Symposium on Visual Languages and Human-Centric Computing. IEEE, 177–184.

Digital Library

[49]

Brad A Myers, Ashley Lai, Tam Minh Le, YoungSeok Yoon, Andrew Faulring, and Joel Brandt. 2015. Selective undo support for painting applications. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems. 4227–4236.

Digital Library

[50]

Mark W Newman and James A Landay. 2000. Sitemaps, storyboards, and specifications: A sketch of web site design practice. In Proceedings of the 3rd conference on Designing interactive systems: processes, practices, methods, and techniques. 263–274.

Digital Library

[51]

Peter O’Donovan, Aseem Agarwala, and Aaron Hertzmann. 2015. Designscape: Design with interactive layout suggestions. In Proceedings of the 33rd annual ACM conference on human factors in computing systems. 1221–1224.

Digital Library

[52]

Stephen Oney and Brad Myers. 2009. FireCrystal: Understanding interactive behaviors in dynamic web pages. In 2009 IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC). IEEE, 105–108.

Digital Library

[53]

Jungkook Park, Yeong Hoon Park, and Alice Oh. 2018. Non-Linear Editing of Text-Based Screencasts. In Proceedings of the 31st Annual ACM Symposium on User Interface Software and Technology. 403–410.

Digital Library

[54]

Jeffrey Stylos, Brad A Myers, and Andrew Faulring. 2004. Citrine: providing intelligent copy-and-paste. In Proceedings of the 17th annual ACM symposium on User interface software and technology. 185–188.

Digital Library

[55]

Amanda Swearngin, Chenglong Wang, Alannah Oleson, James Fogarty, and Amy J Ko. 2020. Scout: Rapid Exploration of Interface Layout Alternatives through High-Level Design Constraints. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems. 1–13.

Digital Library

[56]

Jerry Talton, Lingfeng Yang, Ranjitha Kumar, Maxine Lim, Noah Goodman, and Radomír Měch. 2012. Learning design patterns with bayesian grammar induction. In Proceedings of the 25th annual ACM symposium on User interface software and technology. 63–74.

Digital Library

[57]

Edward R Tufte. 1983. The visual display of quantitative information. Vol. 2.

[58]

April Yi Wang, Zihan Wu, Christopher Brooks, and Steve Oney. 2020. Callisto: Capturing the” Why” by Connecting Conversations with Computational Narratives. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems. 1–13.

Digital Library

[59]

Yifan Wu, Joseph M Hellerstein, and Arvind Satyanarayan. 2020. B2: Bridging Code and Interactive Visualization in Computational Notebooks. In Proceedings of the 33rd Annual ACM Symposium on User Interface Software and Technology. 152–165.

Digital Library

[60]

Xiong Zhang and Philip J Guo. 2018. Fusion: Opportunistic web prototyping with ui mashups. In Proceedings of the 31st Annual ACM Symposium on User Interface Software and Technology. 951–962.

Digital Library

[61]

Nanxuan Zhao, Nam Wook Kim, Laura Mariah Herman, Hanspeter Pfister, Rynson WH Lau, Jose Echevarria, and Zoya Bylinskii. 2020. Iconate: Automatic compound icon generation and ideation. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems. 1–13.

Digital Library

Cited By

Alves SCosta RMontague KGuerreiro T(2024)Citizen-Led Personalization of User Interfaces: Investigating How People Customize Interfaces for Themselves and OthersProceedings of the ACM on Human-Computer Interaction10.1145/36869858:CSCW2(1-23)Online publication date: 8-Nov-2024
https://dl.acm.org/doi/10.1145/3686985
Rahman AHeo S(2023)Frappé: An Ultra Lightweight Mobile UI Framework for Rapid API-based Prototyping and Environmental DeploymentProceedings of the ACM on Human-Computer Interaction10.1145/36042587:MHCI(1-23)Online publication date: 13-Sep-2023
https://dl.acm.org/doi/10.1145/3604258

Recommendations

Scenario-based interactive UI design
CHI '13: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems

Clearly picturing user behavior is one of the key requirements when designing successful interactive software. However, covering all possible user behaviors with one UI is a complex challenge. The Scenario-based Interactive UI Design tool is designed to ...
UI Fin: a process-oriented interface design tool
IUI '09: Proceedings of the 14th international conference on Intelligent user interfaces

Even though over the years a multitude of user interface design tools have been created, designers in practice find themselves limited to a small set of realistic options. These options include interface builders that are attached to development ...
Akin: Generating UI Wireframes From UI Design Patterns Using Deep Learning
IUI '21 Companion: Companion Proceedings of the 26th International Conference on Intelligent User Interfaces

During the User interface (UI) design process, designers use UI design patterns for conceptualizing different UI wireframes for an application. This paper introduces Akin, a UI wireframe generator that allows designers to chose a UI design pattern and ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

UIST '21: The 34th Annual ACM Symposium on User Interface Software and Technology

October 2021

1357 pages

ISBN:9781450386357

DOI:10.1145/3472749

Editors:
Jeffrey Nichols
Apple, USA
,
Ranjitha Kumar
UIUC, USA
,
Michael Nebeling
University of Michigan, USA

Copyright © 2021 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 12 October 2021

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed limited

Funding Sources

National Sciences and Engineering Research Council of Canada

Conference

UIST '21

Sponsor:

UIST '21: The 34th Annual ACM Symposium on User Interface Software and Technology

October 10 - 14, 2021

Virtual Event, USA

Acceptance Rates

Overall Acceptance Rate 561 of 2,567 submissions, 22%

Upcoming Conference

UIST '25

Sponsor:
sigchi
sigchi

The 38th Annual ACM Symposium on User Interface Software and Technology

September 28 - October 1, 2025

Busan , Republic of Korea

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

2
Total Citations
View Citations
554
Total Downloads

Downloads (Last 12 months)69
Downloads (Last 6 weeks)8

Reflects downloads up to 13 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Alves SCosta RMontague KGuerreiro T(2024)Citizen-Led Personalization of User Interfaces: Investigating How People Customize Interfaces for Themselves and OthersProceedings of the ACM on Human-Computer Interaction10.1145/36869858:CSCW2(1-23)Online publication date: 8-Nov-2024
https://dl.acm.org/doi/10.1145/3686985
Rahman AHeo S(2023)Frappé: An Ultra Lightweight Mobile UI Framework for Rapid API-based Prototyping and Environmental DeploymentProceedings of the ACM on Human-Computer Interaction10.1145/36042587:MHCI(1-23)Online publication date: 13-Sep-2023
https://dl.acm.org/doi/10.1145/3604258

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

HTML Format

View this article in HTML Format.

Media

Figures

Other

Tables

View Table of Contents