research-article

Open access

Robust relational layout synthesis from examples for Android

Authors:

Martin VechevAuthors Info & Claims

Proceedings of the ACM on Programming Languages, Volume 2, Issue OOPSLA

Article No.: 156, Pages 1 - 29

https://doi.org/10.1145/3276526

Published: 24 October 2018 Publication History

Abstract

We present a novel approach for synthesizing robust relational layouts from examples. Given an application design consisting of a set of views and their location on the screen, we synthesize a relational layout that when rendered, places the components at that same location.

We present an end-to-end system, called InferUI, that addresses the above challenge in the context of Android. The system is based on the following technical contributions: (i) a formalization of the latest and most efficient ConstraintLayout class, capturing a rich set of relational constraints, (ii) a set of robustness properties designed to prevent common layout generalization errors, (iii) a synthesis algorithm that produces relational layouts that generalize across multiple screen sizes and resolutions, and (iv) a probabilistic model of constraints that guides the synthesizer towards layouts preferred by developers.

Our evaluation shows that InferUI is practically effective: it successfully synthesizes real world complex layouts obtained from top 500 GitHub and top 500 Google Play Store applications, succeeds in 100% of the cases when synthesizing layouts for a single device, and correctly generalizes 92% of the views across multiple devices, all without requiring additional specifications.

Supplementary Material

WEBM File (a156-bielik.webm)

Download
99.44 MB

References

[1]

Miltiadis Allamanis, Earl T. Barr, Christian Bird, and Charles A. Sutton. 2015. Suggesting accurate method and class names. In Proceedings of the 2015 10th Joint Meeting on Foundations of Software Engineering (ESEC/FSE ’15). ACM, New York, NY, USA, 38–49.

Digital Library

[2]

Miltiadis Allamanis, Earl T. Barr, Premkumar Devanbu, and Charles Sutton. 2018. A Survey of Machine Learning for Big Code and Naturalness. ACM Comput. Surv. 51, 4, Article 81 (2018), 37 pages.

Digital Library

[3]

Amigo. 2018. Amigo. https://play.google.com/store/apps/details?id=com.amigotrip.android

[4]

Asobimasu. 2018. Asobimasu. https://github.com/DipanshKhandelwal/Asobimasu

[5]

Matej Balog, Alexander L Gaunt, Marc Brockschmidt, Sebastian Nowozin, and Daniel Tarlow. 2017. DeepCoder: Learning to write programs. In 5th International Conference on Learning Representations (ICLR ’17).

[6]

Tony Beltramelli. 2017. pix2code: Generating Code from a Graphical User Interface Screenshot. CoRR abs/1705.07962 (2017). arXiv: 1705.07962 http://arxiv.org/abs/1705.07962

[7]

Pavol Bielik, Veselin Raychev, and Martin Vechev. 2016. PHOG: Probabilistic Model for Code. In Proceedings of The 33rd International Conference on Machine Learning (ICML ’16), Vol. 48. PMLR, New York, NY, USA, 2933–2942.

Digital Library

[8]

Candid. 2018. Candid. https://play.google.com/store/apps/details?id=in.voiceme.app.voiceme

[9]

Chunyang Chen, Ting Su, Guozhu Meng, Zhenchang Xing, and Yang Liu. 2018. From UI Design Image to GUI Skeleton: A Neural Machine Translator to Bootstrap Mobile GUI Implementation. In Proceedings of the 40th International Conference on Software Engineering (ICSE ’18) . ACM, New York, NY, USA, 665–676.

Digital Library

[10]

Shauvik Roy Choudhary, Mukul R. Prasad, and Alessandro Orso. 2012. CrossCheck: Combining Crawling and Differencing to Better Detect Cross-browser Incompatibilities in Web Applications. In Proceedings of the IEEE 5th International Conference on Software Testing, Verification and Validation (ICST ’12) . IEEE Computer Society, Washington, DC, USA, 171–180.

Digital Library

[11]

Ravi Chugh, Brian Hempel, Mitchell Spradlin, and Jacob Albers. 2016. Programmatic and Direct Manipulation, Together at Last. In Proceedings of the 37th ACM SIGPLAN Conference on Programming Language Design and Implementation (PLDI ’16) . ACM, New York, NY, USA, 341–354.

Digital Library

[12]

Alex Corrado, Avery Lamp, Brendan Walsh, Edward Aryee, Erica Yuen, George Matthews, Jen Madiedo, Jeremie Laval, Luis Torres, Maddy Leger, Paris Hsu, Patrick Chen, Tim Rait, Seth Chong, Wjdan Alharthi, and Xiao Tu. 2018. Ink To Code. https://www.microsoft.com/en-us/garage/profiles/ink-to-code/

[13]

Leonardo De Moura and Nikolaj Bjørner. 2008. Z3: An Efficient SMT Solver. In Proceedings of the Theory and Practice of Software, 14th International Conference on Tools and Algorithms for the Construction and Analysis of Systems (TACAS’08/ETAPS’08) . Springer-Verlag, Berlin, Heidelberg, 337–340. http://dl.acm.org/citation.cfm?id=1792734.1792766

Digital Library

[14]

Kevin Ellis, Daniel Ritchie, Armando Solar-Lezama, and Joshua B. Tenenbaum. 2017. Learning to Infer Graphics Programs from Hand-Drawn Images. CoRR abs/1707.09627 (2017). arXiv: 1707.09627 http://arxiv.org/abs/1707.09627

[15]

Kevin Ellis, Armando Solar-Lezama, and Joshua B. Tenenbaum. 2016. Sampling for Bayesian Program Learning. In Advances in Neural Information Processing Systems 29 . Curran Associates, Inc., Barcelona, Spain, 1297–1305. http: //papers.nips.cc/paper/6082-sampling-for-bayesian-program-learning.pdf

Digital Library

[16]

FBook. 2018. FBook. https://play.google.com/store/apps/details?id=com.framgia.book

[17]

Ruozi Huang, Yonghao Long, and Xiangping Chen. 2016. Automatically Generating Web Page From A Mockup. In The 28th International Conference on Software Engineering and Knowledge Engineering (SEKE ’16) . KSI Research Inc. and Knowledge Systems Institute Graduate School, Redwood City, San Francisco Bay, USA, 589–594.

[18]

Geoffrey Irving, Christian Szegedy, Alexander A Alemi, Niklas Een, Francois Chollet, and Josef Urban. 2016. DeepMath - Deep Sequence Models for Premise Selection. In Advances in Neural Information Processing Systems 29. Curran Associates, Inc., Barcelona, Spain, 2235–2243. http://papers.nips.cc/paper/6280-deepmath-deep-sequence-models-for-premise-selection. pdf

Digital Library

[19]

W. E. Johnson. 1932. Probability: The Deductive and Inductive Problems. Mind 41, 164 (1932), 409–423. http://www.jstor. org/stable/2250183

[20]

Ashwin Kalyan, Abhishek Mohta, Oleksandr Polozov, Dhruv Batra, Prateek Jain, and Sumit Gulwani. 2018. NeuralGuided Deductive Search for Real-Time Program Synthesis from Examples. In 6th International Conference on Learning Representations (ICLR ’18) .

[21]

Omer Katz, Ran El-Yaniv, and Eran Yahav. 2016. Estimating Types in Binaries Using Predictive Modeling. In Proceedings of the 43rd Annual ACM SIGPLAN-SIGACT Symposium on Principles of Programming Languages (POPL ’16) . ACM, New York, NY, USA, 313–326.

Digital Library

[22]

Woosuk Lee, Kihong Heo, Rajeev Alur, and Mayur Naik. 2018. Accelerating Search-based Program Synthesis Using Learned Probabilistic Models. In Proceedings of the 39th ACM SIGPLAN Conference on Programming Language Design and Implementation (PLDI ’18) . ACM, New York, NY, USA, 436–449.

Digital Library

[23]

George James Lidstone. 1920. Note on the general case of the bayes-laplace formula for inductive or a posteriori probabilities. Transactions of the Faculty of Actuaries 8 (1920), 182–192.

[24]

Fan Long and Martin Rinard. 2016. Automatic Patch Generation by Learning Correct Code. In Proceedings of the 43rd Annual ACM SIGPLAN-SIGACT Symposium on Principles of Programming Languages (POPL ’16) . ACM, New York, NY, USA, 298–312.

Digital Library

[25]

Chris Maddison and Daniel Tarlow. 2014. Structured Generative Models of Natural Source Code. In Proceedings of the 31st International Conference on Machine Learning (ICML ’14), Vol. 32-II. PMLR, New York, NY, USA, 649–657. http: //dl.acm.org/citation.cfm?id=3044805.3044965

Digital Library

[26]

Sonal Mahajan, Bailan Li, Pooyan Behnamghader, and William G. J. Halfond. 2016. Using Visual Symptoms for Debugging Presentation Failures in Web Applications. In Proceedings of the IEEE 9th International Conference on Software Testing, Verification and Validation (ICST ’16) . IEEE Computer Society, Washington, DC, USA, 191–201.

[27]

Aditya Menon, Omer Tamuz, Sumit Gulwani, Butler Lampson, and Adam Kalai. 2013. A Machine Learning Framework for Programming by Example. In Proceedings of The 30rd International Conference on Machine Learning (ICML ’13), Vol. 28-I. PMLR, New York, NY, USA, 187–195.

Digital Library

[28]

Kevin Moran, Boyang Li, Carlos Bernal-Cárdenas, Dan Jelf, and Denys Poshyvanyk. 2018. Automated Reporting of GUI Design Violations for Mobile Apps. In Proceedings of the 40th International Conference on Software Engineering (ICSE ’18). ACM, New York, NY, USA, 165–175.

Digital Library

[29]

Tuan Anh Nguyen and Christoph Csallner. 2015. Reverse Engineering Mobile Application User Interfaces with REMAUI (T). In Proceedings of the 30th ACM/IEEE International Conference on Automated Software Engineering (ASE ’15). IEEE Computer Society, Washington, DC, USA, 248–259.

Digital Library

[30]

Pavel Panchekha, Adam T. Geller, Michael D. Ernst, Zachary Tatlock, and Shoaib Kamil. 2018. Verifying That Web Pages Have Accessible Layout. In Proceedings of the 39th ACM SIGPLAN Conference on Programming Language Design and Implementation (PLDI ’18) . ACM, New York, NY, USA, 1–14.

Digital Library

[31]

Veselin Raychev, Martin Vechev, and Andreas Krause. 2015. Predicting Program Properties from "Big Code". In Proceedings of the 42Nd Annual ACM SIGPLAN-SIGACT Symposium on Principles of Programming Languages (POPL ’15) . ACM, New York, NY, USA, 111–124.

Digital Library

[32]

Steven P. Reiss. 2014. Seeking the User Interface. In Proceedings of the 29th ACM/IEEE International Conference on Automated Software Engineering (ASE ’14) . ACM, New York, NY, USA, 103–114.

Digital Library

[33]

Shauvik Roy Choudhary, Mukul R. Prasad, and Alessandro Orso. 2013. X-PERT: Accurate Identification of Cross-browser Issues in Web Applications. In Proceedings of the 2013 International Conference on Software Engineering (ICSE ’13). IEEE Press, Piscataway, NJ, USA, 702–711. http://dl.acm.org/citation.cfm?id=2486788.2486881

Digital Library

[34]

Amanda Swearngin, Mira Dontcheva, Wilmot Li, Joel Brandt, Morgan Dixon, and Andrew J. Ko. 2018. Rewire: Interface Design Assistance from Examples. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems (CHI ’18) . ACM, New York, NY, USA, Article 504, 12 pages.

Digital Library

[35]

Clemens Zeidler, Christof Lutteroth, Wolfgang Stuerzlinger, and Gerald Weber. 2013a. Evaluating Direct Manipulation Operations for Constraint-Based Layout. In 14th IFIP Conference on Human-Computer Interaction (INTERACT ’13). Springer, Berlin, Heidelberg, 513–529.

[36]

Clemens Zeidler, Christof Lutteroth, Wolfgang Sturzlinger, and Gerald Weber. 2013b. The Auckland Layout Editor: An Improved GUI Layout Specification Process. In Proceedings of the 26th Annual ACM Symposium on User Interface Software and Technology (UIST ’13) . ACM, New York, NY, USA, 343–352.

Digital Library

[37]

Clemens Zeidler, Gerald Weber, Wolfgang Stuerzlinger, and Christof Lutteroth. 2017. Automatic Generation of User Interface Layouts for Alternative Screen Orientations. In 16th IFIP Conference on Human-Computer Interaction (INTERACT ’17). Springer, Berlin, Heidelberg, 13–35.

Cited By

Wu JTodi KChan JMyers BLafreniere B(2024)FrameKit: A Tool for Authoring Adaptive UIs Using KeyframesProceedings of the 29th International Conference on Intelligent User Interfaces10.1145/3640543.3645176(660-674)Online publication date: 18-Mar-2024
https://dl.acm.org/doi/10.1145/3640543.3645176
Jiang YLu YKnearem TKliman-Silver CLutteroth CLi TNichols JStuerzlinger W(2024)Computational Methodologies for Understanding, Automating, and Evaluating User InterfacesExtended Abstracts of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613905.3636316(1-7)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613905.3636316
Jiang YZhou CGarg VOulasvirta A(2024)Graph4GUI: Graph Neural Networks for Representing Graphical User InterfacesProceedings of the CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642822(1-18)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3642822
Show More Cited By

Index Terms

Robust relational layout synthesis from examples for Android

Recommendations

Synthesis of web layouts from examples
ESEC/FSE 2021: Proceedings of the 29th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering

We present a new technique for synthesizing dynamic, constraint-based visual layouts from examples. Our technique tackles two major challenges of layout synthesis. First, realistic layouts, especially on the web, often contain hundreds of elements, so ...
Synthesis of layout engines from relational constraints
OOPSLA 2015: Proceedings of the 2015 ACM SIGPLAN International Conference on Object-Oriented Programming, Systems, Languages, and Applications

We present an algorithm for synthesizing efficient document layout engines from compact relational specifications. These specifications are compact in that a single specification can produce multiple engines, each for a distinct layout situation, i.e., ...
Synthesis of layout engines from relational constraints
OOPSLA '15

We present an algorithm for synthesizing efficient document layout engines from compact relational specifications. These specifications are compact in that a single specification can produce multiple engines, each for a distinct layout situation, i.e., ...

Comments

Information & Contributors

Information

Published In

cover image Proceedings of the ACM on Programming Languages

Proceedings of the ACM on Programming Languages Volume 2, Issue OOPSLA

November 2018

1656 pages

EISSN:2475-1421

DOI:10.1145/3288538

Issue’s Table of Contents

Copyright © 2018 Owner/Author.

This work is licensed under a Creative Commons Attribution International 4.0 License.

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 24 October 2018

Published in PACMPL Volume 2, Issue OOPSLA

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

15
Total Citations
View Citations
957
Total Downloads

Downloads (Last 12 months)106
Downloads (Last 6 weeks)16

Reflects downloads up to 10 Aug 2024

Other Metrics

View Author Metrics

Citations

Cited By

Wu JTodi KChan JMyers BLafreniere B(2024)FrameKit: A Tool for Authoring Adaptive UIs Using KeyframesProceedings of the 29th International Conference on Intelligent User Interfaces10.1145/3640543.3645176(660-674)Online publication date: 18-Mar-2024
https://dl.acm.org/doi/10.1145/3640543.3645176
Jiang YLu YKnearem TKliman-Silver CLutteroth CLi TNichols JStuerzlinger W(2024)Computational Methodologies for Understanding, Automating, and Evaluating User InterfacesExtended Abstracts of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613905.3636316(1-7)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613905.3636316
Jiang YZhou CGarg VOulasvirta A(2024)Graph4GUI: Graph Neural Networks for Representing Graphical User InterfacesProceedings of the CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642822(1-18)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3642822
Xiao SChen YSong YChen LSun LZhen YChang YZhou T(2024)UI semantic component group detection: Grouping UI elements with similar semantics in mobile graphical user interfaceDisplays10.1016/j.displa.2024.10267983(102679)Online publication date: Jul-2024
https://doi.org/10.1016/j.displa.2024.102679
Jiang YLu YLutteroth CLi TNichols JStuerzlinger W(2023)The Future of Computational Approaches for Understanding and Adapting User InterfacesExtended Abstracts of the 2023 CHI Conference on Human Factors in Computing Systems10.1145/3544549.3573805(1-5)Online publication date: 19-Apr-2023
https://dl.acm.org/doi/10.1145/3544549.3573805
ÖZÇEVİK Y(2022)Geleneksel İç İçe Yerleşimler ve Kısıt Yerleşimi İçeren Melez Android Ön Yüz Geliştirme YaklaşımıA Hybrid Android Frontend Development Approach with Traditional Nesting Layouts and Constraint LayoutInternational Journal of Innovative Engineering Applications10.46460/ijiea.11162226:2(245-254)Online publication date: 30-Dec-2022
https://doi.org/10.46460/ijiea.1116222
Su YChen CWang JLiu ZWang DLi SWang Q(2022)The Metamorphosis: Automatic Detection of Scaling Issues for Mobile AppsProceedings of the 37th IEEE/ACM International Conference on Automated Software Engineering10.1145/3551349.3556935(1-12)Online publication date: 10-Oct-2022
https://dl.acm.org/doi/10.1145/3551349.3556935
Xie MXing ZFeng SXu XZhu LChen CRoychoudhury ACadar CKim M(2022)Psychologically-inspired, unsupervised inference of perceptual groups of GUI widgets from GUI imagesProceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering10.1145/3540250.3549138(332-343)Online publication date: 7-Nov-2022
https://dl.acm.org/doi/10.1145/3540250.3549138
Li CJiang YXu CDwyer MDamian DZeller A(2022)Push-button synthesis of watch companions for Android appsProceedings of the 44th International Conference on Software Engineering10.1145/3510003.3510056(1793-1804)Online publication date: 21-May-2022
https://dl.acm.org/doi/10.1145/3510003.3510056
Jiang YLu YNichols JStuerzlinger WYu CLutteroth CLi YKumar RLi T(2022)Computational Approaches for Understanding, Generating, and Adapting User InterfacesExtended Abstracts of the 2022 CHI Conference on Human Factors in Computing Systems10.1145/3491101.3504030(1-6)Online publication date: 27-Apr-2022
https://dl.acm.org/doi/10.1145/3491101.3504030
Show More Cited By

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

Media

Figures

Other

Tables

View Issue’s Table of Contents