A Type System for Flexible User Interactions Handling
Article No.: 246, Pages 1 - 27
Abstract
Engineering user interfaces involves the use of multiple user interactions. Developers may struggle with programming and using those user interactions because of a lack of flexibility that affects the current user interface programming approaches. First, developers may want to switch from one user interaction to another close one or combine multiple user interactions without changing much code. Second, developers may also want to use several user interactions to concisely produce the same user command. Third, developers may want to be warned about conflicts between involved user interactions. Currently, developers can hardly perform these first two cases without applying numerous code changes or producing boilerplate code. Regarding the third case, developers can only observe such issues during the execution of the interactive systems, which prolongs the detection time. To overcome these three issues this paper proposes a user interaction type system. This user interaction type system reifies user interactions as first-class concerns with typing facilities for enabling user interactions substitutability and union. It also allows the writing of type checking rules to check for possible issues related to user interactions at compile time. We implemented the type system within the TypeScript version of Interacto, a framework for processing user interactions. We evaluated the soundness and the expressiveness of our approach through several implemented use cases. This demonstrates the feasibility of the proposed approach and its ability to overcome the three mentioned issues.
References
[1]
Saba Alimadadi, Sheldon Sequeira, Ali Mesbah, and Karthik Pattabiraman. 2016. Understanding JavaScript event-based interactions with Clematis. ACM Transactions on Software Engineering and Methodology (TOSEM) 25, 2 (2016), 1--38. https://doi.org/10.1145/2876441
[2]
Caroline Appert and Michel Beaudouin-Lafon. 2008. SwingStates: Adding state machines to Java and the Swing toolkit. Software: Practice and Experience 38, 11 (2008), 1149--1182. https://doi.org/10.1002/spe.867
[3]
Caroline Appert, Olivier Chapuis, Emmanuel Pietriga, and María-Jesús Lobo. 2015. Reciprocal drag-and-drop. ACM Transactions on Computer-Human Interaction (TOCHI) 22, 6 (2015), 1--36. https://doi.org/10.1145/2785670
[4]
Patrick Baudisch, Edward Cutrell, Dan Robbins, Mary Czerwinski, Peter Tandler, Benjamin Bederson, Alex Zierlinger, et al. 2003. Drag-and-pop and drag-and-pick: Techniques for accessing remote screen content on touch-and pen-operated systems. In Proc. of Interact'03. IOSPress, NL, 57--64.
[5]
Olivier Beaudoux, Mickael Clavreul, Arnaud Blouin, Mengqiang Yang, Olivier Barais, and Jean-Marc Jezequel. 2012. Specifying and running rich graphical components with loa. In Proceedings of the 4th ACM SIGCHI symposium on Engineering interactive computing systems. ACM, USA, 169--178. https://doi.org/10.1145/2305484.2305513
[6]
Arnaud Blouin and Olivier Beaudoux. 2010. Improving modularity and usability of interactive systems with Malai. In EICS'10: Proceedings of the 2nd ACM SIGCHI symposium on Engineering interactive computing systems. ACM, USA, 115--124. https://doi.org/10.1145/1822018.1822037
[7]
Arnaud Blouin and Jean-Marc Jézéquel. 2021. Interacto: A Modern User Interaction Processing Model. IEEE Transactions on Software Engineering 48, 9 (2021), 1--20. https://doi.org/10.1109/TSE.2021.3083321
[8]
Arnaud Blouin, Valéria Lelli, Benoit Baudry, and Fabien Coulon. 2018. User Interface Design Smell: Automatic Detection and Refactoring of Blob Listeners. Information and Software Technology 102 (May 2018), 49--64. https://doi.org/10.1016/j.infsof.2018.05.005
[9]
Arnaud Blouin, Brice Morin, Olivier Beaudoux, Grégory Nain, Patrick Albers, and Jean-Marc Jézéquel. 2011. Combining Aspect-Oriented Modeling with Property-Based Reasoning to Improve User Interface Adaptation. In ACM SIGCHI Symposium on Engineering Interactive Computing Systems. ACM, Pisa, Italy, 85--94. https://doi.org/10.1145/1996461.1996500
[10]
José Creissac Campos, Camille Fayollas, Michael D Harrison, Célia Martinie, Paolo Masci, and Philippe Palanque. 2020. Supporting the analysis of safety critical user interfaces: an exploration of three formal tools. ACM Transactions on Computer-Human Interaction (TOCHI) 27, 5 (2020), 1--48. https://doi.org/10.1145/3404199
[11]
José C Campos and Michael D Harrison. 1997. Formally verifying interactive systems: A review. In Design, Specification and Verification of Interactive Systems' 97: Proceedings of the Eurographics Workshop in Granada, Spain, June 4-6, 1997. Springer, Vienna, 109--124. https://doi.org/10.1007/978-3-7091-6878-3_8
[12]
Thomas Degueule, Benoit Combemale, Arnaud Blouin, Olivier Barais, and Jean-Marc Jézéquel. 2017. Safe model polymorphism for flexible modeling. Computer Languages, Systems & Structures 49 (2017), 176--195. https://doi.org/10.1016/j.cl.2016.09.001
[13]
Tore Dybå and Torgeir Dings⊘yr. 2008. Empirical studies of agile software development: A systematic review. Information and software technology 50, 9-10 (2008), 833--859. https://doi.org/10.1016/j.infsof.2008.01.006
[14]
Juan Cruz Gardey, Alejandra Garrido, Sergio Firmenich, Julián Grigera, and Gustavo Rossi. 2020. UX-painter: an approach to explore interaction fixes in the browser. Proceedings of the ACM on Human-Computer Interaction 4, EICS, Article 89 (2020), 21 pages. https://doi.org/10.1145/3397877
[15]
Carl A Gunter. 1992. Semantics of programming languages: structures and techniques. MIT press, USA.
[16]
Clément Guy, Benoit Combemale, Steven Derrien, Jim RH Steel, and Jean-Marc Jézéquel. 2012. On model subtyping. In Modelling Foundations and Applications: 8th European Conference, ECMFA 2012. Springer, Vienna, 400--415. https://doi.org/10.1007/978-3-642-31491-9_30
[17]
Huaxun Huang, Lili Wei, Yepang Liu, and Shing-Chi Cheung. 2018. Understanding and detecting callback compatibility issues for Android applications. In Proceedings of the 33rd ACM/IEEE International Conference on Automated Software Engineering. ACM/IEEE, USA, 532--542. https://doi.org/10.1145/3238147.3238181
[18]
Scott E Hudson, Jennifer Mankoff, and Ian Smith. 2005. Extensible input handling in the subArctic toolkit. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM New York, NY, USA, USA, 381--390. https://doi.org/10.1145/1054972.1055025
[19]
Kenrick Kin, Björn Hartmann, Tony DeRose, and Maneesh Agrawala. 2012. Proton++ a customizable declarative multitouch framework. In Proceedings of the 25th annual ACM symposium on User interface software and technology. ACM, USA, 477--486. https://doi.org/10.1145/2380116.2380176
[20]
Kenrick Kin, Björn Hartmann, Tony DeRose, and Maneesh Agrawala. 2012. Proton: multitouch gestures as regular expressions. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM, USA, 2885--2894. https://doi.org/10.1145/2207676.2208694
[21]
Valéria Lelli, Arnaud Blouin, and Benoit Baudry. 2015. Classifying and Qualifying GUI Defects. In 8th IEEE International Conference on Software Testing, Verification and Validation. IEEE, USA, 1--10. https://doi.org/10.1109/ICST.2015.7102582
[22]
Valéria Lelli, Arnaud Blouin, Benoit Baudry, Fabien Coulon, and Olivier Beaudoux. 2016. Automatic Detection of GUI Design Smells: The Case of Blob Listener. In Proceedings of the 8th ACM SIGCHI Symposium on Engineering Interactive Computing Systems (EICS'16). ACM, USA, 263--274. https://doi.org/10.1145/2933242.2933260
[23]
Yuri Leontiev, M Tamer Özsu, and Duane Szafron. 2002. On type systems for object-oriented database programming languages. ACM Computing Surveys (CSUR) 34, 4 (2002), 409--449. https://doi.org/10.1145/592642.592643
[24]
Mathieu Magnaudet, Stéphane Chatty, Stéphane Conversy, Sébastien Leriche, Celia Picard, and Daniel Prun. 2018. Djnn/Smala: A conceptual framework and a language for interaction-oriented programming. Proceedings of the ACM on Human-Computer Interaction 2, EICS, Article 12 (2018), 27 pages. https://doi.org/10.1145/3229094
[25]
David Navarre, Philippe Palanque, Jean-Francois Ladry, and Eric Barboni. 2009. ICOs: A model-based user interface description technique dedicated to interactive systems addressing usability, reliability and scalability. ACM Trans. on CHI 16, 4 (2009), 1--56. https://doi.org/10.1145/1614390.1614393
[26]
Raquel Oliveira, Sophie Dupuy-Chessa, Gaelle Calvary, and Daniele Dadolle. 2016. Using formal models to cross check an implementation. In Proceedings of the 8th ACM SIGCHI Symposium on Engineering Interactive Computing Systems. ACM, USA, 126--137. https://doi.org/10.1145/2933242.2933257
[27]
Steve Oney, Rebecca Krosnick, Joel Brandt, and Brad Myers. 2019. Implementing multi-touch gestures with touch groups and cross events. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems. ACM, USA, 1--12. https://doi.org/10.1145/3290605.3300585
[28]
Vik Parthiban, Pattie Maes, Quentin Sellier, Arthur Sluÿters, and Jean Vanderdonckt. 2022. Gestural-Vocal Coordinated Interaction on Large Displays. In Companion of the 2022 ACM SIGCHI Symposium on Engineering Interactive Computing Systems. ACM, USA, 26--32. https://doi.org/10.1145/3531706.3536457
[29]
Benjamin C Pierce. 2002. Types and programming languages. MIT press, USA.
[30]
Rachel Potvin and Josh Levenberg. 2016. Why Google stores billions of lines of code in a single repository. Commun. ACM 59, 7 (2016), 78--87. https://doi.org/10.1145/2854146
[31]
Daniel Prun and Pascal Béger. 2022. Formal Verification of Graphical Properties of Interactive Systems. Proceedings of the ACM on Human-Computer Interaction 6, EICS, Article 167 (2022), 30 pages. https://doi.org/10.1145/3534521
[32]
Thibault Raffaillac and Stéphane Huot. 2019. Polyphony: Programming interfaces and interactions with the entity-component-system model. Proceedings of the ACM on Human-Computer Interaction 3, EICS, Article 8 (2019), 22 pages. https://doi.org/10.1145/3331150
[33]
Thibault Raffaillac and Stéphane Huot. 2022. What do Researchers Need when Implementing Novel Interaction Techniques? Proceedings of the ACM on Human-Computer Interaction 6, EICS, Article 159 (2022), 30 pages. https://doi.org/10.1145/3532209
[34]
Thiago Rocha Silva. 2022. Towards a Domain-Specific Language to Specify Interaction Scenarios for Web-Based Graphical User Interfaces. In Companion of the 2022 ACM SIGCHI Symposium on Engineering Interactive Computing Systems. ACM, USA, 48--53. https://doi.org/10.1145/3531706.3536463
[35]
Julia Schwarz, Scott Hudson, Jennifer Mankoff, and Andrew D Wilson. 2010. A framework for robust and flexible handling of inputs with uncertainty. In Proceedings of the 23nd annual ACM symposium on User interface software and technology. ACM New York, NY, USA, USA, 47--56. https://doi.org/10.1145/1866029.1866039
[36]
Jim Steel and Jean-Marc Jézéquel. 2007. On model typing. Software & Systems Modeling 6 (2007), 401--413. https://doi.org/10.1007/s10270-006-0036-6
[37]
Jacob O Wobbrock, Meredith Ringel Morris, and Andrew D Wilson. 2009. User-defined gestures for surface computing. In Proceedings of the SIGCHI conference on human factors in computing systems. ACM, USA, 1083--1092. https://doi.org/10.1145/1518701.1518866
Index Terms
- A Type System for Flexible User Interactions Handling
Recommendations
Yet another BPEL extension for user interactions
ER'11: Proceedings of the 30th international conference on Advances in conceptual modeling: recent developments and new directionsIn this paper, we propose a BPEL extension that deals with the user interactions expression in a Web service composition. This extension defines new data interaction activities to allow user providing, selecting or getting data. Moreover, the extension ...
Semantic web user interactions: exploring hci challenges
CHI EA '08: CHI '08 Extended Abstracts on Human Factors in Computing SystemsThe purpose of this workshop is to engage interaction researchers and designers in the user interaction challenges posed by the Semantic Web. The workshop will be the fifth workshop in the SWUI series - it has previously been hosted at either the World ...
User interaction at various levels of data displays
Data visualizationThe development of interactive statistical graphics in the 1980s switched graphics from a result-presenting device to an analytic tool. But what makes a statistical display interactive? Even within the statistical graphics community, usage and ...
Comments
Information & Contributors
Information
Published In
Copyright © 2024 ACM.
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 17 June 2024
Accepted: 01 March 2024
Revised: 01 January 2024
Received: 01 October 2023
Published in PACMHCI Volume 8, Issue EICS
Check for updates
Author Tags
Qualifiers
- Research-article
- Research
- Refereed
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 67Total Downloads
- Downloads (Last 12 months)67
- Downloads (Last 6 weeks)14
Reflects downloads up to 15 Oct 2024
Other Metrics
Citations
View Options
Get Access
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in