research-article

Public Access

HotStrokes: Word-Gesture Shortcuts on a Trackpad

Authors:

Xiaojun BiAuthors Info & Claims

CHI '19: Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems

Paper No.: 165, Pages 1 - 13

https://doi.org/10.1145/3290605.3300395

Published: 02 May 2019 Publication History

All formats PDF

Abstract

Expert interaction techniques like hotkeys are efficient, but poorly adopted because they are hard to learn. HotStrokes removes the need for learning arbitrary mappings of commands to hotkeys. A user enters a HotStroke by holding a modifier key, then gesture typing a command name on a laptop trackpad as if on an imaginary virtual keyboard. The gestures are recognized using an adaptation of the SHARK2 algorithm with a new spatial model and a refined method for dynamic suggestions. A controlled experiment shows HotStrokes effectively augments the existing "menu and hotkey" command activation paradigm. Results show the method is efficient by reducing command activation time by 43% compared to linear menus. The method is also easy to learn with a high adoption rate, replacing 91% of linear menu usage. Finally, combining linear menus, hotkeys, and HotStrokes leads to 24% faster command activation overall.

Supplementary Material

ZIP File (paper165.zip)

The supplementary file is a PDF including the commands used in collecting HotStrokes and the touch points distribution parameters of all keys obtained in Experiment I.

Download
49.70 KB

MP4 File (paper165.mp4)

Supplemental video

Download
223.17 MB

MP4 File (paper165p.mp4)

Preview video

Download
3.81 MB

References

[1]

Jason Alexander and Andy Cockburn. 2008. An Empirical Characterisation of Electronic Document Navigation. In Proceedings of Graphics Interface 2008 (GI '08). Canadian Information Processing Society, Toronto, Ont., Canada, Canada, 123--130. http://dl.acm.org/citation. cfm?id=1375714.1375736

Digital Library

[2]

Jessalyn Alvina, Carla F. Griggio, Xiaojun Bi, and Wendy E. Mackay. 2017. CommandBoard: Creating a General-Purpose Command Gesture Input Space for Soft Keyboard. In Proceedings of the 30th Annual ACM Symposium on User Interface Software and Technology (UIST '17). ACM, New York, NY, USA, 17--28.

Digital Library

[3]

Fraser Anderson and Walter F. Bischof. 2013. Learning and Performance with Gesture Guides. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '13). ACM, New York, NY, USA, 1109--1118.

Digital Library

[4]

Caroline Appert and Shumin Zhai. 2009. Using Strokes As Command Shortcuts: Cognitive Benefits and Toolkit Support. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI 1https://github.com/cuiwenzhe/HotStrokes '09). ACM, New York, NY, USA, 2289--2298.

Digital Library

[5]

Gilles Bailly, Eric Lecolinet, and Laurence Nigay. 2007. Wave Menus: Improving the Novice Mode of Hierarchical Marking Menus. In Human-Computer Interaction INTERACT 2007, CÃcilia Baranauskas, Philippe Palanque, Julio Abascal, and Simone Diniz Junqueira Barbosa (Eds.). Number 4662 in Lecture Notes in Computer Science. Springer Berlin Heidelberg, 475--488.

Digital Library

[6]

Gilles Bailly, Eric Lecolinet, and Laurence Nigay. 2008. Flower Menus: A New Type of Marking Menu with Large Menu Breadth, Within Groups and Efficient Expert Mode Memorization. In Proceedings of the Working Conference on Advanced Visual Interfaces (AVI '08). ACM, New York, NY, USA, 15--22.

Digital Library

[7]

Olivier Bau and Wendy E. Mackay. 2008. OctoPocus: A Dynamic Guide for Learning Gesture-based Command Sets. In Proceedings of the 21st Annual ACM Symposium on User Interface Software and Technology (UIST '08). ACM, New York, NY, USA, 37--46.

Digital Library

[8]

Xiaojun Bi, Ciprian Chelba, Tom Ouyang, Kurt Partridge, and Shumin Zhai. 2012. Bimanual Gesture Keyboard. In Proceedings of the 25th Annual ACM Symposium on User Interface Software and Technology (UIST '12). ACM, New York, NY, USA, 137--146.

Digital Library

[9]

Andy Cockburn, Carl Gutwin, Joey Scarr, and Sylvain Malacria. 2014. Supporting Novice to Expert Transitions in User Interfaces. ACM Comput. Surv. 47, 2, Article 31 (Nov. 2014), 36 pages. https://doi.org/

Digital Library

[10]

1145/2659796 {10} Sarah M. Donelson and Claire C. Gordon. 1996. 1995 Matched Anthropometric Database of US Marine Corps Personnel: Summary Statistics.

[11]

JÃrÃmie Francone, Gilles Bailly, Eric Lecolinet, Nadine Mandran, and Laurence Nigay. 2010. Wavelet Menus on Handheld Devices: Stacking Metaphor for Novice Mode and Eyes-free Selection for Expert Mode. In Proceedings of the International Conference on Advanced Visual Interfaces (AVI '10). ACM, New York, NY, USA, 173--180.

Digital Library

[12]

Jeremie Francone, Gilles Bailly, Laurence Nigay, and Eric Lecolinet. 2009. Wavelet Menus: A Stacking Metaphor for Adapting Marking Menus to Mobile Devices. In Proceedings of the 11th International Conference on Human-Computer Interaction with Mobile Devices and Services (MobileHCI '09). ACM, New York, NY, USA, 49:1--49:4.

Digital Library

[13]

Bruno Fruchard, Eric Lecolinet, and Olivier Chapuis. 2017. MarkPad: Augmenting Touchpads for Command Selection. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems (CHI '17). ACM, New York, NY, USA, 5630--5642.

Digital Library

[14]

Emmanouil Giannisakis, Gilles Bailly, Sylvain Malacria, and Fanny Chevalier. 2017. IconHK: Using Toolbar Button Icons to Communicate Keyboard Shortcuts. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems (CHI '17). ACM, New York, NY, USA, 4715--4726.

Digital Library

[15]

Tovi Grossman, Pierre Dragicevic, and Ravin Balakrishnan. 2007. Strategies for Accelerating On-line Learning of Hotkeys. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '07). ACM, New York, NY, USA, 1591--1600.

Digital Library

[16]

Sean Gustafson, Daniel Bierwirth, and Patrick Baudisch. 2010. Imaginary Interfaces: Spatial Interaction with Empty Hands and Without Visual Feedback. In Proceedings of the 23Nd Annual ACM Symposium on User Interface Software and Technology (UIST '10). ACM, New York, NY, USA, 3--12.

Digital Library

[17]

Sean Gustafson, Christian Holz, and Patrick Baudisch. 2011. Imaginary Phone: Learning Imaginary Interfaces by Transferring Spatial Memory from a Familiar Device. In Proceedings of the 24th Annual ACM Symposium on User Interface Software and Technology (UIST '11). ACM, New York, NY, USA, 283--292.

Digital Library

[18]

Jeff Hendy, Kellogg S. Booth, and Joanna McGrenere. 2010. Graphically Enhanced Keyboard Accelerators for GUIs. In Proceedings of Graphics Interface 2010 (GI '10). Canadian Information Processing Society, Toronto, Ont., Canada, Canada, 3--10. http://dl.acm.org/citation.cfm? id=1839214.1839217

Digital Library

[19]

Per-Ola Kristensson and Shumin Zhai. 2004. SHARK2: A Large Vocabulary Shorthand Writing System for Pen-based Computers. In Proceedings of the 17th Annual ACM Symposium on User Interface Software and Technology (UIST '04). ACM, New York, NY, USA, 43--52.

Digital Library

[20]

Per Ola Kristensson and Shumin Zhai. 2007. Command Strokes with and Without Preview: Using Pen Gestures on Keyboard for Command Selection. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '07). ACM, New York, NY, USA, 1137--1146.

Digital Library

[21]

Gordon Kurtenbach and William Buxton. 1994. User Learning and Performance with Marking Menus. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '94). ACM, New York, NY, USA, 258--264.

Digital Library

[22]

Gordon Kurtenbach, George W. Fitzmaurice, Russell N. Owen, and Thomas Baudel. 1999. The Hotbox: Efficient Access to a Large Number of Menu-items. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '99). ACM, New York, NY, USA, 231--237.

Digital Library

[23]

G. Kurtenbach, T. P. Moran, and W. Buxton. 1994. Contextual animation of gestural commands. In Computer Graphics Forum. 83--90.

[24]

Gordon Paul Kurtenbach. 1993. The Design and Evaluation of Marking Menus. Technical Report.

[25]

Gordon P. Kurtenbach, Abigail J. Sellen, and William A. S. Buxton. 1993. An Empirical Evaluation of Some Articulatory and Cognitive Aspects of Marking Menus. Hum.-Comput. Interact. 8, 1 (March 1993), 1--23.

Digital Library

[26]

Benjamin Lafreniere, Carl Gutwin, and Andy Cockburn. 2017. Investigating the Post-Training Persistence of Expert Interaction Techniques. ACM Trans. Comput.-Hum. Interact. 24, 4 (Aug. 2017), 29:1-- 29:46.

Digital Library

[27]

David M. Lane, H. Albert Napier, S. Camille Peres, and Aniko Sandor. 2005. Hidden Costs of Graphical User Interfaces: Failure to Make the Transition from Menus and Icon Toolbars to Keyboard Shortcuts. International Journal of Human Computer Interaction 18, 2 (2005), 133-- 144.

[28]

Yang Li. 2010. Gesture Search: A Tool for Fast Mobile Data Access. In UIST'10: Symposium on User Interface Software and Technology. 87--96.

Digital Library

[29]

Hao LÃ and Yang Li. 2011. Gesture Avatar: A Technique for Operating Mobile User Interfaces Using Gestures. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '11). ACM, New York, NY, USA, 207--216.

Digital Library

[30]

Sylvain Malacria, Gilles Bailly, Joel Harrison, Andy Cockburn, and Carl Gutwin. 2013. Promoting Hotkey Use Through Rehearsal with ExposeHK. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '13). ACM, New York, NY, USA, 573--582.

Digital Library

[31]

Anders Markussen, Mikkel Rønne Jakobsen, and Kasper Hornbæk. 2014. Vulture: A Mid-air Word-gesture Keyboard. In Proceedings of the 32Nd Annual ACM Conference on Human Factors in Computing Systems (CHI '14). ACM, New York, NY, USA, 1073--1082.

Digital Library

[32]

Hugh Mcloone, Ken Hinckley, and Edward Cutrell. 2003. Bimanual interaction on the Microsoft Office Keyboard. In In Proceedings of the Conference on HumanComputer Interaction (INTERACT. 49--56.

[33]

Douglas L Nelson, Valerie S Reed, and John R Walling. 1976. Pictorial superiority effect. Journal of Experimental Psychology: Human Learning and Memory 2, 5 (1976), 523.

[34]

Jakob Nielsen. 1992. Finding Usability Problems Through Heuristic Evaluation. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '92). ACM, New York, NY, USA, 373--380.

Digital Library

[35]

Daniel L. Odell, Richard C. Davis, Andrew Smith, and Paul K. Wright. 2004. Toolglasses, Marking Menus, and Hotkeys: A Comparison of One and Two-handed Command Selection Techniques. In Proceedings of Graphics Interface 2004 (GI '04). Canadian Human-Computer Communications Society, School of Computer Science, University of Waterloo, Waterloo, Ontario, Canada, 17--24. http://dl.acm.org/citation.cfm?id= 1006058.1006061

Digital Library

[36]

C. J. Van Rijsbergen. 1979. Information Retrieval (2nd ed.). ButterworthHeinemann, Newton, MA, USA.

Digital Library

[37]

H. Sakoe and S. Chiba. 1978. Dynamic programming algorithm optimization for spoken word recognition. IEEE Transactions on Acoustics, Speech, and Signal Processing 26, 1 (February 1978), 43--49.

[38]

Joey Scarr, Andy Cockburn, Carl Gutwin, and Philip Quinn. 2011. Dips and Ceilings: Understanding and Supporting Transitions to Expertise in User Interfaces. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '11). ACM, New York, NY, USA, 2741--2750.

Digital Library

[39]

Susanne Tak, Piet Westendorp, and Iris van Rooij. 2013. Satisficing and the Use of Keyboard Shortcuts: Being Good Enough Is Enough? Interacting with Computers 25, 5 (2013), 404--416.

[40]

Hui-Shyong Yeo, Xiao-Shen Phang, Steven J. Castellucci, Per Ola Kristensson, and Aaron Quigley. 2017. Investigating Tilt-based Gesture Keyboard Entry for Single-Handed Text Entry on Large Devices. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems (CHI '17). ACM, New York, NY, USA, 4194--4202.

Digital Library

[41]

Chun Yu, Yizheng Gu, Zhican Yang, Xin Yi, Hengliang Luo, and Yuanchun Shi. 2017. Tap, Dwell or Gesture?: Exploring Head-Based Text Entry Techniques for HMDs. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems (CHI '17). ACM, New York, NY, USA, 4479--4488.

Digital Library

[42]

Shumin Zhai and Per-Ola Kristensson. 2003. Shorthand Writing on Stylus Keyboard. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '03). ACM, New York, NY, USA, 97--104.

Digital Library

[43]

Shumin Zhai and Per Ola Kristensson. 2012. The Word-gesture Keyboard: Reimagining Keyboard Interaction. Commun. ACM 55, 9 (Sept. 2012), 91--101.

Digital Library

[44]

Shumin Zhai, Per Ola Kristensson, Caroline Appert, Tue Haste Andersen, and Xiang Cao. 2012. Foundational issues in touch-screen stroke gesture design-an integrative review. Foundations and Trends in Human-Computer Interaction 5, 2 (2012), 97--205. https://hal.inria. fr/hal-00765046/

Digital Library

[45]

Shumin Zhai, Per Ola Kristensson, Pengjun Gong, Michael Greiner, Shilei Allen Peng, Liang Mico Liu, and Anthony Dunnigan. 2009. Shapewriter on the Iphone: From the Laboratory to the Real World. In CHI '09 Extended Abstracts on Human Factors in Computing Systems (CHI EA '09). ACM, New York, NY, USA, 2667--2670.

Digital Library

[46]

Shengdong Zhao and Ravin Balakrishnan. 2004. Simple vs. Compound Mark Hierarchical Marking Menus. In Proceedings of the 17th Annual ACM Symposium on User Interface Software and Technology (UIST '04). ACM, New York, NY, USA, 33--42.

Digital Library

[47]

Jingjie Zheng, Xiaojun Bi, Kun Li, Yang Li, and Shumin Zhai. 2018. M3 Gesture Menu: Design and Experimental Analyses of Marking Menus for Touchscreen Mobile Interaction. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems (CHI '18). ACM, New York, NY, USA, Article 249, 14 pages.

Digital Library

[48]

Jingjie Zheng, Blaine Lewis, Jeff Avery, and Daniel Vogel. 2018. FingerArc and FingerChord: Supporting Novice to Expert Transitions with Guided Finger-Aware Shortcuts. In The 31st Annual ACM Symposium on User Interface Software and Technology. New York, NY, 17p.

Digital Library

[49]

Jingjie Zheng and Daniel Vogel. 2016. Finger-Aware Shortcuts. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (CHI '16). ACM, New York, NY, USA, 4274--4285.

Digital Library

Cited By

Zhang DLi ZAshok VSeiple WRamakrishnan IBi X(2024)Accessible Gesture Typing on Smartphones for People with Low VisionProceedings of the 37th Annual ACM Symposium on User Interface Software and Technology10.1145/3654777.3676447(1-11)Online publication date: 13-Oct-2024
https://dl.acm.org/doi/10.1145/3654777.3676447
Wan TZhang LYang HIrani PYu LLiang H(2024)Exploration of Foot-based Text Entry Techniques for Virtual Reality EnvironmentsProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642757(1-17)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3642757
Chu JAn DMa YCui WZhai SGu XBi X(2023)WordGesture-GAN: Modeling Word-Gesture Movement with Generative Adversarial NetworkProceedings of the 2023 CHI Conference on Human Factors in Computing Systems10.1145/3544548.3581279(1-15)Online publication date: 19-Apr-2023
https://dl.acm.org/doi/10.1145/3544548.3581279
Show More Cited By

Index Terms

HotStrokes: Word-Gesture Shortcuts on a Trackpad
1. Human-centered computing
  1. Human computer interaction (HCI)
    1. Interaction devices
    2. Interaction techniques

Recommendations

ArmTouch: The Forearm as Touchpad for VR Control
MuC '19: Proceedings of Mensch und Computer 2019

Interacting with elements inside a virtual reality (VR) experience is usually done with a controller. However, they tend to be clunky and cumbersome to use. We designed an alternative controller that allows for a rich touch interaction. Our prototype ...
Tabletop sharing of digital photographs for the elderly
CHI '06: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems

We have recently begun to see hardware support for the tabletop user interface, offering a number of new ways for humans to interact with computers. Tabletops offer great potential for face-to-face social interaction; advances in touch technology and ...
Enhancing Pinch-Drag-Flick Paradigm with Two New Gestures: Two-Finger-Tap for Tablets and Tap&Tap for Smartphones
Human-Computer Interaction – INTERACT 2015
Abstract
The mobile versions of services such as Google Maps or Open Street Maps allow the exploration of maps on smartphones and tablets. The gestures used are the pinch to adjust the zoom level and the drag/flick to move the map. In this paper, two new ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

CHI '19: Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems

May 2019

9077 pages

ISBN:9781450359702

DOI:10.1145/3290605

General Chairs:
Stephen Brewster
University of Glasgow, Scotland, UK
,
Geraldine Fitzpatrick
TU Wien, Austria
,
Program Chairs:
Anna Cox
University College London, UK
,
Vassilis Kostakos
University of Melbourne, Australia

Copyright © 2019 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

SIGCHI: ACM Special Interest Group on Computer-Human Interaction

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 02 May 2019

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

National Science Foundation

Conference

CHI '19

Sponsor:

SIGCHI

CHI '19: CHI Conference on Human Factors in Computing Systems

May 4 - 9, 2019

Glasgow, Scotland Uk

Acceptance Rates

CHI '19 Paper Acceptance Rate 703 of 2,958 submissions, 24%;

Overall Acceptance Rate 6,199 of 26,314 submissions, 24%

Upcoming Conference

CHI 2025

Sponsor:
sigchi

ACM CHI Conference on Human Factors in Computing Systems

April 26 - May 1, 2025

Yokohama , Japan

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

17
Total Citations
View Citations
1,442
Total Downloads

Downloads (Last 12 months)267
Downloads (Last 6 weeks)37

Reflects downloads up to 23 Feb 2025

Other Metrics

View Author Metrics

Citations

Cited By

Zhang DLi ZAshok VSeiple WRamakrishnan IBi X(2024)Accessible Gesture Typing on Smartphones for People with Low VisionProceedings of the 37th Annual ACM Symposium on User Interface Software and Technology10.1145/3654777.3676447(1-11)Online publication date: 13-Oct-2024
https://dl.acm.org/doi/10.1145/3654777.3676447
Wan TZhang LYang HIrani PYu LLiang H(2024)Exploration of Foot-based Text Entry Techniques for Virtual Reality EnvironmentsProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642757(1-17)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3642757
Chu JAn DMa YCui WZhai SGu XBi X(2023)WordGesture-GAN: Modeling Word-Gesture Movement with Generative Adversarial NetworkProceedings of the 2023 CHI Conference on Human Factors in Computing Systems10.1145/3544548.3581279(1-15)Online publication date: 19-Apr-2023
https://dl.acm.org/doi/10.1145/3544548.3581279
Cui WLiu RLi ZWang YWang AZhao XRashidian SBaig FRamakrishnan IWang FBi X(2023)GlanceWriter: Writing Text by Glancing Over Letters with GazeProceedings of the 2023 CHI Conference on Human Factors in Computing Systems10.1145/3544548.3581269(1-13)Online publication date: 19-Apr-2023
https://dl.acm.org/doi/10.1145/3544548.3581269
Lemarquis DYilma BLeiva L(2023)What can a swiped word tell us more? Demographic and behavioral correlates from shape-writing text entryNeural Computing and Applications10.1007/s00521-023-08559-035:21(15531-15548)Online publication date: 13-Apr-2023
https://doi.org/10.1007/s00521-023-08559-0
Xu ZMeng YBi XYang X(2022)Phrase-Gesture Typing on SmartphonesProceedings of the 35th Annual ACM Symposium on User Interface Software and Technology10.1145/3526113.3545683(1-11)Online publication date: 29-Oct-2022
https://dl.acm.org/doi/10.1145/3526113.3545683
Chhibber NSurale HMatulic FVogel D(2021)TypealikeProceedings of the ACM on Human-Computer Interaction10.1145/34869525:ISS(1-20)Online publication date: 5-Nov-2021
https://dl.acm.org/doi/10.1145/3486952
Zhao MCui WRamakrishnan IZhai SBi X(2021)Voice and Touch Based Error-tolerant Multimodal Text Editing and Correction for SmartphonesThe 34th Annual ACM Symposium on User Interface Software and Technology10.1145/3472749.3474742(162-178)Online publication date: 10-Oct-2021
https://dl.acm.org/doi/10.1145/3472749.3474742
Keddisseh ESerrano MDubois E(2021)SmartCom: Exploiter un Smartphone pour Interagir avec les Barres d'OutilsProceedings of the 32nd Conference on l'Interaction Homme-Machine10.1145/3450522.3451244(1-10)Online publication date: 13-Apr-2021
https://dl.acm.org/doi/10.1145/3450522.3451244
Keddisseh ESerrano MDubois EKitamura YQuigley AIsbister KIgarashi TBjørn PDrucker S(2021)KeyTch: Combining the Keyboard with a Touchscreen for Rapid Command Selection on ToolbarsProceedings of the 2021 CHI Conference on Human Factors in Computing Systems10.1145/3411764.3445288(1-13)Online publication date: 6-May-2021
https://dl.acm.org/doi/10.1145/3411764.3445288
Show More Cited By

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

HTML Format

View this article in HTML Format.

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Figures

Tables

Media

View Table of Conten