research-article

Creating and Augmenting Keyboards for Extended Reality with the Keyboard Augmentation Toolkit

Authors:

Stephen Brewster,

Daniel Pires De Sa Medeiros,

Mario Gutierrez,

Aidan KehoeAuthors Info & Claims

ACM Transactions on Computer-Human Interaction, Volume 29, Issue 2

Article No.: 15, Pages 1 - 39

https://doi.org/10.1145/3490495

Published: 16 January 2022 Publication History

Abstract

This article discusses the Keyboard Augmentation Toolkit (KAT), which supports the creation of virtual keyboards that can be used both for standalone input (e.g., for mid-air text entry) and to augment physically tracked keyboards/surfaces in mixed reality. In a user study, we firstly examine the impact and pitfalls of visualising shortcuts on a tracked physical keyboard, exploring the utility of virtual per-keycap displays. Supported by this and other recent developments in XR keyboard research, we then describe the design, development, and evaluation-by-demonstration of KAT. KAT simplifies the creation of virtual keyboards (optionally bound to a tracked physical keyboard) that support enhanced display—2D/3D per-key content that conforms to the virtual key bounds; enhanced interactivity—supporting extensible per-key states such as tap, dwell, touch, swipe; flexible keyboard mappings that can encapsulate groups of interaction and display elements, e.g., enabling application-dependent interactions; and flexible layouts—allowing the virtual keyboard to merge with and augment a physical keyboard, or switch to an alternate layout (e.g., mid-air) based on need. Through these features, KAT will assist researchers in the prototyping, creation and replication of XR keyboard experiences, fundamentally altering the keyboard’s form and function.

Supplementary Material

mcgill (mcgill.zip)

Supplemental movie, appendix, image and software files for, Creating and Augmenting Keyboards for Extended Reality with the Keyboard Augmentation Toolkit

Download
79.63 MB

References

[1]

Jonathan Aceituno and Nicolas Roussel. 2014. The hotkey palette: Flexible contextual retrieval of chosen documents and windows. In Proceedings of the 26th Conference on l’Interaction Homme-Machine. 55–59. DOI:https://doi.org/10.1145/2670444.2670452

Digital Library

[2]

Apple. 2018. Apple Touch Bar. Retrieved 10th August 2021 from https://developer.apple.com/macos/touch-bar/.

[3]

Apple. 2018. Use Spotlight on Your Mac. Retrieved from https://support.apple.com/en-gb/HT204014.

[4]

Ahmed Sabbir Arif and Ali Mazalek. 2016. WebTEM: A Web application to record text entry metrics. In Proceedings of the 2016 ACM on Interactive Surfaces and Spaces. DOI:https://doi.org/10.1145/2992154.2996791

Digital Library

[5]

Mahdi Azmandian, Mark Hancock, Hrvoje Benko, Eyal Ofek, and Andrew D. Wilson. 2016. Haptic retargeting. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems. ACM, New York, NY, 1968–1979. DOI: https://doi.org/10.1145/2858036.2858226

Digital Library

[6]

Gilles Bailly, Thomas Pietrzak, Jonathan Deber, and Daniel J. Wigdor. 2013. Métamorphe: Augmenting hotkey usage with actuated keys. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. 563–572. DOI: https://doi.org/10.1145/2470654.2470734

Digital Library

[7]

Florent Berthaut and Alex Jones. 2016. ControllAR: Appropriation of visual feedback on control surfaces. In Proceedings of the 2016 ACM International Conference on Interactive Surfaces and Spaces. DOI:https://doi.org/10.1145/2992154.2998580

Digital Library

[8]

Xiaojun Bi, Tovi Grossman, Justin Matejka, and George Fitzmaurice. 2011. Magic desk: Bringing multi-touch surfaces into desktop work. In Proceedings of the 2011 Annual Conference on Human Factors in Computing Systems. 2511. DOI: https://doi.org/10.1145/1978942.1979309

Digital Library

[9]

Mark Billinghurst, Adrian Clark, and Gun Lee. 2015. A survey of augmented reality. Foundations and Trends in Human-Computer Interaction 8, 2–3 (2015), 73–272. DOI: https://doi.org/10.1561/1100000049

Digital Library

[10]

Florian Block, Hans Gellersen, and Nicolas Villar. 2010. Touch-display keyboards: Transforming keyboards into interactive surfaces. In Proceedings of the 28th International Conference on Human Factors in Computing Systems. 1145. DOI: https://doi.org/10.1145/1753326.1753498

[11]

Sidney Bovet, Noirin Curran, Aidan Kehoe, Mario Gutierrez, Thomas Rouvinez, and Katie Crowley. 2018. Using traditional keyboards in VR: SteamVR developer kit and pilot game user study. In Proceedings of the 2018 IEEE Games, Entertainment, Media Conference. 131–134.

[12]

D. A. Bowman. 2020. Embracing physical keyboards for virtual reality. Computer 53, 09 (Sep 2020), 9–10. DOI: https://doi.org/10.1109/MC.2020.3004605

[13]

J. Brooke. 1996. SUS-A quick and dirty usability scale. Usability Evaluation in Industry 189, 194 (1996), 4–7.

[14]

Daniel Buschek, Bianka Roppelt, and Florian Alt. 2018. Extending keyboard shortcuts with arm and wrist rotation gestures. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. 1–12. DOI: https://doi.org/10.1145/3173574.3173595

[15]

Lung-Pan Cheng, Li Chang, Sebastian Marwecki, and Patrick Baudisch. 2018. iTurk: Turning passive haptics into active haptics by making users reconfigure props in virtual reality. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. ACM, New York, NY. 1–10. DOI: https://doi.org/10.1145/3173574.3173663

Digital Library

[16]

Lung Pan Cheng, Eyal Ofek, Christian Holz, Hrvoje Benko, and Andrew D. Wilson. 2017. Sparse haptic proxy: Touch feedback in virtual environments using a general passive prop. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems. Vol. 2017-May, 3718–3728. DOI:https://doi.org/10.1145/3025453.3025753

Digital Library

[17]

Clodo76. 2018. VR Desktop Mirror. Retrieved from https://github.com/Clodo76/vr-desktop-mirror.

[18]

Andy Cockburn, Carl Gutwin, Joey Scarr, and Sylvain Malacria. 2014. Supporting novice to expert transitions in user interfaces. ACM Computing Surveys 47, 2, Article 31 (2014), 36 pages. DOI: https://doi.org/10.1145/2659796

[19]

Christian Corsten, Ignacio Avellino, Max Möllers, and Jan Borchers. 2013. Instant user interfaces: Repurposing everyday objects as input devices. In Proceedings of the 2013 ACM International Conference on Interactive Tabletops and Surfaces. 71–80. DOI: https://doi.org/10.1145/2512349.2512799

Digital Library

[20]

Steve Draper. 2018. Effect Size. Retrieved from http://www.psy.gla.ac.uk/steve/best/effect.html.

[21]

Tafadzwa Joseph Dube and Ahmed Sabbir Arif. 2020. Impact of key shape and dimension on text entry in virtual reality. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems. ACM, New York, NY, 1–10. DOI:https://doi.org/10.1145/3334480.3382882

[22]

J. Dudley, H. Benko, D. Wigdor, and P. O. Kristensson. 2019. Performance envelopes of virtual keyboard text input strategies in virtual reality. In Proceedings of the 2019 IEEE International Symposium on Mixed and Augmented Reality. 289–300. DOI:https://doi.org/10.1109/ISMAR.2019.00027

[23]

Barrett Ens, Juan David Hincapié-Ramos, and Pourang Irani. 2014. Ethereal planes: A design framework for 2D information space in 3D mixed reality environments. In Proceedings of the 2nd ACM Symposium on Spatial User Interaction. 2–12. DOI:https://doi.org/10.1145/2659766.2659769

Digital Library

[24]

Barrett Ens, Eyal Ofek, Neil Bruce, and Pourang Irani. 2015. Spatial constancy of surface-embedded layouts across multiple environments. In Proceedings of the 3rd ACM Symposium on Spatial User Interaction. ACM, New York, NY, 65–68. DOI: https://doi.org/10.1145/2788940.2788954

Digital Library

[25]

Anna Maria Feit. 2018. Assignment Problems for Optimizing Text Input. Ph.D. Dissertation. Aalto University . Retrieved fromhttp://urn.fi/URN:ISBN:978-952-60-8016-1.

[26]

Anna Maria Feit, Mathieu Nancel, Maximilian John, Andreas Karrenbauer, Daryl Weir, and Antti Oulasvirta. 2021. AZERTY AméLioré: Computational design on a national scale. Communications of the ACM 64, 2 (Jan. 2021), 48–58. DOI: https://doi.org/10.1145/3382035

Digital Library

[27]

Anna Maria Feit, Mathieu Nancel, Daryl Weir, Gilles Bailly, Maximilian John, Andreas Karrenbauer, and Antti Oulasvirta. 2018. Élaboration de la disposition AZERTY modernisée. Retrieved from https://hal.inria.fr/hal-01826476

[28]

Nadia Fereydooni and Bruce N. Walker. 2020. Virtual Reality as a Remote Workspace Platform: Opportunities and Challenges. Retrieved from https://www.microsoft.com/en-us/research/publication/virtual-reality-as-a-remote-workspace-platform-opportunities-and-challenges/.

[29]

Djyp Forest Fortin and Maxime Labelle. 2021. Implémentation de la norme AFNOR. Retrieved from https://github.com/Djyp/azerty_afnor.

[30]

Euan Freeman, Stephen Brewster, and Vuokko Lantz. 2016. Do that, there: An interaction technique for addressing in-air gesture systems. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems. DOI:https://doi.org/10.1145/2858036.2858308

Digital Library

[31]

Bruno Fruchard, Eric Lecolinet, and Olivier Chapuis. 2017. MarkPad: Augmenting touchpads for command selection. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems. Vol. 2017-May, 5630–5642. DOI: https://doi.org/10.1145/3025453.3025486

Digital Library

[32]

H. Gellersen and F. Block. 2012. Novel interactions on the keyboard. Computer 45, 4, 36–40. DOI: https://doi.org/10.1109/MC.2012.112

Digital Library

[33]

Renaud Gervais, Joan Sol Roo, and Martin Hachet. 2016. Tangible viewports: Getting out of flatland in desktop environments. In Proceedings of the 10th Anniversary Conference on Tangible Embedded and Embodied Interaction. 176–184. DOI: https://doi.org/10.1145/2839462.2839468

Digital Library

[34]

Emmanouil Giannisakis, Gilles Bailly, Sylvain Malacria, and Fanny Chevalier. 2017. IconHK: Using toolbar button icons to communicate keyboard shortcuts. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems. DOI:https://doi.org/10.1145/3025453.3025595%3E

Digital Library

[35]

Antonio Gomes, Tristan Trutna, and Roel Vertegaal. 2015. Display cover: A tablet keyboard with an embedded thin-film touchscreen display. In Proceedings of the 17th International Conference on Human-Computer Interaction with Mobile Devices and Services. 531–535. DOI: https://doi.org/10.1145/2785830.2785843

Digital Library

[36]

Keenan R. Gray. 2018. Facilitating Keyboard Use While Wearing a Head-Mounted Display. Master’s thesis. Worcester Polytechnic Institute. Retrieved from https://digitalcommons.wpi.edu/etd-theses/357.

[37]

Jens Grubert, Lukas Witzani, Eyal Ofek, Michel Pahud, Matthias Kranz, and Per Ola Kristensson. 2018. Effects of hand representations for typing in virtual reality. InProceedings of the 2018 IEEE Conference on Virtual Reality and 3D User Interfaces. Retrieved from http://arxiv.org/abs/1802.00613.

[38]

Jens Grubert, Lukas Witzani, Eyal Ofek, Michel Pahud, Matthias Kranz, and Per Ola Kristensson. 2018. Text entry in immersive head-mounted display-based virtual reality using standard keyboards. In Proceedings of the 2018 IEEE Conference on Virtual Reality and 3D User Interfaces.

[39]

Pelin Gul. [n.d.]. Knowledge of the Microsoft Word Keyboard Shortcuts Expert vs. Novice Users. Technical Report.

[40]

Aldo Gunsing et al. 2021. KLFC: Keyboard Layout Files Creator. Retrieved from https://github.com/39aldo39/klfc.

[41]

S. G. Hart and L. E. Staveland. 1988. Development of NASA-TLX (Task Load Index): Results of empirical and theoretical research. In Human Mental Workload. P. A. Hancock & N. Meshkati (Eds.), North-Holland, 139–183. Retrieved from http://humanfactors.arc.nasa.gov/groups/TLX/downloads/NASA-TLXChapter.pdf.

[42]

Doris Hausen, Sebastian Boring, and Saul Greenberg. 2013. The unadorned desk: Exploiting the physical space around a display as an input canvas. In Human-Computer Interaction INTERACT 2013. P. Kotzé, G. Marsden, G. Lindgaard, J. Wesson, and M. Winckler (Eds.), Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), Vol. 8117, Springer. 140–158. DOI:https://doi.org/10.1007/978-3-642-40483-2_10

[43]

Anuruddha Hettiarachchi and Daniel Wigdor. 2016. Annexing reality. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems. ACM, New York, NY, 1957–1967. DOI: https://doi.org/10.1145/2858036.2858134

Digital Library

[44]

Adrian H. Hoppe, Leonard Otto, Florian van de Camp, Rainer Stiefelhagen, and Gabriel Unmüßig. 2018. qVRty: Virtual keyboard with a haptic, real-world representation. In HCI International 2018 Posters’ Extended Abstracts. C. Stephanidis (Ed.), Communications in Computer and Information Science, Vol. 851, Springer, Cham, 266–272. DOI:https://doi.org/10.1007/978-3-319-92279-9_36

[45]

Brent E. Insko. 2019. OpenXR State of the Union. Retrieved from https://www.khronos.org/assets/uploads/developers/library/2019-gdc/OpenXR-Overview-GDC_Mar19.pdf.

[46]

Andrew F. Jarosz and Jennifer Wiley. 2014. What are the odds? A practical guide to computing and reporting bayes factors. The Journal of Problem Solving 7, 1 (2014), 2. DOI:https://doi.org/10.7771/1932-6246.1167

[47]

Shaun K. Kane, Daniel Avrahami, Jacob O. Wobbrock, Beverly Harrison, Adam D. Rea, Matthai Philipose, and Anthony LaMarca. 2009. Bonfire: A nomadic system for hybrid laptop-tabletop interaction. In Proceedings of the 22nd Annual ACM Symposium on User Interface Software and Technology. ACM, New York, NY, 129–138. DOI:https://doi.org/10.1145/1622176.1622202

Digital Library

[48]

Pascal Knierim, Valentin Schwind, Anna Maria Feit, Florian Nieuwenhuizen, and Niels Henze. 2018. Physical keyboards in virtual reality: Analysis of typing performance and effects of avatar hands. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. DOI:https://doi.org/10.1145/3173574.3173919

[49]

Martin Krzywinski and Naomi Altman. 2014. Points of significance: Visualizing samples with box plots. Nature Methods 11, 2 (Jan 2014), 119–120. DOI: https://doi.org/10.1038/nmeth.2813

[50]

Neeraj Kumar. 2015. Finger Swipe Gesture Detection in C#. Retrieved from https://github.com/neervfx/swipe.

[51]

Alton X. Lam. 2018. ImageMSO Microsoft Office Icons (ImageMSO) Gallery & Extraction. Retrieved from https://archive.codeplex.com/?p=imagemso.

[52]

David M. Lane, H. Albert Napier, S. Camille Peres, and Aniko Sandor. 2005. Hidden costs of graphical user interfaces: Failure to make the transition from menus and icon toolbars to keyboard shortcuts. International Journal of Human-Computer Interaction 18, 2 (May 2005), 133–144. DOI: https://doi.org/10.1207/s15327590ijhc1802_1

[53]

David Ledo, Steven Houben, Jo Vermeulen, Nicolai Marquardt, Lora Oehlberg, and Saul Greenberg. 2018. Evaluation strategies for HCI toolkit research. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. ACM, New York, NY. DOI: https://doi.org/10.1145/3173574.3173610

Digital Library

[54]

Heejin Lee. 2018. Hain Launcher. Retrieved from http://hainproject.github.io/hain/.

[55]

R. V. Lenth. 2016. Least-squares means: The R package lsmeans. Journal of Statistical Software 69, 1 (2016). Retrieved from https://www.jstatsoft.org/article/view/v069i01/v69i01.pdf.

[56]

Jia-Wei Lin, Ping-Hsuan Han, Jiun-Yu Lee, Yang-Sheng Chen, Ting-Wei Chang, Kuan-Wen Chen, and Yi-Ping Hung. 2017. Visualizing the keyboard in virtual reality for enhancing immersive experience. In Proceedings of the ACM SIGGRAPH 2017 Posters . ACM, New York, NY, 1–33. DOI:https://doi.org/10.1145/3102163.3102175

Digital Library

[57]

David Lindlbauer, Jörg Müller, and Marc Alexa. 2017. Changing the appearance of real-world objects by modifying their surroundings. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems. DOI:https://doi.org/10.1145/3025453.3025795

[58]

Logitech, Inc.2011. G19 Keyboard for Gaming—Logitech Support. Retrieved from https://support.logitech.com/en%5C_us/product/g19-keyboard-for-gaming/specs.

[59]

I. Scott MacKenzie and R. William Soukoreff. 2003. Phrase sets for evaluating text entry techniques. In Proceedings of the CHI’03 Extended Abstracts on Human Factors in Computing Systems. DOI:https://doi.org/10.1145/765891.765971

Digital Library

[60]

Anindya Maiti, Murtuza Jadliwala, and Chase Weber. 2017. Preventing shoulder surfing using randomized augmented reality keyboards. In Proceedings of the 2017 IEEE International Conference on Pervasive Computing and Communications Workshops. IEEE, 630–635. DOI:https://doi.org/10.1109/PERCOMW.2017.7917636

[61]

Sylvain Malacria, Gilles Bailly, Joel Harrison, Andy Cockburn, and Carl Gutwin. 2013. Promoting hotkey use through rehearsal with ExposeHK. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM, New York, NY, 573–582. DOI:https://doi.org/10.1145/2470654.2470735

Digital Library

[62]

S. a Malacria, G. b Bailly, J. a Harrison, A. a Cockburn, and C. c Gutwin. 2013. Promoting hotkey use through rehearsal with ExposeHK. InProceedings of the SIGCHI Conference on Human Factors in Computing Systems, 573–582. DOI:https://doi.org/10.1145/2470654.2470735

Digital Library

[63]

Sylvain Malacria, Joey Scarr, Andy Cockburn, Carl Gutwin, and Tovi Grossman. 2013. Skillometers: Reflective widgets that motivate and help users to improve performance. In Proceedings of the 26th Annual ACM Symposium on User Interface Software and Technology. ACM, New York, NY, 321–330. DOI:https://doi.org/10.1145/2501988.2501996

Digital Library

[64]

Mark McGill, Daniel Boland, Roderick Murray-Smith, and Stephen Brewster. 2015. A dose of reality: Overcoming usability challenges in VR Head-Mounted Displays. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems. ACM, New York, NY. DOI:https://doi.org/10.1145/2702123.2702382

Digital Library

[65]

Mark Mcgill, Aidan Kehoe, Euan Freeman, and Stephen Brewster. 2020. Expanding the bounds of seated virtual workspaces. ACM Transactions on Computer-Human Interaction 27, 3, Article 13 (May 2020), 40 pages. DOI:https://doi.org/10.1145/3380959

Digital Library

[66]

Craig S. Miller, Svetlin Denkov, and Richard C. Omanson. 2011. Categorization costs for hierarchical keyboard commands. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. 2765–2768. DOI:https://doi.org/10.1145/1978942.1979351

Digital Library

[67]

Miscellaneous. 2018. AutoHotkey. Retrieved from https://www.autohotkey.com/.

[68]

Miscellaneous. 2018. Wox Launcher for Windows. Retrieved from http://www.wox.one/.

[69]

Miscellaneous. 2019. Input System. Retrieved from https://valvesoftware.github.io/steamvr_unity_plugin/articles/SteamVR-Input.html.

[70]

Miscellaneous. 2021. Microsoft Keyboard Layout Creator. Retrieved from https://www.microsoft.com/en-us/download/details.aspx?id=102134.

[71]

Miscellaneous. 2021. Microsoft Mixed Reality Toolkit (MRTK). Retrieved from https://microsoft.github.io/MixedRealityToolkit-Unity/Documentation/WelcomeToMRTK.html.

[72]

Nemeio. 2019. The Customisable, Connected Keyboard That Uses Electronic Ink. Retrieved from https://www.nemeio.com/.

[73]

Jiamu Ni. 2017. A Hotkey Interaction Technique that Promotes Hotkeys. Master’s thesis. Retrieved from https://aaltodoc.aalto.fi/bitstream/handle/123456789/28456/master_Ni_Jiamu_2017.pdf.

[74]

Benjamin Nuernberger, Eyal Ofek, Hrvoje Benko, and Andrew D. Wilson. 2016. SnapToReality: Aligning Augmented Reality to the Real World. ACM, New York, NY, 1233–1244. DOI:https://doi.org/10.1145/2858036.2858250

[75]

‘Objective development Software’. 2018. LaunchBar. Retrieved from https://www.obdev.at/products/launchbar/index.html.

[76]

Oculus. 2020. Infinite Office. Retrieved from https://www.youtube.com/watch?v=5_bVkbG1ZCo.

[77]

Oculus. 2021. Infinite Office Support for Tracked K830 Keyboard. Retrieved from https://www.oculus.com/blog/introducing-oculus-air-link-a-wireless-way-to-play-pc-vr-games-on-oculus-quest-2-plus-infinite-office-updates-support-for-120-hz-on-quest-2-and-more/.

[78]

Eyal Ofek, Jens Grubert, Michel Pahud, Mark Phillips, and Per Ola Kristensson. 2020. Towards a Practical Virtual Office for Mobile Knowledge Workers. https://www.microsoft.com/en-us/research/publication/towards-a-practical-virtual-office-for-mobile-knowledge-workers/.

[79]

Richard C. Omanson, Craig S. Miller, Elizabeth Young, and David Schwantes. 2010. Comparison of mouse and keyboard efficiency. In Proceedings of the Human Factors and Ergonomics Society Annual Meeting. Vol. 54, Sage Publications, Los Angeles, CA, 600–604.

[80]

A. Otte, D. Schneider, T. Menzner, T. Gesslein, P. Gagel, and J. Grubert. 2019. Evaluating text entry in virtual reality using a touch-sensitive physical keyboard. In Proceedings of the 2019 IEEE International Symposium on Mixed and Augmented Reality Adjunct. 387–392. DOI:https://doi.org/10.1109/ISMAR-Adjunct.2019.000-4

[81]

Antti Oulasvirta and Andreas Karrenbauer. 2018. Combinatorial optimization for user interface design. In Computational Interaction, Oxford University Press, 97–119. DOI:https://doi.org/10.1093/oso/9780198799603.003.0005

[82]

Henning Pohl, Christian Domin, and Michael Rohs. 2017. Beyond just text. ACM Transactions on Computer-Human Interaction 24, 1 (Mar 2017), 1–42. DOI:https://doi.org/10.1145/3039685

Digital Library

[83]

Henning Pohl and Michael Rohs. 2014. Around-device devices: My coffee mug is a volume dial. In Proceedings of the 16th ACM International Conference on Human-Computer Interaction with Mobile Devices and Services. 81–90. DOI:https://doi.org/10.1145/2628363.2628401

Digital Library

[84]

Jon Porter. 2019. Report claims that Apple could begin production of iPhone-powered AR glasses this year. Retrieved from https://www.theverge.com/2019/3/8/18256256/apple-ar-glasses-2019-ming-chi-kuo-augmented-reality.

[85]

Ian Prest et al. 2021. KLE: Keyboard Layout Editor. Retrieved from http://www.keyboard-layout-editor.com/.

[86]

Razer. 2012. Razer DeathStalker Ultimate. Retrieved from https://support.razer.com/gaming-keyboards/razer-deathstalker-ultimate.

[87]

‘Running with Crayons’. 2018. Alfred - Productivity App for Mac OS X. Retrieved from https://www.alfredapp.com/.

[88]

Samsung.com. 2018. Samsung Odyssey Mixed Reality HMD. Retrieved from https://www.samsung.com/us/computing/hmd/windows-mixed-reality/xe800zaa-hc1us-xe800zaa-hc1us/.

[89]

Joey Scarr, Andy Cockburn, Carl Gutwin, and Philip Quinn. 2011. Dips and ceilings: Understanding and supporting transitions to expertise in user interfaces. In Proceedings of the 2011 Annual Conference on Human Factors in Computing Systems. 2741. DOI:https://doi.org/10.1145/1978942.1979348

[90]

Daniel Schneider, Alexander Otte, Travis Gesslein, Philipp Gagel, Bastian Kuth, Mohamad Shahm Damlakhi, Oliver Dietz, Eyal Ofek, Michel Pahud, Per Ola Kristensson, Jorg Muller, and Jens Grubert. 2019. ReconViguRation: Reconfiguring physical keyboards in virtual reality. IEEE Transactions on Visualization and Computer Graphics 25, 11 (2019), 3190–3201. DOI:https://doi.org/10.1109/TVCG.2019.2932239

Digital Library

[91]

Dominik Seger. 2013. Not a Question of If, but When? Choosing the Right Trigger to Encourage Keyboard Shortcut Use. Master’s thesis. DOI:https://doi.org/10.1017/CBO9781107415324.004

[92]

Kodai Sekimori, Yusuke Yamasaki, Yuki Takagi, Kazuma Murata, Buntarou Shizuki, and Shin Takahashi. 2018. Ex-Space: Expanded space key by sliding thumb on home position. In Proceedings of the International Conference on Human-Computer Interaction. Springer, Cham, 68–78. DOI:https://doi.org/10.1007/978-3-319-91250-9_6

[93]

Yilei Shi, Tomás Vega Gálvez, Haimo Zhang, and Suranga Nanayakkara. 2017. GestAKey: Get more done with just-a-key on a keyboard. In Proceedings of the Adjunct Publication of the 30th Annual ACM Symposium on User Interface Software and Technology. 73–75. DOI:https://doi.org/10.1145/3131785.3131786

Digital Library

[94]

Ben Shneiderman. 1983. Direct manipulation: A step beyond programming languages. Computer 16, 8 (Aug 1983), 57–69. DOI:https://doi.org/10.1109/MC.1983.1654471

Digital Library

[95]

Adalberto L. Simeone, Eduardo Velloso, and Hans Gellersen. 2015. Substitutional reality: Using the physical environment to design virtual reality experiences. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems. DOI:https://doi.org/10.1145/2702123.2702389

[96]

Tobias Sommer. 2018. Hotkey EVE. Retrieved from http://www.hotkey-eve.com/.

[97]

Marco Speicher, Anna Maria Feit, Pascal Ziegler, and Antonio Krüger. 2018. Selection-based text entry in virtual reality. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. DOI:https://doi.org/10.1145/3173574.3174221

Digital Library

[98]

Art. Lebedev Studio. 2007. Optimus Maximus. Retrieved from https://www.artlebedev.com/optimus/maximus/.

[99]

Sussane Tak. 2007. The Use of Keyboard Shortcuts Optimizing Versus Satisficing in the Use of Complex Technology. Master’s thesis. Retrieved from http://www.efficiencysoftware.co.uk/uploads/nieuws/-1412774825.pdf.

[100]

Stuart Taylor, Cem Keskin, Otmar Hilliges, Shahram Izadi, and John Helmes. 2014. Type-hover-swipe in 96 bytes: A motion sensing mechanical keyboard. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. 1695–1704. DOI:https://doi.org/10.1145/2556288.2557030

Digital Library

[101]

Bruce Tognazzini. 1989. Keyboard vs. The Mouse, pt1. Retrieved from https://www.asktog.com/TOI/toi06KeyboardVMouse1.html.

[102]

Jo Vermeulen, Kris Luyten, Elise van den Hoven, and Karin Coninx. 2013. Crossing the Bridge over Norman’s Gulf of Execution: Revealing Feedforward’s True Identity. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM, New York, NY, 1931–1940. DOI:https://doi.org/10.1145/2470654.2466255

Digital Library

[103]

VIVE Blog. 2018. Introducing the Logitech BRIDGE SDK. Retrieved from https://blog.vive.com/us/2017/11/02/introducing-the-logitech-bridge-sdk/.

[104]

J. A. Wagner Filho, C. M. D. S. Freitas, and L. Nedel. 2018. VirtualDesk: A comfortable and efficient immersive information visualization approach. Computer Graphics Forum 37, 3 (2018), 415–426. DOI:https://doi.org/10.1111/cgf.13430

[105]

James Walker, Bochao Li, Keith Vertanen, and Scott Kuhl. 2017. Efficient typing on a visually occluded physical keyboard. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems. 5457–5461. DOI:https://doi.org/10.1145/3025453.3025783

Digital Library

[106]

James A. Walsh, Stewart von Itzstein, and Bruce H. Thomas. 2014. Ephemeral interaction using everyday objects. In Proceedings of the 15th Australasian User Interface Conference - Volume 150. Australian Computer Society, Inc.29–37.

Digital Library

[107]

Windows Experience Blog. 2018. Windows 10 Tip: Get Started with the Emoji Keyboard Shortcut. Retrieved from https://blogs.windows.com/windowsexperience/2018/02/05/windows-10-tip-get-started-emoji-keyboard-shortcut/.

[108]

Jacob O. Wobbrock, Leah Findlater, Darren Gergle, and James J. Higgins. 2011. The aligned rank transform for nonparametric factorial analyses using only anova procedures. In Proceedings of the 2011 Annual Conference on Human Factors in Computing Systems. ACM, New York, NY, 143. DOI:https://doi.org/10.1145/1978942.1978963

Digital Library

[109]

Chien-Min Wu, Chih-Wen Hsu, Tzu-Kuei Lee, and Shana Smith. 2017. A virtual reality keyboard with realistic haptic feedback in a fully immersive virtual environment. Virtual Reality 21, 1 (Mar 2017), 19–29. DOI:https://doi.org/10.1007/s10055-016-0296-6

Digital Library

[110]

Robert Xiao, Julia Schwarz, Nick Throm, Andrew D. Wilson, and Hrvoje Benko. 2018. MRTouch: Adding touch input to head-mounted mixed reality. IEEE Transactions on Visualization and Computer Graphics 24, 4 (2018). 1653–1660. DOI:https://doi.org/10.1109/TVCG.2018.2794222

Digital Library

[111]

Naoki Yanagihara, Buntarou Shizuki, and Shin Takahashi. 2019. A comparative study of planar surface and spherical surface for 3D pointing using direct touch. In Proceedings of the 25th ACM Symposium on Virtual Reality Software and Technology. ACM, New York, NY. DOI:https://doi.org/10.1145/3359996.3364814

Digital Library

[112]

Zhizhuo Yang, Dongdong Weng, Zhengliang Zhang, Yufeng Li, and Yue Liu. 2016. Perceptual issues of a passive haptics feedback based MR system. In Proceedings of the 2016 IEEE International Symposium on Mixed and Augmented Reality. IEEE, 310–317. DOI:https://doi.org/10.1109/ISMAR-Adjunct.2016.0103

[113]

Haimo Zhang and Yang Li. 2014. GestKeyboard: Enabling gesture-based interaction on ordinary physical keyboard. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. 1675–1684. DOI:https://doi.org/10.1145/2556288.2557362

Digital Library

[114]

Jingjie Zheng. 2017. Enabling Expressive Keyboard Interaction with Finger, Hand, and Hand Posture Identification. Master’s thesis. University of Waterloo. DOI:https://doi.org/10.1145/2858036.2858355

[115]

Jingjie Zheng and Daniel Vogel. 2016. Finger-Aware shortcuts. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems. 4274–4285. DOI:https://doi.org/10.1145/2858036.2858355

Digital Library

[116]

Ryder Ziola, Melanie Kellar, and Kori Inkpen. 2007. DeskJockey: Exploiting passive surfaces to display peripheral information. In Proceedings of the 11th IFIP TC 13 International Conference on Human-Computer Interaction. Vol. 4662, 447–460. DOI:https://doi.org/10.1007/978-3-540-74796-3_43

Cited By

Chiossi FEl Khaoudi YOu CSidenmark LZaky AFeuchtner TMayer S(2024)Evaluating Typing Performance in Different Mixed Reality Manifestations using Physiological FeaturesProceedings of the ACM on Human-Computer Interaction10.1145/36981428:ISS(377-406)Online publication date: 24-Oct-2024
https://dl.acm.org/doi/10.1145/3698142
Abraham MMcgill MKhamis M(2024)What You Experience is What We Collect: User Experience Based Fine-Grained Permissions for Everyday Augmented RealityProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642668(1-24)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3642668
Kern FTschanter JLatoschik M(2024)Handwriting for Text Input and the Impact of XR Displays, Surface Alignments, and Sentence ComplexitiesIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2024.337212430:5(2357-2367)Online publication date: 5-Mar-2024
https://dl.acm.org/doi/10.1109/TVCG.2024.3372124
Show More Cited By

Index Terms

Creating and Augmenting Keyboards for Extended Reality with the Keyboard Augmentation Toolkit
1. Human-centered computing
  1. Human computer interaction (HCI)

Recommendations

Mixed Reality MIDI Keyboard Demonstration
AM '17: Proceedings of the 12th International Audio Mostly Conference on Augmented and Participatory Sound and Music Experiences

The Mixed Reality MIDI Keyboard is a prototype designed to augment virtual reality experiences through the inclusion of a physical interface which aligns the user's senses with the virtual environment. It also serves as a platform on which the uses of ...
Visualizing the keyboard in virtual reality for enhancing immersive experience
SIGGRAPH '17: ACM SIGGRAPH 2017 Posters

Recently, virtual reality (VR) becomes more and more popular and provides users an immersive experience with a head-mounted display (HMD). However, in some applications, users have to interact with physical objects while immersed in VR. With a non-see-...
Move'n'Hold Pro: Consistent Spatial Interaction Techniques for Object Manipulation with Handheld and Head-mounted Displays in Extended Reality
ECCE '24: Proceedings of the European Conference on Cognitive Ergonomics 2024

Extended Reality (XR) technologies are still lacking appropriate interaction methods that enable users to seamlessly switch between different XR devices and degrees of virtuality. Addressing this gap, we present Move’n’Hold Pro – a set of consistent ...

Comments

Information & Contributors

Information

Published In

cover image ACM Transactions on Computer-Human Interaction

ACM Transactions on Computer-Human Interaction Volume 29, Issue 2

April 2022

347 pages

ISSN:1073-0516

EISSN:1557-7325

DOI:10.1145/3505202

Editor:
Kristina Höök
KTH Royal Institute of Technology, Sweden

Issue’s Table of Contents

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 16 January 2022

Accepted: 01 August 2021

Revised: 01 August 2021

Received: 01 January 2021

Published in TOCHI Volume 29, Issue 2

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Refereed

Funding Sources

EPSRC IAA
European Research Council (ERC)
European Union’s Horizon 2020 research and innovation programme

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

13
Total Citations
View Citations
1,385
Total Downloads

Downloads (Last 12 months)311
Downloads (Last 6 weeks)23

Reflects downloads up to 09 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Chiossi FEl Khaoudi YOu CSidenmark LZaky AFeuchtner TMayer S(2024)Evaluating Typing Performance in Different Mixed Reality Manifestations using Physiological FeaturesProceedings of the ACM on Human-Computer Interaction10.1145/36981428:ISS(377-406)Online publication date: 24-Oct-2024
https://dl.acm.org/doi/10.1145/3698142
Abraham MMcgill MKhamis M(2024)What You Experience is What We Collect: User Experience Based Fine-Grained Permissions for Everyday Augmented RealityProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642668(1-24)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3642668
Kern FTschanter JLatoschik M(2024)Handwriting for Text Input and the Impact of XR Displays, Surface Alignments, and Sentence ComplexitiesIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2024.337212430:5(2357-2367)Online publication date: 5-Mar-2024
https://dl.acm.org/doi/10.1109/TVCG.2024.3372124
Khorchi ABoton C(2024)An OpenBIM-based 4D approach to support coordination meetings in virtual reality environmentsJournal of Building Engineering10.1016/j.jobe.2024.10864785(108647)Online publication date: May-2024
https://doi.org/10.1016/j.jobe.2024.108647
Ruvimova AFronchetti FKahn BSusin LHurley ZFritz THancock MShepherd D(2023)Ready Worker One? High-Res VR for the Home OfficeProceedings of the 29th ACM Symposium on Virtual Reality Software and Technology10.1145/3611659.3615680(1-12)Online publication date: 9-Oct-2023
https://dl.acm.org/doi/10.1145/3611659.3615680
Gil HOakley I(2023)ThumbAirProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/35694746:4(1-30)Online publication date: 11-Jan-2023
https://dl.acm.org/doi/10.1145/3569474
Kern FNiebling FLatoschik M(2023)Text Input for Non-Stationary XR Workspaces: Investigating Tap and Word-Gesture Keyboards in Virtual and Augmented RealityIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2023.324709829:5(2658-2669)Online publication date: 1-May-2023
https://dl.acm.org/doi/10.1109/TVCG.2023.3247098
Aggarwal SPaul ABhangale TBharambe SMore J(2023)Gesture-Based Computer Control2023 6th International Conference on Advances in Science and Technology (ICAST)10.1109/ICAST59062.2023.10455034(470-475)Online publication date: 8-Dec-2023
https://doi.org/10.1109/ICAST59062.2023.10455034
Yang YGao YLiu KHe ZCao L(2023)Contactless human–computer interaction system based on three-dimensional holographic display and gesture recognitionApplied Physics B10.1007/s00340-023-08128-2129:12Online publication date: 23-Nov-2023
https://doi.org/10.1007/s00340-023-08128-2
Cui DMousas C(2023)Exploring the effects of virtual hand appearance on midair typing efficiencyComputer Animation and Virtual Worlds10.1002/cav.218934:3-4Online publication date: 25-May-2023
https://doi.org/10.1002/cav.2189
Show More Cited By

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Full Text

View this article in Full Text.

HTML Format

View this article in HTML Format.

Media

Figures

Other

Tables

View full text|Download PDF

View Issue’s Table of Contents