Location via proxy:   [ UP ]  
[Report a bug]   [Manage cookies]                
skip to main content
10.1145/3313831.3376173acmconferencesArticle/Chapter ViewAbstractPublication PageschiConference Proceedingsconference-collections
research-article

EYEditor: Towards On-the-Go Heads-Up Text Editing Using Voice and Manual Input

Published: 23 April 2020 Publication History

Abstract

On-the-go text-editing is difficult, yet frequently done in everyday lives. Using smartphones for editing text forces users into a heads-down posture which can be undesirable and unsafe. We present EYEditor, a heads-up smartglass-based solution that displays the text on a see-through peripheral display and allows text-editing with voice and manual input. The choices of output modality (visual and/or audio) and content presentation were made after a controlled experiment, which showed that sentence-by-sentence visual-only presentation is best for optimizing users' editing and path-navigation capabilities. A second experiment formally evaluated EYEditor against the standard smartphone-based solution for tasks with varied editing complexities and navigation difficulties. The results showed that EYEditor outperformed smartphones as either the path OR the task became more difficult. Yet, the advantage of EYEditor became less salient when both the editing and navigation was difficult. We discuss trade-offs and insights gained for future heads-up text-editing solutions.

Supplemental Material

MP4 File
Preview video
MP4 File
Supplemental video
SRT File
Preview video captions
ZIP File
Video figure captions
ZIP File
EYEditor_withCopyedits.pdf ============== This pdf contains all the copyedits made to prepare the camera ready. Please note that the summary of changes mentioned in the PCS portal includes page numbers. All the page numbers refers to the numbering of this pdf. In this pdf, new contents are in red font, while deleted content is greyed out.

References

[1]
Urs-Vito Albrecht, Ute von Jan, Joachim Kuebler, Christoph Zoeller, Martin Lacher, Oliver J Muensterer, Max Ettinger, Michael Klintschar, and Lars Hagemeier. 2014. Google Glass for documentation of medical findings: evaluation in forensic medicine. Journal of medical Internet research 16, 2 (2014), e53.
[2]
Ouais Alsharif, Tom Ouyang, Françoise Beaufays, Shumin Zhai, Thomas Breuel, and Johan Schalkwyk. 2015. Long short term memory neural network for keyboard gesture decoding. In 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2076--2080.
[3]
Ahmed Sabbir Arif and Ali Mazalek. 2016. A survey of text entry techniques for smartwatches. In International Conference on Human-Computer Interaction. Springer, 255--267.
[4]
Shiri Azenkot and Nicole B Lee. 2013. Exploring the use of speech input by blind people on mobile devices. In Proceedings of the 15th International ACM SIGACCESS Conference on Computers and Accessibility. ACM, 11.
[5]
Ronald T Azuma. 1997. A survey of augmented reality. Presence: Teleoperators & Virtual Environments 6, 4 (1997), 355--385.
[6]
Bartosz Bajer, I Scott MacKenzie, and Melanie Baljko. 2012. Huffman base-4 text entry glove (H4 TEG). In 2012 16th International Symposium on Wearable Computers. IEEE, 41--47.
[7]
Mafkereseb Kassahun Bekele, Roberto Pierdicca, Emanuele Frontoni, Eva Savina Malinverni, and James Gain. 2018. A Survey of Augmented, Virtual, and Mixed Reality for Cultural Heritage. Journal on Computing and Cultural Heritage 11, 2 (March 2018), 1--36.
[8]
Mathilde M Bekker, Floris L van Nes, and James F Juola. 1995. A comparison of mouse and speech input control of a text-annotation system. Behaviour & Information Technology 14, 1 (1995), 14--22.
[9]
Doug A Bowman, Vinh Q Ly, and Joshua M Campbell. 2001. Pinch keyboard: Natural text input for immersive virtual environments. (2001).
[10]
Statistic Brain. 2017. Text Message Statistics - United States. (2017). Retrieved September 14, 2019 from https://www.statisticbrain.com/text-message-statistics/.
[11]
Kenneth Burke. 2018. How Many Texts Do People Send Every Day (2018)? (Nov 2018). Retrieved September 14, 2019 from https://www.textrequest.com/blog/howmany-texts-people-send-per-day/.
[12]
Stuart K Card, Thomas P Moran, and Allen Newell. 1980. Computer text-editing: An information-processing analysis of a routine cognitive skill. Cognitive psychology 12, 1 (1980), 32--74.
[13]
Junhwi Choi, Kyungduk Kim, Sungjin Lee, Seokhwan Kim, Donghyeon Lee, Injae Lee, and Gary Geunbae Lee. 2012. Seamless error correction interface for voice word processor. In 2012 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 4973--4976.
[14]
DD Clark-Carter, AD Heyes, and CI Howarth. 1986. The efficiency and walking speed of visually impaired people. Ergonomics 29, 6 (1986), 779--789.
[15]
Brian Lystgaard Due. 2014. The future of smart glasses: An essay about challenges and possibilities with smart glasses. Working papers on interaction and communication 1, 2 (2014), 1--21.
[16]
Steven Feiner, Blair MacIntyre, Tobias Höllerer, and Anthony Webster. 1997. A touring machine: Prototyping 3D mobile augmented reality systems for exploring the urban environment. Personal Technologies 1, 4 (01 Dec 1997), 208--217.
[17]
Debjyoti Ghosh, Pin Sym Foong, Shengdong Zhao, Di Chen, and Morten Fjeld. 2018. EDITalk: towards designing eyes-free interactions for mobile word processing. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. ACM, 403.
[18]
Mitchell Gordon, Tom Ouyang, and Shumin Zhai. 2016. WatchWriter: Tap and gesture typing on a smartwatch miniature keyboard with statistical decoding. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems. ACM, 3817--3821.
[19]
Tovi Grossman, Xiang Anthony Chen, and George Fitzmaurice. 2015. Typing on glasses: Adapting text entry to smart eyewear. In Proceedings of the 17th International Conference on Human-Computer Interaction with Mobile Devices and Services. ACM, 144--152.
[20]
Aakar Gupta and Ravin Balakrishnan. 2016. DualKey: miniature screen text entry via finger identification. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems. ACM, 59--70.
[21]
Ewa Gustafsson, Sara Thomée, Anna Grimby-Ekman, and Mats Hagberg. 2017. Texting on mobile phones and musculoskeletal disorders in young adults: a five-year cohort study. Applied ergonomics 58 (2017), 208--214.
[22]
Christine A Halverson, Daniel B Horn, Clare-Marie Karat, and John Karat. 1999. The beauty of errors: Patterns of error correction in desktop speech systems. In INTERACT. 133--140.
[23]
Sandra G Hart and Lowell E Staveland. 1988. Development of NASA-TLX (Task Load Index): Results of empirical and theoretical research. In Advances in psychology. Vol. 52. Elsevier, 139--183.
[24]
Jonggi Hong, Seongkook Heo, Poika Isokoski, and Geehyuk Lee. 2015. SplitBoard: A simple split soft keyboard for wristwatch-sized touch screens. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems. ACM, 1233--1236.
[25]
David M Kietrys, Michael J Gerg, Jonathan Dropkin, and Judith E Gold. 2015. Mobile input device type, texting style and screen size influence upper extremity and trapezius muscle activity, and cervical posture while texting. Applied ergonomics 50 (2015), 98--104.
[26]
J Peter Kincaid, Robert P Fishburne Jr, Richard L Rogers, and Brad S Chissom. 1975. Derivation of new readability formulas (automated readability index, fog count and flesch reading ease formula) for navy enlisted personnel. (1975).
[27]
Per Ola Kristensson and Keith Vertanen. 2012. The potential of dwell-free eye-typing for fast assistive gaze communication. In Proceedings of the symposium on eye tracking research and applications. ACM, 241--244.
[28]
Jochen Kuhn, Paul Lukowicz, Michael Hirth, Andreas Poxrucker, Jens Weppner, and Junaid Younas. 2016. gPhysics-Using smart glasses for head-centered, context-aware learning in physics experiments. IEEE Transactions on Learning Technologies 9, 4 (2016), 304--317.
[29]
Andrew Kurauchi, Wenxin Feng, Ajjen Joshi, Carlos Morimoto, and Margrit Betke. 2016. EyeSwipe: Dwell-free text entry using gaze paths. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems. ACM, 1952--1956.
[30]
Pascale Larigauderie, Daniel Gaonach, and Natasha Lacroix. 1998. Working memory and error detection in texts: What are the roles of the central executive and the phonological loop? Applied Cognitive Psychology: The Official Journal of the Society for Applied Research in Memory and Cognition 12, 5 (1998), 505--527.
[31]
Jae Yeol Lee and Guewon Rhee. 2008. Context-aware 3D visualization and collaboration services for ubiquitous cars using augmented reality. The International Journal of Advanced Manufacturing Technology 37, 5 (01 May 2008), 431--442.
[32]
Min Lin, Rich Goldman, Kathleen J Price, Andrew Sears, and Julie Jacko. 2007. How do people tap when walking? An empirical investigation of nomadic data entry. International journal of human-computer studies 65, 9 (2007), 759--769.
[33]
Ming-I Brandon Lin and Yu-Ping Huang. 2017. The impact of walking while using a smartphone on pedestrians' awareness of roadside events. Accident Analysis & Prevention 101 (2017), 87--96.
[34]
Bob W Lord and Ray Velez. 2013. Converge: transforming business at the intersection of marketing and technology. John Wiley & Sons.
[35]
Aliaksandr Malokin, Giovanni Circella, and Patricia L Mokhtarian. 2019. How do activities conducted while commuting influence mode choice? Using revealed preference models to inform public transportation advantage and autonomous vehicle scenarios. Transportation Research Part A: Policy and Practice 124 (2019), 82--114.
[36]
Roderick McCall, Benoît Martin, Andrei Popleteev, Nicolas Louveton, and Thomas Engel. 2015. Text entry on smart glasses. In 2015 8th International Conference on Human System Interaction (HSI). IEEE, 195--200.
[37]
Arthur E McNair and Alex Waibel. 1994. Improving recognizer acceptance through robust, natural speech repair. In Third International Conference on Spoken Language Processing.
[38]
Paul Milgram. 1994. Taxonomy of mixed reality visual displays. IEICE Transactions on Information and Systems E77-D, 12 (1994), 1321--1329.
[39]
Paul Milgram and Fumio Kishino. 1994. A taxonomy of mixed reality visual displays. IEICE TRANSACTIONS on Information and Systems 77, 12 (1994), 1321--1329.
[40]
Stefan Mitrasinovic, Elvis Camacho, Nirali Trivedi, Julia Logan, Colson Campbell, Robert Zilinyi, Bryan Lieber, Eliza Bruce, Blake Taylor, David Martineau, and others. 2015. Clinical and surgical applications of smart glasses. Technology and Health Care 23, 4 (2015), 381--401.
[41]
Hugo Nicolau and Joaquim Jorge. 2012. Touch typing using thumbs: understanding the effect of mobility and hand posture. In Proceedings of the SIGCHI conference on human factors in computing systems. ACM, 2683--2686.
[42]
Sharon Oviatt. 2000. Taming recognition errors with a multimodal interface. Commun. ACM 43, 9 (Sept. 2000), 45--51.
[43]
Sharon Oviatt and Robert VanGent. 1996. Error resolution during multimodal human-computer interaction. In Proceeding of Fourth International Conference on Spoken Language Processing. ICSLP'96, Vol. 1. IEEE, 204--207.
[44]
Physiopedia. 2019. Text Neck. (2019). Retrieved September 18, 2019 from https://www.physio-pedia.com/Text_Neck.
[45]
Antti Pirhonen, Stephen Brewster, Stephen Brewster, and Christopher Holguin. 2002. Gestural and audio metaphors as a means of control for mobile devices. In Proceedings of the SIGCHI conference on Human factors in computing systems. ACM, 291--298.
[46]
Philipp A Rauschnabel, Alexander Brem, and Young Ro. 2015. Augmented reality smart glasses: definition, conceptual insights, and managerial importance. Unpublished Working Paper, The University of Michigan-Dearborn, College of Business (2015).
[47]
Sherry Ruan, Jacob O Wobbrock, Kenny Liou, Andrew Ng, and James Landay. 2016. Speech is 3x faster than typing for english and mandarin text entry on mobile devices. arXiv preprint arXiv:1608.07323 (2016).
[48]
Rufat Rzayev, Paweł W Wożniak, Tilman Dingler, and Niels Henze. 2018. Reading on Smart Glasses: The Effect of Text Position, Presentation Type and Walking. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. ACM, 45.
[49]
Sayan Sarcar, Prateek Panwar, and Tuhin Chakraborty. 2013. EyeK: an efficient dwell-free eye gaze-based text entry system. In Proceedings of the 11th Asia Pacific Conference on computer human interaction. ACM, 215--220.
[50]
Alexander Schick, Daniel Morlock, Christoph Amma, Tanja Schultz, and Rainer Stiefelhagen. 2012. Vision-based handwriting recognition for unrestricted text input in mid-air. In Proceedings of the 14th ACM international conference on Multimodal interaction. ACM, 217--220.
[51]
Andrew Sears, Jinhuan Feng, Kwesi Oseitutu, and Claire-Marie Karat. 2003. Hands-free, speech-based navigation during dictation: difficulties, consequences, and solutions. Human-computer interaction 18, 3 (2003), 229--257.
[52]
Mark Sharp. 2015. Beware the smartphone zombies blindly wandering around Hong Kong. (2 March 2015). Retrieved September 14, 2019 from https://www.scmp.com/lifestyle/technology/article/1725001/smartphone-zombies-are-putting-your-lifeand-theirs-danger.
[53]
Matthias Sperber, Graham Neubig, Christian Fügen, Satoshi Nakamura, and Alex Waibel. 2013. Efficient speech transcription through respeaking. In Interspeech. 1087--1091.
[54]
Srinath Sridhar, Anna Maria Feit, Christian Theobalt, and Antti Oulasvirta. 2015. Investigating the dexterity of multi-finger input for mid-air text entry. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems. ACM, 3643--3652.
[55]
Bernhard Suhm, Brad Myers, and Alex Waibel. 2001. Multimodal error correction for speech user interfaces. ACM transactions on computer-human interaction (TOCHI) 8, 1 (2001), 60--98.
[56]
Husain Sumra. 2019. The best augmented reality glasses 2019: Snap, Vuzix, Microsoft, North & more. (Mar 2019). Retrieved September 14, 2019 from https://www.wareable.com/ar/the-best-smartglassesgoogle-glass-and-the-rest.
[57]
B. Thomas, B. Close, J. Donoghue, J. Squires, P. De Bondi, M. Morris, and W. Piekarski. 2000. ARQuake: an outdoor/indoor augmented reality first person application. In Digest of Papers. Fourth International Symposium on Wearable Computers. 139--146.
[58]
B. Thomas, V. Demczuk, W. Piekarski, D. Hepworth, and B. Gunther. 1998. A wearable computer system with augmented reality to support terrestrial navigation. In Digest of Papers. Second International Symposium on Wearable Computers (Cat. No.98EX215). 168--171.
[59]
Kristin Vadas, Nirmal Patel, Kent Lyons, Thad Starner, and Julie Jacko. 2006. Reading on-the-go: a comparison of audio and hand-held displays. In Proceedings of the 8th conference on Human-computer interaction with mobile devices and services. ACM, 219--226.
[60]
Keith Vertanen and Per Ola Kristensson. 2009. Automatic selection of recognition errors by respeaking the intended text. In 2009 IEEE Workshop on Automatic Speech Recognition & Understanding. IEEE, 130--135.
[61]
Cheng-Yao Wang, Wei-Chen Chu, Po-Tsung Chiu, Min-Chieh Hsiu, Yih-Harn Chiang, and Mike Y Chen. 2015. PalmType: Using palms as keyboards for smart glasses. In Proceedings of the 17th International Conference on Human-Computer Interaction with Mobile Devices and Services. ACM, 153--160.
[62]
X. Wang, S. K. Ong, and A. Y. C. Nee. 2016. A comprehensive survey of augmented reality assembly research. Advances in Manufacturing 4, 1 (March 2016), 1--22.
[63]
MDN web docs. 2019. Web Speech API. (2019). Retrieved August 29, 2019 from https://developer.mozilla.org/en-US/docs/Web/API/ Web_Speech_API.
[64]
Eric Whitmire, Mohit Jain, Divye Jain, Greg Nelson, Ravi Karkar, Shwetak Patel, and Mayank Goel. 2017. Digitouch: Reconfigurable thumb-to-finger input and text entry on head-mounted displays. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 1, 3 (2017), 113.
[65]
Chun Yu, Yizheng Gu, Zhican Yang, Xin Yi, Hengliang Luo, and Yuanchun Shi. 2017. Tap, dwell or gesture?: Exploring head-based text entry techniques for hmds. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems. ACM, 4479--4488.
[66]
Chun Yu, Ke Sun, Mingyuan Zhong, Xincheng Li, Peijun Zhao, and Yuanchun Shi. 2016. One-dimensional handwriting: Inputting letters and words on smart glasses. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems. ACM, 71--82.

Cited By

View all
  • (2024)Augmented Reality on the Move: A Systematic Literature Review for Vulnerable Road UsersProceedings of the ACM on Human-Computer Interaction10.1145/36764908:MHCI(1-30)Online publication date: 24-Sep-2024
  • (2024)Towards Intelligent Wearable AssistantsCompanion of the 2024 on ACM International Joint Conference on Pervasive and Ubiquitous Computing10.1145/3675094.3678989(618-621)Online publication date: 5-Oct-2024
  • (2024)Heads-Up Computing: Opportunities and Challenges of the Next Interaction Paradigm with Wearable Intelligent AssistantsCompanion of the 2024 on ACM International Joint Conference on Pervasive and Ubiquitous Computing10.1145/3675094.3677563(960-963)Online publication date: 5-Oct-2024
  • Show More Cited By

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences
CHI '20: Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems
April 2020
10688 pages
ISBN:9781450367080
DOI:10.1145/3313831
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 23 April 2020

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. eyeditor
  2. heads-up interaction
  3. manual-input
  4. mobile interaction
  5. re-speaking
  6. smart glass
  7. text editing
  8. voice interaction
  9. wearable interaction

Qualifiers

  • Research-article

Funding Sources

  • NUS School of Computing Strategic Initiatives

Conference

CHI '20
Sponsor:

Acceptance Rates

Overall Acceptance Rate 6,199 of 26,314 submissions, 24%

Upcoming Conference

CHI 2025
ACM CHI Conference on Human Factors in Computing Systems
April 26 - May 1, 2025
Yokohama , Japan

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)144
  • Downloads (Last 6 weeks)6
Reflects downloads up to 03 Feb 2025

Other Metrics

Citations

Cited By

View all
  • (2024)Augmented Reality on the Move: A Systematic Literature Review for Vulnerable Road UsersProceedings of the ACM on Human-Computer Interaction10.1145/36764908:MHCI(1-30)Online publication date: 24-Sep-2024
  • (2024)Towards Intelligent Wearable AssistantsCompanion of the 2024 on ACM International Joint Conference on Pervasive and Ubiquitous Computing10.1145/3675094.3678989(618-621)Online publication date: 5-Oct-2024
  • (2024)Heads-Up Computing: Opportunities and Challenges of the Next Interaction Paradigm with Wearable Intelligent AssistantsCompanion of the 2024 on ACM International Joint Conference on Pervasive and Ubiquitous Computing10.1145/3675094.3677563(960-963)Online publication date: 5-Oct-2024
  • (2024)GlassMail: Towards Personalised Wearable Assistant for On-the-Go Email Creation on Smart GlassesProceedings of the 2024 ACM Designing Interactive Systems Conference10.1145/3643834.3660683(372-390)Online publication date: 1-Jul-2024
  • (2024)TouchEditorProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36314547:4(1-29)Online publication date: 12-Jan-2024
  • (2024)Heads-Up Multitasker: Simulating Attention Switching On Optical Head-Mounted DisplaysProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642540(1-18)Online publication date: 11-May-2024
  • (2024)AudioXtend: Assisted Reality Visual Accompaniments for Audiobook Storytelling During Everyday Routine TasksProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642514(1-22)Online publication date: 11-May-2024
  • (2024)Exploring Controller-based Techniques for Precise and Rapid Text Selection in Virtual Reality2024 IEEE Conference Virtual Reality and 3D User Interfaces (VR)10.1109/VR58804.2024.00047(244-253)Online publication date: 16-Mar-2024
  • (2023)GlassMessagingProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36109317:3(1-32)Online publication date: 27-Sep-2023
  • (2023)Gist and Verbatim: Understanding Speech to Inform New Interfaces for Verbal Text CompositionProceedings of the 5th International Conference on Conversational User Interfaces10.1145/3571884.3597134(1-11)Online publication date: 19-Jul-2023
  • Show More Cited By

View Options

Login options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

HTML Format

View this article in HTML Format.

HTML Format

Figures

Tables

Media

Share

Share

Share this Publication link

Share on social media