research-article

Detecting Swipe Errors on Touchscreens using Grip Modulation

Authors:

Mohammad Faizuddin Mohd Noor,

John WilliamsonAuthors Info & Claims

CHI '16: Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems

Pages 1909 - 1920

https://doi.org/10.1145/2858036.2858474

Published: 07 May 2016 Publication History

Abstract

We show that when users make errors on mobile devices they make immediate and distinct physical responses that can be observed with standard sensors. We used three standard cognitive tasks (Flanker, Stroop and SART) to induce errors from 20 participants. Using simple low-resolution capacitive touch sensors placed around a standard mobile device and the built-in accelerometer, we demonstrate that errors can be predicted at low error rates from micro-adjustments to hand grip and movement in the period shortly after swiping the touchscreen. Specifically, when combining features derived from hand grip and movement we obtain a mean AUC of 0.96 (with false accept and reject rates both below 10%). Our results demonstrate that hand grip and movement provide strong and low latency evidence for mistakes. The ability to detect user errors in this way could be a valuable component in future interaction systems, allowing interfaces to make it easier for users to correct erroneous inputs.

Supplementary Material

MP4 File (p1909-mohd-noor.mp4)

Download
210.64 MB

References

[1]

Andrew Campbell, Tanzeem Choudhury, Shaohan Hu, Hong Lu, Matthew K Mukerjee, Mashfiqui Rabbi, and Rajeev DS Raizada. 2010. NeuroPhone: brain-mobile phone interface using a wireless EEG headset. In Proceedings of the second ACM SIGCOMM workshop on Networking, systems, and applications on mobile handhelds. ACM, 3--8.

Digital Library

[2]

Julin Candia, Marta C Gonzlez, Pu Wang, Timothy Schoenharl, Greg Madey, and Albert-Lszl Barabsi. 2008. Uncovering individual and collective human dynamics from mobile phone records. Journal of Physics A: Mathematical and Theoretical 41, 22 (2008), 224015.

[3]

Chih-Chung Chang and Chih-Jen Lin. 2011. LIBSVM: A library for support vector machines. ACM Transactions on Intelligent Systems and Technology 2 (2011), 27:1--27:27. Issue 3.

Digital Library

[4]

Chih-Jen Chang, Chih-Chung; Lin. 2011. LIBSVM: A Library for Support Vector Machines. ACM Trans. Intell. Syst. Technol. 2, 3, Article 27 (May 2011), 27:1--27:27 pages.

Digital Library

[5]

Ricardo Chavarriaga, Pierre W Ferrez, and Jose del R Millan. 2008. To err is human: Learning from error potentials in brain-computer interfaces. In Advances in Cognitive Neurodynamics ICCN 2007. Springer, 777--782.

[6]

James Clawson, Kent Lyons, Alex Rudnick, Robert A Iannucci Jr, and Thad Starner. 2008. Automatic whiteout++: correcting mini-QWERTY typing errors using keypress timing. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM, 573--582.

Digital Library

[7]

Stephen A Coombes, Kelly M Gamble, James H Cauraugh, and Christopher M Janelle. 2008. Emotional states alter force control during a feedback occluded motor task. Emotion 8, 1 (2008), 104.

[8]

Bernardo Dal Seno, Matteo Matteucci, and Luca Mainardi. 2010. Online Detection of P300 and Error Potentials in a BCI Speller. Intell. Neuroscience 2010, Article 11 (Jan. 2010), 1 pages.

Digital Library

[9]

Barbara A Eriksen and Charles W Eriksen. 1974. Effects of noise letters upon the identification of a target letter in a nonsearch task. Perception & psychophysics 16, 1 (1974), 143--149.

[10]

Pierre W Ferrez and Jose del R Millan. 2005. You are wrong!-Automatic detection of interaction errors from brain waves. In International Joint Conference on Artificial Intelligence, Vol. 19. LAWRENCE ERLBAUM ASSOCIATES LTD, 1413.

Digital Library

[11]

WJ Gehring, MGH Coles, DE Meyer, and E Donchin. 1990. The error-related negativity: an event-related brain potential accompanying errors. Psychophysiology 27, 4 (1990), S34.

[12]

Mehmet Gonen and Ethem Alpaydın. 2011. Multiple kernel learning algorithms. The Journal of Machine Learning Research 12 (2011), 2211--2268.

Digital Library

[13]

Joshua Goodman, Gina Venolia, Keith Steury, and Chauncey Parker. 2002. Language modeling for soft keyboards. In Proceedings of the 7th international conference on Intelligent user interfaces. ACM, 194--195.

Digital Library

[14]

Yuta Higuchi and Takashi Okada. 2014. User Interface Using Natural Gripping FeaturesGrip UI. (2014).

[15]

Clay B Holroyd and Michael GH Coles. 2002. The neural basis of human error processing: reinforcement learning, dopamine, and the error-related negativity. Psychological review 109, 4 (2002), 679.

[16]

Per-Ola Kristensson and Shumin Zhai. 2005. Relaxing stylus typing precision by geometric pattern matching. In Proceedings of the 10th international conference on Intelligent user interfaces. ACM, 151--158.

Digital Library

[17]

M. Lehne, K. Ihme, A.-M. Brouwer, J. van Erp, and T.O. Zander. 2009. Error-related EEG patterns during tactile human-machine interaction. In Affective Computing and Intelligent Interaction and Workshops, 2009. ACII 2009. 3rd International Conference on. 1--9.

[18]

Qiang Li, John Stankovic, Mark Hanson, Adam T Barth, John Lach, Gang Zhou, and others. 2009. Accurate, fast fall detection using gyroscopes and accelerometer-derived posture information. In Wearable and Implantable Body Sensor Networks, 2009. BSN 2009. Sixth International Workshop on. IEEE, 138--143.

Digital Library

[19]

I Scott MacKenzie. 2012. Human-computer interaction: An empirical research perspective. Newnes.

Digital Library

[20]

Jillian Madison. 2012. Damn you, autocorrect! Random House.

[21]

Perrin Margaux, Maby Emmanuel, Daligault Sébastien, Bertrand Olivier, and Mattout Jérémie. 2012. Objective and subjective evaluation of online error correction during P300-based spelling. Advances in Human-Computer Interaction 2012 (2012), 4.

Digital Library

[22]

J Timothy Noteboom, Kerry R Barnholt, and Roger M Enoka. 2001. Activation of the arousal response and impairment of performance increase with anxiety and stressor intensity. Journal of applied physiology 91, 5 (2001), 2093--2101.

[23]

Nishkam Ravi, Nikhil Dandekar, Preetham Mysore, and Michael L Littman. 2005. Activity recognition from accelerometer data. In AAAI, Vol. 5. 1541--1546.

Digital Library

[24]

Ian H Robertson, Tom Manly, Jackie Andrade, Bart T Baddeley, and Jenny Yiend. 1997. Oops!: performance correlates of everyday attentional failures in traumatic brain injured and normal subjects. Neuropsychologia 35, 6 (1997), 747--758.

[25]

Marten K Scheffers and Michael GH Coles. 2000. Performance monitoring in a confusing world: error-related brain activity, judgments of response accuracy, and types of errors. Journal of Experimental Psychology: Human Perception and Performance 26, 1 (2000), 141.

[26]

Tom Sharma, Nandita; Gedeon. 2013. Optimal Time Segments for Stress Detection. In Machine Learning and Data Mining in Pattern Recognition, Petra Perner (Ed.). Lecture Notes in Computer Science, Vol. 7988. Springer Berlin Heidelberg, 421--433.

Digital Library

[27]

J Ridley Stroop. 1935. Studies of interference in serial verbal reactions. Journal of experimental psychology 18, 6 (1935), 643.

[28]

Chi Vi and Sriram Subramanian. 2012. Detecting Error-related Negativity for Interaction Design. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '12). ACM, NY, NY, USA, 493--502.

Digital Library

[29]

Chi Thanh Vi, Izdihar Jamil, David Coyle, and Sriram Subramanian. 2014. Error Related Negativity in Observing Interactive Tasks. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '14). ACM, NY, NY, USA, 3787--3796.

Digital Library

[30]

Daryl Weir, Henning Pohl, Simon Rogers, Keith Vertanen, and Per Ola Kristensson. 2014. Uncertain text entry on mobile devices. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM, 2307--2316.

Digital Library

Cited By

Schmitz MMüller FMühlhäuser MRiemann JLe HKitamura YQuigley AIsbister KIgarashi TBjørn PDrucker S(2021)Itsy-Bits: Fabrication and Recognition of 3D-Printed Tangibles with Small Footprints on Capacitive TouchscreensProceedings of the 2021 CHI Conference on Human Factors in Computing Systems10.1145/3411764.3445502(1-12)Online publication date: 6-May-2021
https://dl.acm.org/doi/10.1145/3411764.3445502
Guleria AKaur R(2021)Unintended Notification Swipe Detection System2021 Third International Conference on Inventive Research in Computing Applications (ICIRCA)10.1109/ICIRCA51532.2021.9544898(1614-1620)Online publication date: 2-Sep-2021
https://doi.org/10.1109/ICIRCA51532.2021.9544898
Williamson JQuek MPopescu IRamsay AMurray-Smith R(2020)Efficient human-machine control with asymmetric marginal reliability input devicesPLOS ONE10.1371/journal.pone.023360315:6(e0233603)Online publication date: 1-Jun-2020
https://doi.org/10.1371/journal.pone.0233603
Show More Cited By

Index Terms

Detecting Swipe Errors on Touchscreens using Grip Modulation
1. Human-centered computing
  1. Human computer interaction (HCI)

Recommendations

B2B-Swipe: Swipe Gesture for Rectangular Smartwatches from a Bezel to a Bezel
CHI '16: Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems

We present B2B-Swipe, a single-finger swipe gesture for a rectangular smartwatch that starts at a bezel and ends at a bezel to enrich input vocabulary. There are 16 possible B2B-Swipes because a rectangular smartwatch has four bezels. Moreover, B2B-...
28 frames later: predicting screen touches from back-of-device grip changes
CHI '14: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems

We demonstrate that front-of-screen targeting on mobile phones can be predicted from back-of-device grip manipulations. Using simple, low-resolution capacitive touch sensors placed around a standard phone, we outline a machine learning approach to ...
Pre-interaction identification by dynamic grip classification
IMCOM '17: Proceedings of the 11th International Conference on Ubiquitous Information Management and Communication

We present a novel authentication method to identify users as they pick up a mobile device. We use a combination of back-of-device capacitive sensing and accelerometer measurements to perform classification, and obtain increased performance compared to ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

CHI '16: Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems

May 2016

6108 pages

ISBN:9781450333627

DOI:10.1145/2858036

General Chairs:
Jofish Kaye
Yahoo
,
Allison Druin
University of Maryland / National Park Service
,
Program Chairs:
Cliff Lampe
University of Michigan
,
Dan Morris
Microsoft
,
Juan Pablo Hourcade
University of Iowa

Copyright © 2016 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

SIGCHI: ACM Special Interest Group on Computer-Human Interaction

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 07 May 2016

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

CHI'16

Sponsor:

SIGCHI

CHI'16: CHI Conference on Human Factors in Computing Systems

May 7 - 12, 2016

California, San Jose, USA

Acceptance Rates

CHI '16 Paper Acceptance Rate 565 of 2,435 submissions, 23%;

Overall Acceptance Rate 6,199 of 26,314 submissions, 24%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

16
Total Citations
View Citations
398
Total Downloads

Downloads (Last 12 months)15
Downloads (Last 6 weeks)2

Reflects downloads up to 03 Sep 2024

Other Metrics

View Author Metrics

Citations

Cited By

Schmitz MMüller FMühlhäuser MRiemann JLe HKitamura YQuigley AIsbister KIgarashi TBjørn PDrucker S(2021)Itsy-Bits: Fabrication and Recognition of 3D-Printed Tangibles with Small Footprints on Capacitive TouchscreensProceedings of the 2021 CHI Conference on Human Factors in Computing Systems10.1145/3411764.3445502(1-12)Online publication date: 6-May-2021
https://dl.acm.org/doi/10.1145/3411764.3445502
Guleria AKaur R(2021)Unintended Notification Swipe Detection System2021 Third International Conference on Inventive Research in Computing Applications (ICIRCA)10.1109/ICIRCA51532.2021.9544898(1614-1620)Online publication date: 2-Sep-2021
https://doi.org/10.1109/ICIRCA51532.2021.9544898
Williamson JQuek MPopescu IRamsay AMurray-Smith R(2020)Efficient human-machine control with asymmetric marginal reliability input devicesPLOS ONE10.1371/journal.pone.023360315:6(e0233603)Online publication date: 1-Jun-2020
https://doi.org/10.1371/journal.pone.0233603
Le HMayer SSteuerlein BHenze N(2019)Investigating Unintended Inputs for One-Handed Touch Interaction Beyond the TouchscreenProceedings of the 21st International Conference on Human-Computer Interaction with Mobile Devices and Services10.1145/3338286.3340145(1-14)Online publication date: 1-Oct-2019
https://dl.acm.org/doi/10.1145/3338286.3340145
Sarcar SSilpasuwanchai CDelamare WOno AOulasvirta ARen X(2019)Exploring performance of thumb input for pointing and dragging tasks on mobile deviceProceedings of Asian CHI Symposium 2019: Emerging HCI Research Collection10.1145/3309700.3338434(38-45)Online publication date: 4-May-2019
https://dl.acm.org/doi/10.1145/3309700.3338434
Le HMayer SHenze NFu WPan SBrdiczka OChau PCalvary G(2019)Investigating the feasibility of finger identification on capacitive touchscreens using deep learningProceedings of the 24th International Conference on Intelligent User Interfaces10.1145/3301275.3302295(637-649)Online publication date: 17-Mar-2019
https://dl.acm.org/doi/10.1145/3301275.3302295
Zhang YPahud MHolz CXia HLaput GMcGuffin MTu XMittereder ASu FBuxton WHinckley KBrewster SFitzpatrick GCox AKostakos V(2019)Sensing Posture-Aware Pen+Touch Interaction on TabletsProceedings of the 2019 CHI Conference on Human Factors in Computing Systems10.1145/3290605.3300285(1-14)Online publication date: 2-May-2019
https://dl.acm.org/doi/10.1145/3290605.3300285
Lehmann FKipp MKoike HRatti CTakeuchi YFukuchi KScott SPlasencia D(2018)How to Hold Your Phone When TappingProceedings of the 2018 ACM International Conference on Interactive Surfaces and Spaces10.1145/3279778.3279791(115-127)Online publication date: 19-Nov-2018
https://dl.acm.org/doi/10.1145/3279778.3279791
Le HMayer SHenze NBaudisch PSchmidt AWilson A(2018)InfiniTouchProceedings of the 31st Annual ACM Symposium on User Interface Software and Technology10.1145/3242587.3242605(779-792)Online publication date: 11-Oct-2018
https://dl.acm.org/doi/10.1145/3242587.3242605
Wang JYin JDeng SLi YPu CTang YLuo ZMandryk RHancock MPerry MCox A(2018)Evaluating User Satisfaction with Typography Designs via Mining Touch Interaction Data in Mobile ReadingProceedings of the 2018 CHI Conference on Human Factors in Computing Systems10.1145/3173574.3173687(1-12)Online publication date: 21-Apr-2018
https://dl.acm.org/doi/10.1145/3173574.3173687
Show More Cited By

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents