research-article

PointPose: finger pose estimation for touch input on mobile devices using a depth sensor

Authors:

Maribeth BackAuthors Info & Claims

ITS '13: Proceedings of the 2013 ACM international conference on Interactive tabletops and surfaces

Pages 223 - 230

https://doi.org/10.1145/2512349.2512824

Published: 06 October 2013 Publication History

Abstract

The expressiveness of touch input can be increased by detecting additional finger pose information at the point of touch such as finger rotation and tilt. PointPose is a prototype that performs finger pose estimation at the location of touch using a short-range depth sensor viewing the touch screen of a mobile device. We present an algorithm that extracts finger rotation and tilt from a point cloud generated by a depth sensor oriented towards the device's touchscreen. The results of two user studies we conducted show that finger pose information can be extracted reliably using our proposed method. We show this for controlling rotation and tilt axes separately and also for combined input tasks using both axes. With the exception of the depth sensor, which is mounted directly on the mobile device, our approach does not require complex external tracking hardware, and, furthermore, external computation is unnecessary as the finger pose extraction algorithm can run directly on the mobile device. This makes PointPose ideal for prototyping and developing novel mobile user interfaces that use finger pose estimation.

References

[1]

Bailly, G., Müller, J., Rohs, M., Wigdor, D., and Kratz, S. Shoesense: a new perspective on gestural interaction and wearable applications. In Proceedings of the 2012 ACM annual conference on Human Factors in Computing Systems, ACM (2012), 1239--1248.

Digital Library

[2]

Boring, S., Ledo, D., Chen, X., Marquardt, N., Tang, A., and Greenberg, S. The fat thumb: using the thumb's contact size for single-handed mobile interaction. In Proceedings of the 14th international conference on Human-computer interaction with mobile devices and services, ACM (2012), 39--48.

Digital Library

[3]

Buxton, W. A three-state model of graphical input. In Proc. of the IFIP TC13 Third Intl. Conference on Human-Computer Interaction, North-Holland Publishing Co. (1990), 449--456.

Digital Library

[4]

Buxton, W. Multi-Touch Systems that I have Known and Loved.hboxhttp://www.billbuxton.com/multitouchOverview.html, June 2013.

[5]

Casiez, G., Roussel, N., and Vogel, D. 1 euro filter: a simple speed-based low-pass filter for noisy input in interactive systems. In Proceedings of the 2012 ACM annual conference on Human Factors in Computing Systems, ACM (2012), 2527--2530.

Digital Library

[6]

Enrico Guizzo, IEEE Robotics Blog. Hands-On With the Next Generation Kinect: PrimeSense Capri. http://spectrum.ieee.org/automaton/robotics/robotics-hardware/handson-with-the-next-generation-kinect-primesense-capri, May 2013.

[7]

Fischler, M. A., and Bolles, R. C. Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography. Communications of the ACM 24, 6 (1981), 381--395.

Digital Library

[8]

GSM Arena. HTC Evo 3D (Smartphone with Stereo Camera Pair). http://www.gsmarena.com/htc_evo_3d-3901.php, June 2013.

[9]

Harrison, C., Benko, H., and Wilson, A. D. Omnitouch: wearable multitouch interaction everywhere. In Proceedings of the 24th annual ACM symposium on User interface software and technology, ACM (2011), 441--450.

Digital Library

[10]

Hasan, K., Ahlström, D., and Irani, P. Ad-binning: leveraging around device space for storing, browsing and retrieving mobile device content. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, ACM (2013), 899--908.

Digital Library

[11]

Holz, C., and Baudisch, P. The generalized perceived input point model and how to double touch accuracy by extracting fingerprints. In Proceedings of the 28th international conference on Human factors in computing systems, ACM (2010), 581--590.

Digital Library

[12]

Holz, C., and Baudisch, P. Understanding touch. In Proceedings of the 2011 annual conference on Human factors in computing systems, ACM (2011), 2501--2510.

Digital Library

[13]

Intel. Intel Perceptual Computing SDK. http://software.intel.com/en-us/vcsource/tools/perceptual-computing-sdk, June 2013.

[14]

Ketabdar, H., Yüksel, K. A., Jahnbekam, A., Roshandel, M., and Skirpo, D. Magisign: User identification/authentication based on 3d around device magnetic signatures. In UBICOMM 2010, The Fourth International Conference on Mobile Ubiquitous Computing, Systems, Services and Technologies (2010), 31--34.

[15]

Kratz, S., Rohs, M., Guse, D., Müller, J., Bailly, G., and Nischt, M. Palmspace: continuous around-device gestures vs. multitouch for 3d rotation tasks on mobile devices. In Proceedings of the International Working Conference on Advanced Visual Interfaces, ACM (2012), 181--188.

Digital Library

[16]

Liang, R.-H., Cheng, K.-Y., Su, C.-H., Weng, C.-T., Chen, B.-Y., and Yang, D.-N. Gausssense: Attachable stylus sensing using magnetic sensor grid. In Proceedings of the 25th annual ACM symposium on User interface software and technology, ACM (2012), 319--326.

Digital Library

[17]

Moscovich, T., and Hughes, J. F. Navigating documents with the virtual scroll ring. In Proceedings of the 17th annual ACM symposium on User interface software and technology, ACM (2004), 57--60.

Digital Library

[18]

Olwal, A., Feiner, S., and Heyman, S. Rubbing and tapping for precise and rapid selection on touch-screen displays. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, ACM (2008), 295--304.

Digital Library

[19]

pygame.org. Pygame game programming library. http://www.pygame.org, June 2013.

[20]

Rogers, S., Williamson, J., Stewart, C., and Murray-Smith, R. Anglepose: robust, precise capacitive touch tracking via 3d orientation estimation. In Proceedings of the 2011 annual conference on Human factors in computing systems, ACM (2011), 2575--2584.

Digital Library

[21]

Roudaut, A., Huot, S., and Lecolinet, E. Taptap and magstick: improving one-handed target acquisition on small touch-screens. In Proceedings of the working conference on Advanced visual interfaces, ACM (2008), 146--153.

Digital Library

[22]

Roudaut, A., Lecolinet, E., and Guiard, Y. Microrolls: expanding touch-screen input vocabulary by distinguishing rolls vs. slides of the thumb. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, ACM (2009), 927--936.

Digital Library

[23]

Rusu, R. B., and Cousins, S. 3d is here: Point cloud library (pcl). In 2011 IEEE International Conference on Robotics and Automation (ICRA), IEEE (2011), 1--4.

[24]

Stewart, C., Rohs, M., Kratz, S., and Essl, G. Characteristics of pressure-based input for mobile devices. In Proceedings of the 28th international conference on Human factors in computing systems, ACM (2010), 801--810.

Digital Library

[25]

Takeoka, Y., Miyaki, T., and Rekimoto, J. Z-touch: an infrastructure for 3d gesture interaction in the proximity of tabletop surfaces. In ACM International Conference on Interactive Tabletops and Surfaces, ACM (2010), 91--94.

Digital Library

[26]

von Zezschwitz, E., Koslow, A., De Luca, A., and Hussmann, H. Making graphic-based authentication secure against smudge attacks. In Proceedings of the 2013 international conference on Intelligent user interfaces, ACM (2013), 277--286.

Digital Library

[27]

Wigdor, D., Forlines, C., Baudisch, P., Barnwell, J., and Shen, C. Lucid touch: a see-through mobile device. In Proceedings of the 20th annual ACM symposium on User interface software and technology, ACM (2007), 269--278.

Digital Library

[28]

Williams, L. SmartQuill. British Telecom Labs (1997).

Cited By

Li CYu JHe KFeng JZhou J(2024)SwivelTouch: Boosting Touchscreen Input with 3D Finger Rotation GestureProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36595848:2(1-30)Online publication date: 15-May-2024
https://dl.acm.org/doi/10.1145/3659584
He KLi CDuan YFeng JZhou J(2024)TrackPoseProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36314597:4(1-22)Online publication date: 12-Jan-2024
https://dl.acm.org/doi/10.1145/3631459
Li MZhou YZhang LLi TJiang Y(2024)OneTip: A soft tactile interface for 6-D fingertip pose acquisition in human-computer interactionSensors and Actuators A: Physical10.1016/j.sna.2024.115896379(115896)Online publication date: Dec-2024
https://doi.org/10.1016/j.sna.2024.115896
Show More Cited By

Index Terms

PointPose: finger pose estimation for touch input on mobile devices using a depth sensor
1. Human-centered computing

Recommendations

PointPose demo: using a depth camera for finger pose estimation on touch-based mobile devices
ITS '13: Proceedings of the 2013 ACM international conference on Interactive tabletops and surfaces

We present PointPose, a prototype that allows finger pose information (tilt and rotation) to be obtained at the point of touch on touch-based mobile device, thus adding to the expressiveness of touch input. PointPose uses a short-range depth sensor ...
Extending a mobile device's interaction space through body-centric interaction
MobileHCI '12: Proceedings of the 14th international conference on Human-computer interaction with mobile devices and services

Modern mobile devices rely on the screen as a primary input modality. Yet the small screen real-estate limits interaction possibilities, motivating researchers to explore alternate input techniques. Within this arena, our goal is to develop Body-Centric ...
Mobile interaction with the real world
MobileHCI '06: Proceedings of the 8th conference on Human-computer interaction with mobile devices and services

The main goal of the workshop is to discuss approaches that use a mobile device (e.g. mobile phone, smartphone, PDA) for interactions with objects in the real world. Relevant topics include (but are not limited to) mobile interaction with the real world;...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

ITS '13: Proceedings of the 2013 ACM international conference on Interactive tabletops and surfaces

October 2013

514 pages

ISBN:9781450322713

DOI:10.1145/2512349

General Chairs:
Aaron Quigley
University of St. Andrews, UK
,
Giulio Jacucci
HIIT / University of Helsinki, Finland
,
Program Chairs:
Michael Horn
Northwestern University, USA
,
Miguel Nacenta
University of St. Andrews, UK

Copyright © 2013 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

SIGCHI: ACM Special Interest Group on Computer-Human Interaction

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 06 October 2013

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

ITS '13

Sponsor:

SIGCHI

ITS '13: The ACM International Conference on Interactive Tabletops and Surfaces

October 6 - 9, 2013

St. Andrews, Scotland, United Kingdom

Acceptance Rates

ITS '13 Paper Acceptance Rate 35 of 121 submissions, 29%;

Overall Acceptance Rate 119 of 418 submissions, 28%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

27
Total Citations
View Citations
506
Total Downloads

Downloads (Last 12 months)18
Downloads (Last 6 weeks)2

Reflects downloads up to 25 Dec 2024

Other Metrics

View Author Metrics

Citations

Cited By

Li CYu JHe KFeng JZhou J(2024)SwivelTouch: Boosting Touchscreen Input with 3D Finger Rotation GestureProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36595848:2(1-30)Online publication date: 15-May-2024
https://dl.acm.org/doi/10.1145/3659584
He KLi CDuan YFeng JZhou J(2024)TrackPoseProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36314597:4(1-22)Online publication date: 12-Jan-2024
https://dl.acm.org/doi/10.1145/3631459
Li MZhou YZhang LLi TJiang Y(2024)OneTip: A soft tactile interface for 6-D fingertip pose acquisition in human-computer interactionSensors and Actuators A: Physical10.1016/j.sna.2024.115896379(115896)Online publication date: Dec-2024
https://doi.org/10.1016/j.sna.2024.115896
Duan YYu JFeng JHe KLu JZhou J(2023)3D Finger Rotation Estimation from Fingerprint ImagesProceedings of the ACM on Human-Computer Interaction10.1145/36264677:ISS(114-134)Online publication date: 1-Nov-2023
https://dl.acm.org/doi/10.1145/3626467
Yu JFeng JZhou J(2023)PrintShearProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/35962577:2(1-22)Online publication date: 12-Jun-2023
https://dl.acm.org/doi/10.1145/3596257
Suzuki TManabe H(2023)Estimation of Brush Type Passive Stylus Angles Using Capacitive ImageProceedings of the Seventeenth International Conference on Tangible, Embedded, and Embodied Interaction10.1145/3569009.3573112(1-6)Online publication date: 26-Feb-2023
https://dl.acm.org/doi/10.1145/3569009.3573112
Ullerich JWindl MBulling AMayer S(2022)ThumbPitch: Enriching Thumb Interaction on Mobile Touchscreens using Deep LearningProceedings of the 34th Australian Conference on Human-Computer Interaction10.1145/3572921.3572925(58-66)Online publication date: 29-Nov-2022
https://dl.acm.org/doi/10.1145/3572921.3572925
He KDuan YFeng JZhou J(2022)Estimating 3D Finger Angle via Fingerprint ImageProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/35172436:1(1-22)Online publication date: 29-Mar-2022
https://dl.acm.org/doi/10.1145/3517243
Duan YHe KFeng JLu JZhou J(2022)Estimating 3D Finger Pose via 2D-3D Fingerprint MatchingProceedings of the 27th International Conference on Intelligent User Interfaces10.1145/3490099.3511123(459-469)Online publication date: 22-Mar-2022
https://dl.acm.org/doi/10.1145/3490099.3511123
Vogelsang JKiss FMayer S(2021)A Design Space for User Interface Elements using Finger Orientation InputProceedings of Mensch und Computer 202110.1145/3473856.3473862(1-10)Online publication date: 5-Sep-2021
https://dl.acm.org/doi/10.1145/3473856.3473862
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents