research-article

Open access

Estimating 3D Finger Angle via Fingerprint Image

Authors:

Jianjiang Feng,

Jie ZhouAuthors Info & Claims

Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies, Volume 6, Issue 1

Article No.: 14, Pages 1 - 22

https://doi.org/10.1145/3517243

Published: 29 March 2022 Publication History

Abstract

Touchscreens are the primary input devices for smartphones and tablets. Although widely used, the output of touchscreen controllers is still limited to the two-dimensional position of the contacting finger. Finger angle (or orientation) estimation from touchscreen images has been studied for enriching touch input. However, only pitch and yaw are usually estimated and estimation error is large. One main reason is that touchscreens provide very limited information of finger. With the development of under-screen fingerprint sensing technology, fingerprint images, which contain more information of finger compared with touchscreen images, can be captured when a finger touches the screen. In this paper, we constructed a dataset with fingerprint images and the corresponding ground truth values of finger angle. We contribute with a network architecture and training strategy that harness the strong dependencies among finger angle, finger region, finger type, and fingerprint ridge orientation to produce a top-performing model for finger angle estimation. The experimental results demonstrate the superiority of our method over previous state-of-the-art methods. The mean absolute errors of the three angles are 6.6 degrees for yaw, 7.1 degrees for pitch, and 9.1 degrees for roll, markedly smaller than previously reported errors. Extensive experiments were conducted to examine important factors including image resolution, image size, and finger type. Evaluations on a set of under-screen fingerprints were also performed to explore feasibility in real-world applications. Code and a subset of the data are publicly available.

Supplementary Material

he (he.zip)

Supplemental movie, appendix, image and software files for, Estimating 3D Finger Angle via Fingerprint Image

Download
190.51 MB

References

[1]

Herbert Bay, Andreas Ess, Tinne Tuytelaars, and Luc Van Gool. 2008. Speeded-Up Robust Features (SURF). Computer Vision and Image Understanding 110, 3 (June 2008), 346--359. https://doi.org/10.1016/j.xviu.2007.09.014

[2]

A.M. Bazen and Sabih H. Gerez. 2001. Segmentation of Fingerprint Images. 14th ProRISC Workshop on Circuits, Systems and Signal Processing 2003 (2001), 276--280.

[3]

Tobias Boceck, Sascha Sprott, Huy Viet Le, and Sven Mayer. 2019. Force Touch Detection on Capacitive Sensors Using Deep Neural Networks. In Proceedings of the 21st International Conference on Human-Computer Interaction with Mobile Devices and Services (Taipei, Taiwan) (MobileHCI '19). Association for Computing Machinery, New York, NY, USA, Article 42, 6 pages. https://doi.org/10.1145/3338286.3344389

Digital Library

[4]

Sebastian Boring, David Ledo, Xiang'Anthony' Chen, Nicolai Marquardt, Anthony Tang, and Saul Greenberg. 2012. The fat thumb: using the thumb's contact size for single-handed mobile interaction. In Proceedings of the 14th International Conference on Human-computer Interaction with Mobile Devices and Services. 39--48. https://doi.org/10.1145/2371574.2371582

Digital Library

[5]

Frederick Choi, Sven Mayer, and Chris Harrison. 2021. 3D Hand Pose Estimation on Conventional Capacitive Touchscreens. Proceedings of the 23rd International Conference on Mobile Human-Computer Interaction (2021).

Digital Library

[6]

Chi Tai Dang and Elisabeth André. 2011. Usage and recognition of finger orientation for multi-touch tabletop interaction. In INTERACT'11 Proceedings of the 13th IFIP TC 13 International Conference on Human-computer Interaction - Volume Part III. 409--426.

[7]

Hyunjae Gil, Hongmin Kim, and Ian Oakley. 2018. Fingers and Angles: Exploring the Comfort of Touch Input on Smartwatches. Proc. ACM Interact. Mob. Wearable Ubiquitous Technol. 2, 4, Article 164 (Dec. 2018), 21 pages. https://doi.org/10.1145/3287042

Digital Library

[8]

Alix Goguey, Géry Casiez, Daniel Vogel, and Carl Gutwin. 2018. Characterizing Finger Pitch and Roll Orientation During Atomic Touch Actions. Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems (2018).

Digital Library

[9]

Tobias Grosse-Puppendahl, Christian Holz, Gabe Cohn, Raphael Wimmer, Oskar Bechtold, Steve Hodges, Matthew S. Reynolds, and Joshua R. Smith. 2017. Finding Common Ground: A Survey of Capacitive Sensing in Human-Computer Interaction. Association for Computing Machinery, New York, NY, USA, 3293--3315. https://doi.org/10.1145/3025453.3025808

Digital Library

[10]

Chris Harrison, Julia Schwarz, and Scott E. Hudson. 2011. TapSense: Enhancing Finger Interaction on Touch Surfaces. In Proceedings of the 24th Annual ACM Symposium on User Interface Software and Technology (Santa Barbara, California, USA) (UIST '11). Association for Computing Machinery, New York, NY, USA, 627--636. https://doi.org/10.1145/2047196.2047279

Digital Library

[11]

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep Residual Learning for Image Recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 770--778.

[12]

Christian Holz and Patrick Baudisch. 2010. The generalized perceived input point model and how to double touch accuracy by extracting fingerprints. In Proceedings of the 28th International Conference on Human Factors in Computing Systems - CHI '10. ACM Press. https://doi.org/10.1145/1753326.1753413

Digital Library

[13]

Davood Karimi and Septimiu E. Salcudean. 2020. Reducing the Hausdorff Distance in Medical Image Segmentation With Convolutional Neural Networks. IEEE Transactions on Medical Imaging 39, 2 (2020), 499--513.

[14]

Sven Kratz, Patrick Chiu, and Maribeth Back. 2013. PointPose: Finger Pose Estimation for Touch Input on Mobile Devices Using a Depth Sensor. In Proceedings of the 2013 ACM International Conference on Interactive Tabletops and Surfaces (St. Andrews, Scotland, United Kingdom) (ITS '13). Association for Computing Machinery, New York, NY, USA, 223--230. https://doi.org/10.1145/2512349.2512824

Digital Library

[15]

David G. Lowe. 2004. Distinctive Image Features from Scale-Invariant Keypoints. International Journal of Computer Vision 60, 2 (Nov. 2004), 91--110. https://doi.org/10.1023/b:visi.0000029664.99615.94

Digital Library

[16]

Sven Mayer, Huy Viet Le, and Niels Henze. 2017. Estimating the Finger Orientation on Capacitive Touchscreens Using Convolutional Neural Networks. In Proceedings of the 2017 ACM International Conference on Interactive Surfaces and Spaces. ACM. https://doi.org/10.1145/3132272.3134130

Digital Library

[17]

Sven Mayer, Huy Viet Le, and Niels Henze. 2018. Designing finger orientation input for mobile touchscreens. In Proceedings of the 20th International Conference on Human-Computer Interaction with Mobile Devices and Services. ACM. https://doi.org/10.1145/3229434.3229444

Digital Library

[18]

Sven Mayer, Michael Mayer, and Niels Henze. 2017. Feasibility analysis of detecting the finger orientation with depth cameras. In Proceedings of the 19th International Conference on Human-Computer Interaction with Mobile Devices and Services. ACM. https://doi.org/10.1145/3098279.3122125

Digital Library

[19]

Ian Oakley, Carina Lindahl, Khanh Le, DoYoung Lee, and MD. Rasel Islam. 2016. The Flat Finger: Exploring Area Touches on Smartwatches. Association for Computing Machinery, New York, NY, USA, 4238--4249. https://doi.org/10.1145/2858036.2858179

Digital Library

[20]

Chang Peng, Mengyue Chen, and Xiaoning Jiang. 2021. Under-Display Ultrasonic Fingerprint Recognition With Finger Vessel Imaging. IEEE Sensors Journal 21, 6 (March 2021), 7412--7419. https://doi.org/10.1109/jsen.2021.3051975

[21]

Gabriel Pereyra, George Tucker, Jan Chorowski, Łukasz Kaiser, and Geoffrey Hinton. 2017. Regularizing Neural Networks by Penalizing Confident Output Distributions. In ICLR (Workshop).

[22]

Nalini K. Ratha, Shaoyun Chen, and Anil K. Jain. 1995. Adaptive flow orientation-based feature extraction in fingerprint images. Pattern Recognition 28, 11 (1995), 1657--1672. https://doi.org/10.1016/0031-3203(95)00039-3

[23]

Simon Rogers, John Williamson, Craig Stewart, and Roderick Murray-Smith. 2011. AnglePose: Robust, Precise Capacitive Touch Tracking via 3d Orientation Estimation. Association for Computing Machinery, New York, NY, USA, 2575--2584. https://doi.org/10.1145/1978942.1979318

Digital Library

[24]

Olaf Ronneberger, Philipp Fischer, and Thomas Brox. 2015. U-Net: Convolutional Networks for Biomedical Image Segmentation. In International Conference on Medical Image Computing and Computer-Assisted Intervention. 234--241.

[25]

Anne Roudaut, Eric Lecolinet, and Yves Guiard. 2009. MicroRolls: Expanding Touch-Screen Input Vocabulary by Distinguishing Rolls vs. Slides of the Thumb. Association for Computing Machinery, New York, NY, USA, 927--936. https://doi.org/10.1145/1518701.1518843

Digital Library

[26]

Nataniel Ruiz, Eunji Chong, and James M. Rehg. 2018. Fine-Grained Head Pose Estimation Without Keypoints. In 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW). 2074--2083.

[27]

Yijing Su, Jianjiang Feng, and Jie Zhou. 2016. Fingerprint indexing with pose constraint. Pattern Recognit. 54 (2016), 1--13. https://doi.org/10.1016/j.patcog.2016.01.006

Digital Library

[28]

Carole H. Sudre, Wenqi Li, Tom Vercauteren, Sebastien Ourselin, and M.Jorge Cardoso.2017. Generalised Dice Overlap as a Deep Learning Loss Function for Highly Unbalanced Segmentations. Lecture Notes in Computer Science (2017), 240--248. https://doi.org/10.1007/978-3-319-67558-9_28

[29]

Simon Vandenhende, Stamatios Georgoulis, Wouter Van Gansbeke, Marc Proesmans, Dengxin Dai, and Luc Van Gool. 2021. MultiTask Learning for Dense Prediction Tasks: A Survey. IEEE Transactions on Pattern Analysis and Machine Intelligence (2021), 1--1. https://doi.org/10.1109/tpami.2021.3054719

[30]

Manuel Veit, Antonio Capobianco, and Dominique Bechmann. 2009. Influence of Degrees of Freedom's Manipulation on Performances during Orientation Tasks in Virtual Reality Environments. In Proceedings of the 16th ACM Symposium on Virtual Reality Software and Technology (Kyoto, Japan) (VRST '09). Association for Computing Machinery, New York, NY, USA, 51--58. https://doi.org/10.1145/1643928.1643942

Digital Library

[31]

Jonas Vogelsang, Francisco Kiss, and Sven Mayer. 2021. A Design Space for User Interface Elements Using Finger Orientation Input. In Mensch Und Computer 2021 (Ingolstadt, Germany) (MuC '21). Association for Computing Machinery, New York, NY, USA, 1--10. https://doi.org/10.1145/3473856.3473862

Digital Library

[32]

Feng Wang, Xiang Cao, Xiangshi Ren, and Pourang Irani. 2009. Detecting and leveraging finger orientation for interaction with direct-touch surfaces. In Proceedings of the 22nd Annual ACM Symposium on User Interface Software and Technology - UIST '09. ACM Press. https://doi.org/10.1145/1622176.1622182

Digital Library

[33]

Yoichi Watanabe, Yasutoshi Makino, Katsunari Sato, and Takashi Maeno. 2012. Contact force and finger angles estimation for touch panel by detecting transmitted light on fingernail. In EuroHaptics'12 Proceedings of the 2012 International Conference on Haptics: Perception, Devices, Mobility, and Communication - Volume Part I. 601--612.

Digital Library

[34]

Robert Xiao, Julia Schwarz, and Chris Harrison. 2015. Estimating 3D Finger Angle on Commodity Touchscreens. In Proceedings of the 2015 International Conference on Interactive Tabletops & Surfaces - ITS '15. ACM Press. https://doi.org/10.1145/2817721.2817737

Digital Library

[35]

Ping-Hung Yin, Chih-Wen Lu, Jia-Shyang Wang, Keng-Li Chang, Fu-Kuo Lin, and Poki Chen. 2021. A 368 x 184 Optical Under-Display Fingerprint Sensor Comprising Hybrid Arrays of Global and Rolling Shutter Pixels With Shared Pixel-Level ADCs. IEEE Journal of Solid-State Circuits 56, 3 (March 2021), 763--777. https://doi.org/10.1109/jssc.2020.3042894

[36]

Qihao Yin, Jianjiang Feng, Jiwen Lu, and Jie Zhou. 2021. Joint Estimation of Pose and Singular Points of Fingerprints. IEEE Trans. Inf. Forensics Secur. 16 (2021), 1467--1479. https://doi.org/10.1109/TIFS.2020.3036803

[37]

Vadim Zaliva. 2012. 3D finger posture detection and gesture recognition on touch surfaces. In 2012 12th International Conference on Control Automation Robotics & Vision (ICARCV). IEEE. https://doi.org/10.1109/icarcv.2012.6485185

Cited By

Li CYu JHe KFeng JZhou J(2024)SwivelTouch: Boosting Touchscreen Input with 3D Finger Rotation GestureProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36595848:2(1-30)Online publication date: 15-May-2024
https://dl.acm.org/doi/10.1145/3659584
Feng FBennett DMekler E(2024)Smiles Summon the Warmth of Spring: A Design Framework for Thermal-Affective Interaction based in Chinese Cí PoetryProceedings of the 2024 ACM Designing Interactive Systems Conference10.1145/3643834.3661620(2802-2819)Online publication date: 1-Jul-2024
https://dl.acm.org/doi/10.1145/3643834.3661620
Huang ZGao CWang HDeng XLai YMa CQin SLiu YWang H(2024)SpeciFingersProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36435598:1(1-28)Online publication date: 6-Mar-2024
https://dl.acm.org/doi/10.1145/3643559
Show More Cited By

Index Terms

Estimating 3D Finger Angle via Fingerprint Image
1. Computing methodologies
  1. Machine learning
    1. Learning paradigms
      1. Multi-task learning
2. Human-centered computing
  1. Human computer interaction (HCI)
    1. Interaction techniques

Recommendations

Estimating 3D Finger Pose via 2D-3D Fingerprint Matching
IUI '22: Proceedings of the 27th International Conference on Intelligent User Interfaces

Touchscreens have become the primary input devices for smartphones, tablet computers, and other intelligent devices over the past decades. While for the most pervasive commercial devices, only 2D touch positions on the screen are utilized as interaction ...
Fingerprint image: pre- and post-processing

Automatic Fingerprint Identification Systems (AFIS) are widely used for personal identification due to uniqueness of fingerprints. Minutiae-based fingerprint matching techniques are normally used for fingerprint matching. Fingerprint matching results ...
3D Finger Rotation Estimation from Fingerprint Images

Various touch-based interaction techniques have been developed to make interactions on mobile devices more effective, efficient, and intuitive. Finger orientation, especially, has attracted a lot of attentions since it intuitively brings three additional ...

Comments

Information & Contributors

Information

Published In

cover image Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies

Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies Volume 6, Issue 1

March 2022

1009 pages

EISSN:2474-9567

DOI:10.1145/3529514

Issue’s Table of Contents

Copyright © 2022 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 29 March 2022

Published in IMWUT Volume 6, Issue 1

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed

Funding Sources

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

10
Total Citations
View Citations
688
Total Downloads

Downloads (Last 12 months)162
Downloads (Last 6 weeks)22

Reflects downloads up to 24 Dec 2024

Other Metrics

View Author Metrics

Citations

Cited By

Li CYu JHe KFeng JZhou J(2024)SwivelTouch: Boosting Touchscreen Input with 3D Finger Rotation GestureProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36595848:2(1-30)Online publication date: 15-May-2024
https://dl.acm.org/doi/10.1145/3659584
Feng FBennett DMekler E(2024)Smiles Summon the Warmth of Spring: A Design Framework for Thermal-Affective Interaction based in Chinese Cí PoetryProceedings of the 2024 ACM Designing Interactive Systems Conference10.1145/3643834.3661620(2802-2819)Online publication date: 1-Jul-2024
https://dl.acm.org/doi/10.1145/3643834.3661620
Huang ZGao CWang HDeng XLai YMa CQin SLiu YWang H(2024)SpeciFingersProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36435598:1(1-28)Online publication date: 6-Mar-2024
https://dl.acm.org/doi/10.1145/3643559
He KLi CDuan YFeng JZhou J(2024)TrackPoseProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36314597:4(1-22)Online publication date: 12-Jan-2024
https://doi.org/10.1145/3631459
Li MZhou YZhang LLi TJiang Y(2024)OneTip: A soft tactile interface for 6-D fingertip pose acquisition in human-computer interactionSensors and Actuators A: Physical10.1016/j.sna.2024.115896379(115896)Online publication date: Dec-2024
https://doi.org/10.1016/j.sna.2024.115896
Duan YYu JFeng JHe KLu JZhou J(2023)3D Finger Rotation Estimation from Fingerprint ImagesProceedings of the ACM on Human-Computer Interaction10.1145/36264677:ISS(114-134)Online publication date: 1-Nov-2023
https://dl.acm.org/doi/10.1145/3626467
Torun MMostofi Y(2023)Wi-FlexProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36109307:3(1-27)Online publication date: 27-Sep-2023
https://dl.acm.org/doi/10.1145/3610930
Yu JFeng JZhou J(2023)PrintShearProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/35962577:2(1-22)Online publication date: 12-Jun-2023
https://dl.acm.org/doi/10.1145/3596257
Meng XHe YKunze KWard JMcGill MMarky K(2023)Towards Enhancing a Recorded Concert Experience in Virtual Reality by Visualizing the Physiological Data of the AudienceProceedings of the Augmented Humans International Conference 202310.1145/3582700.3583709(330-333)Online publication date: 12-Mar-2023
https://dl.acm.org/doi/10.1145/3582700.3583709
Suzuki TManabe H(2023)Estimation of Brush Type Passive Stylus Angles Using Capacitive ImageProceedings of the Seventeenth International Conference on Tangible, Embedded, and Embodied Interaction10.1145/3569009.3573112(1-6)Online publication date: 26-Feb-2023
https://dl.acm.org/doi/10.1145/3569009.3573112

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

Media

Figures

Other

Tables

View Issue’s Table of Contents