Location via proxy:   [ UP ]  
[Report a bug]   [Manage cookies]                
skip to main content
research-article
Open access

PrintShear: Shear Input Based on Fingerprint Deformation

Published: 12 June 2023 Publication History

Abstract

Most touch-based input devices, such as touchscreens and touchpads, capture low-resolution capacitive images when a finger touches the device's surface. These devices only output the two-dimensional (2D) positions of contacting points, which are insufficient for complex control tasks, such as the manipulation of 3D objects. To expand the modalities of touch inputs, researchers have proposed a variety of techniques, including finger poses, chording gestures, touch pressure, etc. With the rapid development of fingerprint sensing technology, especially under-screen fingerprint sensors, it has become possible to generate input commands to control multiple degrees of freedom (DOF) at a time using fingerprint images. In this paper, we propose PrintShear, a shear input technique based on fingerprint deformation. Lateral, longitudinal and rotational deformations are extracted from fingerprint images and mapped to 3DOF control commands. Further DOF expansion can be achieved through recognition of the contact region of the touching finger. We conducted a 12-person user study to evaluate the performance of PrintShear on 3D docking tasks. Comparisons with other input methods demonstrated the superiority of our approach. Specifically, a 19.79% reduction in completion time was achieved compared with conventional touch input in a full 6DOF 3D object manipulation task.

Supplemental Material

MP4 File
Supplemental video
MP4 File
Supplemental video

References

[1]
A. Antonelli, R. Cappelli, D. Maio, and D. Maltoni. 2006. Fake Finger Detection by Skin Distortion Analysis. IEEE Transactions on Information Forensics and Security 1, 3 (2006), 360--373. https://doi.org/10.1109/TIFS.2006.879289
[2]
Robert C Barrett, Edwin J Selker, Joseph D Rutledge, and Robert S Olyha. 1995. Negative Inertia: A Dynamic Pointing Function. In Proceedings of the Conference Companion on Human Factors in Computing Systems. Association for Computing Machinery, New York, NY, USA, 316--317.
[3]
Anil Ufuk Batmaz, Mohammad Rajabi Seraji, Johanna Kneifel, and Wolfgang Stuerzlinger. 2021. No Jitter Please: Effects of Rotational and Positional Jitter on 3D Mid-Air Interaction. In Proceedings of the Future Technologies Conference (FTC) 2020, Volume 2, Kohei Arai, Supriya Kapoor, and Rahul Bhatia (Eds.). Springer International Publishing, Cham, 792--808. https://doi.org/10.1007/978--3--030--63089--8_52
[4]
Herbert Bay, Andreas Ess, Tinne Tuytelaars, and Luc Van Gool. 2008. Speeded-Up Robust Features (SURF). Computer Vision and Image Understanding 110, 3 (2008), 346--359. https://doi.org/10.1016/j.cviu.2007.09.014
[5]
Lonni Besançon, Paul Issartel, Mehdi Ammi, and Tobias Isenberg. 2017. Mouse, Tactile, and Tangible Input for 3D Manipulation. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems (Denver, Colorado, USA) (CHI '17). Association for Computing Machinery, New York, NY, USA, 4727--4740. https://doi.org/10.1145/3025453.3025863
[6]
Tobias Boceck, Sascha Sprott, Huy Viet Le, and Sven Mayer. 2019. Force Touch Detection on Capacitive Sensors Using Deep Neural Networks. In Proceedings of the 21st International Conference on Human-Computer Interaction with Mobile Devices and Services (Taipei, Taiwan) (MobileHCI '19). Association for Computing Machinery, New York, NY, USA, Article 42, 6 pages. https://doi.org/10.1145/3338286.3344389
[7]
Sebastian Boring, David Ledo, Xiang 'Anthony' Chen, Nicolai Marquardt, Anthony Tang, and Saul Greenberg. 2012. The Fat Thumb: Using the Thumb's Contact Size for Single-Handed Mobile Interaction. In Proceedings of the 14th International Conference on Human-Computer Interaction with Mobile Devices and Services (San Francisco, California, USA) (MobileHCI '12). Association for Computing Machinery, New York, NY, USA, 39--48. https://doi.org/10.1145/2371574.2371582
[8]
R. Cappelli, D. Maio, and D. Maltoni. 2001. Modelling Plastic Distortion in Fingerprint Images. In Advances in Pattern Recognition --- ICAPR 2001, Sameer Singh, Nabeel Murshed, and Walter Kropatsch (Eds.). Springer Berlin Heidelberg, Berlin, Heidelberg, 371--378.
[9]
Géry Casiez and Daniel Vogel. 2008. The Effect of Spring Stiffness and Control Gain with an Elastic Rate Control Pointing Device. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Florence, Italy) (CHI '08). Association for Computing Machinery, New York, NY, USA, 1709--1718. https://doi.org/10.1145/1357054.1357321
[10]
Géry Casiez, Daniel Vogel, Qing Pan, and Christophe Chaillou. 2007. RubberEdge: Reducing Clutching by Combining Position and Rate Control with Elastic Feedback. In Proceedings of the 20th Annual ACM Symposium on User Interface Software and Technology (Newport, Rhode Island, USA) (UIST '07). Association for Computing Machinery, New York, NY, USA, 129--138. https://doi.org/10.1145/1294211.1294234
[11]
Ashley Colley and Jonna Häkkilä. 2014. Exploring Finger Specific Touch Screen Interaction for Mobile Phone User Interfaces. In Proceedings of the 26th Australian Computer-Human Interaction Conference on Designing Futures: The Future of Design (Sydney, New South Wales, Australia) (OzCHI '14). Association for Computing Machinery, New York, NY, USA, 539--548. https://doi.org/10.1145/2686612.2686699
[12]
Christian Corsten, Marcel Lahaye, Jan Borchers, and Simon Voelker. 2019. ForceRay: Extending Thumb Reach via Force Input Stabilizes Device Grip for Mobile Touch Input. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (Glasgow, Scotland Uk) (CHI '19). Association for Computing Machinery, New York, NY, USA, 1--12. https://doi.org/10.1145/3290605.3300442
[13]
Christian Corsten, Marcel Lahaye, Jan Borchers, and Simon Voelker. 2019. ForceRay: Extending Thumb Reach via Force Input Stabilizes Device Grip for Mobile Touch Input. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (Glasgow, Scotland Uk) (CHI '19). Association for Computing Machinery, New York, NY, USA, 1--12. https://doi.org/10.1145/3290605.3300442
[14]
Zhe Cui, Jianjiang Feng, Shihao Li, Jiwen Lu, and Jie Zhou. 2018. 2-D Phase Demodulation for Deformable Fingerprint Registration. IEEE Transactions on Information Forensics and Security 13, 12 (2018), 3153--3165. https://doi.org/10.1109/TIFS.2018.2841849
[15]
Yongjie Duan, Ke He, Jianjiang Feng, Jiwen Lu, and Jie Zhou. 2022. Estimating 3D Finger Pose via 2D-3D Fingerprint Matching. In 27th International Conference on Intelligent User Interfaces (Helsinki, Finland) (IUI '22). Association for Computing Machinery, New York, NY, USA, 459--469. https://doi.org/10.1145/3490099.3511123
[16]
Jan-Erik Ekberg, Kari Kostiainen, and Nadarajah Asokan. 2013. Trusted Execution Environments on Mobile Devices. In Proceedings of the 2013 ACM SIGSAC Conference on Computer & Communications Security (Berlin, Germany) (CCS '13). Association for Computing Machinery, New York, NY, USA, 1497--1498. https://doi.org/10.1145/2508859.2516758
[17]
Mario Frank, Ralf Biedert, Eugene Ma, Ivan Martinovic, and Dawn Song. 2013. Touchalytics: On the Applicability of Touchscreen Input as a Behavioral Biometric for Continuous Authentication. IEEE Transactions on Information Forensics and Security 8, 1 (2013), 136--148. https://doi.org/10.1109/TIFS.2012.2225048
[18]
Scott Frees, G. Drew Kessler, and Edwin Kay. 2007. PRISM Interaction for Enhancing Control in Immersive Virtual Environments. ACM Trans. Comput.-Hum. Interact. 14, 1 (May 2007), 2-es. https://doi.org/10.1145/1229855.1229857
[19]
Tobias Grosse-Puppendahl, Christian Holz, Gabe Cohn, Raphael Wimmer, Oskar Bechtold, Steve Hodges, Matthew S. Reynolds, and Joshua R. Smith. 2017. Finding Common Ground: A Survey of Capacitive Sensing in Human-Computer Interaction. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems (Denver, Colorado, USA) (CHI '17). Association for Computing Machinery, New York, NY, USA, 3293--3315. https://doi.org/10.1145/3025453.3025808
[20]
Mark Hancock, Thomas ten Cate, and Sheelagh Carpendale. 2009. Sticky Tools: Full 6DOF Force-Based Interaction for Multi-Touch Tables. In Proceedings of the ACM International Conference on Interactive Tabletops and Surfaces (Banff, Alberta, Canada) (ITS '09). Association for Computing Machinery, New York, NY, USA, 133--140. https://doi.org/10.1145/1731903.1731930
[21]
M.S. Hancock, F.D. Vernier, D. Wigdor, S. Carpendale, and Chia Shen. 2006. Rotation and Translation Mechanisms for Tabletop Interaction. In First IEEE International Workshop on Horizontal Interactive Human-Computer Systems (TABLETOP '06). 8 pp.-. https://doi.org/10.1109/TABLETOP.2006.26
[22]
Chris Harrison and Scott Hudson. 2012. Using Shear as a Supplemental Two-Dimensional Input Channel for Rich Touchscreen Interaction. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Austin, Texas, USA) (CHI '12). Association for Computing Machinery, New York, NY, USA, 3149--3152. https://doi.org/10.1145/2207676.2208730
[23]
Chris Harrison, Julia Schwarz, and Scott E. Hudson. 2011. TapSense: Enhancing Finger Interaction on Touch Surfaces. In Proceedings of the 24th Annual ACM Symposium on User Interface Software and Technology (Santa Barbara, California, USA) (UIST '11). Association for Computing Machinery, New York, NY, USA, 627--636. https://doi.org/10.1145/2047196.2047279
[24]
Sandra G. Hart. 2006. Nasa-Task Load Index (NASA-TLX); 20 Years Later. Proceedings of the Human Factors and Ergonomics Society Annual Meeting 50, 9 (2006), 904--908. https://doi.org/10.1177/154193120605000909
[25]
Ke He, Yongjie Duan, Jianjiang Feng, and Jie Zhou. 2022. Estimating 3D Finger Angle via Fingerprint Image. Proc. ACM Interact. Mob. Wearable Ubiquitous Technol. 6, 1, Article 14 (Mar 2022), 22 pages. https://doi.org/10.1145/3517243
[26]
Seongkook Heo and Geehyuk Lee. 2011. Force Gestures: Augmented Touch Screen Gestures Using Normal and Tangential Force. In CHI '11 Extended Abstracts on Human Factors in Computing Systems (Vancouver, BC, Canada) (CHI EA '11). Association for Computing Machinery, New York, NY, USA, 1909--1914. https://doi.org/10.1145/1979742.1979895
[27]
Seongkook Heo and Geehyuk Lee. 2012. ForceDrag: Using Pressure as a Touch Input Modifier. In Proceedings of the 24th Australian Computer-Human Interaction Conference (Melbourne, Australia) (OzCHI '12). Association for Computing Machinery, New York, NY, USA, 204--207. https://doi.org/10.1145/2414536.2414572
[28]
Seongkook Heo and Geehyuk Lee. 2013. Indirect Shear Force Estimation for Multi-Point Shear Force Operations. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Paris, France) (CHI '13). Association for Computing Machinery, New York, NY, USA, 281--284. https://doi.org/10.1145/2470654.2470693
[29]
Da-Yuan Huang, Ming-Chang Tsai, Ying-Chao Tung, Min-Lun Tsai, Yen-Ting Yeh, Liwei Chan, Yi-Ping Hung, and Mike Y. Chen. 2014. TouchSense: Expanding Touchscreen Input Vocabulary Using Different Areas of Users' Finger Pads. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Toronto, Ontario, Canada) (CHI '14). Association for Computing Machinery, New York, NY, USA, 189--192. https://doi.org/10.1145/2556288.2557258
[30]
Mengting Huang, Kazuyuki Fujita, Kazuki Takashima, Taichi Tsuchida, Hiroyuki Manabe, and Yoshifumi Kitamura. 2019. ShearSheet: Low-Cost Shear Force Input with Elastic Feedback for Augmenting Touch Interaction. In Proceedings of the 2019 ACM International Conference on Interactive Surfaces and Spaces (Daejeon, Republic of Korea) (ISS '19). Association for Computing Machinery, New York, NY, USA, 77--87. https://doi.org/10.1145/3343055.3359717
[31]
Sven Kratz, Patrick Chiu, and Maribeth Back. 2013. PointPose: Finger Pose Estimation for Touch Input on Mobile Devices Using a Depth Sensor. In Proceedings of the 2013 ACM International Conference on Interactive Tabletops and Surfaces (St. Andrews, Scotland, United Kingdom) (ITS 13). Association for Computing Machinery, New York, NY, USA, 223--230. https://doi.org/10.1145/2512349.2512824
[32]
Y. Kurita, A. Ikeda, Jun Ueda, and T. Ogasawara. 2005. A Fingerprint Pointing Device Utilizing the Deformation of the Fingertip During the Incipient Slip. IEEE Transactions on Robotics 21, 5 (2005), 801--811. https://doi.org/10.1109/TRO.2005.847568
[33]
Huy Viet Le, Thomas Kosch, Patrick Bader, Sven Mayer, and Niels Henze. 2018. PalmTouch: Using the Palm as an Additional Input Modality on Commodity Smartphones. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems (Montreal QC, Canada) (CHI '18). Association for Computing Machinery, New York, NY, USA, 1--13. https://doi.org/10.1145/3173574.3173934
[34]
Bhoram Lee, Hyunjeong Lee, Soo-Chul Lim, Hyungkew Lee, Seungju Han, and Joonah Park. 2012. Evaluation of Human Tangential Force Input Performance. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Austin, Texas, USA) (CHI '12). Association for Computing Machinery, New York, NY, USA, 3121--3130. https://doi.org/10.1145/2207676.2208727
[35]
Jingbo Liu, Oscar Kin-Chung Au, Hongbo Fu, and Chiew-Lan Tai. 2012. Two-Finger Gestures for 6DOF Manipulation of 3D Objects. Computer Graphics Forum 31, 7 (2012), 2047--2055. https://doi.org/10.1111/j.1467--8659.2012.03197.x
[36]
D.G. Lowe. 1999. Object Recognition from Local Scale-invariant Features. In Proceedings of the Seventh IEEE International Conference on Computer Vision, Vol. 2. 1150--1157 vol.2. https://doi.org/10.1109/ICCV.1999.790410
[37]
Davide Maltoni, Dario Maio, Anil K. Jain, and Jianjiang Feng. 2022. Handbook of Fingerprint Recognition, 3rd Edition. Springer.
[38]
Anthony Martinet, Gery Casiez, and Laurent Grisoni. 2012. Integrality and Separability of Multitouch Interaction Techniques in 3D Manipulation Tasks. IEEE Transactions on Visualization and Computer Graphics 18, 3 (2012), 369--380. https://doi.org/10.1109/TVCG.2011.129
[39]
Sven Mayer, Huy Viet Le, and Niels Henze. 2017. Estimating the Finger Orientation on Capacitive Touchscreens Using Convolutional Neural Networks. In Proceedings of the 2017 ACM International Conference on Interactive Surfaces and Spaces (Brighton, United Kingdom) (ISS '17). Association for Computing Machinery, New York, NY, USA, 220--229. https://doi.org/10.1145/3132272.3134130
[40]
Marius Muja and David G Lowe. 2009. Fast Approximate Nearest Neighbors with Automatic Algorithm Configuration. In Proceedings of International Conference on Computer Vision Theory and Application. 331--340.
[41]
Yuriko Nakai, Shinya Kudo, Ryuta Okazaki, Hiroyuki Kajimoto, and Hidenori Kuribayashi. 2014. Detection of Tangential Force for a Touch Panel Using Shear Deformation of the Gel. In CHI '14 Extended Abstracts on Human Factors in Computing Systems (Toronto, Ontario, Canada) (CHI EA '14). Association for Computing Machinery, New York, NY, USA, 2353--2358. https://doi.org/10.1145/2559206.2581149
[42]
Ian Oakley, Carina Lindahl, Khanh Le, DoYoung Lee, and MD. Rasel Islam. 2016. The Flat Finger: Exploring Area Touches on Smartwatches. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (San Jose, California, USA) (CHI '16). Association for Computing Machinery, New York, NY, USA, 4238--4249. https://doi.org/10.1145/2858036.2858179
[43]
Vishal M. Patel, Rama Chellappa, Deepak Chandra, and Brandon Barbello. 2016. Continuous User Authentication on Mobile Devices: Recent progress and remaining challenges. IEEE Signal Processing Magazine 33, 4 (2016), 49--61. https://doi.org/10.1109/MSP.2016.2555335
[44]
Chang Peng, Mengyue Chen, and Xiaoning Jiang. 2021. Under-Display Ultrasonic Fingerprint Recognition With Finger Vessel Imaging. IEEE Sensors Journal 21, 6 (2021), 7412--7419. https://doi.org/10.1109/JSEN.2021.3051975
[45]
Salil. Prabhakar, Sharath. Pankanti, and Anil K. Jain. 2003. Biometric recognition: security and privacy concerns. IEEE Security & Privacy 1, 2 (2003), 33--42. https://doi.org/10.1109/MSECP.2003.1193209
[46]
Joseph D. Rutledge and Ted Selker. 1990. Force-to-Motion Functions for Pointing. In Proceedings of the IFIP TC13 Third Interational Conference on Human-Computer Interaction (INTERACT '90). North-Holland Publishing Co., NLD, 701--706. https://dl.acm.org/doi/10.5555/647402.725310
[47]
Xuanbin Si, Jianjiang Feng, Jie Zhou, and Yuxuan Luo. 2015. Detection and Rectification of Distorted Fingerprints. IEEE Transactions on Pattern Analysis and Machine Intelligence 37, 3 (2015), 555--568. https://doi.org/10.1109/TPAMI.2014.2345403
[48]
Zdeňka Sitová, Jaroslav Šeděnka, Qing Yang, Ge Peng, Gang Zhou, Paolo Gasti, and Kiran S. Balagani. 2016. HMOG: New Behavioral Biometric Features for Continuous Authentication of Smartphone Users. IEEE Transactions on Information Forensics and Security 11, 5 (2016), 877--892. https://doi.org/10.1109/TIFS.2015.2506542
[49]
Feng Wang, Xiang Cao, Xiangshi Ren, and Pourang Irani. 2009. Detecting and Leveraging Finger Orientation for Interaction with Direct-Touch Surfaces. In Proceedings of the 22nd Annual ACM Symposium on User Interface Software and Technology (Victoria, BC, Canada) (UIST '09). Association for Computing Machinery, New York, NY, USA, 23--32. https://doi.org/10.1145/1622176.1622182
[50]
Frank Wilcoxon. 1992. Individual Comparisons by Ranking Methods. Springer New York, New York, NY, 196--202. https://doi.org/10.1007/978--1--4612--4380--9_16
[51]
Robert Xiao, Gierad Laput, and Chris Harrison. 2014. Expanding the Input Expressivity of Smartwatches with Mechanical Pan, Twist, Tilt and Click. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Toronto, Ontario, Canada) (CHI '14). Association for Computing Machinery, New York, NY, USA, 193--196. https://doi.org/10.1145/2556288.2557017
[52]
Robert Xiao, Julia Schwarz, and Chris Harrison. 2015. Estimating 3D Finger Angle on Commodity Touchscreens. In Proceedings of the 2015 International Conference on Interactive Tabletops & Surfaces (Madeira, Portugal) (ITS '15). Association for Computing Machinery, New York, NY, USA, 47--50. https://doi.org/10.1145/2817721.2817737
[53]
Ping-Hung Yin, Chih-Wen Lu, Jia-Shyang Wang, Keng-Li Chang, Fu-Kuo Lin, and Poki Chen. 2021. A 368 x 184 Optical Under-Display Fingerprint Sensor Comprising Hybrid Arrays of Global and Rolling Shutter Pixels With Shared Pixel-Level ADCs. IEEE Journal of Solid-State Circuits 56, 3 (2021), 763--777. https://doi.org/10.1109/JSSC.2020.3042894
[54]
Vadim Zaliva. 2012. 3D Finger Posture Detection and Gesture Recognition on Touch Surfaces. In 2012 12th International Conference on Control Automation Robotics & Vision (ICARCV). 359--364. https://doi.org/10.1109/ICARCV.2012.6485185
[55]
Shumin Zhai. 1996. Human Performance in Six Degree of Freedom Input Control. University of Toronto.

Cited By

View all
  • (2024)Palmrest+: Expanding Laptop Input Space with Shear Force on Palm-Resting AreaProceedings of the 37th Annual ACM Symposium on User Interface Software and Technology10.1145/3654777.3676371(1-14)Online publication date: 13-Oct-2024
  • (2024)Wearable-based behaviour interpolation for semi-supervised human activity recognitionInformation Sciences: an International Journal10.1016/j.ins.2024.120393665:COnline publication date: 2-Jul-2024
  • (2024)Deep learning based multimodal biomedical data fusion: An overview and comparative reviewInformation Fusion10.1016/j.inffus.2024.102536112(102536)Online publication date: Dec-2024
  • Show More Cited By

Index Terms

  1. PrintShear: Shear Input Based on Fingerprint Deformation

    Recommendations

    Comments

    Information & Contributors

    Information

    Published In

    cover image Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies
    Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies  Volume 7, Issue 2
    June 2023
    969 pages
    EISSN:2474-9567
    DOI:10.1145/3604631
    Issue’s Table of Contents
    This work is licensed under a Creative Commons Attribution International 4.0 License.

    Publisher

    Association for Computing Machinery

    New York, NY, United States

    Publication History

    Published: 12 June 2023
    Published in IMWUT Volume 7, Issue 2

    Check for updates

    Author Tags

    1. 3d object manipulation
    2. shear input
    3. touch input

    Qualifiers

    • Research-article
    • Research
    • Refereed

    Contributors

    Other Metrics

    Bibliometrics & Citations

    Bibliometrics

    Article Metrics

    • Downloads (Last 12 months)207
    • Downloads (Last 6 weeks)19
    Reflects downloads up to 15 Oct 2024

    Other Metrics

    Citations

    Cited By

    View all
    • (2024)Palmrest+: Expanding Laptop Input Space with Shear Force on Palm-Resting AreaProceedings of the 37th Annual ACM Symposium on User Interface Software and Technology10.1145/3654777.3676371(1-14)Online publication date: 13-Oct-2024
    • (2024)Wearable-based behaviour interpolation for semi-supervised human activity recognitionInformation Sciences: an International Journal10.1016/j.ins.2024.120393665:COnline publication date: 2-Jul-2024
    • (2024)Deep learning based multimodal biomedical data fusion: An overview and comparative reviewInformation Fusion10.1016/j.inffus.2024.102536112(102536)Online publication date: Dec-2024
    • (2024)Internet of Things for Emotion Care: Advances, Applications, and ChallengesCognitive Computation10.1007/s12559-024-10327-8Online publication date: 7-Aug-2024

    View Options

    View options

    PDF

    View or Download as a PDF file.

    PDF

    eReader

    View online with eReader.

    eReader

    Get Access

    Login options

    Full Access

    Media

    Figures

    Other

    Tables

    Share

    Share

    Share this Publication link

    Share on social media