Location via proxy:   [ UP ]  
[Report a bug]   [Manage cookies]                
skip to main content
10.1145/3290605.3300373acmconferencesArticle/Chapter ViewAbstractPublication PageschiConference Proceedingsconference-collections
research-article

Multilayer Haptic Feedback for Pen-Based Tablet Interaction

Published: 02 May 2019 Publication History

Abstract

We present a novel, multilayer interaction approach that enables state transitions between spatially above-screen and 2D on-screen feedback layers. This approach supports the exploration of haptic features that are hard to simulate using rigid 2D screens. We accomplish this by adding a haptic layer above the screen that can be actuated and interacted with (pressed on) while the user interacts with on-screen content using pen input. The haptic layer provides variable firmness and contour feedback, while its membrane functionality affords additional tactile cues like texture feedback. Through two user studies, we look at how users can use the layer in haptic exploration tasks, showing that users can discriminate well between different firmness levels, and can perceive object contour characteristics. Demonstrated also through an art application, the results show the potential of multilayer feedback to extend on-screen feedback with additional widget, tool and surface properties, and for user guidance.

Supplementary Material

MP4 File (pn3932.mp4)
Supplemental video

References

[1]
Akamatsu, M. and Sato, S. 1994. A multi-modal mouse with tactile and force feedback. International Journal of Human-Computer Studies. 40, 3 (Mar. 1994), 443--453.
[2]
Annett, M. et al. 2011. Medusa: a proximity-aware multi-touch tabletop. Proceedings of the 24th annual ACM symposium on User interface software and technology - UIST '11 (New York, New York, USA, 2011), 337.
[3]
Batteau, L. et al. 2004. A Study on the Perception of Haptics in Surgical Simulation. Springer Lecture Notes in Computer Science. 3078, (2004), 185--192.
[4]
Baxter, B. et al. 2001. DAB: interactive haptic painting with 3D virtual brushes. Proceedings of the 28th annual conference on Computer graphics and interactive techniques - SIGGRAPH '01 (New York, New York, USA, Aug. 2001), 461--468.
[5]
Benko, H. et al. 2016. NormalTouch and TextureTouch:?: High-fidelity 3D Haptic Shape Rendering on Handheld Virtual Reality Controllers. Proceedings of the 29th Annual Symposium on User Interface Software and Technology - UIST '16 (New York, New York, USA, 2016), 717--728.
[6]
Brewster, S. et al. 2007. Tactile feedback for mobile interactions. Proceedings of the SIGCHI conference on Human factors in computing systems - CHI '07 (New York, New York, USA, Apr. 2007), 159.
[7]
Brumby, D.P. and Zhuang, S. 2015. Visual Grouping in Menu Interfaces. Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems - CHI '15 (New York, New York, USA, 2015), 4203--4206.
[8]
Bullinger, H. et al. 1997. Controls. Handbook of Human Factors and Ergonomics. G. Salvendy, ed. John Wiley & Sons.
[9]
Carrascal, J. and Vertegaal, R. 2017. Effects of Tactile Feedback on the Perception of Virtual Shapes on Non-Planar Display Objects. ACM Conference on Human Factors in Computing Systems (CHI '2017) (2017), 4417--4423.
[10]
Carter, T. et al. 2013. UltraHaptics: multi-point mid-air haptic feedback for touch surfaces. Proceedings of the 26th annual ACM symposium on User interface software and technology - UIST '13 (New York, New York, USA, 2013), 505--514.
[11]
Chen, D. et al. 2015. A novel miniature multi-mode haptic pen for image interaction on mobile terminal. 2015 IEEE International Symposium on Haptic, Audio and Visual Environments and Games (HAVE) (Oct. 2015), 1--6.
[12]
Cheung, V. et al. 2012. Revisiting hovering: interaction guides for interactive surfaces. Proceedings of the 2012 ACM international conference on Interactive tabletops and surfaces - ITS '12 (New York, New York, USA, 2012), 355.
[13]
Cho, Y. et al. 2016. RealPen: Providing Realism in Handwriting Tasks on Touch Surfaces using Auditory-Tactile Feedback. Proceedings of the 29th Annual Symposium on User Interface Software and Technology - UIST '16 (New York, New York, USA, 2016), 195--205.
[14]
Chu, G. et al. 2009. Haptic conviction widgets. Proceedings of Graphics Interface 2009 (2009), 207--210.
[15]
Cockburn, A. and Brewster, S. 2005. Multimodal feedback for the acquisition of small targets. Ergonomics. 48, 9 (Jul. 2005), 1129--50.
[16]
Cornsweet, T. 1962. The staircrase-method in psychophysics. The American journal of psychology. 75, (Sep. 1962), 485--91.
[17]
Culbertson, H. and Kuchenbecker, K.J. 2017. Ungrounded Haptic Augmented Reality System for Displaying Roughness and Friction. IEEE/ASME Transactions on Mechatronics. 22, 4 (Aug. 2017), 1839-- 1849.
[18]
Fellion, N. et al. 2017. FlexStylus:?: Leveraging Bend Input for Pen Interaction. Proceedings of the 30th Annual ACM Symposium on User Interface Software and Technology - UIST '17 (New York, New York, USA, 2017), 375--385.
[19]
Fleureau, J. et al. 2016. Texture Rendering on a Tactile Surface Using Extended Elastic Images and Example-Based Audio Cues. Springer, Cham. 350--359.
[20]
Follmer, S. et al. 2012. Jamming user interfaces: programmable particle stiffness and sensing for malleable and shape-changing devices. Proceedings of the 25th annual ACM symposium on User interface software and technology - UIST '12 (New York, New York, USA, Oct. 2012), 519.
[21]
Fukumoto, M. and Sugimura, T. 2001. Active click: tactile feedback for touch pannels. CHI '01 extended abstracts on Human factors in computing systems - CHI '01 (New York, New York, USA, Mar. 2001), 121.
[22]
Gallant, D. et al. 2008. Towards more paper-like input: flexible input devices for foldable interaction styles. Proceedings of the 21st annual ACM symposium on User interface software and technology - UIST '08 (New York, New York, USA, Oct. 2008), 283.
[23]
Gescheider, G. et al. 1969. Effects of simultaneous auditory stimulation on the detection of tactile stimuli. Journal of Experimental Psychology. 81, 1 (1969), 120--125.
[24]
Grønmo, O. et al. 2007. Learning automata-based solutions to the nonlinear fractional knapsack problem with applications to optimal resource allocation. IEEE Transactions on Systems, Man and Cybernetics. 37, 1 (2007), 166--175.
[25]
Gupta, S. et al. 2010. Squeeze- Block: using virtual springs in mobile devices for eyes-free interaction. Proceedings of the 23nd annual ACM symposium on User interface software and technology - UIST '10 (New York, New York, USA, Oct. 2010), 101.
[26]
Hachet, M. and Kulik, A. 2008. Elastic Control for Navigation Tasks on Pen-based Handheld Computers. 2008 IEEE Symposium on 3D User Interfaces (Mar. 2008), 91--96.
[27]
Hafez, M. 2007. Tactile interfaces: technologies, applications and challenges. The Visual Computer. 23, 4 (Mar. 2007), 267--272.
[28]
Han, J. et al. 2014. Trampoline: a double-sided elastic touch device for creating reliefs. Proceedings of the 27th annual ACM symposium on User interface software and technology - UIST '14 (New York, New York, USA, 2014), 383--388.
[29]
Harrison, C. and Hudson, S.E. 2009. Texture displays: a passive approach to tactile presentation. Proceedings of the 27th international conference on Human factors in computing systems - CHI 09 (New York, New York, USA, Apr. 2009), 2261.
[30]
Hoggan, E. et al. 2009. Audio or tactile feedback: which modality when? Proceedings of the 27th international conference on Human factors in computing systems - CHI 09 (New York, New York, USA, Apr. 2009), 2253.
[31]
Hoggan, E. et al. 2008. Crossmodal congruence: the look, feel and sound of touchscreen widgets. Proceedings of the 10th international conference on Multimodal interfaces - IMCI '08 (New York, New York, USA, Oct. 2008), 157.
[32]
Holman, D. and Vertegaal, R. 2008. Organic user interfaces: designing computers in any way, shape, or form. Communications of the ACM. 51, 6 (Jun. 2008), 48.
[33]
Jansen, Y. et al. 2010. MudPad:?: tactile feedback and haptic texture overlay for touch surfaces. ACM International Conference on Interactive Tabletops and Surfaces - ITS '10 (New York, New York, USA, 2010), 11.
[34]
Jones, L.A. and Tan, H.Z. 2013. Application of Psychophysical Techniques to Haptic Research. IEEE Transactions on Haptics. 6, 3 (Jul. 2013), 268-- 284.
[35]
Kaczmarek, K. et al. 1991. Electrotactile and vibrotactile displays for sensory substitution systems. IEEE transactions on bio-medical engineering. 38, 1 (Jan. 1991), 1--16.
[36]
Kajastila, R. and Lokki, T. 2009. A gesture-based and eyes-free control method for mobile devices. Proceedings of the 27th international conference extended abstracts on Human factors in computing systems - CHI EA '09 (New York, New York, USA, Apr. 2009), 3559.
[37]
Kamuro, S. et al. 2009. Ungrounded kinesthetic pen for haptic interaction with virtual environments. RO-MAN 2009 - The 18th IEEE International Symposium on Robot and Human Interactive Communication (Sep. 2009), 436--441.
[38]
Kildal, J. 2010. 3D-press: haptic illusion of compliance when pressing on a rigid surface. International Conference on Multimodal Interfaces and the Workshop on Machine Learning for Multimodal Interaction on ICMI-MLMI '10 (New York, New York, USA, 2010), 1.
[39]
Kim, S.-C. et al. 2013. Tactile rendering of 3D features on touch surfaces. Proceedings of the 26th annual ACM symposium on User interface software and technology - UIST '13 (New York, New York, USA, Oct. 2013), 531--538.
[40]
Klatzky, R. et al. 2013. Haptic Perception of Material Properties and Implications for Applications. Proceedings of the IEEE. PP, 99 (2013), 1-- 12.
[41]
Klatzky, R. and Lederman, S. 2010. Multisensory Texture Perception. Multisensory Object Perception in the Primate Brain. Springer New York. 211--230.
[42]
Korman, M. et al. 2012. Effects of Order and Sensory Modality in Stiffness Perception. Presence: Teleoperators and Virtual Environments. 21, 3 (Aug. 2012), 295--304.
[43]
Kruijff, E. et al. 2006. Tactylus, a Pen-Input Device exploring Audiotactile Sensory Binding. Proceedings of the ACM Symposium on Virtual Reality Software & Technology 2006 (VRST 2006) (2006).
[44]
Kruijff, E. and Pander, A. 2005. Experiences of using Shockwaves for Haptic Sensations. Proceedings of 3D user interface workshop, IEEE Virtual Reality Conference (VR2005) (2005).
[45]
Kyung, K.-U. and Lee, J.-Y. 2008. wUbi-Pen: windows graphical user interface interacting with haptic feedback stylus. ACM SIGGRAPH 2008 new tech demos on - SIGGRAPH '08 (New York, New York, USA, Aug. 2008), 1--4.
[46]
Lederman, S. and Klatzky, R. 1993. Extracting object properties through haptic exploration. Acta Psychologica. 84, 1 (Oct. 1993), 29--40.
[47]
Lederman, S. and Klatzky, R. 1987. Hand movements: A window into haptic object recognition. Cognitive Psychology. 19, 3 (Jul. 1987), 342-- 368.
[48]
Lee, J. et al. 2004. Haptic pen: a tactile feedback stylus for touch screens. Proceedings of the 17th annual ACM symposium on User interface software and technology - UIST '04 (New York, New York, USA, Oct. 2004), 291.
[49]
Lee, S.-S. et al. 2010. How users manipulate deformable displays as input devices. Proceedings of the 28th international conference on Human factors in computing systems - CHI '10 (New York, New York, USA, Apr. 2010), 1647.
[50]
Lenay, C. et al. 2003. Sensory Substitution: Limits and Perspectives. Touching for Knowing. Y. Hatwell et al., eds. 275--292.
[51]
Leung, R. et al. 2007. Evaluation of haptically augmented touchscreen gui elements under cognitive load. Proceedings of the ninth international conference on Multimodal interfaces - ICMI '07 (New York, New York, USA, Nov. 2007), 374.
[52]
Levesque, V. et al. 2011. Frictional widgets: enhancing touch interfaces with programmable friction. Proceedings of the 2011 annual conference extended abstracts on Human factors in computing systems - CHI EA '11 (New York, New York, USA, May 2011), 1153.
[53]
Lin, L. et al. 2011. Application of Pen-Based Planar Haptic Interface in Physics Education. 2011 12th International Conference on ComputerAided Design and Computer Graphics (Sep. 2011), 375--378.
[54]
Lubos, P. et al. 2015. HoverSpace. Springer International Publishing. 259-- 277.
[55]
Marquardt, A. et al. 2018. Audio-tactile proximity feedback for enhancing 3D manipulation. Proceedings of the 24th ACM Symposium on Virtual Reality Software and Technology - VRST '18 (New York, New York, USA, 2018), 1--10.
[56]
Marquardt, A. et al. 2018. Tactile hand motion and pose guidance for 3D interaction. Proceedings of the 24th ACM Symposium on Virtual Reality Software and Technology - VRST '18 (New York, New York, USA, 2018), 1--10.
[57]
Marquardt, N. et al. 2009. The Haptic Tabletop Puck: tactile feedback for interactive tabletops. Proceedings of the ACM International Conference on Interactive Tabletops and Surfaces - ITS '09 (New York, New York, USA, Nov. 2009), 85.
[58]
Mignonneau, L. and Sommerer, C. 2005. Nano-Scape: Experiencing Aspects of Nanotechnology through a Magnetic Force-Feedback Interface. Proceedings of the ACM SIGCHI International Conference on Advances in Computer Entertainment Technology (2005).
[59]
Mullenbach, J. et al. 2013. Surface haptic interactions with a TPad tablet. Proceedings of the adjunct publication of the 26th annual ACM symposium on User interface software and technology - UIST '13 Adjunct (New York, New York, USA, 2013), 7--8.
[60]
Müller, M. et al. 2014. FlexiWall: Exploring Layered Data with Elastic Displays. Proceedings of the Ninth ACM International Conference on Interactive Tabletops and Surfaces - ITS '14 (New York, New York, USA, 2014), 439--442.
[61]
Peschke, J. et al. 2012. DepthTouch. Proceedings of the International Working Conference on Advanced Visual Interfaces - AVI '12 (New York, New York, USA, May 2012), 770.
[62]
Poupyrev, I. et al. 2004. Haptic feedback for pen computing: directions and strategies. Extended abstracts of the 2004 conference on Human factors and computing systems - CHI '04 (New York, New York, USA, Apr. 2004), 1309.
[63]
Poupyrev, I. and Maruyama, S. 2002. Drawing with feeling: designing tactile display for pen. ACM SIGGRAPH 2002 conference abstracts and applications on - SIGGRAPH '02 (New York, New York, USA, Jul. 2002), 173.
[64]
Pyryeskin, D. et al. 2011. Extending interactions into hoverspace using reflected light. Proceedings of the ACM International Conference on Interactive Tabletops and Surfaces - ITS '11 (New York, New York, USA, 2011), 262--263.
[65]
Rekik, Y. et al. 2017. Localized Haptic Texture: A Rendering Technique based on Taxels for High Density Tactile Feedback. ACM Conference on Human Factors in Computing Systems (CHI '2017) (2017), 5006--5015.
[66]
Ro, T. et al. 2009. Sound enhances touch perception. Experimental Brain Research. 195, 1 (May 2009), 135--143.
[67]
Sahoo, D.R. et al. TableHop: An Actuated Fabric Display Using Transparent Electrodes.
[68]
Schwesig, C. et al. 2004. Gummi: a bendable computer. Proceedings of the 2004 conference on Human factors in computing systems - CHI '04 (New York, New York, USA, Apr. 2004), 263--270.
[69]
Sebastian, F. et al. 2017. Soft Robotic Haptic Interface with Variable Stiffness for Rehabilitation of Neurologically Impaired Hand Function. Frontiers in Robotics and AI. 4, (Dec. 2017), 69.
[70]
Sodhi, R. et al. 2013. AIREAL: tactile gaming experiences in free air. ACM SIGGRAPH 2013 Emerging Technologies on - SIGGRAPH '13 (New York, New York, USA, 2013), 1--1.
[71]
Steimle, J. et al. 2013. Flexpad: a highly flexible handheld display. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems - CHI '13 (New York, New York, USA, Apr. 2013), 237.
[72]
Strohmeier, P. et al. 2016. ReFlex: A Flexible Smartphone with Active Haptic Feedback for Bend Input. Proceedings of the TEI '16: Tenth International Conference on Tangible, Embedded, and Embodied Interaction - TEI '16 (New York, New York, USA, 2016), 185--192.
[73]
Strohmeier, P. and Hornbæk, K. 2017. Generating Haptic Textures with a Vibrotactile Actuator. Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems - CHI '17 (New York, New York, USA, 2017), 4994--5005.
[74]
Subramanian, S. et al. 2006. Multi-layer interaction for digital tables. Proceedings of the 19th annual ACM symposium on User interface software and technology - UIST '06 (New York, New York, USA, 2006), 269.
[75]
Sulaiman, S. et al. 2009. Haptic experience and the design of drawing interfaces. Interacting with Computers. 22, 3 (2009), 193--2005.
[76]
Takahashi, C. and Watt, S.J. 2017. Optimal visual-haptic integration with articulated tools. Experimental brain research. 235, 5 (2017), 1361--1373.
[77]
Tikka, V. and Laitinen, P. 2006. Designing Haptic Feedback for Touch Display: Experimental Study of Perceived Intensity and Integration of Haptic and Audio. Lecture Notes in Computer Science. 4129, (2006), 36-- 44.
[78]
Tsai, H.-R. and Rekimoto, J. 2018. ElasticVR: Providing Multi-level Active and Passive Force Feedback in Virtual Reality Using Elasticity. Extended Abstracts of the 2018 CHI Conference on Human Factors in Computing Systems - CHI '18 (New York, New York, USA, 2018), 1--4.
[79]
Visell, Y. et al. 2014. A Device and Method for Multimodal Haptic Rendering of Volumetric Stiffness. Springer Berlin Heidelberg. 478--486.
[80]
Wang, Q. et al. 2016. EV-Pen: Leveraging Electrovibration Haptic Feedback in Pen Interaction. ACM International Conference on Interactive Surfaces and Spaces (2016), 57--66.
[81]
Weiss, M. et al. 2011. FingerFlux: near-surface haptic feedback on tabletops. Proceedings of the 24th annual ACM symposium on User interface software and technology - UIST '11 (New York, New York, USA, Oct. 2011), 615.

Cited By

View all
  • (2023)FeelPen: A Haptic Stylus Displaying Multimodal Texture Feels on TouchscreensIEEE/ASME Transactions on Mechatronics10.1109/TMECH.2023.326478728:5(2930-2940)Online publication date: Oct-2023
  • (2022)Improving Image Accessibility by Combining Haptic and Auditory FeedbackProceedings of the 24th International ACM SIGACCESS Conference on Computers and Accessibility10.1145/3517428.3550362(1-6)Online publication date: 23-Oct-2022
  • (2022)Hovering and Contact Representation of Laser Contour-Based Hand with Swinging Tablet PC for Distant CommunicationProceedings of the Sixteenth International Conference on Tangible, Embedded, and Embodied Interaction10.1145/3490149.3505566(1-7)Online publication date: 13-Feb-2022
  • Show More Cited By

Index Terms

  1. Multilayer Haptic Feedback for Pen-Based Tablet Interaction

    Recommendations

    Comments

    Information & Contributors

    Information

    Published In

    cover image ACM Conferences
    CHI '19: Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems
    May 2019
    9077 pages
    ISBN:9781450359702
    DOI:10.1145/3290605
    Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

    Sponsors

    Publisher

    Association for Computing Machinery

    New York, NY, United States

    Publication History

    Published: 02 May 2019

    Permissions

    Request permissions for this article.

    Check for updates

    Author Tags

    1. haptic feedback
    2. multilayer interaction
    3. pen interaction

    Qualifiers

    • Research-article

    Conference

    CHI '19
    Sponsor:

    Acceptance Rates

    CHI '19 Paper Acceptance Rate 703 of 2,958 submissions, 24%;
    Overall Acceptance Rate 6,199 of 26,314 submissions, 24%

    Contributors

    Other Metrics

    Bibliometrics & Citations

    Bibliometrics

    Article Metrics

    • Downloads (Last 12 months)50
    • Downloads (Last 6 weeks)8
    Reflects downloads up to 16 Oct 2024

    Other Metrics

    Citations

    Cited By

    View all
    • (2023)FeelPen: A Haptic Stylus Displaying Multimodal Texture Feels on TouchscreensIEEE/ASME Transactions on Mechatronics10.1109/TMECH.2023.326478728:5(2930-2940)Online publication date: Oct-2023
    • (2022)Improving Image Accessibility by Combining Haptic and Auditory FeedbackProceedings of the 24th International ACM SIGACCESS Conference on Computers and Accessibility10.1145/3517428.3550362(1-6)Online publication date: 23-Oct-2022
    • (2022)Hovering and Contact Representation of Laser Contour-Based Hand with Swinging Tablet PC for Distant CommunicationProceedings of the Sixteenth International Conference on Tangible, Embedded, and Embodied Interaction10.1145/3490149.3505566(1-7)Online publication date: 13-Feb-2022
    • (2022)Designing a Game for Haptic Interfaces to Uncover Gestural Pattern in ChildrenProceedings of the International Conference on Ubiquitous Computing & Ambient Intelligence (UCAmI 2022)10.1007/978-3-031-21333-5_93(944-948)Online publication date: 21-Nov-2022
    • (2021)PenShaft: Enabling Pen Shaft Detection and Interaction for Touchscreens12th Augmented Human International Conference10.1145/3460881.3460934(1-9)Online publication date: 27-May-2021
    • (2021)IllumiWand: Improving 3D Interaction with Monoscopic Displays Through a Projected Physical 3D Pointer2021 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops (VRW)10.1109/VRW52623.2021.00046(210-213)Online publication date: Mar-2021
    • (2021)Vibrotactile Feedback Rendering of Patterned Textures Using a Waveform Segment Table MethodIEEE Transactions on Haptics10.1109/TOH.2021.308430414:4(849-861)Online publication date: 1-Oct-2021

    View Options

    Get Access

    Login options

    View options

    PDF

    View or Download as a PDF file.

    PDF

    eReader

    View online with eReader.

    eReader

    HTML Format

    View this article in HTML Format.

    HTML Format

    Media

    Figures

    Other

    Tables

    Share

    Share

    Share this Publication link

    Share on social media