Location via proxy:   [ UP ]  
[Report a bug]   [Manage cookies]                
skip to main content
10.1145/3472749.3474797acmconferencesArticle/Chapter ViewAbstractPublication PagesuistConference Proceedingsconference-collections
research-article
Open access

Weirding Haptics: In-Situ Prototyping of Vibrotactile Feedback in Virtual Reality through Vocalization

Published: 12 October 2021 Publication History

Abstract

Effective haptic feedback in virtual reality (VR) is an essential element for creating convincing immersive experiences. To design such feedback, state-of-the-art VR setups provide APIs for programmatically generating controller vibration patterns. While tools for designing vibrotactile feedback keep evolving, they often require expert knowledge and rarely support direct manipulation methods for mapping feedback to user interactions within the VR environment. To address these challenges, we contribute a novel concept called Weirding Haptics, that supports fast-prototyping by leveraging the user’s voice to design such feedback while manipulating virtual objects in-situ. Through a pilot study (N = 9) focusing on how tactile experiences are vocalized during object manipulation, we identify spatio-temporal mappings and supporting features needed to produce intended vocalizations. To study our concept, we built a VR design tool informed by the results of the pilot study. This tool enables users to design tactile experiences using their voice while manipulating objects, provides a set of modifiers for fine-tuning the created experiences in VR, and allows to rapidly compare various experiences by feeling them. Results from a validation study (N = 8) show that novice hapticians can vocalize experiences and refine their designs with the fine-tuning modifiers to match their intentions. We conclude our work by discussing uncovered design implications for direct manipulation and vocalization of vibrotactile feedback in immersive virtual environments.

References

[1]
Kimi Akita and Mark Dingemanse. 2019. Ideophones (Mimetics, Expressives). https://doi.org/10.1093/acrefore/9780199384655.013.477
[2]
Farah Arab, Sabrina Panëels, Margarita Anastassova, Stéphanie Coeugnet, Fanny Le Morellec, Aurélie Dommes, and Aline Chevalier. 2015. Haptic patterns and older adults: To repeat or not to repeat?. In 2015 IEEE World Haptics Conference (WHC). IEEE, Evanston, IL, USA, 248–253. https://doi.org/10.1109/WHC.2015.7177721
[3]
Sliman Bensmaïa and Mark Hollins. 2005. Pacinian representations of fine surface texture. Perception & Psychophysics 67, 5 (01 Jul 2005), 842–854. https://doi.org/10.3758/BF03193537
[4]
Virginia Braun and Victoria Clarke. 2006. Using thematic analysis in psychology. Qualitative Research in Psychology 3, 2 (2006), 77–101. https://doi.org/10.1191/1478088706qp063oa arXiv:https://www.tandfonline.com/doi/pdf/10.1191/1478088706qp063oa
[5]
Lucie Brunet, Christine Megard, Sabrina Paneels, Gwenael Changeon, Jose Lozada, Marie P. Daniel, and Françoise Darses. 2013. “Invitation to the voyage”: The design of tactile metaphors to fulfill occasional travelers’ needs in transportation networks. In 2013 World Haptics Conference (WHC). 259–264. https://doi.org/10.1109/WHC.2013.6548418
[6]
Youngjun Cho, S Kim, M Joung, and J Lee. 2014. Haptic Cushion: Automatic Generation of Vibro-tactile Feedback Based on Audio Signal for Immersive Interaction with Multimedia. Messe Bremen.
[7]
Alain de Cheveigné and Hideki Kawahara. 2002. YIN, a fundamental frequency estimator for speech and music. The Journal of the Acoustical Society of America 111, 4 (April 2002), 1917–1930. https://doi.org/10.1121/1.1458024
[8]
Donald Degraen, Anna Reindl, Akhmajon Makhsadov, André Zenner, and Antonio Krüger. 2020. Envisioning Haptic Design for Immersive Virtual Environments. In Companion Publication of the 2020 ACM Designing Interactive Systems Conference (Eindhoven, Netherlands) (DIS’ 20 Companion). Association for Computing Machinery, New York, NY, USA, 287–291. https://doi.org/10.1145/3393914.3395870
[9]
Donald Degraen, André Zenner, and Antonio Krüger. 2019. Enhancing Texture Perception in Virtual Reality Using 3D-Printed Hair Structures. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (Glasgow, Scotland Uk) (CHI ’19). Association for Computing Machinery, New York, NY, USA, 1–12. https://doi.org/10.1145/3290605.3300479
[10]
Yves Guiard. 1987. Asymmetric Division of Labor in Human Skilled Bimanual Action. Journal of Motor Behavior 19, 4 (1987), 486–517. https://doi.org/10.1080/00222895.1987.10735426
[11]
Sean A Guynes. 2014. Four-Color Sound: A Peircean Semiotics of Comic Book Onomatopoeia. Public Journal of Semiotics 6, 1 (Dec. 2014), 58–72. https://doi.org/10.37693/pjos.2014.6.11916
[12]
Seongkook Heo, Jaeyeon Lee, and Daniel Wigdor. 2019. PseudoBend: Producing Haptic Illusions of Stretching, Bending, and Twisting Using Grain Vibrations. In Proceedings of the 32nd Annual ACM Symposium on User Interface Software and Technology (New Orleans, LA, USA) (UIST ’19). Association for Computing Machinery, New York, NY, USA, 803–813. https://doi.org/10.1145/3332165.3347941
[13]
Hiroki Hojo, Nozomi Nomachi, Yutaro Tomoto, Tsuyoshi Nakamura, Masayoshi Kanoh, and Koji Yamada. 2014. Fundamental Study for Verbalization of Embodied Expertise based on Pattern Recognition. In 2014 IIAI 3rd International Conference on Advanced Applied Informatics. 925–929. https://doi.org/10.1109/IIAI-AAI.2014.182
[14]
Brent Edward Insko. 2001. Passive Haptics Significantly Enhances Virtual Environments. Ph.D. Dissertation. University of North Carolina at Chapel Hill, USA. Advisor(s) Frederick P. Brooks Jr. http://www.cs.unc.edu/techreports/01-017.pdf
[15]
Interhaptics. 2021. Haptic Composer. https://www.interhaptics.com/haptic-composer
[16]
Roland S Johansson and J Randall Flanagan. 2009. Coding and use of tactile signals from the fingertips in object manipulation tasks. Nature Reviews Neuroscience 10, 5 (2009), 345.
[17]
Johan Kildal. 2010. 3D-Press: Haptic Illusion of Compliance When Pressing on a Rigid Surface. In International Conference on Multimodal Interfaces and the Workshop on Machine Learning for Multimodal Interaction (Beijing, China) (ICMI-MLMI ’10). Association for Computing Machinery, New York, NY, USA, Article 21, 8 pages. https://doi.org/10.1145/1891903.1891931
[18]
Johan Kildal. 2012. Kooboh: Variable Tangible Properties in a Handheld Haptic-Illusion Box. In Proceedings of the 2012 International Conference on Haptics: Perception, Devices, Mobility, and Communication - Volume Part II (Tampere, Finland) (EuroHaptics’12). Springer-Verlag, Berlin, Heidelberg, 191–194. https://doi.org/10.1007/978-3-642-31404-9_33
[19]
Erin Kim and Oliver Schneider. 2020. Defining Haptic Experience: Foundations for Understanding, Communicating, and Evaluating HX. Association for Computing Machinery, New York, NY, USA, 1–13. https://doi.org/10.1145/3313831.3376280
[20]
Robert Kovacs, Eyal Ofek, Mar Gonzalez Franco, Alexa Fay Siu, Sebastian Marwecki, Christian Holz, and Mike Sinclair. 2020. Haptic PIVOT: On-Demand Handhelds in VR. In Proceedings of the 33rd Annual ACM Symposium on User Interface Software and Technology (Virtual Event, USA) (UIST ’20). Association for Computing Machinery, New York, NY, USA, 1046–1059. https://doi.org/10.1145/3379337.3415854
[21]
Susan J Lederman and Roberta L Klatzky. 1987. Hand movements: A window into haptic object recognition. Cognitive Psychology 19, 3 (1987), 342–368. https://doi.org/10.1016/0010-0285(87)90008-9
[22]
Jaebong Lee and Seungmoon Choi. 2013. Real-time perception-level translation from audio signals to vibrotactile effects. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM, Paris France, 2567–2576. https://doi.org/10.1145/2470654.2481354
[23]
Jaeyeon Lee, Mike Sinclair, Mar Gonzalez-Franco, Eyal Ofek, and Christian Holz. 2019. TORC: A Virtual Reality Controller for In-Hand High-Dexterity Finger Interaction. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (Glasgow, Scotland Uk) (CHI ’19). Association for Computing Machinery, New York, NY, USA, 1–13. https://doi.org/10.1145/3290605.3300301
[24]
Gwilym Lockwood and Mark Dingemanse. 2015. Iconicity in the lab: a review of behavioral, developmental, and neuroimaging research into sound-symbolism. Frontiers in Psychology 6 (Aug. 2015). https://doi.org/10.3389/fpsyg.2015.01246
[25]
Lofelt. 2021. Lofelt Studio. https://lofelt.com/design
[26]
Maud Marchal, Gabriel Cirio, Yon Visell, Federico Fontana, Stefania Serafin, Jeremy Cooperstock, and Anatole Lécuyer. 2013. Multimodal Rendering of Walking Over Virtual Grounds. Springer New York, New York, NY, 263–295. https://doi.org/10.1007/978-1-4419-8432-6_12
[27]
David Marino, Paul Bucci, Oliver S. Schneider, and Karon E. MacLean. 2017. Voodle: Vocal Doodling to Sketch Affective Robot Motion. In Proceedings of the 2017 Conference on Designing Interactive Systems (Edinburgh, United Kingdom) (DIS ’17). Association for Computing Machinery, New York, NY, USA, 753–765. https://doi.org/10.1145/3064663.3064668
[28]
Michael Meehan, Brent Insko, Mary Whitton, and Frederick P. Brooks. 2002. Physiological Measures of Presence in Stressful Virtual Environments. ACM Trans. Graph. 21, 3 (July 2002), 645–652. https://doi.org/10.1145/566654.566630
[29]
Marianna Obrist, Sue Ann Seah, and Sriram Subramanian. 2013. Talking about Tactile Experiences. Association for Computing Machinery, New York, NY, USA, 1659–1668. https://doi.org/10.1145/2470654.2466220
[30]
Oculus Developers. [n.d.]. Haptic Feedback | Oculus Developers. https://developer.oculus.com/documentation/native/pc/dg-input-touch-haptic/
[31]
Breakfast Quay. 2021. Rubber Band Library. http://breakfastquay.com/rubberband/
[32]
Davide Rocchesso, Guillaume Lemaitre, Patrick Susini, Sten Ternström, and Patrick Boussard. 2015. Sketching Sound with Voice and Gesture. Interactions 22, 1 (Jan. 2015), 38–41. https://doi.org/10.1145/2685501
[33]
Joseph M. Romano and Katherine J. Kuchenbecker. 2012. Creating Realistic Virtual Textures from Contact Acceleration Data. IEEE Transactions on Haptics 5, 2 (April 2012), 109–119. https://doi.org/10.1109/TOH.2011.38
[34]
Eva-Lotta Sallnäs, Kirsten Rassmus-Gröhn, and Calle Sjöström. 2000. Supporting Presence in Collaborative Environments by Haptic Force Feedback. ACM Trans. Comput.-Hum. Interact. 7, 4 (Dec. 2000), 461–476. https://doi.org/10.1145/365058.365086
[35]
Ryoko Sasamoto. 2019. Onomatopoeia and Relevance. Springer International Publishing. https://doi.org/10.1007/978-3-030-26318-8
[36]
Oliver Schneider, Karon MacLean, Colin Swindells, and Kellogg Booth. 2017. Haptic experience design: What hapticians do and where they need help. International Journal of Human-Computer Studies 107 (2017), 5–21. https://doi.org/10.1016/j.ijhcs.2017.04.004 Multisensory Human-Computer Interaction.
[37]
Oliver S. Schneider, Ali Israr, and Karon E. MacLean. 2015. Tactile Animation by Direct Manipulation of Grid Displays. In Proceedings of the 28th Annual ACM Symposium on User Interface Software and Technology (Charlotte, NC, USA) (UIST ’15). Association for Computing Machinery, New York, NY, USA, 21–30. https://doi.org/10.1145/2807442.2807470
[38]
Oliver S. Schneider and Karon E. MacLean. 2014. Improvising design with a Haptic Instrument. In 2014 IEEE Haptics Symposium (HAPTICS). 327–332. https://doi.org/10.1109/HAPTICS.2014.6775476
[39]
Oliver S. Schneider and Karon E. MacLean. 2016. Studying design process and example use with Macaron, a web-based vibrotactile effect editor. In 2016 IEEE Haptics Symposium (HAPTICS). 52–58. https://doi.org/10.1109/HAPTICS.2016.7463155
[40]
Oliver S. Schneider, Hasti Seifi, Salma Kashani, Matthew Chun, and Karon E. MacLean. 2016. HapTurk: Crowdsourcing Affective Ratings of Vibrotactile Icons. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (San Jose, California, USA) (CHI ’16). Association for Computing Machinery, New York, NY, USA, 3248–3260. https://doi.org/10.1145/2858036.2858279
[41]
Hasti Seifi, Chamila Anthonypillai, and Karon E. MacLean. 2014. End-user customization of affective tactile messages: A qualitative examination of tool parameters. In 2014 IEEE Haptics Symposium (HAPTICS). 251–256. https://doi.org/10.1109/HAPTICS.2014.6775463
[42]
Hasti Seifi, Matthew Chun, Colin Gallacher, Oliver Schneider, and Karon E. MacLean. 2020. How Do Novice Hapticians Design? A Case Study in Creating Haptic Learning Environments. IEEE Transactions on Haptics 13, 4 (2020), 791–805. https://doi.org/10.1109/TOH.2020.2968903
[43]
Hasti Seifi and Karon E. MacLean. 2017. Exploiting haptic facets: Users’ sensemaking schemas as a path to design and personalization of experience. International Journal of Human-Computer Studies 107 (2017), 38–61. https://doi.org/10.1016/j.ijhcs.2017.04.003 Multisensory Human-Computer Interaction.
[44]
David M. Sidhu and Penny M. Pexman. 2017. Five mechanisms of sound symbolic association. Psychonomic Bulletin & Review 25, 5 (Aug. 2017), 1619–1643. https://doi.org/10.3758/s13423-017-1361-1
[45]
Mandayam A. Srinivasan and Cagatay Basdogan. 1997. Haptics in virtual environments: Taxonomy, research status, and challenges. Computers & Graphics 21, 4 (1997), 393–404. https://doi.org/10.1016/S0097-8493(97)00030-7
[46]
Paul Strohmeier, Sebastian Boring, and Kasper Hornbæk. 2018. From Pulse Trains to ”Coloring with Vibrations”: Motion Mappings for Mid-Air Haptic Textures. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems (Montreal QC, Canada) (CHI ’18). Association for Computing Machinery, New York, NY, USA, 1–13. https://doi.org/10.1145/3173574.3173639
[47]
Paul Strohmeier, Seref Güngör, Luis Herres, Dennis Gudea, Bruno Fruchard, and Jürgen Steimle. 2020. BARefoot: Generating Virtual Materials Using Motion Coupled Vibration in Shoes. In Proceedings of the 33rd Annual ACM Symposium on User Interface Software and Technology(Virtual Event, USA) (UIST ’20). Association for Computing Machinery, New York, NY, USA, 579–593. https://doi.org/10.1145/3379337.3415828
[48]
Paul Strohmeier and Kasper Hornbæk. 2017. Generating Haptic Textures with a Vibrotactile Actuator. Association for Computing Machinery, New York, NY, USA, 4994–5005. https://doi.org/10.1145/3025453.3025812
[49]
C. Swindells, E. Maksakov, K.E. MacLean, and V. Chung. [n.d.]. The Role of Prototyping Tools for Haptic Behavior Design. In 2006 14th Symposium on Haptic Interfaces for Virtual Environment and Teleoperator Systems. IEEE. https://doi.org/10.1109/haptic.2006.1627084
[50]
Colin Swindells, Seppo Pietarinen, and Arto Viitanen. 2014. Medium fidelity rapid prototyping of vibrotactile haptic, audio and video effects. In 2014 IEEE Haptics Symposium (HAPTICS). IEEE, Houston, TX, USA, 515–521. https://doi.org/10.1109/HAPTICS.2014.6775509
[51]
Valve Corporation. [n.d.]. SteamVR Unity Plugin. https://valvesoftware.github.io/steamvr_unity_plugin/index.html
[52]
Ronald T Verrillo. 1966. Vibrotactile sensitivity and the frequency response of the Pacinian corpuscle. Psychonomic Science 4, 1 (1966), 135–136.
[53]
Raquel Viciana-Abad, Arcadio Reyes Lecuona, and Matthieu Poyade. 2010. The Influence of Passive Haptic Feedback and Difference Interaction Metaphors on Presence and Task Performance. Presence 19, 3 (2010), 197–212. https://doi.org/10.1162/pres.19.3.197
[54]
Yon Visell, Keerthi Adithya Duraikkannan, and Vincent Hayward. 2014. A Device and Method for Multimodal Haptic Rendering of Volumetric Stiffness. In Haptics: Neuroscience, Devices, Modeling, and Applications, Malika Auvray and Christian Duriez (Eds.). Vol. 8618. Springer Berlin Heidelberg, Berlin, Heidelberg, 478–486. https://doi.org/10.1007/978-3-662-44193-0_60
[55]
Junji Watanabe and Maki Sakamoto. 2012. Comparison between onomatopoeias and adjectives for evaluating tactile sensations. In Proceedings of the 6th International Conference of Soft Computing and Intelligent Systems and the 13th International Symposium on Advanced Intelligent Systems (SCIS-ISIS 2012). 2346–2348.
[56]
Dennis Wittchen, Bruno Fruchard, Paul Strohmeier, and Georg Freitag. 2021. TactJam: A Collaborative Playground for Composing Spatial Tactons. In Proceedings of the Fifteenth International Conference on Tangible, Embedded, and Embodied Interaction(Salzburg, Austria) (TEI ’21). Association for Computing Machinery, New York, NY, USA, Article 64, 4 pages. https://doi.org/10.1145/3430524.3442699
[57]
Junya Yoshino, Akira Yakata, Yuichiro Shimizu, Masaharu Haginoya, and Maki Sakamoto. 2013. Method of evaluating metal textures by the sound symbolism of onomatopoeia. In The Second Asian Conference on Information Systems. 618–624.
[58]
André Zenner and Antonio Krüger. 2017. Shifty: A Weight-Shifting Dynamic Passive Haptic Proxy to Enhance Object Perception in Virtual Reality. IEEE Transactions on Visualization and Computer Graphics 23, 4(2017), 1285–1294. https://doi.org/10.1109/TVCG.2017.2656978
[59]
André Zenner and Antonio Krüger. 2019. Drag:On: A Virtual Reality Controller Providing Haptic Feedback Based on Drag and Weight Shift. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (Glasgow, Scotland Uk) (CHI ’19). Association for Computing Machinery, New York, NY, USA, 1–12. https://doi.org/10.1145/3290605.3300441

Cited By

View all
  • (2024)Towards Designing the Sense of Touch: Innovative Tools for Haptic Feedback in Extended RealityCompanion of the 2024 on ACM International Joint Conference on Pervasive and Ubiquitous Computing10.1145/3675094.3678366(272-275)Online publication date: 5-Oct-2024
  • (2024)HapticPilotProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36314537:4(1-28)Online publication date: 12-Jan-2024
  • (2024)AdapTics: A Toolkit for Creative Design and Integration of Real-Time Adaptive Mid-Air Ultrasound TactonsProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642090(1-15)Online publication date: 11-May-2024
  • Show More Cited By
  1. Weirding Haptics: In-Situ Prototyping of Vibrotactile Feedback in Virtual Reality through Vocalization

      Recommendations

      Comments

      Information & Contributors

      Information

      Published In

      cover image ACM Conferences
      UIST '21: The 34th Annual ACM Symposium on User Interface Software and Technology
      October 2021
      1357 pages
      ISBN:9781450386357
      DOI:10.1145/3472749
      This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike International 4.0 License.

      Sponsors

      Publisher

      Association for Computing Machinery

      New York, NY, United States

      Publication History

      Published: 12 October 2021

      Permissions

      Request permissions for this article.

      Check for updates

      Author Tags

      1. Haptic design
      2. design tool
      3. direct manipulation
      4. vibrotactile feedback
      5. virtual reality
      6. vocalization
      7. voice input

      Qualifiers

      • Research-article
      • Research
      • Refereed limited

      Funding Sources

      Conference

      UIST '21

      Acceptance Rates

      Overall Acceptance Rate 561 of 2,567 submissions, 22%

      Upcoming Conference

      UIST '25
      The 38th Annual ACM Symposium on User Interface Software and Technology
      September 28 - October 1, 2025
      Busan , Republic of Korea

      Contributors

      Other Metrics

      Bibliometrics & Citations

      Bibliometrics

      Article Metrics

      • Downloads (Last 12 months)421
      • Downloads (Last 6 weeks)42
      Reflects downloads up to 25 Dec 2024

      Other Metrics

      Citations

      Cited By

      View all
      • (2024)Towards Designing the Sense of Touch: Innovative Tools for Haptic Feedback in Extended RealityCompanion of the 2024 on ACM International Joint Conference on Pervasive and Ubiquitous Computing10.1145/3675094.3678366(272-275)Online publication date: 5-Oct-2024
      • (2024)HapticPilotProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36314537:4(1-28)Online publication date: 12-Jan-2024
      • (2024)AdapTics: A Toolkit for Creative Design and Integration of Real-Time Adaptive Mid-Air Ultrasound TactonsProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642090(1-15)Online publication date: 11-May-2024
      • (2024)Augmenting the feel of real objectsInternational Journal of Human-Computer Studies10.1016/j.ijhcs.2024.103244185:COnline publication date: 25-Jun-2024
      • (2024)The user experience of distal arm-level vibrotactile feedback for interactions with virtual versus physical displaysVirtual Reality10.1007/s10055-024-00977-228:2Online publication date: 22-Mar-2024
      • (2024)HapMotion: motion-to-tactile framework with wearable haptic devices for immersive VR performance experienceVirtual Reality10.1007/s10055-023-00910-z28:1Online publication date: 9-Jan-2024
      • (2024)Participatory Design for In-Vehicle Vibrotactile Warnings on Driver’s SeatHaptics: Understanding Touch; Technology and Systems; Applications and Interaction10.1007/978-3-031-70061-3_29(358-371)Online publication date: 3-Nov-2024
      • (2023)TactTongue: Prototyping ElectroTactile Stimulations on the TongueProceedings of the 36th Annual ACM Symposium on User Interface Software and Technology10.1145/3586183.3606829(1-14)Online publication date: 29-Oct-2023
      • (2023)VoxelHap: A Toolkit for Constructing Proxies Providing Tactile and Kinesthetic Haptic Feedback in Virtual RealityProceedings of the 36th Annual ACM Symposium on User Interface Software and Technology10.1145/3586183.3606722(1-13)Online publication date: 29-Oct-2023
      • (2023)Feellustrator: A Design Tool for Ultrasound Mid-Air HapticsProceedings of the 2023 CHI Conference on Human Factors in Computing Systems10.1145/3544548.3580728(1-16)Online publication date: 19-Apr-2023
      • Show More Cited By

      View Options

      View options

      PDF

      View or Download as a PDF file.

      PDF

      eReader

      View online with eReader.

      eReader

      HTML Format

      View this article in HTML Format.

      HTML Format

      Login options

      Media

      Figures

      Other

      Tables

      Share

      Share

      Share this Publication link

      Share on social media