poster

UltraBots: Large-Area Mid-Air Haptics for VR with Robotically Actuated Ultrasound Transducers

Authors:

Mehrad Faridan,

Marcus Friedel,

Ryo SuzukiAuthors Info & Claims

UIST '22 Adjunct: Adjunct Proceedings of the 35th Annual ACM Symposium on User Interface Software and Technology

Article No.: 101, Pages 1 - 3

https://doi.org/10.1145/3526114.3561350

Published: 28 October 2022 Publication History

Abstract

We introduce UltraBots, a system that combines ultrasound haptic feedback and robotic actuation for large-area mid-air haptics for VR. Ultrasound haptics can provide precise mid-air haptic feedback and versatile shape rendering, but the interaction area is often limited by the small size of the ultrasound devices, restricting the possible interactions for VR. To address this problem, this paper introduces a novel approach that combines robotic actuation with ultrasound haptics. More specifically, we will attach ultrasound transducer arrays to tabletop mobile robots or robotic arms for scalable, extendable, and translatable interaction areas. We plan to use Sony Toio robots for 2D translation and/or commercially available robotic arms for 3D translation. Using robotic actuation and hand tracking measured by a VR HMD (ex: Oculus Quest), our system can keep the ultrasound transducers underneath the user’s hands to provide on-demand haptics. We demonstrate applications with workspace environments, medical training, education and entertainment.

References

[1]

Bruno Araujo, Ricardo Jota, Varun Perumal, Jia Xian Yao, Karan Singh, and Daniel Wigdor. 2016. Snake Charmer: Physically Enabling Virtual Objects. In Proceedings of the TEI ’16: Tenth International Conference on Tangible, Embedded, and Embodied Interaction(Eindhoven, Netherlands) (TEI ’16). Association for Computing Machinery, New York, NY, USA, 218–226. https://doi.org/10.1145/2839462.2839484

Digital Library

[2]

P Balint and Kaspar Althoefer. 2018. Medical virtual reality palpation training using ultrasound based haptics and image processing. Proc. Jt. Work. New Technol. Comput. Assist. Surg (2018).

[3]

Carlos Bermejo and Pan Hui. 2021. A Survey on Haptic Technologies for Mobile Augmented Reality. ACM Comput. Surv. 54, 9, Article 184 (oct 2021), 35 pages. https://doi.org/10.1145/3465396

Digital Library

[4]

Brygida Dzidek, William Frier, Adam Harwood, and Richard Hayden. 2018. Design and Evaluation of Mid-Air Haptic Interactions in an Augmented Reality Environment. In Haptics: Science, Technology, and Applications, Domenico Prattichizzo, Hiroyuki Shinoda, Hong Z. Tan, Emanuele Ruffaldi, and Antonio Frisoli (Eds.). Springer International Publishing, Cham, 489–499.

[5]

Eric J. Gonzalez, Parastoo Abtahi, and Sean Follmer. 2020. REACH+: Extending the Reachability of Encountered-Type Haptics Devices through Dynamic Redirection in VR. Association for Computing Machinery, New York, NY, USA, 236–248. https://doi.org/10.1145/3379337.3415870

Digital Library

[6]

Inwook Hwang, Hyungki Son, and Jin Ryong Kim. 2017. AirPiano: Enhancing music playing experience in virtual reality with mid-air haptic feedback. In 2017 IEEE World Haptics Conference (WHC). 213–218. https://doi.org/10.1109/WHC.2017.7989903

[7]

Sony Inc. 2018. Sony Toio. https://www.sony.com/en/SonyInfo/design/stories/toio/

[8]

Ultraleap Inc. [n.d.]. Leap Motion Controller. https://www.ultraleap.com/product/leap-motion-controller/

[9]

Ultraleap Inc. 2018. Ultraleap Stratos. https://www.ultraleap.com/haptics/

[10]

Unity Software Inc. [n.d.]. Unity Game Engine. https://unity.com/

[11]

Arata Jingu, Takaaki Kamigaki, Masahiro Fujiwara, Yasutoshi Makino, and Hiroyuki Shinoda. 2021. LipNotif: Use of Lips as a Non-Contact Tactile Notification Interface Based on Ultrasonic Tactile Presentation. In The 34th Annual ACM Symposium on User Interface Software and Technology (Virtual Event, USA) (UIST ’21). Association for Computing Machinery, New York, NY, USA, 13–23. https://doi.org/10.1145/3472749.3474732

Digital Library

[12]

Cédric Kervegant, Félix Raymond, Delphine Graeff, and Julien Castet. 2017. Touch Hologram in Mid-Air. In ACM SIGGRAPH 2017 Emerging Technologies (Los Angeles, California) (SIGGRAPH ’17). Association for Computing Machinery, New York, NY, USA, Article 23, 2 pages. https://doi.org/10.1145/3084822.3084824

Digital Library

[13]

Hannah Limerick. 2020. Call to Interact: Communicating Interactivity and Affordances for Contactless Gesture Controlled Public Displays. In Proceedings of the 9TH ACM International Symposium on Pervasive Displays (Manchester, United Kingdom) (PerDis ’20). Association for Computing Machinery, New York, NY, USA, 63–70. https://doi.org/10.1145/3393712.3395338

Digital Library

[14]

Hannah Limerick, Richard Hayden, David Beattie, Orestis Georgiou, and Jörg Müller. 2019. User Engagement for Mid-Air Haptic Interactions with Digital Signage. In Proceedings of the 8th ACM International Symposium on Pervasive Displays (Palermo, Italy) (PerDis ’19). Association for Computing Machinery, New York, NY, USA, Article 15, 7 pages. https://doi.org/10.1145/3321335.3324944

Digital Library

[15]

Benjamin Long, Sue Ann Seah, Tom Carter, and Sriram Subramanian. 2014. Rendering Volumetric Haptic Shapes in Mid-Air Using Ultrasound. ACM Trans. Graph. 33, 6, Article 181 (nov 2014), 10 pages. https://doi.org/10.1145/2661229.2661257

Digital Library

[16]

Patrizia Marti, Oronzo Parlangeli, Annamaria Recupero, Stefano Guidi, and Matteo Sirizzotti. 2022. Mid-air haptics for shape recognition of virtual objects. Ergonomics 65, 5 (2022), 775–793. https://doi.org/10.1080/00140139.2021.1992019 arXiv:https://doi.org/10.1080/00140139.2021.1992019PMID: 34632962.

[17]

Jonatan Martinez, Daniel Griffiths, Valerio Biscione, Orestis Georgiou, and Tom Carter. 2018. Touchless Haptic Feedback for Supernatural VR Experiences. In 2018 IEEE Conference on Virtual Reality and 3D User Interfaces (VR). 629–630. https://doi.org/10.1109/VR.2018.8446522

[18]

Antti Sand, Ismo Rakkolainen, Poika Isokoski, Jari Kangas, Roope Raisamo, and Karri Palovuori. 2015. Head-Mounted Display with Mid-Air Tactile Feedback. In Proceedings of the 21st ACM Symposium on Virtual Reality Software and Technology (Beijing, China) (VRST ’15). Association for Computing Machinery, New York, NY, USA, 51–58. https://doi.org/10.1145/2821592.2821593

Digital Library

[19]

Vivian Shen, Craig Shultz, and Chris Harrison. 2022. Mouth Haptics in VR Using a Headset Ultrasound Phased Array. In Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems (New Orleans, LA, USA) (CHI ’22). Association for Computing Machinery, New York, NY, USA, Article 275, 14 pages. https://doi.org/10.1145/3491102.3501960

Digital Library

[20]

Anthony Steed, Sebastian Friston, Vijay Pawar, and David Swapp. 2020. Docking Haptics: Dynamic Combinations Of Grounded And Worn Devices. In 2020 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops (VRW). 626–627. https://doi.org/10.1109/VRW50115.2020.00164

[21]

Ryo Suzuki, Hooman Hedayati, Clement Zheng, James L Bohn, Daniel Szafir, Ellen Yi-Luen Do, Mark D Gross, and Daniel Leithinger. 2020. Roomshift: Room-scale dynamic haptics for vr with furniture-moving swarm robots. In Proceedings of the 2020 CHI conference on human factors in computing systems. 1–11.

Digital Library

[22]

Ryo Suzuki, Eyal Ofek, Mike Sinclair, Daniel Leithinger, and Mar Gonzalez-Franco. 2021. HapticBots: Distributed Encountered-Type Haptics for VR with Multiple Shape-Changing Mobile Robots. In The 34th Annual ACM Symposium on User Interface Software and Technology (Virtual Event, USA) (UIST ’21). Association for Computing Machinery, New York, NY, USA, 1269–1281. https://doi.org/10.1145/3472749.3474821

Digital Library

Cited By

Theivendran KWu AFrier WSchneider O(2024)RecHap: An Interactive Recommender System for Navigating a Large Number of Mid-Air Haptic DesignsIEEE Transactions on Haptics10.1109/TOH.2023.327681217:2(165-176)Online publication date: Apr-2024
https://doi.org/10.1109/TOH.2023.3276812
Kim LDomova VYao YRajabi P(2023)SwarmFidget: Exploring Programmable Actuated Fidgeting with Swarm RobotsProceedings of the 36th Annual ACM Symposium on User Interface Software and Technology10.1145/3586183.3606746(1-15)Online publication date: 29-Oct-2023
https://dl.acm.org/doi/10.1145/3586183.3606746

Index Terms

UltraBots: Large-Area Mid-Air Haptics for VR with Robotically Actuated Ultrasound Transducers
1. Human-centered computing
  1. Human computer interaction (HCI)
    1. Interaction devices
      1. Haptic devices
    2. Interaction paradigms
      1. Virtual reality

Recommendations

Extended Mid-air Ultrasound Haptics for Virtual Reality

Mid-air haptics allow bare-hand tactile stimulation; however, it has a constrained workspace, making it unsuitable for room-scale haptics. We present a novel approach to rendering mid-air haptic sensations in a large rendering volume by turning a static ...
TeslaMirror: Multistimulus Encounter-Type Haptic Display for Shape and Texture Rendering in VR
SIGGRAPH '20: ACM SIGGRAPH 2020 Emerging Technologies

We propose a novel concept of a hybrid tactile display with multistimulus feedback, allowing the real-time experience of the position, shape, and texture of the virtual object. TeslaMirror consists of a gross shape display based on 6 DOF robot, fine ...
Explorations of Wrist Haptic Feedback for AR/VR Interactions with Tasbi
CHI EA '20: Extended Abstracts of the 2020 CHI Conference on Human Factors in Computing Systems

Most widespread haptic feedback devices for augmented and virtual reality (AR/VR) fall into one of two categories: simple hand-held controllers with a single vibration actuator, or complex glove systems with several embedded actuators. In this work, we ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

UIST '22 Adjunct: Adjunct Proceedings of the 35th Annual ACM Symposium on User Interface Software and Technology

October 2022

413 pages

ISBN:9781450393218

DOI:10.1145/3526114

Editors:
Maneesh Agrawala
Stanford University, USA
,
Jacob O. Wobbrock
University of Washington, USA
,
Eytan Adar
University of Michigan, USA
,
Vidya Setlur
Tableau Research, USA

Copyright © 2022 Owner/Author.

Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for third-party components of this work must be honored. For all other uses, contact the Owner/Author.

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 28 October 2022

Check for updates

Author Tags

Qualifiers

Poster
Research
Refereed limited

Conference

UIST '22

Sponsor:

UIST '22: The 35th Annual ACM Symposium on User Interface Software and Technology

October 29 - November 2, 2022

OR, Bend, USA

Acceptance Rates

Overall Acceptance Rate 561 of 2,567 submissions, 22%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

2
Total Citations
View Citations
338
Total Downloads

Downloads (Last 12 months)118
Downloads (Last 6 weeks)8

Reflects downloads up to 16 Oct 2024

Other Metrics

View Author Metrics

Citations

Cited By

Theivendran KWu AFrier WSchneider O(2024)RecHap: An Interactive Recommender System for Navigating a Large Number of Mid-Air Haptic DesignsIEEE Transactions on Haptics10.1109/TOH.2023.327681217:2(165-176)Online publication date: Apr-2024
https://doi.org/10.1109/TOH.2023.3276812
Kim LDomova VYao YRajabi P(2023)SwarmFidget: Exploring Programmable Actuated Fidgeting with Swarm RobotsProceedings of the 36th Annual ACM Symposium on User Interface Software and Technology10.1145/3586183.3606746(1-15)Online publication date: 29-Oct-2023
https://dl.acm.org/doi/10.1145/3586183.3606746

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

HTML Format

View this article in HTML Format.

Media

Figures

Other

Tables

View Table of Contents