research-article

Mid-Air Thermo-Tactile Feedback using Ultrasound Haptic Display

Authors:

Yatharth Singhal,

Jin Ryong KimAuthors Info & Claims

VRST '21: Proceedings of the 27th ACM Symposium on Virtual Reality Software and Technology

Article No.: 28, Pages 1 - 11

https://doi.org/10.1145/3489849.3489889

Published: 08 December 2021 Publication History

Abstract

This paper presents a mid-air thermo-tactile feedback system using an ultrasound haptic display. We design a proof-of-concept thermo-tactile feedback system with an open-top chamber, heat modules, and an ultrasound display. Our approach is to provide heated airflow along the path to the focused pressure point created from the ultrasound display to generate thermal and vibrotactile cues in mid-air simultaneously. We confirm that our system can generate the thermo-tactile stimuli up to 54.2°C with 3.43 mN when the ultrasonic haptic signal was set to 100 Hz with a 12 mm radius of the cue size. We also confirm that our system can provide a stable temperature (mean error=0.25%). We measure the warm detection threshold (WDT) and the heat-pain detection threshold (HPDT). The results show that the mean WDT was 32.8°C (SD=1.12), and the mean HPDT was 44.6°C (SD=1.64), which are consistent with the contact-based thermal thresholds. We also found that the accuracy of haptic pattern identification is similar for non-thermal (98.1%) and thermal conditions (97.2%), showing a non-significant effect of high temperature. We finally confirmed that thermo-tactile feedback further enhances the user experiences.

References

[1]

Mayienne Bakkers, Catharina G Faber, Martine JH Peters, Jos PH Reulen, Hessel Franssen, Tanya Z Fischer, and Ingemar SJ Merkies. 2013. Temperature threshold testing: a systematic review. Journal of the Peripheral Nervous System 18, 1 (2013), 7–18.

[2]

Gillian Bartlett, John D Stewart, Robyn Tamblyn, and Michal Abrahamowicz. 1998. Normal distributions of thermal and vibration sensory thresholds. Muscle & Nerve: Official Journal of the American Association of Electrodiagnostic Medicine 21, 3 (1998), 367–374.

[3]

Jas Brooks, Steven Nagels, and Pedro Lopes. 2020. Trigeminal-based Temperature Illusions. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems. 1–12.

Digital Library

[4]

Tom Carter, Sue Ann Seah, Benjamin Long, Bruce Drinkwater, and Sriram Subramanian. 2013. UltraHaptics: multi-point mid-air haptic feedback for touch surfaces. In Proceedings of the 26th annual ACM symposium on User interface software and technology. 505–514.

Digital Library

[5]

Arthur D Craig, Kewei Chen, D Bandy, and Eric M Reiman. 2000. Thermosensory activation of insular cortex. Nature neuroscience 3, 2 (2000), 184–190.

[6]

Ian Darian-Smith and Kenneth O Johnson. 1977. Thermal sensibility and thermoreceptors. Journal of Investigative Dermatology 69, 1 (1977), 146–153.

[7]

Peter James Dyck, I Zimmerman, DA Gillen, D Johnson, JL Karnes, and PC O’Brien. 1993. Cool, warm, and heat-pain detection thresholds: testing methods and inferences about anatomic distribution of receptors. Neurology 43, 8 (1993), 1500–1500.

[8]

F. Alton Everest. 2001. The Master Handbook of Acoustics. McGraw-Hill.

[9]

William Frier, Damien Ablart, Jamie Chilles, Benjamin Long, Marcello Giordano, Marianna Obrist, and Sriram Subramanian. 2018. Using spatiotemporal modulation to draw tactile patterns in mid-air. In International Conference on Human Haptic Sensing and Touch Enabled Computer Applications. Springer, 270–281.

[10]

Simon Gallo, Lucian Cucu, Nicolas Thevenaz, Ali Sengül, and Hannes Bleuler. 2014. Design and control of a novel thermo-tactile multimodal display. In 2014 IEEE Haptics Symposium (Haptics). Ieee, 75–81.

[11]

LR Gavrilov and EM Tsirulnikov. 2012. Focused ultrasound as a tool to input sensory information to humans. Acoustical Physics 58, 1 (2012), 1–21.

[12]

Hyunjae Gil, Hyungki Son, Jin Ryong Kim, and Ian Oakley. 2018. Whiskers: Exploring the use of ultrasonic haptic cues on the face. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. 1–13.

Digital Library

[13]

Barry G Green. 1977. The effect of skin temperature on vibrotactile sensitivity. Perception & Psychophysics 21, 3 (1977), 243–248.

[14]

Sebastian Günther, Florian Müller, Dominik Schön, Omar Elmoghazy, Max Mühlhäuser, and Martin Schmitz. 2020. Therminator: Understanding the Interdependency of Visual and On-Body Thermal Feedback in Virtual Reality. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems. 1–14.

Digital Library

[15]

Ping-Hsuan Han, Yang-Sheng Chen, Kong-Chang Lee, Hao-Cheng Wang, Chiao-En Hsieh, Jui-Chun Hsiao, Chien-Hsing Chou, and Yi-Ping Hung. 2018. Haptic around: multiple tactile sensations for immersive environment and interaction in virtual reality. In Proceedings of the 24th ACM symposium on virtual reality software and technology. 1–10.

Digital Library

[16]

Keisuke Hasegawa and Hiroyuki Shinoda. 2013. A method for distribution control of aerial ultrasound radiation pressure for remote vibrotactile display. In The SICE Annual Conference 2013. IEEE, 223–228.

[17]

Sverre Holm. 2019. Waves with Power-Law Attenuation. Springer.

[18]

Felix Hülsmann, Julia Fröhlich, Nikita Mattar, and Ipke Wachsmuth. 2014. Wind and warmth in virtual reality: implementation and evaluation. In Proceedings of the 2014 Virtual Reality International Conference. 1–8.

Digital Library

[19]

Inwook Hwang, Hyungki Son, and Jin Ryong Kim. 2017. AirPiano: Enhancing music playing experience in virtual reality with mid-air haptic feedback. In 2017 IEEE World Haptics Conference (WHC). IEEE, 213–218.

[20]

Daisuke Iwai, Mei Aoki, and Kosuke Sato. 2018. Non-contact thermo-visual augmentation by IR-RGB projection. IEEE transactions on visualization and computer graphics 25, 4(2018), 1707–1716.

[21]

David G. Foster James Welty, Gregory L. Rorrer. 2019. Fundamentals of Momentum, Heat, and Mass Transfer. Wiley.

[22]

Lynette A Jones and Hsin-Ni Ho. 2008. Warm or cool, large or small? The challenge of thermal displays. IEEE Transactions on Haptics 1, 1 (2008), 53–70.

Digital Library

[23]

Lynette A Jones and Susan J Lederman. 2006. Human hand function. Oxford university press.

[24]

Takaaki Kamigaki, Shun Suzuki, and Hiroyuki Shinoda. 2020. Mid-air Thermal Display via High-intensity Ultrasound. In SIGGRAPH Asia 2020 Emerging Technologies. 1–2.

[25]

Takaaki Kamigaki, Shun Suzuki, and Hiroyuki Shinoda. 2020. Noncontact Thermal and Vibrotactile Display Using Focused Airborne Ultrasound. In International Conference on Human Haptic Sensing and Touch Enabled Computer Applications. Springer, 271–278.

[26]

Peter Kammermeier, Alexander Kron, Jens Hoogen, and Günther Schmidt. 2004. Display of holistic haptic sensations by combined tactile and kinesthetic feedback. Presence: Teleoperators & Virtual Environments 13, 1(2004), 1–15.

Digital Library

[27]

Kiesa Getz Kelly, Thomas Cook, and Misha-Miroslav Backonja. 2005. Pain ratings at the thresholds are necessary for interpretation of quantitative sensory testing. Muscle & Nerve: Official Journal of the American Association of Electrodiagnostic Medicine 32, 2 (2005), 179–184.

[28]

Marius A Kemler, Hubert JA Schouten, and Richard H Gracely. 2000. Diagnosing sensory abnormalities with either normal values or values from contralateral skin: comparison of two approaches in complex regional pain syndrome I. The Journal of the American Society of Anesthesiologists 93, 3(2000), 718–727.

[29]

Jin Ryong Kim, Stephanie Chan, Xiangchao Huang, Kenneth Ng, Limin Paul Fu, and Chen Zhao. 2019. Demonstration of refinity: an interactive holographic signage for new retail shopping experience. In Extended Abstracts of the 2019 CHI Conference on Human Factors in Computing Systems. 1–4.

Digital Library

[30]

Alyssa K Kosturakis, Zijing He, Yan Li, Jessica A Boyette-Davis, Nina Shah, Sheeba K Thomas, Haijun Zhang, Elisabeth G Vichaya, Xin Shelley Wang, Gwen Wendelschafer-Crabb, 2014. Subclinical peripheral neuropathy in patients with multiple myeloma before chemotherapy is correlated with decreased fingertip innervation density. Journal of Clinical Oncology 32, 28 (2014), 3156.

[31]

Clemens JM Lasance. 2008. Ten years of boundary-condition-independent compact thermal modeling of electronic parts: A review. Heat Transfer Engineering 29, 2 (2008), 149–168.

[32]

Marjorie R Leek. 2001. Adaptive procedures in psychophysical research. Perception & psychophysics 63, 8 (2001), 1279–1292.

[33]

HCCH Levitt. 1971. Transformed up-down methods in psychoacoustics. The Journal of the Acoustical society of America 49, 2B (1971), 467–477.

[34]

Richard A Meyer, Ronald E Walker, and Vernon B Mountcastle. 1976. A laser stimulator for the study of cutaneous thermal and pain sensations. IEEE transactions on biomedical engineering1 (1976), 54–60.

[35]

Mitsuru Nakajima, Keisuke Hasegawa, Yasutoshi Makino, and Hiroyuki Shinoda. 2018. Remotely displaying cooling sensation via ultrasound-driven air flow. In 2018 IEEE Haptics Symposium (HAPTICS). IEEE, 340–343.

[36]

Mitsuru Nakajima, Keisuke Hasegawa, Yasutoshi Makino, and Hiroyuki Shinoda. 2021. Spatiotemporal Pinpoint Cooling Sensation Produced by Ultrasound-Driven Mist Vaporization on Skin. IEEE Transactions on Haptics(2021).

Digital Library

[37]

Arinobu Niijima, Toki Takeda, Takafumi Mukouchi, and Takashi Satou. 2020. Thermalbitdisplay: Haptic display providing thermal feedback perceived differently depending on body parts. In Extended Abstracts of the 2020 CHI Conference on Human Factors in Computing Systems. 1–8.

Digital Library

[38]

Roshan Lalintha Peiris, Liwei Chan, and Kouta Minamizawa. 2016. Thermocons: Evaluating the thermal haptic perception of the forehead. In Proceedings of the 29th Annual Symposium on User Interface Software and Technology. 187–188.

Digital Library

[39]

Roshan Lalintha Peiris, Wei Peng, Zikun Chen, Liwei Chan, and Kouta Minamizawa. 2017. Thermovr: Exploring integrated thermal haptic feedback with head mounted displays. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems. 5452–5456.

Digital Library

[40]

John I. Hochstein Philip M. Gerhart, Andrew L. Gerhart. 2015. Munson, Young and Okiishi’s Fundamentals of Fluid Mechanics. Wiley.

[41]

Nicolaas Prins 2016. Psychophysics: a practical introduction. Academic Press.

[42]

Nimesha Ranasinghe, Pravar Jain, Shienny Karwita, David Tolley, and Ellen Yi-Luen Do. 2017. Ambiotherm: enhancing sense of presence in virtual reality by simulating real-world environmental conditions. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems. 1731–1742.

Digital Library

[43]

Ahsan Raza, Waseem Hassan, Tatyana Ogay, Inwook Hwang, and Seokhee Jeon. 2019. Perceptually correct haptic rendering in mid-air using ultrasound phased array. IEEE Transactions on Industrial Electronics 67, 1 (2019), 736–745.

[44]

Thijs Roumen, Simon T Perrault, and Shengdong Zhao. 2015. Notiring: A comparative study of notification channels for wearable interactive rings. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems. 2497–2500.

Digital Library

[45]

Satoshi Saga. 2015. HeatHapt thermal radiation-based haptic display. In Haptic Interaction. Springer, 105–107.

[46]

Emily Shaw, Tessa Roper, Tommy Nilsson, Glyn Lawson, Sue VG Cobb, and Daniel Miller. 2019. The heat is on: Exploring user behaviour in a multisensory virtual environment for fire evacuation. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems. 1–13.

Digital Library

[47]

Anshul Singhal and Lynette A Jones. 2015. Dimensionality of thermal icons. In 2015 IEEE World Haptics Conference (WHC). IEEE, 469–474.

[48]

Anshul Singhal and Lynette A Jones. 2017. Perceptual interactions in thermo-tactile displays. In 2017 IEEE World Haptics Conference (WHC). IEEE, 90–95.

[49]

Joseph C Stevens. 1982. Temperature can sharpen tactile acuity. Perception & psychophysics 31, 6 (1982), 577–580.

[50]

Joseph C Stevens. 1989. Temperature and the two-point threshold. Somatosensory & motor research 6, 3 (1989), 275–284.

[51]

Joseph C Stevens and Barry G Green. 1978. Temperature–touch interaction: Weber’s phenomenon revisited.Sensory processes (1978).

[52]

I A et al. Strigo. 2000. Effect of ambient temperature on human pain and temperature perception. Anesthesiology 92,3(2000), 699–707.

[53]

Ryoko Takahashi, Keisuke Hasegawa, and Hiroyuki Shinoda. 2018. Lateral modulation of midair ultrasound focus for intensified vibrotactile stimuli. In International Conference on Human Haptic Sensing and Touch Enabled Computer Applications. Springer, 276–288.

[54]

Ryoko Takahashi, Keisuke Hasegawa, and Hiroyuki Shinoda. 2019. Tactile stimulation by repetitive lateral movement of midair ultrasound focus. IEEE transactions on haptics 13, 2 (2019), 334–342.

Digital Library

[55]

Peter Weir, Christian Sandor, Matt Swoboda, Thanh Nguyen, Ulrich Eck, Gerhard Reitmayr, and Arindam Day. 2013. Burnar: Involuntary heat sensations in augmented reality. IEEE.

[56]

Reto Wettach, Christian Behrens, Adam Danielsson, and Thomas Ness. 2007. A thermal information display for mobile applications. In Proceedings of the 9th international conference on Human computer interaction with mobile devices and services. 182–185.

Digital Library

[57]

Graham Wilson, Stephen Brewster, Martin Halvey, and Stephen Hughes. 2012. Thermal icons: evaluating structured thermal feedback for mobile interaction. In Proceedings of the 14th international conference on Human-computer interaction with mobile devices and services. 309–312.

Digital Library

[58]

Jiayi Xu, Yoshihiro Kuroda, Shunsuke Yoshimoto, and Osamu Oshiro. 2019. Non-contact cold thermal display by controlling low-temperature air flow generated with vortex tube. In 2019 IEEE World Haptics Conference (WHC). IEEE, 133–138.

[59]

Gi-Hun Yang, Ki-Uk Kyung, Mandayam A Srinivasan, and Dong-Soo Kwon. 2006. Quantitative tactile display device with pin-array type tactile feedback and thermal feedback. In Proceedings 2006 IEEE International Conference on Robotics and Automation, 2006. ICRA 2006. IEEE, 3917–3922.

[60]

Tae-Heon Yang, Jin Ryong Kim, Hanbit Jin, Hyunjae Gil, Jeong-Hoi Koo, and Hye Jin Kim. 2021. Recent Advances and Opportunities of Active Materials for Haptic Technologies in Virtual and Augmented Reality. Advanced Functional Materials(2021), 2008831.

[61]

David Yarnitsky and Elliot Sprecher. 1994. Thermal testing: normative data and repeatability for various test algorithms. Journal of the neurological sciences 125, 1 (1994), 39–45.

[62]

David Yarnitsky, Elhor Sprecher, Ruth Zaslansky, and Jeshayachu A. Hemli. 1995. Heat pain thresholds: normative data and repeatability. Pain 60, 3 (1995), 329–332. https://doi.org/10.1016/0304-3959(94)00132-X

[63]

Kazuma Yoshino and Hiroyuki Shinoda. 2014. Contactless touch interface supporting blind touch interaction by aerial tactile stimulation. In 2014 IEEE Haptics Symposium (HAPTICS). IEEE, 347–350.

Cited By

He SZeng HXue MHuang GYao CYing F(2024)Affective Stroking: Design Thermal Mid-Air Tactile for Assisting People in Stress RegulationApplied Sciences10.3390/app1420949414:20(9494)Online publication date: 17-Oct-2024
https://doi.org/10.3390/app14209494
Nozawa TLiu RSawada H(2024)Displaying Tactile Sensation by SMA-Driven Vibration and Controlled Temperature for Cutaneous Sensation AssessmentActuators10.3390/act1311046313:11(463)Online publication date: 18-Nov-2024
https://doi.org/10.3390/act13110463
Villa SWeiss YHirsch NWiethoff A(2024)An Examination of Ultrasound Mid-air Haptics for Enhanced Material and Temperature Perception in Virtual EnvironmentsProceedings of the ACM on Human-Computer Interaction10.1145/36764888:MHCI(1-21)Online publication date: 24-Sep-2024
https://dl.acm.org/doi/10.1145/3676488
Show More Cited By

Recommendations

Demonstrating FIRE: Mid-Air Thermo-Tactile Display
UIST Adjunct '24: Adjunct Proceedings of the 37th Annual ACM Symposium on User Interface Software and Technology

We present an innovative mid-air thermo-tactile display system based on ultrasound haptics. The system features an open-top chamber, heat modules, and an ultrasound haptic display. Our approach directs heated airflow toward the pressure point created by ...
Head-mounted display with mid-air tactile feedback
VRST '15: Proceedings of the 21st ACM Symposium on Virtual Reality Software and Technology

Virtual and physical worlds are merging. Currently users of head-mounted displays cannot have unobtrusive tactile feedback while touching virtual objects. We present a mid-air tactile feedback system for head-mounted displays. Our prototype uses the ...
FIRE: Mid-Air Thermo-Tactile Display
SIGGRAPH '24: ACM SIGGRAPH 2024 Emerging Technologies

We demonstrate an ultrasound haptic-based mid-air thermo-tactile display system, designed as a proof-of-concept with an open-top chamber, heat modules, and an ultrasound haptic display. Our method involves directing heated airflow toward the focused ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

VRST '21: Proceedings of the 27th ACM Symposium on Virtual Reality Software and Technology

December 2021

563 pages

ISBN:9781450390927

DOI:10.1145/3489849

Editors:
Yuichi Itoh
Aoyama Gakuin University, Japan
,
Kazuki Takashima
Tohoku University, Japan
,
Parinya Punpongsanon
Osaka University, Japan
,
Misha Sra
University of California, Santa Barbara, USA
,
Kazuyuki Fujita
Tohoku University, Japan
,
Shigeo Yoshida
The University of Tokyo, Japan
,
Tham Piumsomboon
University of Canterbury, New Zealand

Copyright © 2021 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 08 December 2021

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed limited

Conference

VRST '21

Sponsor:

VRST '21: 27th ACM Symposium on Virtual Reality Software and Technology

December 8 - 10, 2021

Osaka, Japan

Acceptance Rates

Overall Acceptance Rate 66 of 254 submissions, 26%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

26
Total Citations
View Citations
1,048
Total Downloads

Downloads (Last 12 months)353
Downloads (Last 6 weeks)19

Reflects downloads up to 26 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

He SZeng HXue MHuang GYao CYing F(2024)Affective Stroking: Design Thermal Mid-Air Tactile for Assisting People in Stress RegulationApplied Sciences10.3390/app1420949414:20(9494)Online publication date: 17-Oct-2024
https://doi.org/10.3390/app14209494
Nozawa TLiu RSawada H(2024)Displaying Tactile Sensation by SMA-Driven Vibration and Controlled Temperature for Cutaneous Sensation AssessmentActuators10.3390/act1311046313:11(463)Online publication date: 18-Nov-2024
https://doi.org/10.3390/act13110463
Villa SWeiss YHirsch NWiethoff A(2024)An Examination of Ultrasound Mid-air Haptics for Enhanced Material and Temperature Perception in Virtual EnvironmentsProceedings of the ACM on Human-Computer Interaction10.1145/36764888:MHCI(1-21)Online publication date: 24-Sep-2024
https://dl.acm.org/doi/10.1145/3676488
Singhal YWang HKim J(2024)Demonstrating FIRE: Mid-Air Thermo-Tactile DisplayAdjunct Proceedings of the 37th Annual ACM Symposium on User Interface Software and Technology10.1145/3672539.3686769(1-2)Online publication date: 13-Oct-2024
https://dl.acm.org/doi/10.1145/3672539.3686769
Vikberg ETseng YMoesgen TGowrishankar RXiao Y(2024)ThermoTouch: Exploring a modular design of a programmable wearable thermo-haptic deviceProceedings of the 2024 ACM International Conference on Interactive Media Experiences Workshops10.1145/3672406.3672423(122-126)Online publication date: 12-Jun-2024
https://dl.acm.org/doi/10.1145/3672406.3672423
Wang HSinghal YKim J(2024)Let It Snow: Designing Snowfall Experience in VRProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36595878:2(1-24)Online publication date: 15-May-2024
https://dl.acm.org/doi/10.1145/3659587
Singhal YHonrales DWang HKim J(2024)Thermal In Motion: Designing Thermal Flow Illusions with Tactile and Thermal InteractionProceedings of the 37th Annual ACM Symposium on User Interface Software and Technology10.1145/3654777.3676460(1-13)Online publication date: 13-Oct-2024
https://dl.acm.org/doi/10.1145/3654777.3676460
Wang HSinghal YGil HKim J(2024)Fiery Hands: Designing Thermal Glove through Thermal and Tactile Integration for Virtual Object ManipulationProceedings of the 37th Annual ACM Symposium on User Interface Software and Technology10.1145/3654777.3676457(1-15)Online publication date: 13-Oct-2024
https://dl.acm.org/doi/10.1145/3654777.3676457
Ichihashi SInami MHo HHowell N(2024)Hydroptical Thermal Feedback: Spatial Thermal Feedback Using Visible Lights and WaterProceedings of the 37th Annual ACM Symposium on User Interface Software and Technology10.1145/3654777.3676453(1-19)Online publication date: 13-Oct-2024
https://dl.acm.org/doi/10.1145/3654777.3676453
Kang SKim GHwang SPark JElsharkawy AKim S(2024)Flip-Pelt: Motor-Driven Peltier Elements for Rapid Thermal Stimulation and Congruent Pressure Feedback in Virtual RealityProceedings of the 37th Annual ACM Symposium on User Interface Software and Technology10.1145/3654777.3676363(1-15)Online publication date: 13-Oct-2024
https://dl.acm.org/doi/10.1145/3654777.3676363
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

HTML Format

View this article in HTML Format.

Figures

Tables

Media

View Table of Conten