research-article

Open access

ForceSight: Non-Contact Force Sensing with Laser Speckle Imaging

Authors:

Pradyumna Chari,

Achuta Kadambi,

Yang ZhangAuthors Info & Claims

UIST '22: Proceedings of the 35th Annual ACM Symposium on User Interface Software and Technology

Article No.: 25, Pages 1 - 11

https://doi.org/10.1145/3526113.3545622

Published: 28 October 2022 Publication History

All formats PDF

Abstract

Force sensing has been a key enabling technology for a wide range of interfaces such as digitally enhanced body and world surfaces for touch interactions. Additionally, force often contains rich contextual information about user activities and can be used to enhance machine perception for improved user and environment awareness. To sense force, conventional approaches rely on contact sensors made of pressure-sensitive materials such as piezo films/discs or force-sensitive resistors. We present ForceSight, a non-contact force sensing approach using laser speckle imaging. Our key observation is that object surfaces deform in the presence of force. This deformation, though very minute, manifests as observable and discernible laser speckle shifts, which we leverage to sense the applied force. This non-contact force-sensing capability opens up new opportunities for rich interactions and can be used to power user-/environment-aware interfaces. We first built and verified the model of laser speckle shift with surface deformations. To investigate the feasibility of our approach, we conducted studies on metal, plastic, wood, along with a wide variety of materials. Additionally, we included supplementary tests to fully tease out the performance of our approach. Finally, we demonstrated the applicability of ForceSight with several demonstrative example applications.

References

[1]

David Briers, Donald D Duncan, Evan R Hirst, Sean J Kirkpatrick, Marcus Larsson, Wiendelt Steenbergen, Tomas Stromberg, and Oliver B Thompson. 2013. Laser speckle contrast imaging: theoretical and practical limitations. Journal of biomedical optics 18, 6 (2013), 066018.

[2]

J David Briers and Sian Webster. 1996. Laser speckle contrast analysis (LASCA): a nonscanning, full-field technique for monitoring capillary blood flow. Journal of biomedical optics 1, 2 (1996), 174–179.

[3]

Álvaro Cassinelli, Stéphane Perrin, and Masatoshi Ishikawa. 2005. Smart laser-scanner for 3D human-machine interface. In CHI’05 Extended Abstracts on Human Factors in Computing Systems. 1138–1139.

[4]

Abe Davis, Katherine L Bouman, Justin G Chen, Michael Rubinstein, Fredo Durand, and William T Freeman. 2015. Visual vibrometry: Estimating material properties from small motion in video. In Proceedings of the ieee conference on computer vision and pattern recognition. 5335–5343.

[5]

Home Depot. 2022. Plywood – Columbia Forest Products. https://www.homedepot.com/p/Columbia-Forest-Products-1-2-in-x-2-ft-x-2-ft-PureBond-Red-Oak-Plywood-Project-Panel-Free-Custom-Cut-Available-2847/204771237 Last accessed 24 July 2022.

[6]

Mustafa Doga Dogan, Steven Vidal Acevedo Colon, Varnika Sinha, Kaan Akşit, and Stefanie Mueller. 2021. SensiCut: Material-Aware Laser Cutting Using Speckle Sensing and Deep Learning. In The 34th Annual ACM Symposium on User Interface Software and Technology. 24–38.

[7]

Andrew K Dunn, Hayrunnisa Bolay, Michael A Moskowitz, and David A Boas. 2001. Dynamic imaging of cerebral blood flow using laser speckle. Journal of Cerebral Blood Flow & Metabolism 21, 3(2001), 195–201.

[8]

Berthy T Feng, Alexander C Ogren, Chiara Daraio, and Katherine L Bouman. 2022. Visual vibration tomography: Estimating interior material properties from monocular video. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 16231–16240.

[9]

Wido Heeman, Wiendelt Steenbergen, Gooitzen M van Dam, and E Christiaan Boerma. 2019. Clinical applications of laser speckle contrast imaging: a review. Journal of biomedical optics 24, 8 (2019), 080901.

[10]

Roy Henderson and Karl Schulmeister. 2003. Laser safety. CRC Press.

[11]

Yu-Chih Huang, Tyson L Ringold, J Stuart Nelson, and Bernard Choi. 2008. Noninvasive blood flow imaging for real-time feedback during laser therapy of port wine stain birthmarks. Lasers in Surgery and Medicine: The Official Journal of the American Society for Laser Medicine and Surgery 40, 3 (2008), 167–173.

[12]

Vikram Iyer, Elyas Bayati, Rajalakshmi Nandakumar, Arka Majumdar, and Shyamnath Gollakota. 2018. Charging a smartphone across a room using lasers. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 1, 4 (2018), 1–21.

Digital Library

[13]

Kensei Jo, Mohit Gupta, and Shree K Nayar. 2015. Spedo: 6 dof ego-motion sensor using speckle defocus imaging. In Proceedings of the IEEE International Conference on Computer Vision. 4319–4327.

Digital Library

[14]

David Kim, Otmar Hilliges, Shahram Izadi, Alex D Butler, Jiawen Chen, Iason Oikonomidis, and Patrick Olivier. 2012. Digits: freehand 3D interactions anywhere using a wrist-worn gloveless sensor. In Proceedings of the 25th annual ACM symposium on User interface software and technology. 167–176.

Digital Library

[15]

KyungDuk Kim, HyeonSeung Yu, Joonyoung Koh, Jung H Shin, Wonhee Lee, and YongKeun Park. 2016. Remote sensing of pressure inside deformable microchannels using light scattering in Scotch tape. Optics Letters 41, 8 (2016), 1837–1840.

[16]

Gierad Laput and Chris Harrison. 2019. SurfaceSight: a new spin on touch, user, and object sensing for IoT experiences. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems. 1–12.

Digital Library

[17]

Dengfeng Li, Xinhong Liu, Yizhi Liang, Jun Fan, and Lidai Wang. 2019. A Low-Cost Portable Nanophotonic Sensor Based on a Smartphone: A system readily available for many applications. IEEE Nanotechnology Magazine 13, 3 (2019), 6–12.

[18]

Camillo Lugaresi, Jiuqiang Tang, Hadon Nash, Chris McClanahan, Esha Uboweja, Michael Hays, Fan Zhang, Chuo-Ling Chang, Ming Yong, Juhyun Lee, 2019. Mediapipe: A framework for perceiving and processing reality. In Third Workshop on Computer Vision for AR/VR at IEEE Computer Vision and Pattern Recognition (CVPR), Vol. 2019.

[19]

Thomas Henry Gordon Megson. 2019. Structural and stress analysis. Butterworth-Heinemann.

[20]

Alex Olwal, Andrew Bardagjy, Jan Zizka, and Ramesh Raskar. 2012. SpeckleEye: gestural interaction for embedded electronics in ubiquitous computing. In CHI’12 Extended Abstracts on Human Factors in Computing Systems. 2237–2242.

[21]

Joseph A Paradiso. 1997. The LaserWall. The LaserWall (1997). http://paradiso.media.mit.edu/SpectrumWeb/captions/Laser.html

[22]

Joseph A Paradiso, Kai-Yuh Hsiao, Joshua Strickon, and Peter Rice. 2000. New sensor and music systems for large interactive surfaces. In ICMC. Citeseer.

[23]

Ashwin B Parthasarathy, Erica L Weber, Lisa M Richards, Douglas J Fox, and Andrew K Dunn. 2010. Laser speckle contrast imaging of cerebral blood flow in humans during neurosurgery: a pilot clinical study. Journal of biomedical optics 15, 6 (2010), 066030.

[24]

Santa Monica Plastics. 2022. Plastic – Santa Monica Plastics. https://santamonicaplastics.com/shop/acrylic-sheets-more-cut-to-size/cut-to-size-clear-white/acrylic-sheets-cut-to-size-opaque-white-7328/ Last accessed 24 July 2022.

[25]

Munehiko Sato, Shigeo Yoshida, Alex Olwal, Boxin Shi, Atsushi Hiyama, Tomohiro Tanikawa, Michitaka Hirose, and Ramesh Raskar. 2015. Spectrans: Versatile material classification for interaction with textureless, specular and transparent surfaces. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems. 2191–2200.

Digital Library

[26]

Valkyrie Savage, Colin Chang, and Björn Hartmann. 2013. Sauron: embedded single-camera sensing of printed physical user interfaces. In Proceedings of the 26th annual ACM symposium on User interface software and technology. 447–456.

Digital Library

[27]

Martin Schmitz, Mohammadreza Khalilbeigi, Matthias Balwierz, Roman Lissermann, Max Mühlhäuser, and Jürgen Steimle. 2015. Capricate: A fabrication pipeline to design and 3D print capacitive touch sensors for interactive objects. In Proceedings of the 28th Annual ACM Symposium on User Interface Software & Technology. 253–258.

Digital Library

[28]

Mark Sheinin, Dorian Chan, Matthew O’Toole, and Srinivasa G Narasimhan. 2022. Dual-Shutter Optical Vibration Sensing. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 16324–16333.

[29]

CJR Sheppard and M Hrynevych. 1992. Diffraction by a circular aperture: a generalization of Fresnel diffraction theory. JOSA A 9, 2 (1992), 274–281.

[30]

Yi Chang Shih, Abe Davis, Samuel W Hasinoff, Frédo Durand, and William T Freeman. 2012. Laser speckle photography for surface tampering detection. In 2012 IEEE Conference on Computer Vision and Pattern Recognition. IEEE, 33–40.

[31]

Brandon M Smith, Pratham Desai, Vishal Agarwal, and Mohit Gupta. 2017. CoLux: Multi-object 3d micro-motion analysis using speckle imaging. ACM Transactions on Graphics (TOG) 36, 4 (2017), 1–12.

Digital Library

[32]

Brandon M Smith, Matthew O’Toole, and Mohit Gupta. 2018. Tracking multiple objects outside the line of sight using speckle imaging. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 6258–6266.

[33]

Metal Supermarkets. 2022. Metal – Metal Supermarkets. https://www.metalsupermarkets.com/product/aluminum-sheet-5052/ Last accessed 24 July 2022.

[34]

Nobukatsu Takai, T Iwai, and Toshimitsu Asakura. 1983. Correlation distance of dynamic speckles. Applied Optics 22, 1 (1983), 170–177.

[35]

Yasuhiro Tamaki, Makoto Araie, Eizo Kawamoto, Shuichiro Eguchi, and Hitoshi Fujii. 1994. Noncontact, two-dimensional measurement of retinal microcirculation using laser speckle phenomenon.Investigative ophthalmology & visual science 35, 11 (1994), 3825–3834.

[36]

Will J Warren, Erik A Moro, Matthew E Briggs, and Eric B Flynn. 2014. Simulating translation-induced laser speckle dynamics in photon Doppler velocimetry. Applied Optics 53, 21 (2014), 4661–4668.

[37]

Karl Willis, Eric Brockmeyer, Scott Hudson, and Ivan Poupyrev. 2012. Printed optics: 3D printing of embedded optical elements for interactive devices. In Proceedings of the 25th annual ACM symposium on User interface software and technology. 589–598.

Digital Library

[38]

Nan Wu and S Haruyama. 2020. Real-time audio detection and regeneration of moving sound source based on optical flow algorithm of laser speckle images. Optics Express 28, 4 (2020), 4475–4488.

[39]

Robert Xiao, Chris Harrison, and Scott E Hudson. 2013. WorldKit: rapid and easy creation of ad-hoc interactive applications on everyday surfaces. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. 879–888.

Digital Library

[40]

Robert Xiao, Chris Harrison, Karl DD Willis, Ivan Poupyrev, and Scott E Hudson. 2013. Lumitrack: low cost, high precision, high speed tracking with projected m-sequences. In Proceedings of the 26th annual ACM symposium on User interface software and technology. 3–12.

Digital Library

[41]

Robert Xiao, Scott Hudson, and Chris Harrison. 2016. Direct: Making touch tracking on ordinary surfaces practical with hybrid depth-infrared sensing. In Proceedings of the 2016 ACM International Conference on Interactive Surfaces and Spaces. 85–94.

Digital Library

[42]

Robert Xiao, Scott Hudson, and Chris Harrison. 2017. Supporting responsive cohabitation between virtual interfaces and physical objects on everyday surfaces. Proceedings of the ACM on Human-Computer Interaction 1, EICS(2017), 1–17.

Digital Library

[43]

Yang Zhang, Wolf Kienzle, Yanjun Ma, Shiu S Ng, Hrvoje Benko, and Chris Harrison. 2019. ActiTouch: Robust touch detection for on-skin AR/VR interfaces. In Proceedings of the 32nd Annual ACM Symposium on User Interface Software and Technology. 1151–1159.

Digital Library

[44]

Yang Zhang, Gierad Laput, and Chris Harrison. 2018. Vibrosight: Long-range vibrometry for smart environment sensing. In Proceedings of the 31st Annual ACM Symposium on User Interface Software and Technology. 225–236.

Digital Library

[45]

Yang Zhang, Sven Mayer, Jesse T Gonzalez, and Chris Harrison. 2021. Vibrosight++: City-Scale Sensing Using Existing Retroreflective Signs and Markers. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems. 1–14.

Digital Library

[46]

Jan Zizka, Alex Olwal, and Ramesh Raskar. 2011. SpeckleSense: fast, precise, low-cost and compact motion sensing using laser speckle. In Proceedings of the 24th annual ACM symposium on User interface software and technology. 489–498.

Digital Library

Cited By

Min TXia CYamamoto TSugiura Y(2023)Seeing the Wind: An Interactive Mist Interface for Airflow InputProceedings of the ACM on Human-Computer Interaction10.1145/36264807:ISS(398-419)Online publication date: 1-Nov-2023
https://dl.acm.org/doi/10.1145/3626480
Abdelzaher TCaesar MMendis CNahrstedt KSrivastava MYu M(2023)Challenges in Metaverse Research: An Internet of Things Perspective2023 IEEE International Conference on Metaverse Computing, Networking and Applications (MetaCom)10.1109/MetaCom57706.2023.00042(161-170)Online publication date: Jun-2023
https://doi.org/10.1109/MetaCom57706.2023.00042
Zhang TSheinin MChan DRau MO'Toole MNarasimhan S(2023)Analyzing Physical Impacts Using Transient Surface Wave Imaging2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)10.1109/CVPR52729.2023.00422(4339-4348)Online publication date: Jun-2023
https://doi.org/10.1109/CVPR52729.2023.00422

Index Terms

ForceSight: Non-Contact Force Sensing with Laser Speckle Imaging
1. Hardware
  1. Communication hardware, interfaces and storage
    1. Sensor applications and deployments
    2. Sensors and actuators
2. Human-centered computing
  1. Human computer interaction (HCI)

Recommendations

Microinstrument contact force sensing based on cable tension using BLSTM–MLP network
Abstract
Minimally invasive surgical robotic systems established the foundation for precise and refined surgery, and the intelligentization of robotic systems is an important direction for future development. Among the methods of intelligentization, ...
Accurate estimation of the inhibition zone of antibiotics based on laser speckle imaging and multiple random speckle illumination
Abstract
The antimicrobial susceptibility test (AST) plays a crucial role in selecting appropriate antibiotics for the treatment of bacterial infections in patients. The diffusion disk method is widely adopted AST method due to its simplicity, cost-...
Highlights
- Laser speckle imaging enables the dynamic changes in a turbid medium via analyzing temporal correlation.
- Multiple speckle illumination with the rotation of the diffuser allows independent illumination patterns.
- A correlation map ...
Indirect shear force estimation for multi-point shear force operations
CHI EA '13: CHI '13 Extended Abstracts on Human Factors in Computing Systems

As a way to enrich touch screen interaction, the possibility of using shear forces is being explored recently. However, most of the studies are restricted to the case of single-point shear forces possibly due to the difficulty of sensing shear forces at ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

UIST '22: Proceedings of the 35th Annual ACM Symposium on User Interface Software and Technology

October 2022

1363 pages

ISBN:9781450393201

DOI:10.1145/3526113

Editors:
Maneesh Agrawala
Stanford University, USA
,
Jacob O. Wobbrock
University of Washington, USA
,
Eytan Adar
University of Michigan, USA
,
Vidya Setlur
Tableau Research, USA

Copyright © 2022 Owner/Author.

This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike International 4.0 License.

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 28 October 2022

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed limited

Conference

UIST '22

Sponsor:

UIST '22: The 35th Annual ACM Symposium on User Interface Software and Technology

October 29 - November 2, 2022

OR, Bend, USA

Acceptance Rates

Overall Acceptance Rate 842 of 3,967 submissions, 21%

Upcoming Conference

UIST '24

Sponsor:
sigchi
sigchi

The 37th Annual ACM Symposium on User Interface Software and Technology

October 13 - 16, 2024

Pittsburgh , PA , USA

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

3
Total Citations
View Citations
2,765
Total Downloads

Downloads (Last 12 months)1,832
Downloads (Last 6 weeks)226

Reflects downloads up to 11 Aug 2024

Other Metrics

View Author Metrics

Citations

Cited By

Min TXia CYamamoto TSugiura Y(2023)Seeing the Wind: An Interactive Mist Interface for Airflow InputProceedings of the ACM on Human-Computer Interaction10.1145/36264807:ISS(398-419)Online publication date: 1-Nov-2023
https://dl.acm.org/doi/10.1145/3626480
Abdelzaher TCaesar MMendis CNahrstedt KSrivastava MYu M(2023)Challenges in Metaverse Research: An Internet of Things Perspective2023 IEEE International Conference on Metaverse Computing, Networking and Applications (MetaCom)10.1109/MetaCom57706.2023.00042(161-170)Online publication date: Jun-2023
https://doi.org/10.1109/MetaCom57706.2023.00042
Zhang TSheinin MChan DRau MO'Toole MNarasimhan S(2023)Analyzing Physical Impacts Using Transient Surface Wave Imaging2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)10.1109/CVPR52729.2023.00422(4339-4348)Online publication date: Jun-2023
https://doi.org/10.1109/CVPR52729.2023.00422

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

HTML Format

View this article in HTML Format.

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Media

Figures

Other

Tables

View Table of Contents