research-article

Open access

RadarNet: Efficient Gesture Recognition Technique Utilizing a Miniature Radar Sensor

Authors:

Nicholas Gillian,

Leonardo Giusti,

Ivan PoupyrevAuthors Info & Claims

CHI '21: Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems

Article No.: 5, Pages 1 - 14

https://doi.org/10.1145/3411764.3445367

Published: 07 May 2021 Publication History

All formats PDF

Abstract

Gestures are a promising candidate as an input modality for ambient computing where conventional input modalities such as touchscreens are not available. Existing works have focused on gesture recognition using image sensors. However, their cost, high battery consumption, and privacy concerns made cameras challenging as an always-on solution. This paper introduces an efficient gesture recognition technique using a miniaturized 60 GHz radar sensor. The technique recognizes four directional swipes and an omni-swipe using a radar chip (6.5 × 5.0 mm) integrated into a mobile phone. We developed a convolutional neural network model efficient enough for battery powered and computationally constrained processors. Its model size and inference time is less than 1/5000 compared to an existing gesture recognition technique using radar. Our evaluations with large scale datasets consisting of 558,000 gesture samples and 3,920,000 negative samples demonstrated our algorithm’s efficiency, robustness, and readiness to be deployed outside of research laboratories.

References

[1]

Amazon. 2014. Echo. https://www.amazon.com/smart-home-devices/b?ie=UTF8&node=9818047011. Retrieved September 10, 2020.

[2]

Apple. 2018. Siri. https://www.apple.com/siri/. Retrieved September 10, 2020.

[3]

Richard A Bolt. 1980. “Put-that-there” Voice and gesture at the graphics interface. In Proceedings of the 7th annual conference on Computer graphics and interactive techniques. 262–270.

Digital Library

[4]

Alex Butler, Shahram Izadi, and Steve Hodges. 2008. SideSight: multi-” touch” interaction around small devices. In Proceedings of the 21st annual ACM symposium on User interface software and technology. 201–204.

Digital Library

[5]

Kelly E Caine, Arthur D Fisk, and Wendy A Rogers. 2006. Benefits and privacy concerns of a home equipped with a visual sensing system: A perspective from older adults. In Proceedings of the human factors and ergonomics society annual meeting, Vol. 50. SAGE Publications Sage CA: Los Angeles, CA, 180–184.

[6]

Ting-Han Chen, Sok-Ian Sou, and Yinman Lee. 2019. WiTrack: Human-to-Human Mobility Relationship Tracking in Indoor Environments Based on Spatio-Temporal Wireless Signal Strength(Proceedings of the IEEE International Conference on Dependable, Autonomic and Secure Computing, Vol. 00). 788–795.

[7]

Jae-Woo Choi, Si-Jung Ryu, and Jong-Hwan Kim. 2019. Short-range radar based real-time hand gesture recognition using LSTM encoder. IEEE Access 7(2019), 33610–33618.

[8]

Gabe Cohn, Daniel Morris, Shwetak Patel, and Desney Tan. 2012. Humantenna: using the body as an antenna for real-time whole-body interaction(Proceedings of the SIGCHI Conference on Human Factors in Computing Systems). 1901–1910.

[9]

Facebook. 2012. Oculus. https://www.oculus.com/. Retrieved September 10, 2020.

[10]

Google. 2015. TensorFlow Lite. https://www.tensorflow.org/lite. Retrieved September 10, 2020.

[11]

Google. 2016. Google Assistant. https://assistant.google.com/. Retrieved September 10, 2020.

[12]

Google. 2016. Nest Home Hub. https://store.google.com/product/google_nest_hub. Retrieved September 10, 2020.

[13]

Google. 2017. Nest Thermostat. https://nest.com/. Retrieved September 10, 2020.

[14]

Google. 2019. Jacquard. https://atap.google.com/jacquard/. Retrieved September 10, 2020.

[15]

Sidhant Gupta, Daniel Morris, Shwetak Patel, and Desney Tan. 2012. Soundwave: using the doppler effect to sense gestures(Proceedings of the SIGCHI Conference on Human Factors in Computing Systems). 1911–1914.

[16]

Chris Harrison and Scott E Hudson. 2009. Abracadabra: wireless, high-precision, and unpowered finger input for very small mobile devices(Proceedings of the 22nd annual ACM symposium on User interface software and technology). 121–124.

[17]

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep Residual Learning for Image Recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[18]

Marigo Heijboer, Elise Hoven, Bert Bongers, and Saskia Bakker. 2016. Facilitating Peripheral Interaction: Design and Evaluation of Peripheral Interaction for a Gesture-Based Lighting Control with Multimodal Feedback. 20, 1 (2016).

[19]

Andrew G Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam. 2017. Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv:1704.04861(2017).

[20]

Infenion. 2016. Infenion XENSIV. https://www.infineon.com/cms/en/product/sensor/radar-sensors/. Retrieved September 10, 2020.

[21]

Bryce Kellogg, Vamsi Talla, and Shyamnath Gollakota. 2014. Bringing gesture recognition to all devices. In 11th {USENIX} Symposium on Networked Systems Design and Implementation ({NSDI} 14). 303–316.

[22]

Cem Keskin, Furkan Kıraç, Yunus Emre Kara, and Lale Akarun. 2012. Computer Vision – ECCV 2012, 12th European Conference on Computer Vision, Florence, Italy, October 7-13, 2012, Proceedings, Part VI. (2012), 852–863.

[23]

Jaime Lien, Nicholas Gillian, M. Emre Karagozler, Patrick Amihood, Carsten Schwesig, Erik Olson, Hakim Raja, and Ivan Poupyrev. 2016. Soli: ubiquitous gesture sensing with millimeter wave radar. ACM Transactions on Graphics (TOG) 35, 4, 142.

Digital Library

[24]

Microsoft. 2010. Kinect. https://developer.microsoft.com/en-us/windows/kinect/. Retrieved September 10, 2020.

[25]

Microsoft. 2019. Hololens: Handtracking. https://microsoft.github.io/MixedRealityToolkit-Unity/Documentation/Input/HandTracking.html. Retrieved September 10, 2020.

[26]

GRS Murthy and RS Jadon. 2009. A review of vision based hand gestures recognition. International Journal of Information Technology and Knowledge Management 2, 2 (2009), 405–410.

[27]

I. Nasr, R. Jungmaier, A. Baheti, D. Noppeney, J. S. Bal, M. Wojnowski, E. Karagozler, H. Raja, J. Lien, I. Poupyrev, and S. Trotta. 2016. A Highly Integrated 60 GHz 6-Channel Transceiver With Antenna in Package for Smart Sensing and Short-Range Communications. IEEE Journal of Solid-State Circuits 51, 9 (2016), 2066–2076. https://doi.org/10.1109/JSSC.2016.2585621

[28]

Antti Pirhonen, Stephen Brewster, and Christopher Holguin. 2002. Gestural and audio metaphors as a means of control for mobile devices. In Proceedings of the SIGCHI conference on Human factors in computing systems. 291–298.

Digital Library

[29]

Qifan Pu, Sidhant Gupta, Shyamnath Gollakota, and Shwetak Patel. 2013. Whole-home gesture recognition using wireless signals(Proceedings of the 13th Annual International Conference on Mobile Computing and Networking). 27–38.

[30]

Toby Sharp, Cem Keskin, Duncan Robertson, Jonathan Taylor, Jamie Shotton, David Kim, Christoph Rhemann, Ido Leichter, Alon Vinnikov, Yichen Wei, 2015. Accurate, robust, and flexible real-time hand tracking. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems. 3633–3642.

Digital Library

[31]

Christian Szegedy, Sergey Ioffe, Vincent Vanhoucke, and Alex Alemi. 2016. Inception-v4, inception-resnet and the impact of residual connections on learning. arXiv preprint arXiv:1602.07261(2016).

[32]

Sheng Tan and Jie Yang. 2016. WiFinger: leveraging commodity WiFi for fine-grained finger gesture recognition(Proceedings of the 17th International Symposium on Mobile Ad Hoc Networking and Computing). 201–210.

[33]

S. Trotta, D. Weber, R. W. Jungmaier, A. Baheti, J. Lien, D. Noppeney, M. Tabesh, C. Rumpler, M. Aichner, S. Albel, J. S. Bal, and I. Poupyrev. 2021 (to appear). SOLI: A Tiny Device for a New Human Machine Interface. IEEE International Solid-State Circuits Conference (2021 (to appear)).

[34]

Radu-Daniel Vatavu. 2012. Nomadic gestures: A technique for reusing gesture commands for frequent ambient interactions. Journal of Ambient Intelligence and Smart Environments 4, 2(2012), 79–93.

Digital Library

[35]

Saiwen Wang, Jie Song, Jaime Lien, Ivan Poupyrev, and Otmar Hilliges. 2016. Interacting with soli: Exploring fine-grained dynamic gesture recognition in the radio-frequency spectrum. In Proceedings of the 29th Annual Symposium on User Interface Software and Technology. 851–860.

Digital Library

[36]

Mark Weiser and John Seely Brown. 1997. The coming age of calm technology. In Beyond calculation. Springer, 75–85.

[37]

Eric Zeng, Shrirang Mare, and Franziska Roesner. 2017. End User Security and Privacy Concerns with Smart Homes. In Thirteenth Symposium on Usable Privacy and Security (SOUPS 2017). USENIX Association, Santa Clara, CA, 65–80.

[38]

Fan Zhang, Valentin Bazarevsky, Andrey Vakunov, Andrei Tkachenka, George Sung, Chuo-Ling Chang, and Matthias Grundmann. 2020. MediaPipe Hands: On-device Real-time Hand Tracking. arXiv preprint arXiv:2006.10214(2020).

Cited By

Sukianto TWagner MSeifi SCarbonelli CHuemer M(2024)An Uncertainty Aware Semi-Supervised Federated Learning Framework for Radar-based Hand Gesture Recognition2024 21st European Radar Conference (EuRAD)10.23919/EuRAD61604.2024.10734896(168-171)Online publication date: 25-Sep-2024
https://doi.org/10.23919/EuRAD61604.2024.10734896
Affortunati SZagar B(2024)Analysis of a capacitance tomography system for object detection and localizationtm - Technisches Messen10.1515/teme-2024-001391:6(345-355)Online publication date: 1-Apr-2024
https://doi.org/10.1515/teme-2024-0013
Mei LLiu RYin ZZhao QJiang WWang SWang SLu KHe T(2024)mmSpyVR: Exploiting mmWave Radar for Penetrating Obstacles to Uncover Privacy Vulnerability of Virtual RealityProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36997728:4(1-29)Online publication date: 21-Nov-2024
https://dl.acm.org/doi/10.1145/3699772
Show More Cited By

Index Terms

RadarNet: Efficient Gesture Recognition Technique Utilizing a Miniature Radar Sensor
1. Computing methodologies
  1. Machine learning
    1. Machine learning approaches
2. Human-centered computing
  1. Human computer interaction (HCI)

Index terms have been assigned to the content through auto-classification.

Recommendations

Multi-scenario gesture recognition using Kinect
CGAMES '12: Proceedings of the 2012 17th International Conference on Computer Games: AI, Animation, Mobile, Interactive Multimedia, Educational & Serious Games (CGAMES)

Hand gesture recognition (HGR) is an important research topic because some situations require silent communication with sign languages. Computational HGR systems assist silent communication, and help people learn a sign language. In this article, a ...
Smart Hand Device Gesture Recognition with Dynamic Time-Warping Method
BDIOT '17: Proceedings of the International Conference on Big Data and Internet of Thing

In this paper, we present a smart wearable hand-gesture recognition system based on the movement of the hand and fingers. The proposed smart wearable system is built using the fewest sensors necessary for gesture recognition. Thus, motion sensors are ...
Evaluating Gesture Recognition by Multiple-Sensor-Containing Mobile Devices
ISWC '11: Proceedings of the 2011 15th Annual International Symposium on Wearable Computers

Mobile phones or video game controllers using gesture recognition technologies enable easy and intuitive operations. However, usually only one of each type of sensor is installed in each device, and the effect of multiple sensors on recognition accuracy ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

CHI '21: Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems

May 2021

10862 pages

ISBN:9781450380966

DOI:10.1145/3411764

General Chairs:
Yoshifumi Kitamura
Tohoku University, Japan
,
Aaron Quigley
University of New South Wales, Australia
,
Program Chairs:
Katherine Isbister
University of California Santa Cruz, USA
,
Takeo Igarashi
The University of Tokyo, Japan
,
Publications Chairs:
Pernille Bjørn
University of Copenhagen, Denmark
,
Steven Drucker
Microsoft Research, USA

Copyright © 2021 Owner/Author.

This work is licensed under a Creative Commons Attribution International 4.0 License.

Sponsors

SIGCHI: ACM Special Interest Group on Computer-Human Interaction

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 07 May 2021

Check for updates

Badges

Best Paper

Author Tags

Qualifiers

Research-article
Research
Refereed limited

Conference

CHI '21

Sponsor:

SIGCHI

CHI '21: CHI Conference on Human Factors in Computing Systems

May 8 - 13, 2021

Yokohama, Japan

Acceptance Rates

Overall Acceptance Rate 6,199 of 26,314 submissions, 24%

Upcoming Conference

CHI 2025

Sponsor:
sigchi

ACM CHI Conference on Human Factors in Computing Systems

April 26 - May 1, 2025

Yokohama , Japan

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

65
Total Citations
View Citations
9,200
Total Downloads

Downloads (Last 12 months)1,527
Downloads (Last 6 weeks)162

Reflects downloads up to 24 Dec 2024

Other Metrics

View Author Metrics

Citations

Cited By

Sukianto TWagner MSeifi SCarbonelli CHuemer M(2024)An Uncertainty Aware Semi-Supervised Federated Learning Framework for Radar-based Hand Gesture Recognition2024 21st European Radar Conference (EuRAD)10.23919/EuRAD61604.2024.10734896(168-171)Online publication date: 25-Sep-2024
https://doi.org/10.23919/EuRAD61604.2024.10734896
Affortunati SZagar B(2024)Analysis of a capacitance tomography system for object detection and localizationtm - Technisches Messen10.1515/teme-2024-001391:6(345-355)Online publication date: 1-Apr-2024
https://doi.org/10.1515/teme-2024-0013
Mei LLiu RYin ZZhao QJiang WWang SWang SLu KHe T(2024)mmSpyVR: Exploiting mmWave Radar for Penetrating Obstacles to Uncover Privacy Vulnerability of Virtual RealityProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36997728:4(1-29)Online publication date: 21-Nov-2024
https://dl.acm.org/doi/10.1145/3699772
Čopič Pucihar KMeneghello FSalami DSluÿters AVatavu R(2024)Beyond Radar Waves: The First Workshop on Radar-Based Human-Computer InteractionCompanion Proceedings of the 16th ACM SIGCHI Symposium on Engineering Interactive Computing Systems10.1145/3660515.3662836(97-102)Online publication date: 24-Jun-2024
https://dl.acm.org/doi/10.1145/3660515.3662836
Danyluk KKlueber SNittala AWillett W(2024)Understanding Gesture and Microgesture Inputs for Augmented Reality MapsProceedings of the 2024 ACM Designing Interactive Systems Conference10.1145/3643834.3661630(409-423)Online publication date: 1-Jul-2024
https://dl.acm.org/doi/10.1145/3643834.3661630
Borts DLiang EBroedermann TRamazzina AWalz SPalladin ESun JBrueggemann DSakaridis CVan Gool LBijelic MHeide F(2024)Radar Fields: Frequency-Space Neural Scene Representations for FMCW RadarACM SIGGRAPH 2024 Conference Papers10.1145/3641519.3657510(1-10)Online publication date: 13-Jul-2024
https://dl.acm.org/doi/10.1145/3641519.3657510
Liu HLiu XXie XTong XLi K(2024)PmTrackProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36314337:4(1-30)Online publication date: 12-Jan-2024
https://dl.acm.org/doi/10.1145/3631433
Hajika RGunasekaran THaigh CPai YHayashi ELien JLottridge DBillinghurst M(2024)RadarHand: A Wrist-Worn Radar for On-Skin Touch-Based Proprioceptive GesturesACM Transactions on Computer-Human Interaction10.1145/361736531:2(1-36)Online publication date: 29-Jan-2024
https://dl.acm.org/doi/10.1145/3617365
Lee SSutthiwanna SLee JBae S(2024)DirActor: Creating Interaction Illustrations by Oneself through Directing and Acting Simultaneously in VRExtended Abstracts of the CHI Conference on Human Factors in Computing Systems10.1145/3613905.3650768(1-6)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613905.3650768
Strobel MSchoenfeldt SDaugalas J(2024)Gesture Recognition for FMCW Radar on the Edge2024 IEEE Topical Conference on Wireless Sensors and Sensor Networks (WiSNeT)10.1109/WiSNeT59910.2024.10438579(45-48)Online publication date: 21-Jan-2024
https://doi.org/10.1109/WiSNeT59910.2024.10438579
Show More Cited By

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

HTML Format

View this article in HTML Format.

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Media

Figures

Other

Tables

View Table of Contents