research-article

Open access

Soli: ubiquitous gesture sensing with millimeter wave radar

Authors:

Nicholas Gillian,

M. Emre Karagozler,

Patrick Amihood,

Carsten Schwesig,

Ivan PoupyrevAuthors Info & Claims

ACM Transactions on Graphics (TOG), Volume 35, Issue 4

Article No.: 142, Pages 1 - 19

https://doi.org/10.1145/2897824.2925953

Published: 11 July 2016 Publication History

Abstract

This paper presents Soli, a new, robust, high-resolution, low-power, miniature gesture sensing technology for human-computer interaction based on millimeter-wave radar. We describe a new approach to developing a radar-based sensor optimized for human-computer interaction, building the sensor architecture from the ground up with the inclusion of radar design principles, high temporal resolution gesture tracking, a hardware abstraction layer (HAL), a solid-state radar chip and system architecture, interaction models and gesture vocabularies, and gesture recognition. We demonstrate that Soli can be used for robust gesture recognition and can track gestures with sub-millimeter accuracy, running at over 10,000 frames per second on embedded hardware.

Supplementary Material

MP4 File (a142.mp4)

Download
377.37 MB

References

[1]

Axelsson, S. R. 2004. Noise radar using random phase and frequency modulation. IEEE Transactions on Geoscience and Remote Sensing 42, 11, 2370--2384.

[2]

Azevedo, S., and McEwan, T. 1996. Micropower impulse radar. Science and Technology Review, 17--29.

[3]

Barnes, S. B. 1997. Douglas Carl Engelbart: Developing the underlying concepts for contemporary computing. IEEE Annals of the History of Computing 19, 3, 16--26.

Digital Library

[4]

Baum, C. E., Rothwell, E. J., Chen, K.-M., and Nyquist, D. P. 1991. The singularity expansion method and its application to target identification. Proceedings of the IEEE 79, 10, 1481--1492.

[5]

Benezeth, Y., Jodoin, P. M., Emile, B., Laurent, H., and Rosenberger, C. 2008. Review and evaluation of commonly-implemented background subtraction algorithms. In ICPR 2008.

[6]

Bishop, C. M. 2006. Pattern recognition. Machine Learning.

Digital Library

[7]

Bolt, R. a. 1980. Put-that-there. SIGGRAPH '80, 262--270.

Digital Library

[8]

Breiman, L. 2001. Random forests. Machine Learning 45, 1, 5--32.

Digital Library

[9]

Brookner, E. 1985. Phased-array radars. Scientific American 252, 2, 94--102.

[10]

Brown, L. 1999. A Radar History of World War II: Technical and Military Imperatives. Institute of Physics Publishing.

[11]

Brunnbauer, M., Meyer, T., Ofner, G., Mueller, K., and Hagen, R. 2008. Embedded wafer level ball grid array (eWLB). In IEMT 2008, IEEE, 1--6.

[12]

Card, S., Moran, T., and Newell, A. 1983. The Psychology of Human-Computer Interaction. L. Erlbaum Associates.

Digital Library

[13]

Chan, L., Chen, C.-h. H. Y.-l., and Yang, S. 2015. Cyclops: Wearable and single-piece full-body gesture input devices. In CHI 2015, ACM, 3001--3009.

Digital Library

[14]

Comscore Inc. 2014. The US mobile app report. Tech. rep.

[15]

Cooperstock, J. R., Fels, S. S., Buxton, W., and Smith, K. C. 1997. Reactive environments. Communications of the ACM 40, 9 (Sep), 65--73.

Digital Library

[16]

Dardas, N. H., and Georganas, N. D. 2011. Real-time hand gesture detection and recognition using bag-of-features and support vector machine techniques. IEEE Transactions on Instrumentation and Measurement 60, 11, 3592--3607.

[17]

Dietz, P., and Leigh, D. 2001. Diamond Touch. In UIST '01, 219.

[18]

Dorfmuller-Ulhaas, K., and Schmalstieg, D. 2001. Finger tracking for interaction in augmented environments. ISMAR 2001, 55--64.

Digital Library

[19]

Duffner, S., Berlemont, S., Lefebvre, G., and Garcia, C. 2014. 3D gesture classification with convolutional neural networks. In ICASSP 2014, IEEE, 5432--5436.

[20]

FCC, 2016. FCC online table of frequency allocations.

[21]

Geng, S., Kivinen, J., Zhao, X., and Vainikainen, P. 2009. Millimeter-wave propagation channel characterization for short-range wireless communications. IEEE Transactions on Vehicular Technology 58, 1 (Jan), 3--13.

[22]

Gillian, N., Knapp, R. B., and O'Modhrain, S. 2011. Recognition of multivariate temporal musical gestures using n-dimensional dynamic time warping. In Proceedings of the 2011 International Conference on New Interfaces for Musical Expression (NIME 11), Oslo, Norway.

[23]

Gillian, N. 2011. Gesture Recognition for Musician Computer Interaction. PhD thesis, School of Music and Sonic Arts, Queen's University Belfast.

[24]

Gustafson, S., Holz, C., and Baudisch, P. 2011. Imaginary phone: Learning imaginary interfaces by transferring spatial memory from a familiar device. UIST '11, 283--292.

Digital Library

[25]

Guyon, I., and Elisseeff, A. 2003. An introduction to variable and feature selection. The Journal of Machine Learning Research 3.

Digital Library

[26]

Hansen, C. J. 2011. WiGiG: Multi-gigabit wireless communications in the 60 GHz band. IEEE Wireless Communications 18, 6, 6--7.

[27]

Harrison, C., Tan, D., and Morris, D. 2010. Skinput: Appropriating the body as an input surface. CHI 2010, 453--462.

Digital Library

[28]

Holleis, P., Schmidt, A., Paasovaara, S., Puikkonen, A., and Häkkilä, J. 2008. Evaluating capacitive touch input on clothes. In MobileHCI '08, ACM Press, New York, New York, USA, 81--90.

Digital Library

[29]

Holz, C., and Wilson, A. 2011. Data miming: Inferring spatial object descriptions from human gesture. CHI 2011, 811--820.

Digital Library

[30]

Hussain, M. G. 1998. Ultra-wideband impulse radar -- an overview of the principles. IEEE Aerospace and Electronic Systems Magazine 13, 9, 9--14.

[31]

Ji, S., Xu, W., Yang, M., and Yu, K. 2013. 3D convolutional neural networks for human action recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence 35, 1, 221--231.

Digital Library

[32]

Junker, H., Amft, O., Lukowicz, P., and Tröster, G. 2008. Gesture spotting with body-worn inertial sensors to detect user activities. Pattern Recognition 41, 6, 2010--2024.

Digital Library

[33]

Keller, J. B. 1962. Geometrical theory of diffraction. JOSA 52, 2, 116--130.

[34]

Kellogg, B., Talla, V., and Gollakota, S. 2014. Bringing gesture recognition to all devices. In 11th USENIX Symposium on Networked Systems Design and Implementation (NSDI 14), 303--316.

Digital Library

[35]

Keskin, C., Kiraç, F., Kara, Y. E., and Akarun, L. 2013. Real time hand pose estimation using depth sensors. In Consumer Depth Cameras for Computer Vision. Springer, 119--137.

[36]

Khoshelham, K., and Elberink, S. O. 2012. Accuracy and resolution of Kinect depth data for indoor mapping applications. Sensors 12, 2, 1437--1454.

[37]

Kim, D., Hilliges, O., Izadi, S., Butler, A. D., Chen, J., Oikonomidis, I., and Olivier, P. 2012. Digits. UIST '12, 167--176.

Digital Library

[38]

Knott, E. F. 2012. Radar Cross Section Measurements. Springer Science & Business Media.

[39]

Kurakin, A., Zhang, Z., and Liu, Z. 2012. A real time system for dynamic hand gesture recognition with a depth sensor. In 2012 Proceedings of the 20th European Signal Processing Conference (EUSIPCO), 1975--1979.

[40]

Lee, S., and Buxton, W. 1985. A multi-touch three dimensional touch-sensitive tablet. In CHI'85, 21--25.

Digital Library

[41]

Levanon, N. 2000. Multifrequency complementary phase-coded radar signal. IEE Proceedings - Radar, Sonar and Navigation 147, 6, 276--284.

[42]

LoPresti, L., LaCascia, M., Sclaroff, S., and Camps, O. 2015. Gesture modeling by Hanklet-based hidden Markov model. In Computer Vision -- ACCV 2014, D. Cremers, I. Reid, H. Saito, and M.-H. Yang, Eds., vol. 9005 of Lecture Notes in Computer Science. Springer International Publishing, 529--546.

[43]

Melgarejo, P., Zhang, X., Ramanathan, P., and Chu, D. 2014. Leveraging directional antenna capabilities for fine-grained gesture recognition. In Proceedings of the 2014 ACM International Joint Conference on Pervasive and Ubiquitous Computing, ACM, 541--551.

Digital Library

[44]

Molchanov, P., Gupta, S., Kim, K., and Kautz, J. 2015. Hand gesture recognition with 3D convolutional neural networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, 1--7.

[45]

Molchanov, P., Gupta, S., Kim, K., and Pulli, K. 2015. Multi-sensor system for driver's hand-gesture recognition. In IEEE Conference on Automatic Face and Gesture Recognition.

[46]

Molchanov, P., Gupta, S., Kim, K., and Pulli, K. 2015. Short-range FMCW monopulse radar for hand-gesture sensing. In IEEE Radar Conference (RadarCon 2015), IEEE, 1491--1496.

[47]

Nasr, I., Karagozler, E., Poupyrev, I., and Trotta, S. 2015. A highly integrated 60-GHz 6-channel transceiver chip in 0.35 um SiGe technology for smart sensing and short-range communications. In IEEE CSICS 2015, IEEE, 1--4.

[48]

Nasr, I., Jungmaier, R., Baheti, A., Noppeney, D., Bal, J. S., Wojnowski, M., Karagozler, E., Raja, H., Lien, J., Poupyrev, I., and Trotta, S. 2016. A highly integrated 60 GHz 6-channel transceiver with antenna in package for smart sensing and short range communications. submitted to IEEE Journal of Solid State Circuits.

[49]

Nathanson, F. E., Reilly, J. P., and Cohen, M. N. 1991. Radar Design Principles -- Signal Processing and the Environment, vol. 91.

[50]

Novelda. Novelda Xethru NVA620x IC.

[51]

Otero, M. 2005. Application of a continuous wave radar for human gait recognition. Proceedings of SPIE 5809, 538--548.

[52]

Paradiso, J., Abler, C., Hsiao, K.-y., and Reynolds, M. 1997. The Magic Carpet: Physical sensing for immersive environments. In CHI'97 Extended Abstracts on Human Factors in Computing Systems, ACM, 277--278.

Digital Library

[53]

Paradiso, J. A. 1999. The Brain Opera technology: New instruments and gestural sensors for musical interaction and performance. Journal of New Music Research 28, 2, 130--149.

[54]

Park, J.-I., and Kim, K.-T. 2010. A comparative study on ISAR imaging algorithms for radar target identification. Progress In Electromagnetics Research 108, 155--175.

[55]

Potter, L. C., Chiang, D.-M., Carriere, R., and Gerry, M. J. 1995. A GTD-based parametric model for radar scattering. IEEE Transactions on Antennas and Propagation 43, 10, 1058--1067.

[56]

Pu, Q., Gupta, S., Gollakota, S., and Patel, S. 2013. Whole-home gesture recognition using wireless signals. In Proceedings of the 19th Annual International Conference on Mobile Computing & Networking, ACM, 27--38.

Digital Library

[57]

Rahman, T., Adams, A. T., Zhang, M., and Choudhury, T. 2015. DoppleSleep: A contactless unobtrusive sleep sensing system using short-range doppler radar. In Ubicomp '15, ACM.

Digital Library

[58]

Ralston, T. S., Charvat, G. L., and Peabody, J. E. 2010. Real-time through-wall imaging using an ultrawideband MIMO phased array radar system. In IEEE International Symposium on Phased Array Systems and Technology (ARRAY), IEEE.

[59]

Rekimoto, J. 2001. GestureWrist and GesturePad: Unobtrusive wearable interaction devices. ISWC 2001, 21--27.

Digital Library

[60]

Richards, M. A., Scheer, J., Holm, W. A., et al. 2010. Principles of modern radar: basic principles. SciTech Pub.

[61]

Romero, J., Kjellström, H., Ek, C. H., and Kragic, D. 2013. Non-parametric hand pose estimation with object context. Image and Vision Computing 31, 8, 555--564.

Digital Library

[62]

Russell, D., Streitz, N., and Winograd, T. 2005. Building disappearing computers. Communications of the ACM 48, 3, 42--48.

Digital Library

[63]

Saponas, T. S., Tan, D. S., Morris, D., Balakrishnan, R., Turner, J., and Landay, J. A. 2009. Enabling always-available input with muscle-computer interfaces. UIST 2009, 167--176.

Digital Library

[64]

Sharp, T., Keskin, C., Robertson, D., Taylor, J., Shotton, J., Kim, D., Rhemann, C., Leichter, I., Vinnikov, A., Wei, Y., Freedman, D., Kohli, P., Krupka, E., Fitzgibbon, A., and Izadi, S. 2015. Accurate, robust, and flexible real-time hand tracking. CHI 2015, 3633--3642.

Digital Library

[65]

Shotton, J., Sharp, T., Kipman, A., Fitzgibbon, A., Finocchio, M., Blake, A., Cook, M., and Moore, R. 2013. Real-time human pose recognition in parts from single depth images. Communications of the ACM 56, 1, 116--124.

Digital Library

[66]

Shuey, D., Bailey, D., and Morrissey, T. P. 1986. PHIGS: A standard, dynamic, interactive graphics interface. IEEE Computer Graphics and Applications 6, 8, 50--57.

Digital Library

[67]

Skolnik, M. I. 1962. Introduction to radar. Radar Handbook 2.

[68]

Smith, C., and Goggans, P. 1993. Radar target identification. IEEE Antennas and Propagation Magazine 35, 2 (April), 27--38.

[69]

Smith, J., White, T., Dodge, C., Paradiso, J., Gershenfeld, N., and Allport, D. 1998. Electric field sensing for graphical interfaces. IEEE Computer Graphics and Applications 18, June, 54--59.

Digital Library

[70]

Smulders, P. 2002. Exploiting the 60 GHz band for local wireless multimedia access: Prospects and future directions. IEEE Communications Magazine 40, 1, 140--147.

Digital Library

[71]

Song, J., Sörös, G., Pece, F., Fanello, S. R., Izadi, S., Keskin, C., and Hilliges, O. 2014. In-air gestures around unmodified mobile devices. UIST 2014, 319--329.

Digital Library

[72]

Storcheus, D., Rostamizadeh, A., and Kumar, S. 2015. A survey of modern questions and challenges in feature extraction. In Proceedings of the 1st International Workshop on Feature Extraction: Modern Questions and Challenges, NIPS, 1--18.

[73]

Stove, A. G. 1992. Linear FMCW radar techniques. In IEEE Proceedings on Radar and Signal Processing, vol. 139, IET, 343--350.

[74]

Strickon, J., and Paradiso, J. 1998. Tracking hands above large interactive surfaces with a low-cost scanning laser rangefinder. CHI'98, 231--232.

Digital Library

[75]

Sutherland, I. E., Blackwell, A., and Rodden, K. 1963. Sketchpad: A man-machine graphical communication system. Tech. Rep. TK296, Lincoln Laboratory, MIT, Lexington, MA.

Digital Library

[76]

Wan, Q., Li, Y., Li, C., and Pal, R. 2014. Gesture recognition for smart home applications using portable radar sensors. In 36th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC 2014), IEEE, 6414--6417.

[77]

Wang, Y., and Fathy, A. E. 2011. Micro-doppler signatures for intelligent human gait recognition using a UWB impulse radar. In IEEE Antennas and Propagation Society, AP-S International Symposium (Digest), 2103--2106.

[78]

Wang, R. Y., and Popović, J. 2009. Real-time hand-tracking with a color glove. In SIGGRAPH 2009, ACM Press, New York, New York, USA, 1.

Digital Library

[79]

Watson-Watt, R. 1945. Radar in war and in peace. Nature 155, 3935, 319--324.

[80]

Wei, T., and Zhang, X. 2015. mTrack: High-precision passive tracking using millimeter wave radios. In Proceedings of the 21st Annual International Conference on Mobile Computing and Networking, ACM, 117--129.

Digital Library

[81]

Weichert, F., Bachmann, D., Rudak, B., and Fisseler, D. 2013. Analysis of the accuracy and robustness of the Leap Motion controller. Sensors 13, 5, 6380--6393.

[82]

Weigel, M., Lu, T., Bailly, G., Oulasvirta, A., Majidi, C., and Steimle, J. 2015. iSkin: Flexible, stretchable and visually customizable on-body touch sensors for mobile computing. In CHI 2015, ACM, 1--10.

Digital Library

[83]

Wu, J., Konrad, J., and Ishwar, P. 2013. Dynamic time warping for gesture-based user identification and authentication with Kinect. In ICASSP 2013, 2371--2375.

[84]

Yatani, K., and Truong, K. N. 2012. BodyScope: A wearable acoustic sensor for activity recognition. In Ubicomp '12, ACM, 341--350.

Digital Library

[85]

Zhai, S., Milgram, P., and Buxton, W. 1996. The influence of muscle groups on performance of multiple degree-of-freedom input. CHI '96, 308--315.

Digital Library

[86]

Zhuang, Y., Song, C., Wang, A., Lin, F., Li, Y., Gu, C., Li, C., and Xu, W. 2015. SleepSense: Non-invasive sleep event recognition using an electromagnetic probe. In IEEE 12th International Conference on Wearable and Implantable Body Sensor Networks (BSN 2015), 1--6.

Cited By

Zhang HLiu KZhang YLin J(2024)TRANS-CNN-Based Gesture Recognition for mmWave RadarSensors10.3390/s2406180024:6(1800)Online publication date: 11-Mar-2024
https://doi.org/10.3390/s24061800
Cheng YYeh WLiao Y(2024)The Implementation of a Gesture Recognition System with a Millimeter Wave and Thermal ImagerSensors10.3390/s2402058124:2(581)Online publication date: 17-Jan-2024
https://doi.org/10.3390/s24020581
Shi YQiao LShu YLi BXiao BLi WGao X(2024)Semi-Supervised FMCW Radar Hand Gesture Recognition via Pseudo-Label Consistency LearningRemote Sensing10.3390/rs1613226716:13(2267)Online publication date: 21-Jun-2024
https://doi.org/10.3390/rs16132267
Show More Cited By

Index Terms

Soli: ubiquitous gesture sensing with millimeter wave radar
1. Hardware
  1. Communication hardware, interfaces and storage
    1. Sensor devices and platforms
2. Human-centered computing
  1. Human computer interaction (HCI)
    1. Interaction devices
    2. Interaction techniques
      1. Gestural input

Recommendations

LeapLook: a free-hand gestural travel technique using the leap motion finger tracker
SUI '14: Proceedings of the 2nd ACM symposium on Spatial user interaction

Contactless motion sensing devices enable a new form of input that does not encumber the user with wearable tracking equipment. We present a novel travel technique using the Leap Motion finger tracker which adopts a 2DOF steering metaphor used in ...
MicroRolls: expanding touch-screen input vocabulary by distinguishing rolls vs. slides of the thumb
CHI '09: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems

The input vocabulary for touch-screen interaction on handhelds is dramatically limited, especially when the thumb must be used. To enrich that vocabulary we propose to discriminate, among thumb gestures, those we call MicroRolls, characterized by zero ...
Designing user-, hand-, and handpart-aware tabletop interactions with the TouchID toolkit
ITS '11: Proceedings of the ACM International Conference on Interactive Tabletops and Surfaces

Recent work in multi-touch tabletop interaction introduced many novel techniques that let people manipulate digital content through touch. Yet most only detect touch blobs. This ignores richer interactions that would be possible if we could identify (1) ...

Comments

Information & Contributors

Information

Published In

cover image ACM Transactions on Graphics

ACM Transactions on Graphics Volume 35, Issue 4

July 2016

1396 pages

ISSN:0730-0301

EISSN:1557-7368

DOI:10.1145/2897824

Issue’s Table of Contents

Copyright © 2016 Owner/Author.

This work is licensed under a Creative Commons Attribution-NonCommercial International 4.0 License.

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 11 July 2016

Published in TOG Volume 35, Issue 4

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

737
Total Citations
View Citations
19,041
Total Downloads

Downloads (Last 12 months)1,981
Downloads (Last 6 weeks)245

Reflects downloads up to 04 Sep 2024

Other Metrics

View Author Metrics

Citations

Cited By

Zhang HLiu KZhang YLin J(2024)TRANS-CNN-Based Gesture Recognition for mmWave RadarSensors10.3390/s2406180024:6(1800)Online publication date: 11-Mar-2024
https://doi.org/10.3390/s24061800
Cheng YYeh WLiao Y(2024)The Implementation of a Gesture Recognition System with a Millimeter Wave and Thermal ImagerSensors10.3390/s2402058124:2(581)Online publication date: 17-Jan-2024
https://doi.org/10.3390/s24020581
Shi YQiao LShu YLi BXiao BLi WGao X(2024)Semi-Supervised FMCW Radar Hand Gesture Recognition via Pseudo-Label Consistency LearningRemote Sensing10.3390/rs1613226716:13(2267)Online publication date: 21-Jun-2024
https://doi.org/10.3390/rs16132267
Tseng TDing J(2024)Adaptive Trajectories’ Constant False Alarm Rate Mirror Filters and Elevation Angle Evaluation for Multiple-Input Multiple-Output Radar-Based Hand Gesture RecognitionElectronics10.3390/electronics1304068213:4(682)Online publication date: 7-Feb-2024
https://doi.org/10.3390/electronics13040682
Ma JChen PXiong XZhang LYu SZhang D(2024)Research on Vision-Based Servoing and Trajectory Prediction Strategy for Capturing Illegal DronesDrones10.3390/drones80401278:4(127)Online publication date: 28-Mar-2024
https://doi.org/10.3390/drones8040127
Bai YWang JChen PGong ZXiong Q(2024)Hand Trajectory Recognition by Radar with a Finite-State Machine and a Bi-LSTMApplied Sciences10.3390/app1415678214:15(6782)Online publication date: 3-Aug-2024
https://doi.org/10.3390/app14156782
Gayathri RNam S(2024)Enhancing User Experience in Virtual Museums: Impact of Finger Vibrotactile FeedbackApplied Sciences10.3390/app1415659314:15(6593)Online publication date: 28-Jul-2024
https://doi.org/10.3390/app14156593
ZHANG JYU YWU YZHAO CZHANG QKANG K(2024)Developments and challenges of mm-Wave integrated circuits on siliconSCIENTIA SINICA Informationis10.1360/SSI-2023-033454:1(68)Online publication date: 8-Jan-2024
https://doi.org/10.1360/SSI-2023-0334
Abdelnasser HHeggo MPang OKovac MMcCann J(2024)RaDro: Indoor Drone Tracking Using Millimeter Wave RadarProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36785498:3(1-23)Online publication date: 22-Aug-2024
https://dl.acm.org/doi/10.1145/3678549
Čopič Pucihar KMeneghello FSalami DSluÿters AVatavu R(2024)Beyond Radar Waves: The First Workshop on Radar-Based Human-Computer InteractionCompanion Proceedings of the 16th ACM SIGCHI Symposium on Engineering Interactive Computing Systems10.1145/3660515.3662836(97-102)Online publication date: 24-Jun-2024
https://dl.acm.org/doi/10.1145/3660515.3662836
Show More Cited By

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

Media

Figures

Other

Tables

View Issue’s Table of Contents