research-article

Public Access

Leveraging the Properties of mmWave Signals for 3D Finger Motion Tracking for Interactive IoT Applications

Authors:

Srihari NelakuditiAuthors Info & Claims

Proceedings of the ACM on Measurement and Analysis of Computing Systems, Volume 6, Issue 3

Article No.: 52, Pages 1 - 28

https://doi.org/10.1145/3570613

Published: 08 December 2022 Publication History

Abstract

mmWave signals form a critical component of 5G and next-generation wireless networks, which are also being increasingly considered for sensing the environment around us to enable ubiquitous IoT applications. In this context, this paper leverages the properties of mmWave signals for tracking 3D finger motion for interactive IoT applications. While conventional vision-based solutions break down under poor lighting, occlusions, and also suffer from privacy concerns, mmWave signals work under typical occlusions and non-line-of-sight conditions, while being privacy-preserving. In contrast to prior works on mmWave sensing that focus on predefined gesture classification, this work performs continuous 3D finger motion tracking. Towards this end, we first observe via simulations and experiments that the small size of fingers coupled with specular reflections do not yield stable mmWave reflections. However, we make an interesting observation that focusing on the forearm instead of the fingers can provide stable reflections for 3D finger motion tracking. Muscles that activate the fingers extend through the forearm, whose motion manifests as vibrations on the forearm. By analyzing the variation in phases of reflected mmWave signals from the forearm, this paper designs mm4Arm, a system that tracks 3D finger motion. Nontrivial challenges arise due to the high dimensional search space, complex vibration patterns, diversity across users, hardware noise, etc. mm4Arm exploits anatomical constraints in finger motions and fuses them with machine learning architectures based on encoder-decoder and ResNets in enabling accurate tracking. A systematic performance evaluation with 10 users demonstrates a median error of 5.73° (location error of 4.07 mm) with robustness to multipath and natural variation in hand position/orientation. The accuracy is also consistent under non-line-of-sight conditions and clothing that might occlude the forearm. mm4Arm runs on smartphones with a latency of 19ms and low energy overhead.

References

[1]

5dt data glove ultra - 5dt. https://5dt.com/5dt-data-glove-ultra/.

[2]

American manual alphabet. https://en.wikipedia.org/wiki/American_manual_alphabet.

[3]

Cyberglove systems llc. http://www.cyberglovesystems.com/.

[4]

Forearm muscles : Attachment, nerve supply action. https://anatomyinfo.com/forearm-muscles/.

[5]

Hanes men's full-zip eco-smart hoodie. https://www.amazon.com/Hanes-EcoSmart-Fleece-Hoodie-Black/dp/B00JUM4CT4/.

[6]

How many deaf people are there in united states. https://research.gallaudet.edu/Demographics/deaf-US.php.

[7]

Iwr6843isk. https://www.ti.com/tool/IWR6843ISK.

[8]

Leap motion developer. https://developer.leapmotion.com/.

[9]

Microsoft kinect2.0. https://developer.microsoft.com/en-us/windows/kinect.

[10]

mm4arm demo video. https://www.dropbox.com/s/j14oh4udhaxqa05/mm4Arm_demo.mp4?dl=0.

[11]

Myo official tutorial. https://support.getmyo.com/hc/en-us/articles/203910089-Warm-up-while-wearing-your-Myoarmband.

[12]

patch camelyon tensorflow datasets. https://www.tensorflow.org/datasets/catalog/patch_camelyon.

[13]

Profile battery usage with batterystats and battery historian. https://developer.android.com/topic/performance/power/setup-battery-historian.

[14]

Real-time data-capture adapter for radar sensing evaluation module. https://www.ti.com/tool/DCA1000EVM.

[15]

Ti mmwave radar. https://dev.ti.com/tirex/explore/node.

[16]

tiuserguide. https://www.ti.com/lit/ug/spruix8/spruix8.pdf?ts=1631234356918.

[17]

Using wavefarer automotive radar simulation software and chirp doppler to assess radar performance for drive scenario. https://resources.remcom.com/automotive-radar/publications-wavefarer-chirp-doppler-for-drive-scenarioscomcas2019.

[18]

Wavefarer radar simulation software. https://www.remcom.com/wavefarer-automotive-radar-software.

[19]

Yasrkml 3 panel room divider. https://www.amazon.com/YASRKML-Partition-Separators-Freestanding-102x71--3/dp/B092HQ5W7D/.

[20]

Abadi, M., et al. Tensorflow: A system for large-scale machine learning. In OSDI (2016).

Digital Library

[21]

Adib, F., Hsu, C.-Y., Mao, H., Katabi, D., and Durand, F. Capturing the human figure through a wall. ACM Transactions on Graphics (TOG) 34, 6 (2015), 1--13.

Digital Library

[22]

Ahmed, M. A., et al. A review on systems-based sensory gloves for sign language recognition state of the art between 2007 and 2017. Sensors 18, 7 (2018), 2208.

[23]

An, S., and Ogras, U. Y. Mars: mmwave-based assistive rehabilitation system for smart healthcare. ACM Transactions on Embedded Computing Systems (TECS) 20, 5s (2021), 1--22.

[24]

Ashwini, A. Everything you need to know about iot prototyping, Mar 2020. https://medium.com/swlh/everythingyou-need-to-know-about-iot-prototyping-e4ad2739bc6a.

[25]

Bansal, K., Rungta, K., Zhu, S., and Bharadia, D. Pointillism: Accurate 3d bounding box estimation with multiradars. In Proceedings of the 18th Conference on Embedded Networked Sensor Systems (2020), pp. 340--353.

Digital Library

[26]

Bansal, K., Rungta, K., Zhu, S., and Bharadia, D. Pointillism: accurate 3d bounding box estimation with multi-radars. In Proceedings of the 18th Conference on Embedded Networked Sensor Systems (2020), pp. 340--353.

Digital Library

[27]

Bertero, M., De Mol, C., and Viano, G. A. The stability of inverse problems. In Inverse scattering problems in optics. Springer, 1980, pp. 161--214.

[28]

Cai, Y., Ge, L., Cai, J., and Yuan, J. Weakly-supervised 3d hand pose estimation from monocular rgb images. In ECCV (2018).

Digital Library

[29]

Cao, Z., Radosavovic, I., Kanazawa, A., and Malik, J. Reconstructing hand-object interactions in the wild. In Proceedings of the IEEE/CVF International Conference on Computer Vision (2021), pp. 12417--12426.

[30]

Chang, Z., Zhang, F., Xiong, J., Ma, J., Jin, B., and Zhang, D. Sensor-free soil moisture sensing using lora signals. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 6, 2 (2022), 1--27.

[31]

Chen, A. T.-Y., Biglari-Abhari, M., and Wang, K. I.-K. Context is king: Privacy perceptions of camera-based surveillance. In 2018 15th IEEE International Conference on Advanced Video and Signal Based Surveillance (AVSS) (2018), pp. 1--6.

[32]

Chen Chen, F., et al. Constraint study for a hand exoskeleton: human hand kinematics and dynamics. Journal of Robotics 2013 (2013).

[33]

Chintalapudi, K., et al. Indoor localization without the pain. In ACM MobiCom (2010).

Digital Library

[34]

Connolly, J., et al. Imu sensor-based electronic goniometric glove for clinical finger movement analysis. IEEE Sensors Journal (2017).

[35]

Davis, T. S., Wark, H. A., Hutchinson, D., Warren, D. J., O'neill, K., Scheinblum, T., Clark, G. A., Normann, R. A., and Greger, B. Restoring motor control and sensory feedback in people with upper extremity amputations using arrays of 96 microelectrodes implanted in the median and ulnar nerves. Journal of neural engineering 13, 3 (2016), 036001.

[36]

De Silva, A., et al. Real-time hand gesture recognition using temporal muscle activation maps of multi-channel semg signals. arXiv:2002.03159 (2020).

[37]

Deng, J., et al. Imagenet: A large-scale hierarchical image database. In IEEE CVPR (2009).

[38]

Devlin, J., et al. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 (2018).

[39]

Du, G., Zhang, P., Mai, J., and Li, Z. Markerless kinect-based hand tracking for robot teleoperation. International Journal of Advanced Robotic Systems 9, 2 (2012), 36.

[40]

Du, Y., et al. Semi-supervised learning for surface emg-based gesture recognition. In IJCAI (2017).

[41]

Fang, B., Co, J., and Zhang, M. Deepasl: Enabling ubiquitous and non-intrusive word and sentence-level sign language translation. In Proceedings of the 15th ACM Conference on Embedded Network Sensor Systems (2017), pp. 1--13.

Digital Library

[42]

George, J. A., et al. Bilaterally mirrored movements improve the accuracy and precision of training data for supervised learning of neural or myoelectric prosthetic control. arXiv preprint (2020).

[43]

Google. Deploy machine learning models on mobile and IoT devices. "https://www.tensorflow.org/lite", 2019.

[44]

Ha, U., Leng, J., Khaddaj, A., and Adib, F. Food and liquid sensing in practical environments using {RFIDs}. In 17th USENIX Symposium on Networked Systems Design and Implementation (NSDI 20) (2020), pp. 1083--1100.

[45]

Ha, U., Madani, S., and Adib, F. Wistress: Contactless stress monitoring using wireless signals. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 5, 3 (2021), 1--37.

[46]

Han, X., et al. Pre-trained alexnet architecture with pyramid pooling and supervision for high spatial resolution remote sensing image scene classification. Remote Sensing (2017).

[47]

Hasan, S., and Linte, C. A. U-netplus: a modified encoder-decoder u-net architecture for semantic and instance segmentation of surgical instrument. arXiv preprint arXiv:1902.08994 (2019).

[48]

He, K., et al. Deep residual learning for image recognition. In IEEE CVPR (2016).

[49]

Hu, F., et al. Fingertrak: Continuous 3d hand pose tracking by deep learning hand silhouettes captured by miniature thermal cameras on wrist. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies (2020).

Digital Library

[50]

Ioffe, S., et al. Batch normalization: Accelerating deep network training by reducing internal covariate shift. arXiv preprint arXiv:1502.03167 (2015).

Digital Library

[51]

Iqbal, U., et al. Hand pose estimation via latent 2.5 d heatmap regression. In ECCV (2018).

[52]

Iravantchi, Y., Zhang, Y., Bernitsas, E., Goel, M., and Harrison, C. Interferi: Gesture sensing using on-body acoustic interferometry. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (2019), pp. 1--13.

Digital Library

[53]

Jiang, C., Guo, J., He, Y., Jin, M., Li, S., and Liu, Y. mmvib: micrometer-level vibration measurement with mmwave radar. In Proceedings of the 26th Annual International Conference on Mobile Computing and Networking (2020), pp. 1--13.

Digital Library

[54]

Kandel, I., and Castelli, M. How deeply to fine-tune a convolutional neural network: a case study using a histopathology dataset. Applied Sciences 10, 10 (2020), 3359.

[55]

Khamis, A., Kusy, B., Chou, C. T., McLaws, M.-L., and Hu,W. Rfwash: a weakly supervised tracking of hand hygiene technique. In Proceedings of the 18th Conference on Embedded Networked Sensor Systems (2020), pp. 572--584.

Digital Library

[56]

Kienzle, W., Whitmire, E., Rittaler, C., and Benko, H. Electroring: Subtle pinch and touch detection with a ring. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems (2021), pp. 1--12.

Digital Library

[57]

Kim, D., et al. Digits: freehand 3d interactions anywhere using a wrist-worn gloveless sensor. In ACM UIST (2012).

Digital Library

[58]

Kim, D., Hilliges, O., Izadi, S., Butler, A. D., Chen, J., Oikonomidis, I., and Olivier, P. Digits: freehand 3d interactions anywhere using a wrist-worn gloveless sensor. In Proceedings of the 25th annual ACM symposium on User interface software and technology (2012), pp. 167--176.

Digital Library

[59]

Kim, J. H., Chun, H. J., Hong, I. P., Kim, Y. J., and Park, Y. B. Analysis of fss radomes based on physical optics method and ray tracing technique. IEEE Antennas and Wireless Propagation Letters 13 (2014), 868--871.

[60]

Kingma, D. P., and Ba, J. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014).

[61]

Kong, H., Xu, X., Yu, J., Chen, Q., Ma, C., Chen, Y., Chen, Y.-C., and Kong, L. m3track: mmwave-based multi-user 3d posture tracking. In Proceedings of the 20th Annual International Conference on Mobile Systems, Applications and Services (2022), pp. 491--503.

Digital Library

[62]

Kouyoumjian, R. G., and Pathak, P. H. A uniform geometrical theory of diffraction for an edge in a perfectly conducting surface. Proceedings of the IEEE 62, 11 (1974), 1448--1461.

[63]

Krizhevsky, A., et al. Imagenet classification with deep convolutional neural networks. In NIPS (2012).

Digital Library

[64]

Li, H., Yang, W., Wang, J., Xu, Y., and Huang, L. Wifinger: talk to your smart devices with finger-grained gesture. In Proceedings of the 2016 ACM International Joint Conference on Pervasive and Ubiquitous Computing (2016), ACM, pp. 250--261.

Digital Library

[65]

Li, Y., Wang, N., Shi, J., Hou, X., and Liu, J. Adaptive batch normalization for practical domain adaptation. Pattern Recognition 80 (2018), 109--117.

[66]

Li, Y., Wang, N., Shi, J., Liu, J., and Hou, X. Revisiting batch normalization for practical domain adaptation. arXiv preprint arXiv:1603.04779 (2016).

[67]

Liew, S. S., et al. Bounded activation functions for training stability of deep neural networks on visual pattern recognition problems. Neurocomputing (2016).

[68]

Lin, B.-S., et al. Design of an inertial-sensor-based data glove for hand function evaluation. Sensors (2018).

[69]

Lin, J., and Wu, T. S. H. Modeling the constraints of human hand motion.

[70]

Lin, J., Wu, Y., and Huang, T. S. Modeling the constraints of human hand motion. In Proceedings workshop on human motion (2000), IEEE, pp. 121--126.

[71]

Ling, H., Chou, R.-C., and Lee, S.-W. Shooting and bouncing rays: Calculating the rcs of an arbitrarily shaped cavity. IEEE Transactions on Antennas and propagation 37, 2 (1989), 194--205.

[72]

Liu, H., Wang, Y., Zhou, A., He, H., Wang, W., Wang, K., Pan, P., Lu, Y., Liu, L., and Ma, H. Real-time arm gesture recognition in smart home scenarios via millimeter wave sensing. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 4, 4 (2020), 1--28.

Digital Library

[73]

Liu, J., et al. uwave: Accelerometer-based personalized gesture recognition and its applications. Pervasive and Mobile Computing (2009).

[74]

Liu, Y., Lin, C., and Li, Z. Wr-hand: Wearable armband can track user's hand. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 5, 3 (2021), 1--27.

[75]

Liu, Y., Zhang, S., and Gowda, M. Neuropose: 3d hand pose tracking using emg wearables. In Proceedings of the Web Conference 2021 (2021), pp. 1471--1482.

Digital Library

[76]

Liu, Y., Zhang, S., and Gowda, M. When video meets inertial sensors: Zero-shot domain adaptation for finger motion analytics with inertial sensors. In Proceedings of the International Conference on Internet-of-Things Design and Implementation (2021), pp. 182--194.

Digital Library

[77]

Ma, Y., Zhou, G., Wang, S., Zhao, H., and Jung, W. Signfi: Sign language recognition using wifi. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 2, 1 (2018), 23.

Digital Library

[78]

Melgarejo, P., Zhang, X., Ramanathan, P., and Chu, D. Leveraging directional antenna capabilities for fine-grained gesture recognition. In Proceedings of the 2014 ACM International Joint Conference on Pervasive and Ubiquitous Computing (2014), ACM, pp. 541--551.

Digital Library

[79]

Michaeli, A. Equivalent edge currents for arbitrary aspects of observation. IEEE Transactions on Antennas and Propagation 32, 3 (1984), 252--258.

[80]

Mitchell, R. E. How many deaf people are there in the united states? estimates from the survey of income and program participation. Journal of deaf studies and deaf education 11, 1 (2005), 112--119.

[81]

Mueller, F., Bernard, F., Sotnychenko, O., Mehta, D., Sridhar, S., Casas, D., and Theobalt, C. Ganerated hands for real-time 3d hand tracking from monocular rgb. In 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition (2018), pp. 49--59.

[82]

Mueller, F., et al. Ganerated hands for real-time 3d hand tracking from monocular rgb. In IEEE CVPR (2018).

[83]

Nandakumar, R., et al. Fingerio: Using active sonar for fine-grained finger tracking. In ACM CHI (2016).

Digital Library

[84]

Nawaz, W., et al. Classification of breast cancer histology images using alexnet. In International conference image analysis and recognition (2018), Springer.

[85]

Nielsen, J. L., et al. Simultaneous and proportional force estimation for multifunction myoelectric prostheses using mirrored bilateral training. IEEE Transactions on Biomedical Engineering (2010).

[86]

Organization, W. H. Deafness and hearing loss. https://www.who.int/news-room/fact-sheets/detail/deafness-andhearing- loss.

[87]

Pan, L., et al. Continuous estimation of finger joint angles under different static wrist motions from semg signals. Biomedical Signal Processing and Control (2014).

[88]

Parate, A., et al. Risq: Recognizing smoking gestures with inertial sensors on a wristband. In ACM MobiSys (2014).

Digital Library

[89]

Parizi, F. S., Whitmire, E., and Patel, S. Auraring: Precise electromagnetic finger tracking. ACM IMWUT (2019).

Digital Library

[90]

Peña Pitarch, E. Virtual human hand: Grasping strategy and simulation. Universitat Politècnica de Catalunya, 2008.

[91]

Peng, Y., Yan, S., and Lu, Z. Transfer learning in biomedical natural language processing: an evaluation of bert and elmo on ten benchmarking datasets. arXiv preprint arXiv:1906.05474 (2019).

[92]

Pu, Q., Gupta, S., Gollakota, S., and Patel, S. Whole-home gesture recognition using wireless signals. In Proceedings of the 19th annual international conference on Mobile computing & networking (2013), ACM, pp. 27--38.

Digital Library

[93]

Qian, K., He, Z., and Zhang, X. 3d point cloud generation with millimeter-wave radar. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 4, 4 (2020), 1--23.

Digital Library

[94]

Qu, C., et al. Bert with history answer embedding for conversational question answering. In ACM SIGIR Conference on Research and Development in Information Retrieval (2019).

Digital Library

[95]

Quivira, F., et al. Translating semg signals to continuous hand poses using recurrent neural networks. In 2018 IEEE EMBS International Conference on Biomedical & Health Informatics (BHI) (2018), IEEE.

[96]

Rao, S. Introduction to mmwave sensing: Fmcw radars. Texas Instruments (TI) mmWave Training Series (2017).

[97]

Raurale, S., et al. Emg acquisition and hand pose classification for bionic hands from randomly-placed sensors. In IEEE ICASSP (2018).

Digital Library

[98]

Regani, S. D., Wu, C., Wang, B., Wu, M., and Liu, K. R. mmwrite: Passive handwriting tracking using a single millimeter wave radio. IEEE Internet of Things Journal (2021).

[99]

Ren, Y., Lu, J., Beletchi, A., Huang, Y., Karmanov, I., Fontijne, D., Patel, C., and Xu, H. Hand gesture recognition using 802.11 ad mmwave sensor in the mobile device. In 2021 IEEE Wireless Communications and Networking Conference Workshops (WCNCW) (2021), IEEE, pp. 1--6.

[100]

Roda-Sales, A., et al. Effect on manual skills of wearing instrumented gloves during manipulation. Journal of biomechanics (2020).

[101]

Santhalingam, P. S., Du, Y., Wilkerson, R., Zhang, D., Pathak, P., Rangwala, H., Kushalnagar, R., et al. Expressive asl recognition using millimeter-wave wireless signals. In 2020 17th Annual IEEE International Conference on Sensing, Communication, and Networking (SECON) (2020), IEEE, pp. 1--9.

Digital Library

[102]

Santhalingam, P. S., Hosain, A. A., Zhang, D., Pathak, P., Rangwala, H., and Kushalnagar, R. mmasl: Environment-independent asl gesture recognition using 60 ghz millimeter-wave signals. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 4, 1 (2020), 1--30.

Digital Library

[103]

Shang, J., and Wu, J. A robust sign language recognition system with multiple wi-fi devices. In Proceedings of the Workshop on Mobility in the Evolving Internet Architecture (2017), ACM, pp. 19--24.

Digital Library

[104]

Sherman, M., et al. User-generated free-form gestures for authentication: Security and memorability. In ACM MobiSys (2014).

Digital Library

[105]

Shi, C., Lu, L., Liu, J., Wang, Y., Chen, Y., and Yu, J. mpose: Environment-and subject-agnostic 3d skeleton posture reconstruction leveraging a single mmwave device. Smart Health (2021), 100228.

[106]

Skidmore, G., Chawla, T., and Bedrosian, G. Combining physical optics and method of equivalent currents to create unique near-field propagation and scattering technique for automotive radar applications. In 2019 IEEE International Conference on Microwaves, Antennas, Communications and Electronic Systems (COMCAS) (2019), IEEE, pp. 1--6.

[107]

Song, J., Sörös, G., Pece, F., Fanello, S. R., Izadi, S., Keskin, C., and Hilliges, O. In-air gestures around unmodified mobile devices. In Proceedings of the 27th annual ACM symposium on User interface software and technology (2014), pp. 319--329.

Digital Library

[108]

Sosin, I., et al. Continuous gesture recognition from semg sensor data with recurrent neural networks and adversarial domain adaptation. In International Conference on Control, Automation, Robotics and Vision (ICARCV) (2018), IEEE.

[109]

Sun, W., Li, F. M., Huang, C., Lei, Z., Steeper, B., Tao, S., Tian, F., and Zhang, C. Thumbtrak: Recognizing micro-finger poses using a ring with proximity sensing. arXiv preprint arXiv:2105.14680 (2021).

[110]

Tomasi, C., Petrov, S., and Sastry, A. 3d tracking= classification interpolation. In ICCV (2003), vol. 3, p. 1441.

[111]

Torres, T. Myo gesture control armband review. https://www.pcmag.com/reviews/myo-gesture-control-armband, 2015.

[112]

Truong, H., et al. Capband: Battery-free successive capacitance sensing wristband for hand gesture recognition. In ACM SenSys (2018).

Digital Library

[113]

Tung, Y.-C., and Shin, K. G. Echotag: Accurate infrastructure-free indoor location tagging with smartphones. In ACM MobiCom (2015).

Digital Library

[114]

Uttner, I., Kraft, E., Nowak, D. A., Müller, F., Philipp, J., Zierdt, A., and Hermsdörfer, J. Mirror movements and the role of handedness: isometric grip forces changes. Motor control 11, 1 (2007).

[115]

Wager, S., Wang, S., and Liang, P. S. Dropout training as adaptive regularization. In Advances in neural information processing systems (2013), pp. 351--359.

Digital Library

[116]

Wang, J., et al. Ubiquitous keyboard for mobile devices: harnessing multipath fading for fine-grained keystroke localization. In ACM MobiCom (2014).

Digital Library

[117]

Wang, S., Song, J., Lien, J., Poupyrev, I., and Hilliges, O. Interacting with soli: Exploring fine-grained dynamic gesture recognition in the radio-frequency spectrum. In Proceedings of the 29th Annual Symposium on User Interface Software and Technology (2016), pp. 851--860.

Digital Library

[118]

Wang, W., Liu, A. X., and Sun, K. Device-free gesture tracking using acoustic signals. Association for Computing Machinery.

[119]

Winkel, J., et al. Significance of skin temperature changes in surface electromyography. European journal of applied physiology and occupational physiology (1991).

[120]

Wu, E., Yuan, Y., Yeo, H.-S., Quigley, A., Koike, H., and Kitani, K. M. Back-hand-pose: 3d hand pose estimation for a wrist-worn camera via dorsum deformation network. In Proceedings of the 33rd Annual ACM Symposium on User Interface Software and Technology (2020), pp. 1147--1160.

Digital Library

[121]

Xia, Z., Luomei, Y., Zhou, C., and Xu, F. Multidimensional feature representation and learning for robust hand-gesture recognition on commercial millimeter-wave radar. IEEE Transactions on Geoscience and Remote Sensing 59, 6 (2020), 4749--4764.

[122]

Xiong, J., and Jamieson, K. Arraytrack: A fine-grained indoor location system. In USENIX NSDI (2013).

Digital Library

[123]

Xue, H., Ju, Y., Miao, C., Wang, Y., Wang, S., Zhang, A., and Su, L. mmmesh: Towards 3d real-time dynamic human mesh construction using millimeter-wave. In Proceedings of the 19th Annual International Conference on Mobile Systems, Applications, and Services (2021), pp. 269--282.

Digital Library

[124]

Zhai, S., Milgram, P., and Buxton,W. The influence of muscle groups on performance of multiple degree-of-freedom input. In Proceedings of the SIGCHI conference on Human factors in computing systems (1996), pp. 308--315.

Digital Library

[125]

Zhang, C., et al. Fingerping: Recognizing fine-grained hand poses using active acoustic on-body sensing. In ACM CHI (2018).

Digital Library

[126]

Zhang, F., Bazarevsky, V., Vakunov, A., Tkachenka, A., Sung, G., Chang, C.-L., and Grundmann, M. Mediapipe hands: On-device real-time hand tracking. arXiv preprint arXiv:2006.10214 (2020).

[127]

Zhang, H., Xu, J., andWang, J. Pretraining-based natural language generation for text summarization. arXiv preprint arXiv:1902.09243 (2019).

[128]

Zhang, Y., and Harrison, C. Tomo:Wearable, low-cost electrical impedance tomography for hand gesture recognition. In Proceedings of the 28th Annual ACM Symposium on User Interface Software & Technology (2015), pp. 167--173.

Digital Library

[129]

Zhang, Z., Tian, Z., Zhou, M., Nie, W., and Li, Z. Riddle: Real-time interacting with hand description via millimeterwave sensor. In 2018 IEEE International Conference on Communications (ICC) (2018), IEEE, pp. 1--6.

[130]

Zhao, M., et al. Through-wall human mesh recovery using radio signals. In IEEE CVPR (2019).

[131]

Zhao, Y., Sark, V., Krstic, M., and Grass, E. Novel approach for gesture recognition using mmwave fmcw radar. In 2022 IEEE 95th Vehicular Technology Conference:(VTC2022-Spring) (2022), IEEE, pp. 1--6.

[132]

Zhou, P., et al. Use it free: Instantly knowing your phone attitude. In ACM MobiCom (2014).

Digital Library

[133]

Zhou, Z., et al. Fine-tuning convolutional neural networks for biomedical image analysis: actively and incrementally. In IEEE CVPR (2017).

Cited By

Ling YZhao DDeng KYin KZheng WMa H(2024)Uranus: Empowering Generalized Gesture Recognition with Mobility through Generating Large-scale mmWave Radar DataProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36997548:4(1-28)Online publication date: 21-Nov-2024
https://dl.acm.org/doi/10.1145/3699754
Stein BChang BSridharan M(2024)Interactive Abstract Interpretation with Demanded SummarizationACM Transactions on Programming Languages and Systems10.1145/364844146:1(1-40)Online publication date: 29-Mar-2024
https://dl.acm.org/doi/10.1145/3648441
Liu HLiu XXie XTong XLi K(2024)PmTrackProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36314337:4(1-30)Online publication date: 12-Jan-2024
https://dl.acm.org/doi/10.1145/3631433
Show More Cited By

Index Terms

Leveraging the Properties of mmWave Signals for 3D Finger Motion Tracking for Interactive IoT Applications
1. Computing methodologies
  1. Machine learning
    1. Machine learning approaches
      1. Neural networks
2. Human-centered computing
  1. Ubiquitous and mobile computing
    1. Ubiquitous and mobile computing design and evaluation methods
    2. Ubiquitous and mobile devices
      1. Mobile devices

Recommendations

Leveraging the Properties of mmWave Signals for 3D Finger Motion Tracking for Interactive IoT Applications
SIGMETRICS '23: Abstract Proceedings of the 2023 ACM SIGMETRICS International Conference on Measurement and Modeling of Computer Systems

Wireless signals, which are mainly used for communication networks, also have the potential to extend our senses, enabling us to see behind closed doors and track moving objects through walls [3, 10]. Accordingly, there is a growing interest in the ...
Leveraging the Properties of mmWave Signals for 3D Finger Motion Tracking for Interactive IoT Applications
SIGMETRICS '23

Wireless signals, which are mainly used for communication networks, also have the potential to extend our senses, enabling us to see behind closed doors and track moving objects through walls [3, 10]. Accordingly, there is a growing interest in the ...
Finger identification and hand gesture recognition techniques for natural user interface
APCHI '13: Proceedings of the 11th Asia Pacific Conference on Computer Human Interaction

The natural user interface using hand gesture have been popular field in Human-Computer-Interaction(HCI). Many research papers have been proposed in this field. They proposed vision-based, glove-based and depth-based approach for hand gesture ...

Comments

Information & Contributors

Information

Published In

cover image Proceedings of the ACM on Measurement and Analysis of Computing Systems

Proceedings of the ACM on Measurement and Analysis of Computing Systems Volume 6, Issue 3

POMACS

December 2022

534 pages

EISSN:2476-1249

DOI:10.1145/3576048

Editors:
Augustin Chaintreau
Columbia University
,
Leana Golubchik
University of Southern California, United States
,
Zhi-Li Zhang
University of Minnesota, United States

Issue’s Table of Contents

Copyright © 2022 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 08 December 2022

Published in POMACS Volume 6, Issue 3

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

National Science Foundation

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

24
Total Citations
View Citations
1,039
Total Downloads

Downloads (Last 12 months)604
Downloads (Last 6 weeks)87

Reflects downloads up to 25 Dec 2024

Other Metrics

View Author Metrics

Citations

Cited By

Ling YZhao DDeng KYin KZheng WMa H(2024)Uranus: Empowering Generalized Gesture Recognition with Mobility through Generating Large-scale mmWave Radar DataProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36997548:4(1-28)Online publication date: 21-Nov-2024
https://dl.acm.org/doi/10.1145/3699754
Stein BChang BSridharan M(2024)Interactive Abstract Interpretation with Demanded SummarizationACM Transactions on Programming Languages and Systems10.1145/364844146:1(1-40)Online publication date: 29-Mar-2024
https://dl.acm.org/doi/10.1145/3648441
Liu HLiu XXie XTong XLi K(2024)PmTrackProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36314337:4(1-30)Online publication date: 12-Jan-2024
https://dl.acm.org/doi/10.1145/3631433
Kyu AMao HZhu JGoel MAhuja K(2024)EITPose: Wearable and Practical Electrical Impedance Tomography for Continuous Hand Pose EstimationProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642663(1-10)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3642663
Chen JQin XZhu GJi BLi B(2024)Motion-Prediction-Based Wireless Scheduling for Interactive Panoramic Scene DeliveryIEEE Transactions on Network Science and Engineering10.1109/TNSE.2023.332542011:2(1566-1579)Online publication date: Mar-2024
https://doi.org/10.1109/TNSE.2023.3325420
Cao JLiu YHan LLi Z(2024)Finger Tracking Using Wrist-Worn EMG SensorsIEEE Transactions on Mobile Computing10.1109/TMC.2024.343901823:12(14099-14110)Online publication date: Dec-2024
https://doi.org/10.1109/TMC.2024.3439018
Lin CSun ZAhmad AFan XWang YWang LFan XWu G(2024)AirWrite: An Aerial Handwriting Trajectory Tracking and Recognition System With mmWaveIEEE Transactions on Mobile Computing10.1109/TMC.2024.342570923:12(13325-13341)Online publication date: Dec-2024
https://doi.org/10.1109/TMC.2024.3425709
Jin CMeng XLi XWang JPan MFang Y(2024)Rodar: Robust Gesture Recognition Based on mmWave Radar Under Human Activity InterferenceIEEE Transactions on Mobile Computing10.1109/TMC.2024.340235623:12(11735-11749)Online publication date: Dec-2024
https://doi.org/10.1109/TMC.2024.3402356
Kong HLyu HYu JKong LYang JRen YLiu HChen Y(2024)mmHand: 3D Hand Pose Estimation Leveraging mmWave Signals2024 IEEE 44th International Conference on Distributed Computing Systems (ICDCS)10.1109/ICDCS60910.2024.00102(1062-1073)Online publication date: 23-Jul-2024
https://doi.org/10.1109/ICDCS60910.2024.00102
Xiao ZChen YZhou XHe MLiu LYu FJiang M(2024)Human action recognition in immersive virtual reality based on multi‐scale spatio‐temporal attention networkComputer Animation and Virtual Worlds10.1002/cav.229335:5Online publication date: 23-Sep-2024
https://doi.org/10.1002/cav.2293
Show More Cited By

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

Media

Figures

Other

Tables

View Issue’s Table of Contents