research-article

MilliBack: Real-Time Plug-n-Play Millimeter Level Tracking Using Wireless Backscattering

Authors:

Hao ZhouAuthors Info & Claims

Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies, Volume 3, Issue 3

Article No.: 112, Pages 1 - 23

https://doi.org/10.1145/3351270

Published: 09 September 2019 Publication History

Abstract

Real-time handwriting tracking is important for many emerging applications such as Artificial Intelligence assisted education and healthcare. Existing movement tracking systems, including those based on vision, ultrasound or wireless technologies, fail to offer high tracking accuracy, no learning/training/calibration process, low tracking latency, low cost and easy to deploy at the same time. In this work, we design and evaluate a wireless backscattering based handwriting tracking system, called MilliBack, that satisfies all these requirements. At the heart of MilliBack are two Phase Differential Iterative (PDI) schemes that can infer the position of the backscatter tag (which is attached to a writing tool) from the change in the signal phase. By adopting carefully-designed differential techniques in an iterative manner, we can take the diversity of devices out of the equation. The resulting position calculation has a linear complexity with the number of samples, ensuring fast and accurate tracking.

We have put together a MilliBack prototype and conducted comprehensive experiments. We show that our system can track various handwriting traces accurately, in some testings it achieve a median error of 4.9 mm. We can accurately track and reconstruct arbitrary writing/drawing trajectories such as equations, Chinese characters or just random shapes. We also show that MilliBack can support relatively high writing speed and smoothly adapt to the changes of working environment.

Supplementary Material

xiao (xiao.zip)

Supplemental movie, appendix, image and software files for, MilliBack: Real-Time Plug-n-Play Millimeter Level Tracking Using Wireless Backscattering

Download
9.79 MB

References

[1]

Heba Abdelnasser, Moustafa Youssef, and Khaled A Harras. 2015. Wigest: A ubiquitous wifi-based gesture recognition system. In Computer Communications (INFOCOM), 2015 IEEE Conference on. IEEE, 1472--1480.

[2]

Fadel Adib, Zachary Kabelac, and Dina Katabi. 2014. Multi-Person Motion Tracking via RF Body Reflections. In MIT CSAIL Technical Report.

[3]

Fadel Adib, Zachary Kabelac, and Dina Katabi. 2015. Multi-Person Localization via RF Body Reflections. In NSDI. 279--292.

Digital Library

[4]

Fadel Adib, Zachary Kabelac, Dina Katabi, and Robert C Miller. 2014. 3D Tracking via Body Radio Reflections. In NSDI, Vol. 14. 317--329.

Digital Library

[5]

Fadel Adib and Dina Katabi. 2013. See Through Walls with Wi-Fi!. In ACM SIGCOMM, Vol. 43. 75--86.

Digital Library

[6]

Kamran Ali, Alex X Liu, Wei Wang, and Muhammad Shahzad. 2015. Keystroke recognition using wifi signals. In Proceedings of the 21st Annual International Conference on Mobile Computing and Networking. ACM, 90--102.

Digital Library

[7]

Apollonius's theorem. https://en.wikipedia.org/wiki/Apollonius's_theorem.

[8]

Liqiong Chang, Jie Xiong, Ju Wang, Xiaojiang Chen, Yu Wang, Zhanyong Tang, and Dingyi Fang. 2018. RF-Copybook: A Millimeter Level Calligraphy Copybook based on commodity RFID. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 1, 4 (2018), 128.

Digital Library

[9]

Chirp-Z Transform. https://en.wikipedia.org/wiki/Chirp_Z-transform.

[10]

Gabe Cohn, Daniel Morris, Shwetak Patel, and Desney Tan. 2012. Humantenna: using the body as an antenna for real-time whole-body interaction. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM, 1901--1910.

Digital Library

[11]

Cuneiform. https://en.wikipedia.org/wiki/Cuneiform_script.

[12]

Andrea Giovanni Cutti, Andrea Giovanardi, Laura Rocchi, Angelo Davalli, and Rinaldo Sacchetti. 2008. Ambulatory measurement of shoulder and elbow kinematics through inertial and magnetic sensors. Medical & biological engineering & computing 46, 2 (2008), 169--178.

[13]

Mahmoud El-Gohary and James McNames. 2012. Shoulder and elbow joint angle tracking with inertial sensors. IEEE Transactions on Biomedical Engineering 59, 9 (2012), 2635--2641.

[14]

Google project soli. https://www.google.com/atap/project-soli/.

[15]

Sidhant Gupta, Daniel Morris, Shwetak Patel, and Desney Tan. 2012. Soundwave: using the doppler effect to sense gestures. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM, 1911--1914.

Digital Library

[16]

Chris Harrison, Desney Tan, and Dan Morris. 2010. Skinput: appropriating the body as an input surface. In Proceedings of the SIGCHI conference on human factors in computing systems. ACM, 453--462.

Digital Library

[17]

Daniel Hauschildt and Nicolaj Kirchhof. 2010. Advances in thermal infrared localization: Challenges and solutions. In Indoor Positioning and Indoor Navigation (IPIN), 2010 International Conference on. IEEE, 1--8.

[18]

Ngoc-Huynh Ho, Phuc Truong, and Gu-Min Jeong. 2016. Step-detection and adaptive step-length estimation for pedestrian dead-reckoning at various walking speeds using a smartphone. Sensors 16, 9 (2016), 1423.

[19]

Horion. https://www.horion.com.

[20]

Zhenhua Jia, Musaab Alaziz, Xiang Chi, Richard E Howard, Yanyong Zhang, Pei Zhang, Wade Trappe, Anand Sivasubramaniam, and Ning An. 2016. HB-phone: a bed-mounted geophone-based heartbeat monitoring system. In Proceedings of the 15th International Conference on Information Processing in Sensor Networks. IEEE Press, 22.

Digital Library

[21]

Wonho Kang and Youngnam Han. 2015. SmartPDR: Smartphone-based pedestrian dead reckoning for indoor localization. IEEE Sensors journal 15, 5 (2015), 2906--2916.

[22]

Bryce Kellogg, Vamsi Talla, Shyamnath Gollakota, and Joshua R Smith. 2016. Passive Wi-Fi: Bringing Low Power to Wi-Fi Transmissions. In NSDI, Vol. 16. 151--164.

Digital Library

[23]

Jürgen Kemper and Daniel Hauschildt. 2010. Passive infrared localization with a probability hypothesis density filter. In Positioning Navigation and Communication (WPNC), 2010 7th Workshop on. IEEE, 68--76.

[24]

David Kim, Otmar Hilliges, Shahram Izadi, Alex D Butler, Jiawen Chen, Iason Oikonomidis, and Patrick Olivier. 2012. Digits: freehand 3D interactions anywhere using a wrist-worn gloveless sensor. In Proceedings of the 25th annual ACM symposium on User interface software and technology. ACM, 167--176.

Digital Library

[25]

Manikanta Kotaru, Kiran Joshi, Dinesh Bharadia, and Sachin Katti. 2015. Spotfi: Decimeter level localization using wifi. In ACM SIGCOMM Computer Communication Review, Vol. 45. ACM, 269--282.

Digital Library

[26]

Manikanta Kotaru, Pengyu Zhang, and Sachin Katti. 2017. Localizing Low-power Backscatter Tags Using Commodity WiFi. In Proceedings of the 13th International Conference on emerging Networking EXperiments and Technologies. ACM, 251--262.

Digital Library

[27]

Leap Motiont. https://www.leapmotion.com/.

[28]

Tianxing Li, Chuankai An, Zhao Tian, Andrew T Campbell, and Xia Zhou. 2015. Human sensing using visible light communication. In Proceedings of the 21st Annual International Conference on Mobile Computing and Networking. ACM, 331--344.

Digital Library

[29]

Xiang Li, Daqing Zhang, Qin Lv, Jie Xiong, Shengjie Li, Yue Zhang, and Hong Mei. 2017. IndoTrack: Device-free indoor human tracking with commodity Wi-Fi. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 1, 3 (2017), 72.

Digital Library

[30]

Hongbo Liu, Yu Gan, Jie Yang, Simon Sidhom, Yan Wang, Yingying Chen, and Fan Ye. 2012. Push the limit of WiFi based localization for smartphones. In Proceedings of the 18th annual international conference on Mobile computing and networking. ACM, 305--316.

Digital Library

[31]

Jia Liu, Min Chen, Shigang Chen, Qingfeng Pan, and Lijun Chen. 2017. Tag-compass: Determining the spatial direction of an object with small dimensions. In IEEE INFOCOM 2017-IEEE Conference on Computer Communications. IEEE, 1--9.

[32]

Vincent Liu, Aaron Parks, Vamsi Talla, Shyamnath Gollakota, David Wetherall, and Joshua R Smith. 2013. Ambient backscatter: wireless communication out of thin air. In ACM SIGCOMM Computer Communication Review, Vol. 43. ACM, 39--50.

Digital Library

[33]

Jullawadee Maneesilp, Chong Wang, Hongyi Wu, and Nian-Feng Tzeng. 2012. RFID support for accurate 3D localization. IEEE Trans. Comput. 62, 7 (2012), 1447--1459.

Digital Library

[34]

Microsoft Kinect. http://www.microsoft.com/en-us/kinectforwindows/.

[35]

Words per minute. https://en.wikipedia.org/wiki/Words_per_minute.

[36]

John Kangchun Perng, Brian Fisher, Seth Hollar, and Kristofer SJ Pister. 1999. Acceleration sensing glove (ASG). In Digest of Papers. Third International Symposium on Wearable Computers. IEEE, 178--180.

Digital Library

[37]

Kun Qian, Chenshu Wu, Zheng Yang, Yunhao Liu, and Kyle Jamieson. 2017. Widar: Decimeter-level passive tracking via velocity monitoring with commodity Wi-Fi. In Proceedings of the 18th ACM International Symposium on Mobile Ad Hoc Networking and Computing. ACM, 6.

Digital Library

[38]

Dong Ren, Yizhuo Zhang, Ning Xiao, Hao Zhou, Xiangyang Li, Jianwei Qian, and Panlong Yang. 2018. Word-Fi: Accurate handwrite system empowered by wireless backscattering and machine learning. IEEE Network 32, 4 (2018), 47--53.

[39]

Sheng Shen, Mahanth Gowda, and Romit Roy Choudhury. 2018. Closing the Gaps in Inertial Motion Tracking. In Proceedings of the 24th Annual International Conference on Mobile Computing and Networking. ACM, 429--444.

Digital Library

[40]

Jamie Shotton, Toby Sharp, Alex Kipman, Andrew Fitzgibbon, Mark Finocchio, Andrew Blake, Mat Cook, and Richard Moore. 2013. Real-Time Human Pose Recognition in Parts from Single Depth Images. Commun. ACM 56, 1 (2013), 116--124.

Digital Library

[41]

Elahe Soltanaghaei, Avinash Kalyanaraman, and Kamin Whitehouse. 2018. Multipath triangulation: Decimeter-level wifi localization and orientation with a single unaided receiver. In Proceedings of the 16th Annual International Conference on Mobile Systems, Applications, and Services. ACM, 376--388.

Digital Library

[42]

Li Sun, Souvik Sen, Dimitrios Koutsonikolas, and Kyu-Han Kim. 2015. Widraw: Enabling hands-free drawing in the air on commodity wifi devices. In Proceedings of the 21st Annual International Conference on Mobile Computing and Networking. ACM, 77--89.

Digital Library

[43]

Zheng Sun, Aveek Purohit, Kaifei Chen, Shijia Pan, Trevor Pering, and Pei Zhang. 2011. PANDAA: physical arrangement detection of networked devices through ambient-sound awareness. In Proceedings of the 13th international conference on Ubiquitous computing. ACM, 425--434.

Digital Library

[44]

Sheng Tan and Jie Yang. 2016. WiFinger: leveraging commodity WiFi for fine-grained finger gesture recognition. In Proceedings of the 17th ACM International Symposium on Mobile Ad Hoc Networking and Computing. ACM, 201--210.

Digital Library

[45]

Chuyu Wang, Jian Liu, Yingying Chen, Hongbo Liu, Lei Xie, Wei Wang, Bingbing He, and Sanglu Lu. 2018. Multi-Touch in the Air: Device-Free Finger Tracking and Gesture Recognition via COTS RFID. In IEEE INFOCOM 2018-IEEE Conference on Computer Communications. IEEE, 1691--1699.

Digital Library

[46]

He Wang, Souvik Sen, Ahmed Elgohary, Moustafa Farid, Moustafa Youssef, and Romit Roy Choudhury. 2012. No need to war-drive: Unsupervised indoor localization. In Proceedings of the 10th international conference on Mobile systems, applications, and services. ACM, 197--210.

Digital Library

[47]

Jue Wang, Deepak Vasisht, and Dina Katabi. 2014. RF-IDraw: virtual touch screen in the air using RF signals. In ACM SIGCOMM Computer Communication Review, Vol. 44. ACM, 235--246.

Digital Library

[48]

Ju Wang, Jie Xiong, Hongbo Jiang, Xiaojiang Chen, and Dingyi Fang. 2017. D-Watch: Embracing "Bad" multipaths for device-free localization with COTS RFID devices. IEEE/ACM Transactions on Networking (TON) 25, 6 (2017), 3559--3572.

Digital Library

[49]

Lei Wang, Ke Sun, Haipeng Dai, Alex X Liu, and Xiaoyu Wang. 2018. WiTrace: Centimeter-Level Passive Gesture Tracking Using WiFi Signals. In 2018 15th Annual IEEE International Conference on Sensing, Communication, and Networking (SECON). IEEE, 1--9.

[50]

Teng Wei and Xinyu Zhang. 2015. mtrack: High-precision passive tracking using millimeter wave radios. In Proceedings of the 21st Annual International Conference on Mobile Computing and Networking. ACM, 117--129.

Digital Library

[51]

Teng Wei and Xinyu Zhang. 2016. Gyro in the air: tracking 3D orientation of batteryless internet-of-things. In Proceedings of the 22nd Annual International Conference on Mobile Computing and Networking. ACM, 55--68.

Digital Library

[52]

Ning Xiao, Panlong Yang, Yubo Yan, Hao Zhou, and Xiang-Yang Li. 2018. Motion-Fi: Recognizing and Counting Repetitive Motions with Passive Wireless Backscattering. In IEEE INFOCOM 2018-IEEE Conference on Computer Communications. IEEE, 2024--2032.

[53]

Jie Xiong and Kyle Jamieson. 2013. Arraytrack: A fine-grained indoor location system. In Presented as part of the 10th {USENIX} Symposium on Networked Systems Design and Implementation ({NSDI} 13). 71--84.

Digital Library

[54]

Lei Yang, Yekui Chen, Xiang-Yang Li, Chaowei Xiao, Mo Li, and Yunhao Liu. 2014. Tagoram: Real-time tracking of mobile RFID tags to high precision using COTS devices. In Proceedings of the 20th annual international conference on Mobile computing and networking. ACM, 237--248.

Digital Library

[55]

Lei Yang, Qiongzheng Lin, Xiangyang Li, Tianci Liu, and Yunhao Liu. 2015. See through walls with cots rfid system!. In Proceedings of the 21st Annual International Conference on Mobile Computing and Networking. ACM, 487--499.

Digital Library

[56]

Nan Yu, Wei Wang, Alex X Liu, and Lingtao Kong. 2018. QGesture: Quantifying Gesture Distance and Direction with WiFi Signals. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 2, 1 (2018), 51.

Digital Library

[57]

Sangki Yun, Yi-Chao Chen, and Lili Qiu. 2015. Turning a mobile device into a mouse in the air. In Proceedings of the 13th Annual International Conference on Mobile Systems, Applications, and Services. ACM, 15--29.

Digital Library

[58]

Yunze Zeng, Parth H Pathak, and Prasant Mohapatra. 2016. WiWho: wifi-based person identification in smart spaces. In Proceedings of the 15th International Conference on Information Processing in Sensor Networks. IEEE Press, 4.

Digital Library

[59]

Lan Zhang, Kebin Liu, Yonghang Jiang, Xiang-Yang Li, Yunhao Liu, Panlong Yang, and Zhenhua Li. 2017. Montage: Combine frames with movement continuity for realtime multi-user tracking. IEEE Transactions on Mobile Computing 16, 4 (2017), 1019--1031.

Digital Library

[60]

Yunting Zhang, Jiliang Wang, Weiyi Wang, Zhao Wang, and Yunhao Liu. 2018. Vernier: Accurate and Fast Acoustic Motion Tracking Using Mobile Devices. In IEEE INFOCOM 2018-IEEE Conference on Computer Communications. IEEE, 1709--1717.

[61]

Huiyu Zhou and Huosheng Hu. 2007. Upper limb motion estimation from inertial measurements. International Journal of Information Technology 13, 1 (2007), 1--14.

[62]

Huiyu Zhou, Huosheng Hu, and Yaqin Tao. 2006. Inertial measurements of upper limb motion. Medical and Biological Engineering and Computing 44, 6 (2006), 479--487.

Cited By

Liu SMao ZLi XPan MLiu PHuang YYou X(2025)Model-Driven Deep Neural Network for Enhancing Direction Finding with Commodity 5G gNodeBACM Transactions on Sensor Networks10.1145/3712305Online publication date: 14-Jan-2025
https://doi.org/10.1145/3712305
Mrs. S. Ponnarasi Mr. A. Saravanan Mr. R. Sathish Mr. S. R. Siva Sarathi (2024)Deep Sea Fisherman’s App using WWSNInternational Journal of Advanced Research in Science, Communication and Technology10.48175/IJARSCT-18033(190-193)Online publication date: 3-May-2024
https://doi.org/10.48175/IJARSCT-18033
Ma JZhang FJin BSu CLi SWang ZNi J(2024)Push the Limit of Highly Accurate Ranging on Commercial UWB DevicesProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36596028:2(1-27)Online publication date: 15-May-2024
https://dl.acm.org/doi/10.1145/3659602
Show More Cited By

Index Terms

MilliBack: Real-Time Plug-n-Play Millimeter Level Tracking Using Wireless Backscattering
1. Human-centered computing
  1. Human computer interaction (HCI)
  2. Ubiquitous and mobile computing

Recommendations

Advancements in 3D interactive devices for virtual environments
EGVE '03: Proceedings of the workshop on Virtual environments 2003

New commercially available interactive 3D tracking devices and systems for use in virtual environments are discussed. InterSense originally introduced the IS-900 scalable-area hybrid tracking system for virtual environments in 1999. In response to ...
Catadioptric Stereo Using Planar Mirrors

By using mirror reflections of a scene, stereo images can be captured with a single camera (catadioptric stereo). In addition to simplifying data acquisition single camera stereo provides both geometric and radiometric advantages over traditional two ...
Omnistereo: Panoramic Stereo Imaging

An Omnistereo panorama consists of a pair of panoramic images, where one panorama is for the left eye and another panorama is for the right eye. The panoramic stereo pair provides a stereo sensation up to a full 360 degrees. Omnistereo panoramas cannot ...

Comments

Information & Contributors

Information

Published In

cover image Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies

Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies Volume 3, Issue 3

September 2019

1415 pages

EISSN:2474-9567

DOI:10.1145/3361560

Issue’s Table of Contents

Copyright © 2019 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 09 September 2019

Published in IMWUT Volume 3, Issue 3

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed limited

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

21
Total Citations
View Citations
754
Total Downloads

Downloads (Last 12 months)34
Downloads (Last 6 weeks)4

Reflects downloads up to 03 Feb 2025

Other Metrics

View Author Metrics

Citations

Cited By

Liu SMao ZLi XPan MLiu PHuang YYou X(2025)Model-Driven Deep Neural Network for Enhancing Direction Finding with Commodity 5G gNodeBACM Transactions on Sensor Networks10.1145/3712305Online publication date: 14-Jan-2025
https://doi.org/10.1145/3712305
Mrs. S. Ponnarasi Mr. A. Saravanan Mr. R. Sathish Mr. S. R. Siva Sarathi (2024)Deep Sea Fisherman’s App using WWSNInternational Journal of Advanced Research in Science, Communication and Technology10.48175/IJARSCT-18033(190-193)Online publication date: 3-May-2024
https://doi.org/10.48175/IJARSCT-18033
Ma JZhang FJin BSu CLi SWang ZNi J(2024)Push the Limit of Highly Accurate Ranging on Commercial UWB DevicesProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36596028:2(1-27)Online publication date: 15-May-2024
https://dl.acm.org/doi/10.1145/3659602
Cao YDhekne AAmmar MOkoshi TKo JLiKamWa R(2024)UTrack3D: 3D Tracking Using Ultra-wideband (UWB) RadiosProceedings of the 22nd Annual International Conference on Mobile Systems, Applications and Services10.1145/3643832.3661881(345-358)Online publication date: 3-Jun-2024
https://dl.acm.org/doi/10.1145/3643832.3661881
Xia SXing TWu CLiu GYang JLi K(2023)AQMon: A Fine-grained Air Quality Monitoring System Based on UAV Images for Smart CitiesACM Transactions on Sensor Networks10.1145/363876620:2(1-20)Online publication date: 29-Dec-2023
https://dl.acm.org/doi/10.1145/3638766
Luo ZYan MWang WZhang Q(2023)Non-intrusive Anomaly Detection of Industrial Robot Operations by Exploiting Nonlinear EffectProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/35694776:4(1-27)Online publication date: 11-Jan-2023
https://dl.acm.org/doi/10.1145/3569477
Wang LWan HZhao TSun KShi SDai HChen GLiu HWang W(2023)SCALAR: Self-Calibrated Acoustic Ranging for Distributed Mobile DevicesIEEE Transactions on Mobile Computing10.1109/TMC.2023.3241304(1-15)Online publication date: 2023
https://doi.org/10.1109/TMC.2023.3241304
Li XYin MZhang YYang PWan CGuo XTan H(2023)Back-Guard: Wireless Backscattering Based User Sensing With Parallel Attention ModelIEEE Transactions on Mobile Computing10.1109/TMC.2022.321501222:12(7466-7481)Online publication date: Dec-2023
https://doi.org/10.1109/TMC.2022.3215012
Han ZLu ZWen XGuo LZhao J(2023)Towards 3D Centimeter-Level Passive Gesture Tracking With Two WiFi LinksIEEE Transactions on Mobile Computing10.1109/TMC.2021.312369422:5(3031-3045)Online publication date: 1-May-2023
https://doi.org/10.1109/TMC.2021.3123694
Han ZLu ZChen YWen XJiang J(2023)MagicInput: Virtual Handwriting Interface Using Ubiquitous WiFi SignalsIEEE Transactions on Industrial Informatics10.1109/TII.2022.322887219:9(9515-9524)Online publication date: Sep-2023
https://doi.org/10.1109/TII.2022.3228872
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Issue’s Table of Contents