research-article

Approaching the Real-World: Supporting Activity Recognition Training with Virtual IMU Data

Authors:

Hyeokhyen Kwon,

Gregory D. Abowd,

Thomas PlötzAuthors Info & Claims

Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies, Volume 5, Issue 3

Article No.: 111, Pages 1 - 32

https://doi.org/10.1145/3478096

Published: 14 September 2021 Publication History

Abstract

Recently, IMUTube introduced a paradigm change for bootstrapping human activity recognition (HAR) systems for wearables. The key idea is to utilize videos of activities to support training activity recognizers based on inertial measurement units (IMUs). This system retrieves video from public repositories and subsequently generates virtual IMU data from this. The ultimate vision for such a system is to make large amounts of weakly labeled videos accessible for model training in HAR and, as such, to overcome one of the most pressing issues in the field: the lack of significant amounts of labeled sample data. In this paper we present the first in-detail exploration of IMUTube in a realistic assessment scenario: the analysis of free-weight gym exercises. We make significant progress towards a flexible, fully-functional IMUTube system by extending it such that it can handle a range of artifacts that are common in unrestricted online videos, including various forms of video noise, non-human poses, body part occlusions, and extreme camera and human motion. By overcoming these real-world challenges, we are able to generate high-quality virtual IMU data, which allows us to employ IMUTube for practical analysis tasks. We show that HAR systems trained by incorporating virtual sensor data generated by IMUTube significantly outperform baseline models trained only with real IMU data. In doing so we demonstrate the practical utility of IMUTube and the progress made towards the final vision of the new bootstrapping paradigm.

References

[1]

M. Bächlin, M. Plotnik, and G. Tröster. 2010. Wearable assistant for Parkinson's disease patients with the freezing of gait symptom. IEEE Trans. Inf. Technol. Biomed. 14, 2 (2010), 436--446.

Digital Library

[2]

G. Batista, E. J. Keogh, O. M. Tataw, and V. De Souza. 2014. CID: an efficient complexity-invariant distance for time series. Data Mining and Knowledge Discovery 28, 3 (2014), 634--669.

Digital Library

[3]

A. Bewley, Z. Ge, L. Ott, F. Ramos, and B. Upcroft. 2016. Simple online and realtime tracking. In IEEE International Conference on Image Processing (ICIP). 3464--3468.

[4]

O. Bogdan, V. Eckstein, F. Rameau, and J. Bazin. 2018. DeepCalib: a deep learning approach for automatic intrinsic calibration of wide field-of-view cameras. In Proceedings of the ACM SIGGRAPH European Conference on Visual Media Production. ACM, 6:1-6:10.

[5]

A. Bulling, U. Blanke, and B. Schiele. 2014. A tutorial on human activity recognition using body-worn inertial sensors. ACM CSUR 46, 3 (2014), 33.

Digital Library

[6]

Z. Cao, G. Hidalgo Martinez, T. Simon, S. Wei, and Y. A. Sheikh. 2019. OpenPose: Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields. IEEE Transactions on Pattern Analysis and Machine Intelligence (2019).

[7]

J. Carreira, P. Agrawal, K. Fragkiadaki, and J. Malik. 2016. Human pose estimation with iterative error feedback. In Proceedings of the IEEE conference on computer vision and pattern recognition. 4733--4742.

[8]

Larry Chan, Vedant Das Swain, Christina Kelley, Kaya de Barbaro, Gregory D. Abowd, and Lauren Wilcox. 2018. Students' Experiences with Ecological Momentary Assessment Tools to Report on Emotional Well-Being. IMWUT 2, 1, Article 3 (March 2018), 20 pages.

[9]

R. Chavarriaga, H. Sagha, and D. Roggen. 2013. The Opportunity challenge: A benchmark database for on-body sensor-based activity recognition. Pattern Recognition Letter 34, 15 (2013), 2033--2042.

Digital Library

[10]

M. Christ, A. W. Kempa-Liehr, and M. Feindt. 2016. Distributed and parallel time series feature extraction for industrial big data applications. https://tsfresh.readthedocs.io. arXiv preprint arXiv:1610.07717 (2016).

[11]

Blender Online Community. 2018. Blender - a 3D modelling and rendering package. Blender Foundation, Stichting Blender Foundation, Amsterdam. http://www.blender.org

[12]

W. Conover and R. Iman. 1981. Rank transformations as a bridge between parametric and nonparametric statistics. The American Statistician 35, 3 (1981), 124--129.

[13]

S. Consolvo and M. Walker. 2003. Using the experience sampling method to evaluate ubicomp applications. IEEE Pervasive Computing 2, 2 (2003), 24--31. https://doi.org/10.1109/MPRV.2003.1203750

Digital Library

[14]

Jifeng Dai, Kaiming He, Yi Li, Shaoqing Ren, and Jian Sun. 2016. Instance-sensitive fully convolutional networks. In European Conference on Computer Vision. Springer, 534--549.

[15]

Jifeng Dai, Kaiming He, and Jian Sun. 2015. Convolutional feature masking for joint object and stuff segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition. 3992--4000.

[16]

Alexey Dosovitskiy, Philipp Fischer, Eddy Ilg, Philip Hausser, Caner Hazirbas, Vladimir Golkov, Patrick Van Der Smagt, Daniel Cremers, and Thomas Brox. 2015. Flownet: Learning optical flow with convolutional networks. In Proceedings of the IEEE international conference on computer vision. 2758--2766.

Digital Library

[17]

Hao-Shu Fang, Shuqin Xie, Yu-Wing Tai, and Cewu Lu. 2017. RMPE: Regional Multi-person Pose Estimation. In ICCV.

[18]

H. Fawaz, G. Forestier, J. Weber, L. Idoumghar, and P. Muller. 2018. Data augmentation using synthetic data for time series classification with deep residual networks. arXiv preprint arXiv:1808.02455 (2018).

[19]

D. Figo, P. C. Diniz, D. R. Ferreira, and J. M. Cardoso. 2010. Preprocessing techniques for context recognition from accelerometer data. Pers. Ubiquitous Comput. 14, 7 (2010), 645--662.

Digital Library

[20]

Vitor Fortes Rey, Kamalveer Kaur Garewal, and Paul Lukowicz. 2020. Yet it moves: Learning from Generic Motions to Generate IMU data from YouTube videos. arXiv e-prints (2020), arXiv-2011.

[21]

Andreas Geiger, Philip Lenz, and Raquel Urtasun. 2012. Are we ready for Autonomous Driving? The KITTI Vision Benchmark Suite. In Conference on Computer Vision and Pattern Recognition (CVPR).

Digital Library

[22]

Ross Girshick, Forrest Iandola, Trevor Darrell, and Jitendra Malik. 2015. Deformable part models are convolutional neural networks. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition. 437--446.

[23]

Ke Gong, Yiming Gao, Xiaodan Liang, Xiaohui Shen, Meng Wang, and Liang Lin. 2019. Graphonomy: Universal human parsing via graph transfer learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 7450--7459.

[24]

Ke Gong, Xiaodan Liang, Yicheng Li, Yimin Chen, Ming Yang, and Liang Lin. 2018. Instance-level human parsing via part grouping network. In Proceedings of the European Conference on Computer Vision (ECCV). 770--785.

Digital Library

[25]

I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio. 2014. Generative adversarial networks. arXiv preprint arXiv:1406.2661 (2014).

[26]

A. Gordon, H. Li, R. Jonschkowski, and A. Angelova. 2019. Depth From Videos in the Wild: Unsupervised Monocular Depth Learning From Unknown Cameras. In IEEE International Conference on Computer Vision (ICCV). IEEE.

[27]

Tobias Gruber, Mario Bijelic, Felix Heide, Werner Ritter, and Klaus Dietmayer. 2019. Pixel-accurate depth evaluation in realistic driving scenarios. In 2019 International Conference on 3D Vision (3DV). IEEE, 95--105.

[28]

Shuosen Guan, Haoxin Li, and Wei-Shi Zheng. 2019. Unsupervised learning for optical flow estimation using pyramid convolution lstm. In 2019 IEEE International Conference on Multimedia and Expo (ICME). IEEE, 181--186.

[29]

John K Haas. 2014. A history of the unity game engine. (2014).

[30]

N. Hammerla, R. Kirkham, P. Andras, and T. Ploetz. 2013. On preserving statistical characteristics of accelerometry data using their empirical cumulative distribution. In Proceedings of the ACM International Symposium on Wearable Computers. 65--68.

[31]

S. Haradal, H. Hayashi, and S. Uchida. 2018. Biosignal data augmentation based on generative adversarial networks. In Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC). IEEE, 368--371.

[32]

Bharath Hariharan, Pablo Arbeláez, Ross Girshick, and Jitendra Malik. 2014. Simultaneous detection and segmentation. In European Conference on Computer Vision. Springer, 297--312.

[33]

Bharath Hariharan, Pablo Arbeláez, Ross Girshick, and Jitendra Malik. 2015. Hypercolumns for object segmentation and fine-grained localization. In Proceedings of the IEEE conference on computer vision and pattern recognition. 447--456.

[34]

R. I. Hartley and A. Zisserman. 2004. Multiple View Geometry in Computer Vision (second ed.). Cambridge University Press, ISBN: 0521540518.

[35]

Haoyu He, Jing Zhang, Qiming Zhang, and Dacheng Tao. 2020. Grapy-ML: Graph Pyramid Mutual Learning for Cross-dataset Human Parsing. In Proceedings of the AAAI Conference on Artificial Intelligence.

[36]

Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. 2017. Mask r-cnn. In Proceedings of the IEEE international conference on computer vision. 2961--2969.

[37]

K. He, X. Zhang, S. Ren, and J. Sun. 2016. Deep residual learning for image recognition. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 770--778.

[38]

GV Horn and Pietro Perona. 2017. The devil is in the tails: Fine-grained classification in the wild. arXiv preprint arXiv:1709.01450 2 (2017).

[39]

H. Huang, Y. Sun, and M. Liu. 2019. Reliable Monocular Ego-Motion Estimation System in Rainy Urban Environments. In 2019 IEEE Intelligent Transportation Systems Conference (ITSC). 1290--1297. https://doi.org/10.1109/ITSC.2019.8916977

[40]

Junhwa Hur and Stefan Roth. 2019. Iterative residual refinement for joint optical flow and occlusion estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 5754--5763.

[41]

Tae Hyun Kim, Kyoung Mu Lee, Bernhard Scholkopf, and Michael Hirsch. 2017. Online video deblurring via dynamic temporal blending network. In Proceedings of the IEEE International Conference on Computer Vision. 4038--4047.

[42]

Eddy Ilg, Nikolaus Mayer, Tonmoy Saikia, Margret Keuper, Alexey Dosovitskiy, and Thomas Brox. 2017. Flownet 2.0: Evolution of optical flow estimation with deep networks. In Proceedings of the IEEE conference on computer vision and pattern recognition. 2462--2470.

[43]

Joel Janai, Fatma Guney, Anurag Ranjan, Michael Black, and Andreas Geiger. 2018. Unsupervised learning of multi-frame optical flow with occlusions. In Proceedings of the European Conference on Computer Vision (ECCV). 690--706.

[44]

J Yu Jason, Adam W Harley, and Konstantinos G Derpanis. 2016. Back to basics: Unsupervised learning of optical flow via brightness constancy and motion smoothness. In European Conference on Computer Vision. Springer, 3--10.

[45]

I. Joel, A.and Stergios. 2011. A Direct Least-Squares (DLS) method for PnP. In IEEE International Conference on Computer Vision (ICCV). IEEE.

[46]

C. Kang, H. Jung, and Y. Lee. 2019. Towards Machine Learning with Zero Real-World Data. In The ACM Workshop on Wearable Systems and Applications. 41--46.

[47]

H. Koskimäki, P. Siirtola, and J. Röning. 2017. MyoGym: Introducing an Open Gym Data Set for Activity Recognition Collected Using Myo Armband. In Proceedings of the 2017 ACM International Joint Conference on Pervasive and Ubiquitous Computing and Proceedings of the 2017 ACM International Symposium on Wearable Computers. ACM, 537--546.

[48]

H. Koskimäki and P. Siirtola. 2014. Recognizing gym exercises using acceleration data from wearable sensors. In 2014 IEEE Symposium on Computational Intelligence and Data Mining (CIDM). 321--328. https://doi.org/10.1109/CIDM.2014.7008685

[49]

Hyeokhyen Kwon, Gregory D. Abowd, and Thomas Plötz. 2019. Handling Annotation Uncertainty in Human Activity Recognition. In Proceedings of the 23rd International Symposium on Wearable Computers (ISWC '19). ACM, 109--117.

Digital Library

[50]

H. Kwon, C. Tong, H. Haresamudram, Y. Gao, G. D. Abowd, N. D. Lane, and T. Plötz. 2020. IMUTube: Automatic Extraction of Virtual on-Body Accelerometry from Video for Human Activity Recognition. Proc. ACM Interact. Mob. Wearable Ubiquitous Technol. 4, 3, Article 87 (Sept. 2020), 29 pages.

[51]

Carnegie Mellon Graphics Lab. 2008. Carnegie Mellon Motion Capture Database. http://mocap.cs.cmu.edu/

[52]

O. D. Lara and M. A Labrador. 2012. A survey on human activity recognition using wearable sensors. IEEE communications surveys & tutorials 15, 3 (2012), 1192--1209.

[53]

A. Le Guennec, S. Malinowski, and R. Tavenard. 2016. Data Augmentation for Time Series Classification using Convolutional Neural Networks. In ECML/PKDD Workshop on Advanced Analytics and Learning on Temporal Data.

[54]

X. Li, J. Luo, and R. Younes. 2020. ActivityGAN: Generative Adversarial Networks for Data Augmentation in Sensor-Based Human Activity Recognition (UbiComp-ISWC '20). ACM, 249--254.

[55]

Yi Li, Haozhi Qi, Jifeng Dai, Xiangyang Ji, and Yichen Wei. 2017. Fully convolutional instance-aware semantic segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition. 2359--2367.

[56]

D. Liaqat, M. Abdalla, Pegah Abed-Esfahani, Moshe Gabel, Tatiana Son, Robert Wu, Andrea Gershon, Frank Rudzicz, and Eyal De Lara. 2019. WearBreathing: Real World Respiratory Rate Monitoring Using Smartwatches. Proceedings of the ACM on interactive, mobile, wearable and ubiquitous technologies (IMWUT) 3, 2 (2019), 1--22.

Digital Library

[57]

Liang Liu, Jiangning Zhang, Ruifei He, Yong Liu, Yabiao Wang, Ying Tai, Donghao Luo, Chengjie Wang, Jilin Li, and Feiyue Huang. 2020. Learning by Analogy: Reliable Supervision from Transformations for Unsupervised Optical Flow Estimation. In IEEE Conference on Computer Vision and Pattern Recognition(CVPR).

[58]

Pengpeng Liu, Irwin King, Michael R Lyu, and Jia Xu. 2019. Ddflow: Learning optical flow with unlabeled data distillation. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 33. 8770--8777.

Digital Library

[59]

Pengpeng Liu, Michael Lyu, Irwin King, and Jia Xu. 2019. Selflow: Self-supervised learning of optical flow. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 4571--4580.

[60]

Shu Liu, Jiaya Jia, Sanja Fidler, and Raquel Urtasun. 2017. Sgn: Sequential grouping networks for instance segmentation. In Proceedings of the IEEE International Conference on Computer Vision. 3496--3504.

[61]

Ziwei Liu, Zhongqi Miao, Xiaohang Zhan, Jiayun Wang, Boqing Gong, and Stella X Yu. 2019. Large-scale long-tailed recognition in an open world. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2537--2546.

[62]

N. Mahmood, N. Ghorbani, N. Troje, G. Pons-Moll, and M. Black. 2019. AMASS: Archive of motion capture as surface shapes. In IEEE International Conference on Computer Vision (ICCV). 5442--5451.

[63]

Xiao-Jiao Mao, Chunhua Shen, and Yu-Bin Yang. 2016. Image restoration using very deep convolutional encoder-decoder networks with symmetric skip connections. arXiv preprint arXiv:1603.09056 (2016).

[64]

J. Martinez, R. Hossain, J. Romero, and J. Little. 2017. A simple yet effective baseline for 3d human pose estimation. In IEEE International Conference on Computer Vision (ICCV). 2640--2649.

[65]

A. Mathur, T. Zhang, S. Bhattacharya, P. Velickovic, L.Joffe, N. Lane, F. Kawsar, and P. Lió. 2018. Using deep data augmentation training to address software and hardware heterogeneities in wearable and smartphone sensing devices. In IEEE International Conference on Information Processing in Sensor Networks (IPSN). IEEE, 200--211.

[66]

Nikolaus Mayer, Eddy Ilg, Philip Hausser, Philipp Fischer, Daniel Cremers, Alexey Dosovitskiy, and Thomas Brox. 2016. A large dataset to train convolutional networks for disparity, optical flow, and scene flow estimation. In Proceedings of the IEEE conference on computer vision and pattern recognition. 4040--4048.

[67]

D. Mehta, O. Sotnychenko, F. Mueller, W. Xu, M. Elgharib, P. Fua, H. Seidel, H. Rhodin, G. Pons-Moll, and C. Theobalt. 2020. XNect: Real-Time Multi-Person 3D Motion Capture with a Single RGB Camera. ToG. 39, 4, Article 82 (July 2020), 17 pages.

[68]

Simon Meister, Junhwa Hur, and Stefan Roth. 2018. Unflow: Unsupervised learning of optical flow with a bidirectional census loss. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 32.

[69]

Dan Morris, T Scott Saponas, Andrew Guillory, and Ilya Kelner. 2014. RecoFit: using a wearable sensor to find, recognize, and count repetitive exercises. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. 3225--3234.

Digital Library

[70]

Saypraseuth Mounsaveng, David Vazquez, Ismail Ben Ayed, and Marco Pedersoli. 2019. Adversarial learning of general transformations for data augmentation. arXiv preprint arXiv:1909.09801 (2019).

[71]

A. Newell, K. Yang, and J. Deng. 2016. Stacked hourglass networks for human pose estimation. In European conference on computer vision. Springer, 483--499.

[72]

Xuecheng Nie, Jiashi Feng, and Shuicheng Yan. 2018. Mutual Learning to Adapt for Joint Human Parsing and Pose Estimation. In Proceedings of the European Conference on Computer Vision (ECCV).

[73]

F. J. Ordóñez and D. Roggen. 2016. Deep convolutional and lstm recurrent neural networks for multimodal wearable activity recognition. Sensors 16, 1 (2016), 115.

[74]

Jinshan Pan, Haoran Bai, and Jinhui Tang. 2020. Cascaded Deep Video Deblurring Using Temporal Sharpness Prior. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[75]

George Papandreou, Tyler Zhu, Liang-Chieh Chen, Spyros Gidaris, Jonathan Tompson, and Kevin Murphy. 2018. Personlab: Person pose estimation and instance segmentation with a bottom-up, part-based, geometric embedding model. In Proceedings of the European Conference on Computer Vision (ECCV). 269--286.

Digital Library

[76]

J. Park, Q. Zhou, and V. Koltun. 2017. Colored Point Cloud Registration Revisited. In IEEE International Conference on Computer Vision (ICCV). 143--152.

[77]

Deepak Pathak, Philipp Krahenbuhl, Jeff Donahue, Trevor Darrell, and Alexei A Efros. 2016. Context encoders: Feature learning by inpainting. In Proceedings of the IEEE conference on computer vision and pattern recognition. 2536--2544.

[78]

D. Pavllo, C. Feichtenhofer, D. Grangier, and M. Auli. 2019. 3D human pose estimation in video with temporal convolutions and semi-supervised training. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 7753--7762.

[79]

L. Pei, S. Xia, L. Chu, F. Xiao, Q. Wu, W. Yu, and R. Qiu. 2021. MARS: Mixed Virtual and Real Wearable Sensors for Human Activity Recognition with Multi-Domain Deep Learning Model. IEEE Internet of Things Journal (2021).

[80]

L. Pishchulin, E. Insafutdinov, S. Tang, B. Andres, M. Andriluka, P. Gehler, and B. Schiele. 2016. Deepcut: Joint subset partition and labeling for multi person pose estimation. In Proceedings of the IEEE conference on computer vision and pattern recognition. 4929--4937.

[81]

Joseph Redmon and Ali Farhadi. 2018. YOLOv3: An Incremental Improvement. arXiv (2018).

[82]

A. Reiss and D. Stricker. 2012. Introducing a new benchmarked dataset for activity monitoring. In Proceedings of the ACM International Symposium on Wearable Computers. IEEE, 108--109.

[83]

V. Rey, P. Hevesi, O. Kovalenko, and P. Lukowicz. 2019. Let there be IMU data: generating training data for wearable, motion sensor based activity recognition from monocular RGB videos. In Adjunct Proceedings of the ACM International Joint Conference on Pervasive and Ubiquitous Computing and Proceedings of the ACM International Symposium on Wearable Computers. 699--708.

[84]

Daniel J. Rough and Aaron Quigley. 2020. End-User Development of Experience Sampling Smartphone Apps-Recommendations and Requirements. IMWUT 4, 2, Article 56 (June 2020), 19 pages.

[85]

P. M. Scholl, M. Wille, and K. Van Laerhoven. 2015. Wearables in the wet lab: a laboratory system for capturing and guiding experiments. In Proceedings of the International Conference on Ubiquitous Computing. ACM, 589--599.

[86]

T. Schreiber and A. Schmitz. 1997. Discrimination power of measures for nonlinearity in a time series. Physical Review E 55, 5 (1997), 5443.

[87]

K. Shoemake. 1985. Animating rotation with quaternion curves. In Proceedings of the 12th annual conference on Computer graphics and interactive techniques. 245--254.

Digital Library

[88]

C. Shorten and T. Khoshgoftaar. 2019. A survey on image data augmentation for deep learning. Journal of Big Data 6, 1 (2019), 60.

[89]

K. Simonyan and A. Zisserman. 2014. Very deep convolutional networks for large-scale image recognition. arXiv:1409.1556 (2014).

[90]

D. Strömbäck, S. Huang, and V. Radu. 2020. MM-Fit: Multimodal Deep Learning for Automatic Exercise Logging across Sensing Devices. IMWUT 4, 4, Article 168 (Dec. 2020), 22 pages.

[91]

J. Sturm, N. Engelhard, F. Endres, W. Burgard, and D. Cremers. 2012. A Benchmark for the Evaluation of RGB-D SLAM Systems. In Proc. of the International Conference on Intelligent Robot Systems (IROS).

[92]

S. Suh, H. Lee, J. Jo, P. Lukowicz, and Y. Lee. 2019. Generative oversampling method for imbalanced data on bearing fault detection and diagnosis. Applied Sciences 9, 4 (2019), 746.

[93]

Deqing Sun, Xiaodong Yang, Ming-Yu Liu, and Jan Kautz. 2018. Pwc-net: Cnns for optical flow using pyramid, warping, and cost volume. In Proceedings of the IEEE conference on computer vision and pattern recognition. 8934--8943.

[94]

T. Sztyler and H. Stuckenschmidt. 2016. On-body localization of wearable devices: An investigation of position-aware activity recognition. In IEEE International Conference on Pervasive Computing and Communications (PerCom). IEEE, 1--9.

[95]

S. Takeda, T. Okita, P. Lago, and S. Inoue. 2018. A multi-sensor setting activity recognition simulation tool. In Proceedings of the ACM International Joint Conference and International Symposium on Pervasive and Ubiquitous Computing and Wearable Computers. 1444--1448.

[96]

Subarna Tripathi, Maxwell Collins, Matthew Brown, and Serge Belongie. 2017. Pose2instance: Harnessing keypoints for person instance segmentation. arXiv preprint arXiv:1704.01152 (2017).

[97]

T. Um, F. Pfister, D. Pichler, S. Endo, M. Lang, S. Hirche, U. Fietzek, and D. Kulić. 2017. Data augmentation of wearable sensor data for parkinson's disease monitoring using convolutional neural networks. In Proceedings of the ACM International Conference on Multimodal Interaction. 216--220.

[98]

Terry Taewoong Um, Vahid Babakeshizadeh, and Dana Kulić. 2017. Exercise motion classification from large-scale wearable sensor data using convolutional neural networks. In 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE, 2385--2390.

Digital Library

[99]

Niels van Berkel, Jorge Goncalves, Simo Hosio, and Vassilis Kostakos. 2017. Gamification of Mobile Experience Sampling Improves Data Quality and Quantity. IMWUT 1, 3, Article 107 (Sept. 2017), 21 pages.

[100]

Madhu Vankadari, Sourav Garg, Anima Majumder, Swagat Kumar, and Ardhendu Behera. 2020. Unsupervised monocular depth estimation for night-time images using adversarial domain feature adaptation. In European Conference on Computer Vision. Springer, 443--459.

Digital Library

[101]

A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin. 2017. Attention is all you need. arXiv:1706.03762 (2017).

[102]

Xintao Wang, Kelvin CK Chan, Ke Yu, Chao Dong, and Chen Change Loy. 2019. Edvr: Video restoration with enhanced deformable convolutional networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops. 0--0.

[103]

Yang Wang, Yi Yang, Zhenheng Yang, Liang Zhao, Peng Wang, and Wei Xu. 2018. Occlusion aware unsupervised learning of optical flow. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 4884--4893.

[104]

E. Wilson. 1927. Probable inference, the law of succession, and statistical inference. J. Amer. Statist. Assoc. 22, 158 (1927), 209--212.

[105]

Chaowei Xiao, Jun-Yan Zhu, Bo Li, Warren He, Mingyan Liu, and Dawn Song. 2018. Spatially transformed adversarial examples. arXiv preprint arXiv:1801.02612 (2018).

[106]

F. Xiao, L. Pei, L. Chu, D. Zou, W. Yu, Y. Zhu, and T. Li. 2020. A Deep Learning Method for Complex Human Activity Recognition Using Virtual Wearable Sensors. arXiv preprint arXiv:2003.01874 (2020).

[107]

Tete Xiao, Yingcheng Liu, Bolei Zhou, Yuning Jiang, and Jian Sun. 2018. Unified perceptual parsing for scene understanding. In Proceedings of the European Conference on Computer Vision (ECCV). 418--434.

Digital Library

[108]

Qizhe Xie, Zihang Dai, Eduard Hovy, Minh-Thang Luong, and Quoc V Le. 2019. Unsupervised data augmentation for consistency training. arXiv preprint arXiv:1904.12848 (2019).

[109]

Angela Yao, Dominique Uebersax, Juergen Gall, and Luc Van Gool. 2010. Tracking people in broadcast sports. In Joint Pattern Recognition Symposium. Springer, 151--161.

[110]

S. Yao, Y. Zhao, H. Shao, C. Zhang, A. Zhang, S. Hu, D. Liu, S. Liu, Lu Su, and T. Abdelzaher. 2018. Sensegan: Enabling deep learning for internet of things with a semi-supervised framework. Proceedings of the ACM on interactive, mobile, wearable and ubiquitous technologies (IMWUT) 2, 3 (2018), 1--21.

[111]

Zhichao Yin and Jianping Shi. 2018. Geonet: Unsupervised learning of dense depth, optical flow and camera pose. In Proceedings of the IEEE conference on computer vision and pattern recognition. 1983--1992.

[112]

A. Young, M. Ling, and D. Arvind. 2011. IMUSim: A simulation environment for inertial sensing algorithm design and evaluation. In Proceedings of the International Conference on Information Processing in Sensor Networks (IPSN). IEEE, 199--210.

[113]

Kaihao Zhang, Wenhan Luo, Yiran Zhong, Lin Ma, Wei Liu, and Hongdong Li. 2018. Adversarial spatio-temporal learning for video deblurring. IEEE Transactions on Image Processing 28, 1 (2018), 291--301.

Digital Library

[114]

M. Zhang and A. A. Sawchuk. 2012. USC-HAD: a daily activity dataset for ubiquitous activity recognition using wearable sensors. In Proceedings of the International Conference on Ubiquitous Computing.

[115]

Song-Hai Zhang, Ruilong Li, Xin Dong, Paul Rosin, Zixi Cai, Xi Han, Dingcheng Yang, Haozhi Huang, and Shi-Min Hu. 2019. Pose2seg: Detection free human instance segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 889--898.

[116]

Wang Zhao, Shaohui Liu, Yezhi Shu, and Yong-Jin Liu. 2020. Towards Better Generalization: Joint Depth-Pose Learning without PoseNet. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[117]

Kaiyang Zhou, Yongxin Yang, Andrea Cavallaro, and Tao Xiang. 2019. Omni-Scale Feature Learning for Person Re-Identification. In ICCV.

[118]

Bingke Zhu, Yingying Chen, Ming Tang, and Jinqiao Wang. 2018. Progressive cognitive human parsing. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 32.

[119]

Yuliang Zou, Zelun Luo, and Jia-Bin Huang. 2018. Df-net: Unsupervised joint learning of depth and flow using cross-task consistency. In Proceedings of the European conference on computer vision (ECCV). 36--53.

Cited By

Leng ZBhattacharjee ARajasekhar HZhang LBruda EKwon HPlötz T(2024)IMUGPT 2.0: Language-Based Cross Modality Transfer for Sensor-Based Human Activity RecognitionProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36785458:3(1-32)Online publication date: 9-Sep-2024
https://dl.acm.org/doi/10.1145/3678545
Hwang SLeng ZOh SKim KPlötz TKostakos VKay JHoang T(2024)More Data for People with Disabilities! Comparing Data Collection Efforts for Wheelchair Transportation Mode DetectionProceedings of the 2024 ACM International Symposium on Wearable Computers10.1145/3675095.3676617(82-88)Online publication date: 5-Oct-2024
https://dl.acm.org/doi/10.1145/3675095.3676617
Leng ZJung MHwang SOh SZhang LPlötz TKim KKostakos VKay JHoang T(2024)Emotion Recognition on the Go: Utilizing Wearable IMUs for Personalized Emotion RecognitionCompanion of the 2024 on ACM International Joint Conference on Pervasive and Ubiquitous Computing10.1145/3675094.3678452(537-544)Online publication date: 5-Oct-2024
https://dl.acm.org/doi/10.1145/3675094.3678452
Show More Cited By

Index Terms

Approaching the Real-World: Supporting Activity Recognition Training with Virtual IMU Data
1. Computing methodologies
  1. Artificial intelligence
    1. Computer vision
  2. Machine learning
    1. Learning paradigms
      1. Supervised learning
2. Human-centered computing
  1. Ubiquitous and mobile computing
    1. Ubiquitous and mobile computing systems and tools

Recommendations

Let there be IMU data: generating training data for wearable, motion sensor based activity recognition from monocular RGB videos
UbiComp/ISWC '19 Adjunct: Adjunct Proceedings of the 2019 ACM International Joint Conference on Pervasive and Ubiquitous Computing and Proceedings of the 2019 ACM International Symposium on Wearable Computers

Recent advances in Machine Learning, in particular Deep Learning have been driving rapid progress in fields such as computer vision and natural language processing. Human activity recognition (HAR) using wearable sensors, which has been a thriving ...
Towards global aerobic activity monitoring
PETRA '11: Proceedings of the 4th International Conference on PErvasive Technologies Related to Assistive Environments

With recent progress in wearable sensing it becomes reasonable for individuals to wear different sensors all day, thus global activity monitoring is establishing. The goals in global activity monitoring systems are amongst others to tell the type of ...
Real world activity summary for senior home monitoring
ICME '11: Proceedings of the 2011 IEEE International Conference on Multimedia and Expo

From a senior person's daily activities, one can tell a lot about the health condition of the senior person. Thus we believe that senior home activity analysis will play an important role in the health care of senior people. Toward this goal, we propose ...

Comments

Information & Contributors

Information

Published In

cover image Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies

Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies Volume 5, Issue 3

Sept 2021

1443 pages

EISSN:2474-9567

DOI:10.1145/3486621

Issue’s Table of Contents

Copyright © 2021 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 14 September 2021

Published in IMWUT Volume 5, Issue 3

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

20
Total Citations
View Citations
1,207
Total Downloads

Downloads (Last 12 months)357
Downloads (Last 6 weeks)31

Reflects downloads up to 22 Sep 2024

Other Metrics

View Author Metrics

Citations

Cited By

Leng ZBhattacharjee ARajasekhar HZhang LBruda EKwon HPlötz T(2024)IMUGPT 2.0: Language-Based Cross Modality Transfer for Sensor-Based Human Activity RecognitionProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36785458:3(1-32)Online publication date: 9-Sep-2024
https://dl.acm.org/doi/10.1145/3678545
Hwang SLeng ZOh SKim KPlötz TKostakos VKay JHoang T(2024)More Data for People with Disabilities! Comparing Data Collection Efforts for Wheelchair Transportation Mode DetectionProceedings of the 2024 ACM International Symposium on Wearable Computers10.1145/3675095.3676617(82-88)Online publication date: 5-Oct-2024
https://dl.acm.org/doi/10.1145/3675095.3676617
Leng ZJung MHwang SOh SZhang LPlötz TKim KKostakos VKay JHoang T(2024)Emotion Recognition on the Go: Utilizing Wearable IMUs for Personalized Emotion RecognitionCompanion of the 2024 on ACM International Joint Conference on Pervasive and Ubiquitous Computing10.1145/3675094.3678452(537-544)Online publication date: 5-Oct-2024
https://dl.acm.org/doi/10.1145/3675094.3678452
Deng KZhao DZhang ZWang SZheng WMa H(2024)Midas++: Generating Training Data of mmWave Radars From Videos for Privacy-Preserving Human Sensing With MobilityIEEE Transactions on Mobile Computing10.1109/TMC.2023.332539923:6(6650-6666)Online publication date: Jun-2024
https://doi.org/10.1109/TMC.2023.3325399
Singh DRay LZhou BSuh SLukowicz P(2024)A Novel Local-Global Feature Fusion Framework for Body-Weight Exercise Recognition with Pressure Mapping SensorsICASSP 2024 - 2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)10.1109/ICASSP48485.2024.10447226(6375-6379)Online publication date: 14-Apr-2024
https://doi.org/10.1109/ICASSP48485.2024.10447226
Uhlenberg LDerungs AAmft O(2023)Co-simulation of human digital twins and wearable inertial sensors to analyse gait event estimationFrontiers in Bioengineering and Biotechnology10.3389/fbioe.2023.110400011Online publication date: 12-Apr-2023
https://doi.org/10.3389/fbioe.2023.1104000
Li JHuang LShah SJones SJin YWang DRussell AChoi SGao YYuan JJin Z(2023)SignRingProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36108817:3(1-29)Online publication date: 27-Sep-2023
https://dl.acm.org/doi/10.1145/3610881
Leng ZJain YKwon HPloetz T(2023)On the Utility of Virtual On-body Acceleration Data for Fine-grained Human Activity RecognitionProceedings of the 2023 ACM International Symposium on Wearable Computers10.1145/3594738.3611364(55-59)Online publication date: 8-Oct-2023
https://dl.acm.org/doi/10.1145/3594738.3611364
Xue HCao QMiao CJu YHu HZhang ASu LCosta XAl Hassanieh HAsadi ACox LPerino DWidmer JGiustiniano D(2023)Towards Generalized mmWave-based Human Pose Estimation through Signal AugmentationProceedings of the 29th Annual International Conference on Mobile Computing and Networking10.1145/3570361.3613302(1-15)Online publication date: 2-Oct-2023
https://dl.acm.org/doi/10.1145/3570361.3613302
Xu HZhou PTan RLi MCosta XAl Hassanieh HAsadi ACox LPerino DWidmer JGiustiniano D(2023)Practically Adopting Human Activity RecognitionProceedings of the 29th Annual International Conference on Mobile Computing and Networking10.1145/3570361.3613299(1-15)Online publication date: 2-Oct-2023
https://dl.acm.org/doi/10.1145/3570361.3613299
Show More Cited By

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Issue’s Table of Contents