research-article

Open access

IMUGPT 2.0: Language-Based Cross Modality Transfer for Sensor-Based Human Activity Recognition

Authors:

Amitrajit Bhattacharjee,

Hrudhai Rajasekhar,

Elizabeth Bruda,

Hyeokhyen Kwon,

Thomas PlötzAuthors Info & Claims

Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies, Volume 8, Issue 3

Article No.: 112, Pages 1 - 32

https://doi.org/10.1145/3678545

Published: 09 September 2024 Publication History

Abstract

One of the primary challenges in the field of human activity recognition (HAR) is the lack of large labeled datasets. This hinders the development of robust and generalizable models. Recently, cross modality transfer approaches have been explored that can alleviate the problem of data scarcity. These approaches convert existing datasets from a source modality, such as video, to a target modality, such as inertial measurement units (IMUs). With the emergence of generative AI models such as large language models (LLMs) and text-driven motion synthesis models, language has become a promising source data modality as well - as shown in proof of concepts such as IMUGPT. In this work, we conduct a large-scale evaluation of language-based cross modality transfer to determine their effectiveness for HAR. Based on this study, we introduce two new extensions for IMUGPT that enhance its use for practical HAR application scenarios: a motion filter capable of filtering out irrelevant motion sequences to ensure the relevance of the generated virtual IMU data, and a set of metrics that measure the diversity of the generated data facilitating the determination of when to stop generating virtual IMU data for both effective and efficient processing. We demonstrate that our diversity metrics can reduce the effort needed for the generation of virtual IMU data by at least 50%, which opens up IMUGPT for practical use cases beyond a mere proof of concept.

References

[1]

2021. sentence-transformers/all-mpnet-base-v2. https://huggingface.co/sentence-transformers/all-mpnet-base-v2 (2024, Feb 1).

[2]

2022. GPT-3.5. https://platform.openai.com/docs/models/gpt-3-5 (2024, Feb 1).

[3]

2023. REDUCELRONPLATEAU. https://pytorch.org/docs/stable/generated/torch.optim.lr_scheduler.ReduceLROnPlateau.html (2024, Feb 1).

[4]

Karan Ahuja, Yue Jiang, Mayank Goel, and Chris Harrison. 2021. Vid2Doppler: Synthesizing Doppler radar data from videos for training privacy-preserving activity recognition. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems. 1--10.

Digital Library

[5]

Rohan Anil, Andrew M. Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, Eric Chu, Jonathan H. Clark, Laurent El Shafey, Yanping Huang, Kathy Meier-Hellstern, et al. 2023. PaLM 2 Technical Report. arXiv:2305.10403 [cs.CL]

[6]

Matthias Bächlin, Meir Plotnik, Daniel Roggen, Nir Giladi, Jeffrey M Hausdorff, and Gerhard Tröster. 2010. Wearable assistant for Parkinson's disease patients with the freezing of gait symptom. IEEE Transactions on Information Technology in Biomedicine 14, 2 (2010), 436--446. https://doi.org/10.1109/TITB.2009.2036165

Digital Library

[7]

Lei Bai, Lina Yao, Xianzhi Wang, Salil S. Kanhere, Bin Guo, and Zhiwen Yu. 2020. Adversarial Multi-view Networks for Activity Recognition. Proc. ACM Interact. Mob. Wearable Ubiquitous Technol. 4, 2, Article 42 (jun 2020), 22 pages. https://doi.org/10.1145/3397323

Digital Library

[8]

Lei Bai, Lina Yao, Xianzhi Wang, Salil S. Kanhere, and Yang Xiao. 2020. Prototype Similarity Learning for Activity Recognition. In Advances in Knowledge Discovery and Data Mining, Hady W. Lauw, Raymond Chi-Wing Wong, Alexandros Ntoulas, Ee-Peng Lim, See-Kiong Ng, and Sinno Jialin Pan (Eds.). Springer International Publishing, Cham, 649--661.

[9]

Dmitrijs Balabka. 2019. Semi-supervised learning for human activity recognition using adversarial autoencoders. In Adjunct Proceedings of the 2019 ACM International Joint Conference on Pervasive and Ubiquitous Computing and Proceedings of the 2019 ACM International Symposium on Wearable Computers (London, United Kingdom) (UbiComp/ISWC '19 Adjunct). Association for Computing Machinery, New York, NY, USA, 685--688. https://doi.org/10.1145/3341162.3344854

Digital Library

[10]

Sejal Bhalla, Mayank Goel, and Rushil Khurana. 2021. IMU2Doppler: Cross-Modal Domain Adaptation for Doppler-based Activity Recognition Using IMU Data. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 5, 4 (2021), 1--20.

Digital Library

[11]

Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, et al. 2020. Language Models are Few-Shot Learners. In Advances in Neural Information Processing Systems, Vol. 33. Curran Associates, Inc., 1877--1901.

[12]

Ricardo Chavarriaga, Hesam Sagha, Alberto Calatroni, Sundara Tejaswi Digumarti, Gerhard Tröster, José del R Millán, and Daniel Roggen. 2013. The opportunity challenge: a benchmark database for on-body sensor-based activity recognition. Pattern Recognition Letters 34, 15 (2013), 2033--2042. https://doi.org/10.1016/j.patrec.2012.12.014

Digital Library

[13]

Wenqiang Chen, Shupei Lin, Elizabeth Thompson, and John Stankovic. 2021. SenseCollect: We Need Efficient Ways to Collect On-body Sensor-based Human Activity Data! Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 5, 3 (2021), 1--27.

Digital Library

[14]

Dongzhou Cheng, Lei Zhang, Can Bu, Xing Wang, Hao Wu, and Aiguo Song. 2023. ProtoHAR: Prototype Guided Personalized Federated Learning for Human Activity Recognition. IEEE Journal of Biomedical and Health Informatics 27, 8 (2023), 3900--3911. https://doi.org/10.1109/JBHI.2023.3275438

[15]

L. Cilliers. 2020. Wearable devices in healthcare: Privacy and information security issues. Health information management journal 49, 2--3 (2020), 150--156.

[16]

Richard O. Duda, Peter E. Hart, and David G. Stork. 2000. Pattern Classification (2nd Edition) (2 ed.). Wiley-Interscience.

Digital Library

[17]

Floyd Els and Liezel Cilliers. 2017. Improving the information security of personal electronic health records to protect a patient's health information. In 2017 Conference on Information Communication Technology and Society (ICTAS). 1--6. https://doi.org/10.1109/ICTAS.2017.7920658

[18]

Siwei Feng and Marco F. Duarte. 2019. Few-shot learning-based human activity recognition. Expert Systems with Applications 138 (2019), 112782. https://doi.org/10.1016/j.eswa.2019.06.070

[19]

Arthur Gretton, Karsten M. Borgwardt, Malte J. Rasch, Bernhard Schölkopf, and Alexander Smola. 2012. A Kernel Two-Sample Test. Journal of Machine Learning Research 13, 25 (2012), 723--773. http://jmlr.org/papers/v13/gretton12a.html

Digital Library

[20]

Chuan Guo, Shihao Zou, Xinxin Zuo, Sen Wang, Wei Ji, Xingyu Li, and Li Cheng. 2022. Generating Diverse and Natural 3D Human Motions From Text. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 5152--5161.

[21]

Chuan Guo, Xinxin Zuo, Sen Wang, and Li Cheng. 2022. TM2T: Stochastic and Tokenized Modeling for the Reciprocal Generation of 3D Human Motions and Texts. In ECCV.

[22]

N. Y. Hammerla, R. Kirkham, P. Andras, and T. Ploetz. 2013. On preserving statistical characteristics of accelerometry data using their empirical cumulative distribution. In Proceedings of the 2013 international symposium on wearable computers. 65--68.

[23]

Harish Haresamudram, Apoorva Beedu, Varun Agrawal, Patrick L. Grady, Irfan Essa, Judy Hoffman, and Thomas Plötz. 2020. Masked reconstruction based self-supervision for human activity recognition. In Proceedings of the 2020 ACM International Symposium on Wearable Computers (Virtual Event, Mexico) (ISWC '20). Association for Computing Machinery, New York, NY, USA, 45--49. https://doi.org/10.1145/3410531.3414306

Digital Library

[24]

Harish Haresamudram, Irfan Essa, and Thomas Plötz. 2021. Contrastive Predictive Coding for Human Activity Recognition. 5, 2, Article 65 (jun 2021), 26 pages. https://doi.org/10.1145/3463506

Digital Library

[25]

Harish Haresamudram, Irfan Essa, and Thomas Plötz. 2022. Assessing the state of self-supervised human activity recognition using wearables. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 6, 3 (2022), 1--47.

Digital Library

[26]

Biao Jiang, Xin Chen, Wen Liu, Jingyi Yu, Gang Yu, and Tao Chen. 2023. MotionGPT: Human Motion as a Foreign Language. arXiv preprint arXiv:2306.14795 (2023).

[27]

D. Jiang and G. Shi. 2021. Research on data security and privacy protection of wearable equipment in healthcare. Journal of Healthcare Engineering 2021 (2021).

[28]

Takeshi Kojima, Shixiang (Shane) Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. 2022. Large Language Models are Zero-Shot Reasoners. In Advances in Neural Information Processing Systems, S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (Eds.), Vol. 35. Curran Associates, Inc., 22199--22213. https://proceedings.neurips.cc/paper_files/paper/2022/file/8bb0d291acd4acf06ef112099c16f326-Paper-Conference.pdf

[29]

Heli Koskimäki, Pekka Siirtola, and Juha Röning. 2017. MyoGym: Introducing an Open Gym Data Set for Activity Recognition Collected Using Myo Armband. In Proceedings of the 2017 ACM International Joint Conference on Pervasive and Ubiquitous Computing and Proceedings of the 2017 ACM International Symposium on Wearable Computers. Association for Computing Machinery, New York, NY, USA. https://doi.org/10.1145/3123024.3124400

Digital Library

[30]

Taku Kudo and John Richardson. 2018. SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing. arXiv:1808.06226 [cs.CL]

[31]

Hyeokhyen Kwon, Gregory D Abowd, and Thomas Plötz. 2019. Handling annotation uncertainty in human activity recognition. In Proceedings of the 23rd International Symposium on Wearable Computers. 109--117.

Digital Library

[32]

Hyeokhyen Kwon, Gregory D Abowd, and Thomas Plötz. 2021. Complex Deep Neural Networks from Large Scale Virtual IMU Data for Effective Human Activity Recognition Using Wearables. Sensors 21, 24 (2021), 8337.

[33]

Hyeokhyen Kwon, Catherine Tong, Harish Haresamudram, Yan Gao, Gregory D Abowd, Nicholas D Lane, and Thomas Ploetz. 2020. IMUTube: Automatic extraction of virtual on-body accelerometry from video for human activity recognition. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 4, 3 (2020), 1--29.

Digital Library

[34]

Hyeokhyen Kwon, Bingyao Wang, Gregory D Abowd, and Thomas Plötz. 2021. Approaching the Real-World: Supporting Activity Recognition Training with Virtual IMU Data. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 5, 3 (2021), 1--32.

Digital Library

[35]

Julia Lahn, Heiko Peter, and Peter Braun. 2015. Car Crash Detection on Smartphones (iWOAR '15). Association for Computing Machinery, New York, NY, USA. https://doi.org/10.1145/2790044.2790049

Digital Library

[36]

Yi-An Lai, Xuan Zhu, Yi Zhang, and Mona Diab. [n. d.]. Diversity, Density, and Homogeneity: Quantitative Characteristic Metrics for Text Collections. ([n. d.]).

[37]

Clayton Frederick Souza Leite and Yu Xiao. 2020. Improving Cross-Subject Activity Recognition via Adversarial Learning. IEEE Access 8 (2020), 90542--90554. https://doi.org/10.1109/ACCESS.2020.2993818

[38]

Zikang Leng, Yash Jain, Hyeokhyen Kwon, and Thomas Ploetz. 2023. On the Utility of Virtual On-body Acceleration Data for Fine-grained Human Activity Recognition. In Proceedings of the 2023 ACM International Symposium on Wearable Computers (ISWC '23). Association for Computing Machinery, New York, NY, USA. https://doi.org/10.1145/3594738.3611364

Digital Library

[39]

Zikang Leng, Hyeokhyen Kwon, and Thomas Ploetz. 2023. Generating Virtual On-Body Accelerometer Data from Virtual Textual Descriptions for Human Activity Recognition. In Proceedings of the 2023 ACM International Symposium on Wearable Computers. Association for Computing Machinery, New York, NY, USA. https://doi.org/10.1145/3594738.3611361

Digital Library

[40]

Jiyang Li, Lin Huang, Siddharth Shah, Sean J. Jones, Yincheng Jin, Dingran Wang, Adam Russell, Seokmin Choi, Yang Gao, Junsong Yuan, and Zhanpeng Jin. 2023. SignRing: Continuous American Sign Language Recognition Using IMU Rings and Virtual IMU Data. Proc. ACM Interact. Mob. Wearable Ubiquitous Technol. 7, 3, Article 107 (sep 2023), 29 pages. https://doi.org/10.1145/3610881

Digital Library

[41]

Dawei Liang, Guihong Li, Rebecca Adaimi, Radu Marculescu, and Edison Thomaz. 2022. AudioIMU: Enhancing Inertial Sensing-Based Activity Recognition with Acoustic Models. In Proceedings of the 2022 ACM International Symposium on Wearable Computers. 44--48.

Digital Library

[42]

Daniyal Liaqat, Mohamed Abdalla, Pegah Abed-Esfahani, Moshe Gabel, Tatiana Son, Robert Wu, Andrea Gershon, Frank Rudzicz, and Eyal De Lara. 2019. WearBreathing: Real World Respiratory Rate Monitoring Using Smartwatches. Proc. ACM Interact. Mob. Wearable Ubiquitous Technol. 3, 2, Article 56 (jun 2019), 22 pages. https://doi.org/10.1145/3328927

Digital Library

[43]

Jing Lin, Ailing Zeng, Shunlin Lu, Yuanhao Cai, Ruimao Zhang, Haoqian Wang, and Lei Zhang. 2023. Motion-X: A Large-scale 3D Expressive Whole-body Human Motion Dataset. Advances in Neural Information Processing Systems (2023).

[44]

Jiawei Liu, Xiaohu Li, Shanshan Huang, Rui Chao, Zhidong Cao, Shu Wang, Aiguo Wang, and Li Liu. 2023. A review of wearable sensors based fall-related recognition systems. Engineering Applications of Artificial Intelligence 121 (2023), 105993. https://doi.org/10.1016/j.engappai.2023.105993

Digital Library

[45]

Yilin Liu, Fengyang Jiang, and Mahanth Gowda. 2020. Finger Gesture Tracking for Interactive Applications: A Pilot Study with Sign Languages. 4, 3 (2020). https://doi.org/10.1145/3414117

Digital Library

[46]

Yilin Liu, Shijia Zhang, and Mahanth Gowda. 2021. When Video Meets Inertial Sensors: Zero-Shot Domain Adaptation for Finger Motion Analytics with Inertial Sensors. In Proceedings of the International Conference on Internet-of-Things Design and Implementation (Charlottesvle, VA, USA) (IoTDI '21). Association for Computing Machinery, New York, NY, USA, 182--194. https://doi.org/10.1145/3450268.3453537

Digital Library

[47]

Matthew Loper, Naureen Mahmood, Javier Romero, Gerard Pons-Moll, and Michael J. Black. 2015. SMPL: A Skinned Multi-Person Linear Model. ACM Trans. Graph. 34, 6, Article 248 (nov 2015), 16 pages. https://doi.org/10.1145/2816795.2818013

Digital Library

[48]

MinYen Lu, ChenHao Chen, Shigemi Ishida, Yugo Nakamura, and Yutaka Arakawa. 2022. A study on estimating the accurate head IMU motion from Video. Proceedings of the Symposium on Multimedia, Distributed, Cooperative, and Mobile (DICOMO) 2022 2022 (07 2022), 918--923. https://cir.nii.ac.jp/crid/1050011771467456512

[49]

David Martin, Zikang Leng, Tan Gemicioglu, Jon Womack, Jocelyn Heath, William C Neubauer, Hyeokhyen Kwon, Thomas Ploetz, and Thad Starner. 2023. FingerSpeller: Camera-Free Text Entry Using Smart Rings for American Sign Language Fingerspelling Recognition. In Proceedings of the 25th International ACM SIGACCESS Conference on Computers and Accessibility. Association for Computing Machinery, New York, NY, USA. https://doi.org/10.1145/3597638.3614491

Digital Library

[50]

Sara Mohammed, Reda Elbasiony, and Walid Gomaa. 2018. An LSTM-based Descriptor for Human Activities Recognition using IMU Sensors. 504--511. https://doi.org/10.5220/0006902405040511

[51]

F. Mohd-Yasin, C.E. Korman, and D.J. Nagel. 2001. Measurement of noise characteristics of MEMS accelerometers. In 2001 International Semiconductor Device Research Symposium. Symposium Proceedings (Cat. No.01EX497). 190--193. https://doi.org/10.1109/ISDRS.2001.984472

[52]

OpenAI, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, Red Avila, Igor Babuschkin, Suchir Balaji, Valerie Balcom, et al. 2023. GPT-4 Technical Report. arXiv:2303.08774 [cs.CL]

[53]

Francisco Javier Ordóñez and Daniel Roggen. 2016. Deep Convolutional and LSTM Recurrent Neural Networks for Multimodal Wearable Activity Recognition. Sensors (2016).

[54]

Thomas Plötz. 2023. If only we had more data!: Sensor-Based Human Activity Recognition in Challenging Scenarios. In 2023 IEEE International Conference on Pervasive Computing and Communications Workshops and other Affiliated Events (PerCom Workshops). 565--570. https://doi.org/10.1109/PerComWorkshops56833.2023.10150267

[55]

Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer. Journal of Machine Learning Research 21, 140 (2020), 1--67. http://jmlr.org/papers/v21/20-074.html

[56]

Daniele Ravi, Charence Wong, Benny Lo, and Guang-Zhong Yang. 2016. Deep learning for human activity recognition: A resource efficient implementation on low-power devices. In 2016 IEEE 13th International Conference on Wearable and Implantable Body Sensor Networks (BSN). 71--76. https://doi.org/10.1109/BSN.2016.7516235

[57]

Nils Reimers and Iryna Gurevych. 2019. Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics. http://arxiv.org/abs/1908.10084

[58]

Attila Reiss and Didier Stricker. 2012. Introducing a New Benchmarked Dataset for Activity Monitoring (ISWC '12). IEEE Computer Society. https://doi.org/10.1109/ISWC.2012.13

Digital Library

[59]

Vitor Fortes Rey, Peter Hevesi, Onorina Kovalenko, and Paul Lukowicz. 2019. Let There Be IMU Data: Generating Training Data for Wearable, Motion Sensor Based Activity Recognition from Monocular RGB Videos. In Adjunct Proceedings of the 2019 ACM International Joint Conference on Pervasive and Ubiquitous Computing and Proceedings of the 2019 ACM International Symposium on Wearable Computers. Association for Computing Machinery, 699--708. https://doi.org/10.1145/3341162.3345590

Digital Library

[60]

Aaqib Saeed, Tanir Ozcelebi, and Johan Lukkien. 2019. Multi-task Self-Supervised Learning for Human Activity Detection. Proc. ACM Interact. Mob. Wearable Ubiquitous Technol. 3, 2, Article 61 (jun 2019), 30 pages. https://doi.org/10.1145/3328932

Digital Library

[61]

Panneer Selvam Santhalingam, Parth Pathak, Huzefa Rangwala, and Jana Kosecka. 2023. Synthetic Smartwatch IMU Data Generation from In-the-Wild ASL Videos. Proc. ACM Interact. Mob. Wearable Ubiquitous Technol. (2023).

Digital Library

[62]

Allah Bux Sargano, Xiaofeng Wang, Plamen Angelov, and Zulfiqar Habib. 2017. Human action recognition using transfer learning with deep representations. In 2017 International Joint Conference on Neural Networks (IJCNN). 463--469. https://doi.org/10.1109/IJCNN.2017.7965890

[63]

Yongliang Shen, Kaitao Song, Xu Tan, Dongsheng Li, Weiming Lu, and Yueting Zhuang. 2023. HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in HuggingFace. arXiv:2303.17580 [cs.CL]

[64]

Satya P. Singh, Madan Kumar Sharma, Aimé Lay-Ekuakille, Deepak Gangwar, and Sukrit Gupta. 2021. Deep ConvLSTM With Self-Attention for Human Activity Decoding Using Wearable Sensors. IEEE Sensors Journal 21, 6 (2021), 8575--8582. https://doi.org/10.1109/JSEN.2020.3045135

[65]

Timo Sztyler and Heiner Stuckenschmidt. 2016. On-body localization of wearable devices: An investigation of position-aware activity recognition. In 2016 IEEE International Conference on Pervasive Computing and Communications (PerCom). 1--9. https://doi.org/10.1109/PERCOM.2016.7456521

[66]

Chi Ian Tang, Ignacio Perez-Pozuelo, Dimitris Spathis, Soren Brage, Nick Wareham, and Cecilia Mascolo. 2021. SelfHAR: Improving Human Activity Recognition through Self-training with Unlabeled Data. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 5, 1 (March 2021), 1--30. https://doi.org/10.1145/3448112

Digital Library

[67]

Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M. Dai, Anja Hauth, Katie Millican, David Silver, Slav Petrov, Melvin Johnson, and Ioannis Antonoglou others. 2023. Gemini: A Family of Highly Capable Multimodal Models. arXiv:2307.09288 [cs.CL]

[68]

Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023. LLaMA: Open and Efficient Foundation Language Models. arXiv:2302.13971 [cs.CL]

[69]

Lena Uhlenberg and Oliver Amft. 2022. Comparison of Surface Models and Skeletal Models for Inertial Sensor Data Synthesis. In 2022 IEEE-EMBS International Conference on Wearable and Implantable Body Sensor Networks (BSN). 1--5. https://doi.org/10.1109/BSN56160.2022.9928504

[70]

Aaron van den Oord, Oriol Vinyals, and Koray Kavukcuoglu. 2017. Neural Discrete Representation Learning. In Proceedings of the 31st International Conference on Neural Information Processing Systems (NIPS'17). 6309--6318.

Digital Library

[71]

Vincent T. van Hees, Zhou Fang, Joss Langford, Felix Assah, Anwar Mohammad, Inacio C. M. da Silva, Michael I. Trenell, Tom White, Nicholas J. Wareham, and Søren Brage. 2014. Autocalibration of accelerometer data for free-living physical activity assessment using local gravity and temperature: an evaluation on four continents. Journal of Applied Physiology 117, 7 (2014), 738--744. https://doi.org/10.1152/japplphysiol.00421.2014 arXiv:https://doi.org/10.1152/japplphysiol.00421.2014 25103964.

[72]

Chenfei Wu, Shengming Yin, Weizhen Qi, Xiaodong Wang, Zecheng Tang, and Nan Duan. 2023. Visual ChatGPT: Talking, Drawing and Editing with Visual Foundation Models. arXiv:2303.04671 [cs.CV]

[73]

Chengshuo Xia, Ayane Saito, and Yuta Sugiura. 2022. Using the virtual data-driven measurement to support the prototyping of hand gesture recognition interface with distance sensor. Sensors and Actuators A: Physical 338 (2022), 113463.

[74]

Chengshuo Xia and Yuta Sugiura. 2022. Virtual IMU Data Augmentation by Spring-Joint Model for Motion Exercises Recognition without Using Real Data. In Proceedings of the 2022 ACM International Symposium on Wearable Computers (ISWC '22). Association for Computing Machinery, 79--83. https://doi.org/10.1145/3544794.3558460

Digital Library

[75]

Fanyi Xiao, Ling Pei, Lei Chu, Danping Zou, Wenxian Yu, Yifan Zhu, and Tao Li. 2021. A Deep Learning Method for Complex Human Activity Recognition Using Virtual Wearable Sensors. In Spatial Data and Intelligence. Springer International Publishing. https://doi.org/10.1007/978-3-030-69873-7_19

Digital Library

[76]

Chenhan Xu, Huining Li, Zhengxiong Li, Xingyu Chen, Aditya Singh Rathore, Hanbin Zhang, Kun Wang, and Wenyao Xu. 2022. The Visual Accelerometer: A High-fidelity Optic-to-Inertial Transformation Framework for Wearable Health Computing. In 2022 IEEE 10th International Conference on Healthcare Informatics (ICHI). IEEE, 319--329.

[77]

Hyungjun Yoon, Hyeongheon Cha, Canh Hoang Nguyen, Taesik Gong, and Sung-Ju Lee. 2022. IMG2IMU: Applying Knowledge from Large-Scale Images to IMU Applications via Contrastive Learning. arXiv preprint arXiv:2209.00945 (2022).

[78]

A. D. Young, M. J. Ling, and D. K. Arvind. 2011. IMUSim: A simulation environment for inertial sensing algorithm design and evaluation. In Proceedings of the 10th ACM/IEEE International Conference on Information Processing in Sensor Networks. 199--210.

[79]

Jianrong Zhang, Yangsong Zhang, Xiaodong Cun, Shaoli Huang, Yong Zhang, Hongwei Zhao, Hongtao Lu, and Xi Shen. 2023. T2M-GPT: Generating Human Motion from Textual Descriptions with Discrete Representations. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[80]

Mingyuan Zhang, Zhongang Cai, Liang Pan, Fangzhou Hong, Xinying Guo, Lei Yang, and Ziwei Liu. 2022. MotionDiffuse: Text-Driven Human Motion Generation with Diffusion Model. arXiv preprint arXiv:2208.15001 (2022).

[81]

Mingyuan Zhang, Xinying Guo, Liang Pan, Zhongang Cai, Fangzhou Hong, Huirong Li, Lei Yang, and Ziwei Liu. 2023. ReMoDiffuse: Retrieval-Augmented Motion Diffusion Model. arXiv preprint arXiv:2304.01116 (2023).

[82]

Mi Zhang and Alexander A. Sawchuk. 2012. USC-HAD: A Daily Activity Dataset for Ubiquitous Activity Recognition Using Wearable Sensors. Association for Computing Machinery.

[83]

Shibo Zhang and Nabil Alshurafa. 2020. Deep Generative Cross-Modal on-Body Accelerometer Data Synthesis from Videos. In Adjunct Proceedings of the 2020 ACM International Joint Conference on Pervasive and Ubiquitous Computing and Proceedings of the 2020 ACM International Symposium on Wearable Computers (UbiComp/ISWC '20 Adjunct). Association for Computing Machinery, 223--227.

Cited By

Fan YWang YZheng HShi Z(2025)Video2mmPoint: Synthesizing mmWave Point Cloud Data From Videos for Gait RecognitionIEEE Sensors Journal10.1109/JSEN.2024.348383525:1(773-782)Online publication date: 1-Jan-2025
https://doi.org/10.1109/JSEN.2024.3483835
Miao SChen L(2024)GOAT: A Generalized Cross-Dataset Activity Recognition Framework with Natural Language SupervisionProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36997368:4(1-28)Online publication date: 21-Nov-2024
https://dl.acm.org/doi/10.1145/3699736
Hwang SLeng ZOh SKim KPlötz TKostakos VKay JHoang T(2024)More Data for People with Disabilities! Comparing Data Collection Efforts for Wheelchair Transportation Mode DetectionProceedings of the 2024 ACM International Symposium on Wearable Computers10.1145/3675095.3676617(82-88)Online publication date: 5-Oct-2024
https://dl.acm.org/doi/10.1145/3675095.3676617
Show More Cited By

Index Terms

IMUGPT 2.0: Language-Based Cross Modality Transfer for Sensor-Based Human Activity Recognition
1. Computing methodologies
  1. Artificial intelligence
2. Human-centered computing
  1. Ubiquitous and mobile computing

Recommendations

Generating Virtual On-body Accelerometer Data from Virtual Textual Descriptions for Human Activity Recognition
ISWC '23: Proceedings of the 2023 ACM International Symposium on Wearable Computers

The development of robust, generalized models for human activity recognition (HAR) has been hindered by the scarcity of large-scale, labeled data sets. Recent work has shown that virtual IMU data extracted from videos using computer vision techniques ...
A CNN Based Transfer Learning Model for Automatic Activity Recognition from Accelerometer Sensors
Machine Learning and Data Mining in Pattern Recognition
Abstract
Accelerometers are become ubiquitous and available in several devices such as smartphones, smartwaches, fitness trackers, and wearable devices. Accelerometers are increasingly used to monitor human activities of daily living in different contexts ...
Activity Recognition based on High-Level Reasoning
ICPRAM 2016: Proceedings of the 5th International Conference on Pattern Recognition Applications and Methods

In the context of Ambient Assisted Living (AAL), the detection of daily activities is an active field of research. In this study, we present an algorithm for the performed Activities of Daily Living (ADLs) related to personal hygiene, which is based on ...

Comments

Information & Contributors

Information

Published In

cover image Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies

Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies Volume 8, Issue 3

September 2024

1782 pages

EISSN:2474-9567

DOI:10.1145/3695755

Issue’s Table of Contents

Copyright © 2024 Owner/Author.

This work is licensed under a Creative Commons Attribution International 4.0 License.

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 09 September 2024

Published in IMWUT Volume 8, Issue 3

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

5
Total Citations
View Citations
1,795
Total Downloads

Downloads (Last 12 months)1,795
Downloads (Last 6 weeks)352

Reflects downloads up to 01 Mar 2025

Other Metrics

View Author Metrics

Citations

Cited By

Fan YWang YZheng HShi Z(2025)Video2mmPoint: Synthesizing mmWave Point Cloud Data From Videos for Gait RecognitionIEEE Sensors Journal10.1109/JSEN.2024.348383525:1(773-782)Online publication date: 1-Jan-2025
https://doi.org/10.1109/JSEN.2024.3483835
Miao SChen L(2024)GOAT: A Generalized Cross-Dataset Activity Recognition Framework with Natural Language SupervisionProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36997368:4(1-28)Online publication date: 21-Nov-2024
https://dl.acm.org/doi/10.1145/3699736
Hwang SLeng ZOh SKim KPlötz TKostakos VKay JHoang T(2024)More Data for People with Disabilities! Comparing Data Collection Efforts for Wheelchair Transportation Mode DetectionProceedings of the 2024 ACM International Symposium on Wearable Computers10.1145/3675095.3676617(82-88)Online publication date: 5-Oct-2024
https://dl.acm.org/doi/10.1145/3675095.3676617
Leng ZJung MHwang SOh SZhang LPlötz TKim KKostakos VKay JHoang T(2024)Emotion Recognition on the Go: Utilizing Wearable IMUs for Personalized Emotion RecognitionCompanion of the 2024 on ACM International Joint Conference on Pervasive and Ubiquitous Computing10.1145/3675094.3678452(537-544)Online publication date: 5-Oct-2024
https://dl.acm.org/doi/10.1145/3675094.3678452
Uhlenberg LOle Haeusler LAmft O(2024)SynHAR: Augmenting Human Activity Recognition With Synthetic Inertial Sensor Data Generated From Human Surface ModelsIEEE Access10.1109/ACCESS.2024.351347712(194839-194858)Online publication date: 2024
https://doi.org/10.1109/ACCESS.2024.3513477

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

Figures

Tables

Media

View Issue’s Table of Contents