research-article

Open access

Multimodal Attention Networks for Human Activity Recognition From Earable Devices

Authors:

Jake Stuchbury-Wass,

Andrea Ferlini,

Cecilia MascoloAuthors Info & Claims

UbiComp/ISWC '22 Adjunct: Adjunct Proceedings of the 2022 ACM International Joint Conference on Pervasive and Ubiquitous Computing and the 2022 ACM International Symposium on Wearable Computers

Pages 258 - 260

https://doi.org/10.1145/3544793.3563422

Published: 24 April 2023 Publication History

All formats PDF

Abstract

Earables (a.k.a ear-worn wearable devices) are gaining traction in the wearables ecosystem for monitoring user health. Human activity recognition (HAR) is a promising use case of earables due to their placement on the head and the combination of sensors. In this paper, we explore using multimodal attention-based neural networks for HAR from the ear. Attention networks have had a large impact on other disciplines’ machine learning tasks and we believe they present opportunities in HAR from earable data. Different methods of utilising attention mechanisms in the literature are discussed as well as the benefits and challenges of using such networks in the context of HAR on real systems.

References

[1]

Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2016. Neural Machine Translation by Jointly Learning to Align and Translate. https://doi.org/10.48550/arXiv.1409.0473 arxiv:1409.0473 [cs, stat]

[2]

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers). Association for Computational Linguistics, Minneapolis, Minnesota, 4171–4186. https://doi.org/10.18653/v1/N19-1423

[3]

Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. 2021. An Image Is Worth 16x16 Words: Transformers for Image Recognition at Scale. https://doi.org/10.48550/arXiv.2010.11929 arxiv:2010.11929 [cs]

[4]

Andrea Ferlini, Dong Ma, Lorena Qendro, and Cecilia Mascolo. 2022. Mobile Health With Head-Worn Devices: Challenges and Opportunities. IEEE Pervasive Computing01 (2022), 1–9.

[5]

Wenbin Gao, Lei Zhang, Qi Teng, Jun He, and Hao Wu. 2021. DanHAR: Dual Attention Network for Multimodal Human Activity Recognition Using Wearable Sensors. Applied Soft Computing 111 (Nov. 2021), 107728. https://doi.org/10.1016/j.asoc.2021.107728

Digital Library

[6]

Ali Hassani, Steven Walton, Nikhil Shah, Abulikemu Abuduweili, Jiachen Li, and Humphrey Shi. 2021. Escaping the Big Data Paradigm with Compact Transformers. CoRR abs/2104.05704(2021). arXiv:2104.05704https://arxiv.org/abs/2104.05704

[7]

Tahera Hossain, Md Shafiqul Islam, Md Atiqur Rahman Ahad, and Sozo Inoue. 2019. Human Activity Recognition Using Earable Device. In Adjunct Proceedings of the 2019 ACM International Joint Conference on Pervasive and Ubiquitous Computing and Proceedings of the 2019 ACM International Symposium on Wearable Computers(UbiComp/ISWC ’19 Adjunct). Association for Computing Machinery, New York, NY, USA, 81–84. https://doi.org/10.1145/3341162.3343822

Digital Library

[8]

Tailin Liang, John Glossner, Lei Wang, Shaobo Shi, and Xiaotong Zhang. 2021. Pruning and quantization for deep neural network acceleration: A survey. Neurocomputing 461(2021), 370–403.

Digital Library

[9]

Xin Liu, Josh Fromm, Shwetak Patel, and Daniel McDuff. 2020. Multi-Task Temporal Shift Attention Networks for On-Device Contactless Vitals Measurement. In Advances in Neural Information Processing Systems, Vol. 33. Curran Associates, Inc., 19400–19411.

[10]

Zhaoyang Niu, Guoqiang Zhong, and Hui Yu. 2021. A Review on the Attention Mechanism of Deep Learning. Neurocomputing 452 (Sept. 2021), 48–62. https://doi.org/10.1016/j.neucom.2021.03.091

[11]

Valentin Radu, Catherine Tong, Sourav Bhattacharya, Nicholas D. Lane, Cecilia Mascolo, Mahesh K. Marina, and Fahim Kawsar. 2018. Multimodal Deep Learning for Activity and Context Recognition. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 1, 4 (Jan. 2018), 157:1–157:27. https://doi.org/10.1145/3161174

Digital Library

[12]

Nishkam Ravi, Nikhil Dandekar, Preetham Mysore, and Michael L. Littman. 2005. Activity Recognition from Accelerometer Data. In Proceedings of the 17th Conference on Innovative Applications of Artificial Intelligence - Volume 3 (Pittsburgh, Pennsylvania) (IAAI’05). AAAI Press, 1541–1546.

Digital Library

[13]

Blaž Škrlj, Sašo Džeroski, Nada Lavrač, and Matej Petkovič. 2020. Feature Importance Estimation with Self-Attention Networks. https://doi.org/10.3233/FAIA200256 arxiv:2002.04464 [cs, stat]

[14]

Achyut Mani Tripathi and Rashmi Dutta Baruah. 2020. Multivariate Time Series Classification With An Attention-Based Multivariate Convolutional Neural Network. In 2020 International Joint Conference on Neural Networks (IJCNN). 1–8. https://doi.org/10.1109/IJCNN48605.2020.9206725

[15]

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention Is All You Need. arXiv:1706.03762 [cs] (Dec. 2017). arxiv:1706.03762 [cs]

[16]

Qingsong Wen, Tian Zhou, Chaoli Zhang, Weiqi Chen, Ziqing Ma, Junchi Yan, and Liang Sun. 2022. Transformers in Time Series: A Survey. https://doi.org/10.48550/arXiv.2202.07125 arxiv:2202.07125 [cs, eess, stat]

[17]

Amir Yazdanbakhsh, Kiran Seshadri, Berkin Akin, James Laudon, and Ravi Narayanaswami. 2021. An evaluation of edge tpu accelerators for convolutional neural networks. arXiv preprint arXiv:2102.10423(2021).

Cited By

Wang JCui WZhu TNing HLiu Z(2024)An Improved Masking Strategy for Self- Supervised Masked Reconstruction in Human Activity RecognitionIEEE Sensors Journal10.1109/JSEN.2024.339075524:11(18699-18709)Online publication date: 1-Jun-2024
https://doi.org/10.1109/JSEN.2024.3390755
Aggarwal SBhola GVishwakarma D(2024)Weighted voting ensemble of hybrid CNN-LSTM Models for vision-based human activity recognitionMultimedia Tools and Applications10.1007/s11042-024-19582-1Online publication date: 8-Jun-2024
https://doi.org/10.1007/s11042-024-19582-1

Index Terms

Multimodal Attention Networks for Human Activity Recognition From Earable Devices
1. Computing methodologies
  1. Machine learning
2. Human-centered computing
  1. Ubiquitous and mobile computing
    1. Ubiquitous and mobile computing theory, concepts and paradigms
      1. Mobile computing

Recommendations

Self-Gated Recurrent Neural Networks for Human Activity Recognition on Wearable Devices
Thematic Workshops '17: Proceedings of the on Thematic Workshops of ACM Multimedia 2017

This paper develops a self-gated recurrent neural network (SGRNN), and applies it to human activity recognition (HAR), using time-series signals collected from embedded sensors of wearable devices. Recurrent neural networks (RNNs) are very powerful for ...
Human Activity Recognition: Using Wearable Sensors and Smartphones
Human activity recognition using earable device
UbiComp/ISWC '19 Adjunct: Adjunct Proceedings of the 2019 ACM International Joint Conference on Pervasive and Ubiquitous Computing and Proceedings of the 2019 ACM International Symposium on Wearable Computers

Wearable sensors are monumental for human activity recognition. Researchers are continuously inventing new technology to detect human activity properly. Earable opens up interesting possibilities of monitoring personal scale behavioral activities. In ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

UbiComp/ISWC '22 Adjunct: Adjunct Proceedings of the 2022 ACM International Joint Conference on Pervasive and Ubiquitous Computing and the 2022 ACM International Symposium on Wearable Computers

September 2022

538 pages

ISBN:9781450394239

DOI:10.1145/3544793

Copyright © 2022 Owner/Author.

This work is licensed under a Creative Commons Attribution International 4.0 License.

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 24 April 2023

Check for updates

Qualifiers

Research-article
Research
Refereed limited

Conference

UbiComp/ISWC '22

Sponsor:

UbiComp/ISWC '22: The 2022 ACM International Joint Conference on Pervasive and Ubiquitous Computing

September 11 - 15, 2022

Cambridge, United Kingdom

Acceptance Rates

Overall Acceptance Rate 764 of 2,912 submissions, 26%

Upcoming Conference

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

2
Total Citations
View Citations
293
Total Downloads

Downloads (Last 12 months)182
Downloads (Last 6 weeks)38

Reflects downloads up to 25 Feb 2025

Other Metrics

View Author Metrics

Citations

Cited By

Wang JCui WZhu TNing HLiu Z(2024)An Improved Masking Strategy for Self- Supervised Masked Reconstruction in Human Activity RecognitionIEEE Sensors Journal10.1109/JSEN.2024.339075524:11(18699-18709)Online publication date: 1-Jun-2024
https://doi.org/10.1109/JSEN.2024.3390755
Aggarwal SBhola GVishwakarma D(2024)Weighted voting ensemble of hybrid CNN-LSTM Models for vision-based human activity recognitionMultimedia Tools and Applications10.1007/s11042-024-19582-1Online publication date: 8-Jun-2024
https://doi.org/10.1007/s11042-024-19582-1

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

HTML Format

View this article in HTML Format.

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Figures

Tables

Media

View Table of Conten