research-article

Generalized Zero-Shot Activity Recognition with Embedding-Based Method

Authors:

Qingzhong LiAuthors Info & Claims

ACM Transactions on Sensor Networks, Volume 19, Issue 3

Article No.: 72, Pages 1 - 25

https://doi.org/10.1145/3582690

Published: 05 April 2023 Publication History

Abstract

Sensor-based human activity recognition aims to recognize the activities performed by people with the sensor readings. Most of existing works in this area rely on supervised classification algorithms, and can only recognize activities covered by the training data. Whereas, in many practical applications, while performing activity recognition, not only the activities covered by the training data, but also some previously unseen activities need to be recognized. In this paper, we study the problem of generalized zero-shot activity recognition. In this problem, the activities that need to be recognized contain both the activities covered by the training data and the previously unseen activities. We firstly give a formulation of this problem, and then propose an embedding-based method to address it. In this method, an embedding-compatibility model is learned. When performing activity recognition, the learned model and the calibrated stacking mechanism are employed. Extensive experiments on publicly available datasets demonstrate the effectiveness of our method.

References

[1]

Rebecca Adaimi and Edison Thomaz. 2019. Leveraging active learning and conditional mutual information to minimize data annotation in human activity recognition. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 3, 3 (2019), 70:1–70:23.

Digital Library

[2]

Zeynep Akata, Florent Perronnin, Zaid Harchaoui, and Cordelia Schmid. 2016. Label-embedding for image classification. IEEE Transactions on Pattern Analysis and Machine Intelligence 38, 7 (2016), 1425–1438.

[3]

Mohammad Al-Naser, Hiroki Ohashi, Sheraz Ahmed, Katsuyuki Nakamura, Takayuki Akiyama, Takuto Sato, Phong Nguyen, and Andreas Dengel. 2018. Hierarchical model for zero-shot activity recognition using wearable sensors. In International Conference on Agents and Artificial Intelligence. 478–485.

[4]

Xiang Ao, Xu-Yao Zhang, and Cheng-Lin Liu. 2022. Cross-modal prototype learning for zero-shot handwritten character recognition. Pattern Recognition 131 (2022), 108859.

Digital Library

[5]

David Belanger and Andrew McCallum. 2016. Structured prediction energy networks. In International Conference on Machine Learning. 983–992.

[6]

Sejal Bhalla, Mayank Goel, and Rushil Khurana. 2021. IMU2Doppler: Cross-modal domain adaptation for doppler-based activity recognition using IMU data. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 5, 4 (2021), 145:1–145:20.

Digital Library

[7]

Soravit Changpinyo, Wei-Lun Chao, Boqing Gong, and Fei Sha. 2020. Classifier and exemplar synthesis for zero-shot learning. International Journal of Computer Vision 128, 1 (2020), 166–201.

Digital Library

[8]

Wei-Lun Chao, Soravit Changpinyo, Boqing Gong, and Fei Sha. 2016. An empirical study and analysis of generalized zero-shot learning for object recognition in the wild. In European Conference on Computer Vision. 52–68.

[9]

Chen Chen, Roozbeh Jafari, and Nasser Kehtarnavaz. 2015. UTD-MHAD: A multimodal dataset for human action recognition utilizing a depth camera and a wearable inertial sensor. In IEEE International Conference on Image Processing. 168–172.

Digital Library

[10]

Hong Chen, Yongtan Luo, Liujuan Cao, Baochang Zhang, Guodong Guo, Cheng Wang, Jonathan Li, and Rongrong Ji. 2019. Generalized zero-shot vehicle detection in remote sensing imagery via coarse-to-fine framework. In International Joint Conference on Artificial Intelligence. 687–693.

[11]

Kaixuan Chen, Dalin Zhang, Lina Yao, Bin Guo, Zhiwen Yu, and Yunhao Liu. 2021. Deep learning for sensor-based human activity recognition: Overview, challenges, and opportunities. Comput. Surveys 54, 4 (2021), 77:1–77:40.

[12]

Liming Chen, Jesse Hoey, Chris D. Nugent, Diane J. Cook, and Zhiwen Yu. 2012. Sensor-based activity recognition. IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews) 42, 6 (2012), 790–808.

Digital Library

[13]

Xingyu Chen, Jin Li, Xuguang Lan, and Nanning Zheng. 2022. Generalized zero-shot learning via multi-modal aggregated posterior aligning neural network. IEEE Transactions on Multimedia 24 (2022), 177–187.

Digital Library

[14]

Zhi Chen, Yadan Luo, Ruihong Qiu, Sen Wang, Zi Huang, Jingjing Li, and Zheng Zhang. 2021. Semantics disentangling for generalized zero-shot learning. In IEEE/CVF International Conference on Computer Vision. 8692–8700.

[15]

Heng-Tze Cheng, Feng-Tso Sun, Martin Griss, Paul Davis, Jianguo Li, and Di You. 2013. NuActiv: Recognizing unseen new activities using semantic attribute-based learning. In International Conference on Mobile Systems, Applications, and Services. 361–374.

Digital Library

[16]

Zhiyu Fang, Xiaobin Zhu, Chun Yang, Zheng Han, Jingyan Qin, and Xu-Cheng Yin. 2022. Learning aligned cross-modal representation for generalized zero-shot classification. In The Thirty-sixth AAAI Conference on Artificial Intelligence. 6605–6613.

[17]

Siwei Feng and Marco F. Duarte. 2019. Few-shot learning-based human activity recognition. Expert Systems with Applications 138 (2019), 112782.

[18]

Colin Graber, Ofer Meshi, and Alexander Schwing. 2018. Deep structured prediction with nonlinear output transformations. In Advances in Neural Information Processing Systems. 6323–6334.

[19]

Fuqiang Gu, Mu-Huan Chung, Mark Chignell, Shahrokh Valaee, Baoding Zhou, and Xue Liu. 2021. A survey on deep learning for human activity recognition. Comput. Surveys 54, 8 (2021), 177:1–177:34.

[20]

Michael Gygli, Mohammad Norouzi, and Anelia Angelova. 2017. Deep value networks learn to evaluate and iteratively refine structured outputs. In International Conference on Machine Learning. 1341–1351.

[21]

Mingyao Hong, Guorong Li, Xinfeng Zhang, and Qingming Huang. 2020. Generalized zero-shot video classification via generative adversarial networks. In ACM International Conference on Multimedia. 2419–2426.

Digital Library

[22]

H. M. Sajjad Hossain, MD. Abdullah Al Haiz Khan, and Nirmalya Roy. 2018. DeActive: Scaling activity recognition with active deep learning. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 2, 2 (2018), 66:1–66:23.

Digital Library

[23]

He Huang, Changhu Wang, Philip S. Yu, and Chang-Dong Wang. 2019. Generative dual adversarial network for generalized zero-shot learning. In IEEE/CVF Conference on Computer Vision and Pattern Recognition. 801–810.

[24]

Panagiotis Kasnesis, Christos Chatzigeorgiou, Charalampos Z. Patrikakis, and Maria Rangoussi. 2021. Modality-wise relational reasoning for one-shot sensor-based activity recognition. Pattern Recognition Letters 146 (2021), 90–99.

[25]

Junhan Kim, Kyuhong Shim, and Byonghyo Shim. 2022. Semantic feature extraction for generalized zero-shot learning. In The Thirty-sixth AAAI Conference on Artificial Intelligence. 1166–1173.

[26]

Diederik P. Kingma and Jimmy Lei Ba. 2015. Adam: A method for stochastic optimization. In International Conference on Learning Representations. 1–15.

[27]

Xia Kong, Zuodong Gao, Xiaofan Li, Ming Hong, Jun Liu, Chengjie Wang, Yuan Xie, and Yanyun Qu. 2022. En-compactness: Self-distillation embedding & contrastive generation for generalized zero-shot learning. In IEEE/CVF Conference on Computer Vision and Pattern Recognition. 9296–9305.

[28]

Heike Leutheuser, Dominik Schuldhaus, and Bjoern M. Eskofier. 2013. Hierarchical, multi-sensor based classification of daily life activities: Comparison with state-of-the-art algorithms using a benchmark dataset. PloS ONE 8, 10 (2013), e75196.

[29]

Fang Li and Mei-Chen Yeh. 2021. Generalized zero-shot recognition through image-guided semantic classification. In IEEE International Conference on Image Processing. 2483–2487.

[30]

Jin Li, Xuguang Lan, Yang Long, Yang Liu, Xingyu Chen, Ling Shao, and Nanning Zheng. 2020. A joint label space for generalized zero-shot classification. IEEE Transactions on Image Processing 29 (2020), 5817–5831.

[31]

Xinyu Li, Yuan He, Francesco Fioranelli, and Xiaojun Jing. 2021. Semisupervised human activity recognition with radar micro-doppler signatures. IEEE Transactions on Geoscience and Remote Sensing 60 (2021), 5103112.

[32]

Xiangyu Li, Zhe Xu, Kun Wei, and Cheng Deng. 2021. Generalized zero-shot learning via disentangled representation. In The Thirty-fifth AAAI Conference on Artificial Intelligence. 1966–1974.

[33]

Mingqi Lv, Ling Chen, Tieming Chen, and Gencai Chen. 2018. Bi-view semi-supervised learning based semantic human activity recognition using accelerometers. IEEE Transactions on Mobile Computing 17, 9 (2018), 1991–2001.

Digital Library

[34]

Fadi Al Machot, Mohammed R. Elkobaisi, and Kyandoghere Kyamakya. 2020. Zero-shot human activity recognition using non-visual sensors. Sensors 20, 3 (2020), 825.

[35]

Devraj Mandal, Sanath Narayan, Saikumar Dwivedi, Vikram Gupta, Shuaib Ahmed, Fahad Shahbaz Khan, and Ling Shao. 2019. Out-of-distribution detection for generalized zero-shot action recognition. In IEEE/CVF Conference on Computer Vision and Pattern Recognition. 9977–9985.

[36]

Moe Matsuki and Sozo Inoue. 2016. Recognizing unknown activities using semantic word vectors and Twitter timestamps. In ACM International Joint Conference on Pervasive and Ubiquitous Computing: Adjunct. 823–830.

Digital Library

[37]

Moe Matsuki, Paula Lago, and Sozo Inoue. 2019. Characterizing word embeddings for zero-shot sensor-based human activity recognition. Sensors 19, 22 (2019), 5043.

[38]

Hiroki Ohashi, Mohammad Al-Naser, Sheraz Ahmed, Katsuyuki Nakamura, Takuto Sato, and Andreas Dengel. 2018. Attributes’ importance for zero-shot pose-classification based on wearable sensors. Sensors 18, 8 (2018), 2485.

[39]

Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf, Edward Yang, Zach DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. 2019. Pytorch: An imperative style, high-performance deep learning library. In Advances in Neural Information Processing Systems. 8024–8035.

[40]

Jeffrey Pennington, Richard Socher, and Christopher D. Manning. 2014. GloVe: Global vectors for word representation. In The 2014 Conference on Empirical Methods in Natural Language Processing. 1532–1543.

[41]

Farhad Pourpanah, Moloud Abdar, Yuxuan Luo, Xinlei Zhou, Ran Wang, Chee Peng Lim, Xi-Zhao Wang, and Q. M. Jonathan Wu. 2022. A review of generalized zero-shot learning methods. IEEE Transactions on Pattern Analysis and Machine Intelligence (2022), 1–20. DOI:

Digital Library

[42]

Aria Ghora Prabono, Bernardo Nugroho Yahya, and Seok-Lyong Lee. 2022. Multiple-instance domain adaptation for cost-effective sensor-based human activity recognition. Future Generation Computer Systems 133 (2022), 114–123.

Digital Library

[43]

Attila Reiss and Didier Stricker. 2012. Introducing a new benchmarked dataset for activity monitoring. In International Symposium on Wearable Computers. 108–109.

[44]

Farzad Shahabi, Yang Gao, and Nabil Alshurafa. 2022. ActiveSense: A novel active learning framework for human activity recognition. In IEEE International Conference on Pervasive Computing and Communications Workshops and other Affiliated Events. 224–229.

[45]

Richard Socher, Milind Ganjoo, Christopher D. Manning, and Andrew Y. Ng. 2013. Zero-shot learning through cross-modal transfer. In Advances in Neural Information Processing Systems. 935–943.

Digital Library

[46]

Hongzu Su, Jingjing Li, Zhi Chen, Lei Zhu, and Ke Lu. 2022. Distinguishing unseen from seen for generalized zero-shot learning. In IEEE/CVF Conference on Computer Vision and Pattern Recognition. 7875–7884.

[47]

Catherine Tong, Jinchen Ge, and Nicholas D. Lane. 2021. Zero-shot learning for IMU-based activity recognition using video embeddings. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 5, 4 (2021), 180:1–180:23.

Digital Library

[48]

Vinay Kumar Verma, Gundeep Arora, Ashish Mishra, and Piyush Rai. 2018. Generalized zero-shot learning via synthesized examples. In IEEE/CVF Conference on Computer Vision and Pattern Recognition. 4281–4289.

[49]

Chaoqun Wang, Xuejin Chen, Shaobo Min, Xiaoyan Sun, and Houqiang Li. 2021. Task-independent knowledge makes for transferable representations for generalized zero-shot learning. In The Thirty-fifth AAAI Conference on Artificial Intelligence. 2710–2718.

[50]

Wei Wang and Chunyan Miao. 2018. Activity recognition in new smart home environments. In ACM International Conference on Multimedia Workshop: International Workshop on Multimedia for Personal Health and Health Care. 29–37.

Digital Library

[51]

Wei Wang and Chunyan Miao. 2018. Multi-resident activity recognition with unseen classes in smart homes. In IEEE International Conference on Ubiquitous Intelligence and Computing. 780–787.

[52]

Wei Wang, Chunyan Miao, and Shuji Hao. 2017. Zero-shot human activity recognition via nonlinear compatibility based method. In IEEE/WIC/ACM International Conference on Web Intelligence. 322–330.

Digital Library

[53]

Wei Wang, Vincent W. Zheng, Han Yu, and Chunyan Miao. 2019. A survey of zero-shot learning: Settings, methods, and applications. ACM Transactions on Intelligent Systems and Technology 10, 2 (2019), 13:1–13:37.

Digital Library

[54]

Tong Wu, Yiqiang Chen, Yang Gu, Jiwei Wang, Siyu Zhang, and Zhanghu Zhechen. 2020. Multi-layer cross loss model for zero-shot human activity recognition. In Pacific-Asia Conference on Knowledge Discovery and Data Mining. 210–221.

Digital Library

[55]

Yongqin Xian, Christoph H. Lampert, Bernt Schiele, and Zeynep Akata. 2019. Zero-shot learning-a comprehensive evaluation of the good, the bad and the ugly. IEEE Transactions on Pattern Analysis and Machine Intelligence 41, 9 (2019), 2251–2265.

[56]

Wenjia Xu, Yongqin Xian, Jiuniu Wang, Bernt Schiele, and Zeynep Akata. 2020. Attribute prototype network for zero-shot learning. In Advances in Neural Information Processing Systems. 21969–21980.

[57]

Xun Xu, Timothy Hospedales, and Shaogang Gong. 2015. Semantic embedding space for zero-shot action recognition. In IEEE International Conference on Image Processing. 63–67.

Digital Library

[58]

Xun Xu, Timothy Hospedales, and Shaogang Gong. 2017. Transductive zero-shot action recognition by word-vector embedding. International Journal of Computer Vision 123, 3 (2017), 309–333.

Digital Library

[59]

Guanyu Yang, Kaizhu Huang, Rui Zhang, John Y. Goulermas, and Amir Hussain. 2021. Coarse-grained generalized zero-shot learning with efficient self-focus mechanism. Neurocomputing 463 (2021), 400–410.

Digital Library

[60]

Fei Zhang and Guangming Shi. 2019. Co-representation network for generalized zero-shot learning. In International Conference on Machine Learning. 7434–7443.

[61]

Yong Zhang, Yang Chen, Yujie Wang, Qingqing Liu, and Andong Cheng. 2022. CSI-based human activity recognition with graph few-shot learning. IEEE Internet of Things Journal 9, 6 (2022), 4139–4151.

[62]

Ziming Zhang and Venkatesh Saligrama. 2016. Zero-shot learning via joint latent similarity embedding. In IEEE Conference on Computer Vision and Pattern Recognition. 6034–6042.

[63]

Jiachen Zhao, Fang Deng, Haibo He, and Jie Chen. 2021. Local domain adaptation for cross-domain activity recognition. IEEE Transactions on Human-Machine Systems 51, 1 (2021), 12–21.

[64]

Qingchang Zhu, Zhenghua Chen, and Yeng Chai Soh. 2019. A novel semisupervised deep learning method for human activity recognition. IEEE Transactions on Industrial Informatics 15, 7 (2019), 3821–3830.

Cited By

Zhang PLiu MSong XCao DGao ZNie L(2024)Universal Relocalizer for Weakly Supervised Referring Expression GroundingACM Transactions on Multimedia Computing, Communications, and Applications10.1145/365604520:7(1-23)Online publication date: 16-May-2024
https://dl.acm.org/doi/10.1145/3656045
Ben HWang SWang MHong RGurrin CKongkachandra RSchoeffmann KDang-Nguyen DRossetto LSatoh SZhou L(2024)Pseudo Content Hallucination for Unpaired Image CaptioningProceedings of the 2024 International Conference on Multimedia Retrieval10.1145/3652583.3658080(320-329)Online publication date: 30-May-2024
https://dl.acm.org/doi/10.1145/3652583.3658080
Antil ADhiman C(2024)MF2ShrT: Multimodal Feature Fusion Using Shared Layered Transformer for Face Anti-spoofingACM Transactions on Multimedia Computing, Communications, and Applications10.1145/364081720:6(1-21)Online publication date: 8-Mar-2024
https://dl.acm.org/doi/10.1145/3640817
Show More Cited By

Index Terms

Generalized Zero-Shot Activity Recognition with Embedding-Based Method
1. Computing methodologies
  1. Machine learning
    1. Learning paradigms
2. Human-centered computing
  1. Ubiquitous and mobile computing
    1. Ubiquitous and mobile computing theory, concepts and paradigms

Recommendations

ConvNet-based performers attention and supervised contrastive learning for activity recognition
Abstract
Human activity recognition based on generated sensor data plays a major role in a large number of applications such as healthcare monitoring and surveillance system. Yet, accurately recognizing human activities is still challenging and active ...
Activity Recognition based on High-Level Reasoning
ICPRAM 2016: Proceedings of the 5th International Conference on Pattern Recognition Applications and Methods

In the context of Ambient Assisted Living (AAL), the detection of daily activities is an active field of research. In this study, we present an algorithm for the performed Activities of Daily Living (ADLs) related to personal hygiene, which is based on ...
Activity recognition with hand-worn magnetic sensors

Activity recognition is a key technology for realizing ambient assisted living applications such as care of the elderly and home automation. This paper proposes a new activity recognition method that employs hand-worn magnetic sensors to recognize a ...

Comments

Information & Contributors

Information

Published In

cover image ACM Transactions on Sensor Networks

ACM Transactions on Sensor Networks Volume 19, Issue 3

August 2023

597 pages

ISSN:1550-4859

EISSN:1550-4867

DOI:10.1145/3584865

Editor:
Yunhao Liu
Tsinghua University, China

Issue’s Table of Contents

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Publisher

Association for Computing Machinery

New York, NY, United States

Journal Family

ACM Journals for the Design of Smart and Connected Systems

Publication History

Published: 05 April 2023

Online AM: 01 February 2023

Accepted: 25 January 2023

Revised: 19 December 2022

Received: 22 August 2022

Published in TOSN Volume 19, Issue 3

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

8
Total Citations
View Citations
607
Total Downloads

Downloads (Last 12 months)344
Downloads (Last 6 weeks)11

Reflects downloads up to 30 Aug 2024

Other Metrics

View Author Metrics

Citations

Cited By

Zhang PLiu MSong XCao DGao ZNie L(2024)Universal Relocalizer for Weakly Supervised Referring Expression GroundingACM Transactions on Multimedia Computing, Communications, and Applications10.1145/365604520:7(1-23)Online publication date: 16-May-2024
https://dl.acm.org/doi/10.1145/3656045
Ben HWang SWang MHong RGurrin CKongkachandra RSchoeffmann KDang-Nguyen DRossetto LSatoh SZhou L(2024)Pseudo Content Hallucination for Unpaired Image CaptioningProceedings of the 2024 International Conference on Multimedia Retrieval10.1145/3652583.3658080(320-329)Online publication date: 30-May-2024
https://dl.acm.org/doi/10.1145/3652583.3658080
Antil ADhiman C(2024)MF2ShrT: Multimodal Feature Fusion Using Shared Layered Transformer for Face Anti-spoofingACM Transactions on Multimedia Computing, Communications, and Applications10.1145/364081720:6(1-21)Online publication date: 8-Mar-2024
https://dl.acm.org/doi/10.1145/3640817
Li MZhou THuang ZYang JYang JGong C(2024)Dynamic Weighted Adversarial Learning for Semi-Supervised Classification under Intersectional Class MismatchACM Transactions on Multimedia Computing, Communications, and Applications10.1145/363531020:4(1-24)Online publication date: 11-Jan-2024
https://dl.acm.org/doi/10.1145/3635310
Shi PHu MShi XRen F(2024)Deep Modular Co-Attention Shifting Network for Multimodal Sentiment AnalysisACM Transactions on Multimedia Computing, Communications, and Applications10.1145/363470620:4(1-23)Online publication date: 11-Jan-2024
https://dl.acm.org/doi/10.1145/3634706
Nai KChen S(2024)Learning a Novel Ensemble Tracker for Robust Visual TrackingIEEE Transactions on Multimedia10.1109/TMM.2023.330793926(3194-3206)Online publication date: 1-Jan-2024
https://dl.acm.org/doi/10.1109/TMM.2023.3307939
Anand SDevulapally NBhattacharjee SYuan JEl Saddik AMei TCucchiara RBertini MTobon Vallejo DAtrey PHossain M(2023)Multi-label Emotion Analysis in Conversation via Multimodal Knowledge DistillationProceedings of the 31st ACM International Conference on Multimedia10.1145/3581783.3612517(6090-6100)Online publication date: 26-Oct-2023
https://dl.acm.org/doi/10.1145/3581783.3612517
Lu JWang SZhang XHao YHe XEl Saddik AMei TCucchiara RBertini MTobon Vallejo DAtrey PHossain M(2023)Semantic-based Selection, Synthesis, and Supervision for Few-shot LearningProceedings of the 31st ACM International Conference on Multimedia10.1145/3581783.3611784(3569-3578)Online publication date: 26-Oct-2023
https://dl.acm.org/doi/10.1145/3581783.3611784

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Full Text

View this article in Full Text.

HTML Format

View this article in HTML Format.

Media

Figures

Other

Tables

View full text|Download PDF

View Issue’s Table of Contents