Deep Heterogeneous Contrastive Hyper-Graph Learning for In-the-Wild Context-Aware Human Activity Recognition
Article No.: 159, Pages 1 - 23
Abstract
Human Activity Recognition (HAR) is a challenging, multi-label classification problem as activities may co-occur and sensor signals corresponding to the same activity may vary in different contexts (e.g., different device placements). This paper proposes a Deep Heterogeneous Contrastive Hyper-Graph Learning (DHC-HGL) framework that captures heterogenous Context-Aware HAR (CA-HAR) hypergraph properties in a message-passing and neighborhood-aggregation fashion. Prior work only explored homogeneous or shallow-node-heterogeneous graphs. DHC-HGL handles heterogeneous CA-HAR data by innovatively 1) Constructing three different types of sub-hypergraphs that are each passed through different custom HyperGraph Convolution (HGC) layers designed to handle edge-heterogeneity and 2) Adopting a contrastive loss function to ensure node-heterogeneity. In rigorous evaluation on two CA-HAR datasets, DHC-HGL significantly outperformed state-of-the-art baselines by 5.8% to 16.7% on Matthews Correlation Coefficient (MCC) and 3.0% to 8.4% on Macro F1 scores. UMAP visualizations of learned CA-HAR node embeddings are also presented to enhance model explainability. Our code is publicly available1 to encourage further research.
References
[1]
2017. Lightgbm: A highly efficient gradient boosting decision tree. Proc. NIPS 30 (2017).
[2]
Abdulaziz Alajaji, Walter Gerych, Kavin Chandrasekaran, Luke Buquicchio, Emmanuel Agu, and Elke Rundensteiner. 2020. Deepcontext: Parameterized compatibility-based attention cnn for human context recognition. In 2020 IEEE 14th International Conference on Semantic Computing (ICSC). IEEE, 53--60.
[3]
Lei Bai, Lina Yao, Xianzhi Wang, Salil S Kanhere, Bin Guo, and Zhiwen Yu. 2020. Adversarial multi-view networks for activity recognition. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 4, 2 (2020), 1--22.
[4]
Song Bai, Feihu Zhang, and Philip HS Torr. 2021. Hypergraph convolution and hypergraph attention. Pattern Recognition 110 (2021), 107637.
[5]
Valentina Bianchi, Marco Bassoli, Gianfranco Lombardo, Paolo Fornacciari, Monica Mordonini, and Ilaria De Munari. 2019. IoT wearable sensor and deep learning: An integrated approach for personalized human activity recognition in a smart home environment. IEEE Internet of Things Journal 6, 5 (2019), 8553--8562.
[6]
Bhaskar Chakraborty, Ognjen Rudovic, and Jordi Gonzalez. 2008. View-invariant human-body detection with extension to human action recognition using component-wise HMM of body parts. In Int'l Conf. Automatic Face & Gesture Rec. IEEE, 1--6.
[7]
Youngjae Chang, Akhil Mathur, Anton Isopoussu, Junehwa Song, and Fahim Kawsar. 2020. A systematic study of unsupervised domain adaptation for robust human-activity recognition. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 4, 1 (2020), 1--30.
[8]
Kuang-Hsuan Chen, Yu-Wei Hsu, Jing-Jung Yang, and Fu-Shan Jaw. 2018. Evaluating the specifications of built-in accelerometers in smartphones on fall detection performance. Inst. Sci. & Tech. 46, 2 (2018), 194--206.
[9]
Ling Chen, Yi Zhang, and Liangying Peng. 2020. METIER: A deep multi-task learning based activity and user recognition model using wearable sensors. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 4, 1 (2020), 1--18.
[10]
Davide Chicco and Giuseppe Jurman. 2020. The advantages of the Matthews correlation coefficient (MCC) over F1 score and accuracy in binary classification evaluation. BMC genomics 21 (2020), 1--13.
[11]
Ada Dogrucu, Alex Perucic, Anabella Isaro, Damon Ball, Ermal Toto, Elke A Rundensteiner, Emmanuel Agu, Rachel Davis-Martin, and Edwin Boudreaux. 2020. Moodable: On feasibility of instantaneous depreswension assessment using machine learning on voice samples with retrospectively harvested smartphone and social media data. Smart Health 17 (2020), 100118.
[12]
Haoyi Fan, Fengbin Zhang, Yuxuan Wei, Zuoyong Li, Changqing Zou, Yue Gao, and Qionghai Dai. 2021. Heterogeneous hypergraph variational autoencoder for link prediction. IEEE Transactions on Pattern Analysis and Machine Intelligence 44, 8 (2021), 4125--4138.
[13]
Xile Gao, Haiyong Luo, Qu Wang, Fang Zhao, Langlang Ye, and Yuexia Zhang. 2019. A human activity recognition algorithm based on stacking denoising autoencoder and lightGBM. Sensors 19, 4 (2019), 947.
[14]
Wen Ge and Emmanuel Agu. 2020. CRUFT: Context recog. under uncertainty using fusion and temporal learning. In Proc. ICMLA. IEEE, 747--52.
[15]
Wen Ge and Emmanuel O Agu. 2022. QCRUFT: Quaternion Context Recog. under Uncertainty using Fusion & Temporal Learning. In Proc. ICSC. IEEE, 41--50.
[16]
Wen Ge, Guanyi Mou, Emmanuel O Agu, and Kyumin Lee. 2023. Heterogeneous Hyper-Graph Neural Networks for Context-aware Human Activity Recognition. In PerCom.
[17]
Raia Hadsell, Sumit Chopra, and Yann LeCun. 2006. Dimensionality reduction by learning an invariant mapping. In 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'06), Vol. 2. IEEE, 1735--1742.
[18]
Will Hamilton, Zhitao Ying, and Jure Leskovec. 2017. Inductive representation learning on large graphs. Proc. NIPS 30 (2017).
[19]
Hua Kang, Qianyi Huang, and Qian Zhang. 2022. Augmented Adversarial Learning for Human Activity Recognition with Partial Sensor Sets. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 6, 3 (2022), 1--30.
[20]
Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. 2020. Supervised contrastive learning. Advances in neural information processing systems 33 (2020), 18661--18673.
[21]
Thomas N Kipf and Max Welling. 2016. Semi-supervised classification with graph convolutional networks. arXiv preprint arXiv:1609.02907 (2016).
[22]
Qimai Li, Zhichao Han, and Xiao-Ming Wu. 2018. Deeper insights into graph convolutional networks for semi-supervised learning. In Proceedings of the AAAI conference on artificial intelligence, Vol. 32.
[23]
Yinfeng Li, Chen Gao, Quanming Yao, Tong Li, Depeng Jin, and Yong Li. 2022. DisenHCN: Disentangled Hypergraph Convolutional Networks for Spatiotemporal Activity Prediction. arXiv preprint arXiv:2208.06794 (2022).
[24]
Janne Lindqvist and Jason Hong. 2011. Undistracted driving: A mobile phone that doesn't distract. In Proc. Hotmobile. 70--75.
[25]
Liyuan Liu, Haoming Jiang, Pengcheng He, Weizhu Chen, Xiaodong Liu, Jianfeng Gao, and Jiawei Han. 2019. On the variance of the adaptive learning rate and beyond. arXiv preprint arXiv:1908.03265 (2019).
[26]
Shinan Liu, Tarun Mangla, Ted Shaowang, Jinjin Zhao, John Paparrizos, Sanjay Krishnan, and Nick Feamster. 2023. AMIR: Active Multimodal Interaction Recognition from Video and Network Traffic in Connected Environments. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 7, 1 (2023), 1--26.
[27]
Weifeng Liu, Sichao Fu, Yicong Zhou, Zheng-Jun Zha, and Liqiang Nie. 2021. Human activity recognition by manifold regularization based dynamic graph convolutional networks. Neurocomputing 444 (2021), 217--225.
[28]
Jing Ma, Mengting Wan, Longqi Yang, Jundong Li, Brent Hecht, and Jaime Teevan. 2022. Learning causal effects on hypergraphs. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining. 1202--1212.
[29]
Henry Martin, Dominik Bucher, Esra Suel, Pengxiang Zhao, Fernando Perez-Cruz, and Martin Raubal. 2018. Graph convolutional neural networks for human activity purpose imputation. In Spatiotemporal workshop co-located with NIPS.
[30]
Leland McInnes, John Healy, and James Melville. 2018. Umap: Uniform manifold approximation and projection for dimension reduction. arXiv preprint arXiv:1802.03426 (2018).
[31]
Abduallah Mohamed, Fernando Lejarza, Stephanie Cahail, Christian Claudel, and Edison Thomaz. 2022. HAR-GCNN: Deep Graph CNNs for Human Activity Recog. From Highly Unlabeled Mobile Sensor Data. In Proc. PerCom. IEEE, 335--40.
[32]
Kazushige Ouchi and Miwako Doi. 2013. Smartphone-based monitoring system for activities of daily living for elderly people and their relatives etc. In Proceedings of the 2013 ACM conference on Pervasive and ubiquitous computing adjunct publication. 103--106.
[33]
Leif E Peterson. 2009. K-nearest neighbor. Scholarpedia 4, 2 (2009), 1883.
[34]
Sreenivasan Ramasamy Ramamurthy, Soumyajit Chatterjee, Elizabeth Galik, Aryya Gangopadhyay, Nirmalya Roy, Bivas Mitra, and Sandip Chakraborty. 2022. CogAx: Early Assessment of Cognitive and Functional Impairment from Accelerometry. In PerCom. IEEE, 66--76.
[35]
Tifenn Rault, Abdelmadjid Bouabdallah, Yacine Challal, and Frédéric Marin. 2017. A survey of energy-efficient context recognition systems using wearable sensors for healthcare applications. Perv. & Mob. Comp. 37 (2017), 23--44.
[36]
Chuan Shi, Binbin Hu, Wayne Xin Zhao, and S Yu Philip. 2018. Heterogeneous information network embedding for recommendation. IEEE Transactions on Knowledge and Data Engineering 31, 2 (2018), 357--370.
[37]
Lei Shi, Yifan Zhang, Jian Cheng, and Hanqing Lu. 2020. Skeleton-based action recognition with multi-stream adaptive graph convolutional networks. IEEE Transactions on Image Processing 29 (2020), 9532--9545.
[38]
Guillaume St-Onge, Iacopo Iacopini, Vito Latora, Alain Barrat, Giovanni Petri, Antoine Allard, and Laurent Hébert-Dufresne. 2022. Influential groups for seeding and sustaining nonlinear contagion in heterogeneous hypergraphs. Communications Physics 5, 1 (2022), 25.
[39]
Xiangguo Sun, Hongzhi Yin, Bo Liu, Hongxu Chen, Jiuxin Cao, Yingxia Shao, and Nguyen Quoc Viet Hung. 2021. Heterogeneous hypergraph embedding for graph classification. In Proceedings of the 14th ACM international conference on web search and data mining. 725--733.
[40]
Yonatan Vaizman, Katherine Ellis, and Gert Lanckriet. 2017. Recognizing detailed human context in the wild from smartphones and smartwatches. IEEE Pervasive Computing 16, 4 (2017), 62--74.
[41]
Yonatan Vaizman, Nadir Weibel, and Gert Lanckriet. 2018. Context recognition in-the-wild: Unified model for multi-modal sensors and multi-label classification. ACM IMWUT 1, 4 (2018), 1--22.
[42]
Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, and Yoshua Bengio. 2017. Graph attention networks. arXiv preprint arXiv:1710.10903 (2017).
[43]
Jianchao Wu, Limin Wang, Li Wang, Jie Guo, and Gangshan Wu. 2019. Learning actor relation graphs for group activity recognition. In Proceedings of the IEEE/CVF Conference on computer vision and pattern recognition. 9964--9974.
[44]
Dingqi Yang, Bingqing Qu, Jie Yang, and Philippe Cudre-Mauroux. 2019. Revisiting user mobility and social relationships in lbsns: a hypergraph embedding approach. In The WWW Conf. 2147--2157.
[45]
Dingqi Yang, Bingqing Qu, Jie Yang, and Philippe Cudré-Mauroux. 2020. Lbsn2vec++: Heterogeneous hypergraph embedding for location-based social networks. Trans. Knowledge and Data Engineering (2020).
[46]
Shuochao Yao, Yiran Zhao, Huajie Shao, Aston Zhang, Chao Zhang, Shen Li, and Tarek Abdelzaher. 2018. Rdeepsense: Reliable deep mobile computing models with uncertainty estimations. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 1, 4 (2018), 1--26.
[47]
Chuxu Zhang, Dongjin Song, Chao Huang, Ananthram Swami, and Nitesh V Chawla. 2019. Heterogeneous graph neural network. In Proc. ACM SIGKDD. 793--803.
[48]
Ye Zhang, Longguang Wang, Huiling Chen, Aosheng Tian, Shilin Zhou, and Yulan Guo. 2022. IF-ConvTransformer: A framework for human activity recognition using IMU fusion and ConvTransformer. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 6, 2 (2022), 1--26.
[49]
Wenbo Zheng, Lan Yan, Chao Gou, and Fei-Yue Wang. 2022. Meta-learning meets the Internet of Things: Graph prototypical models for sensor-based human activity recognition. Information Fusion 80 (2022), 1--22.
[50]
Jie Zhou, Ganqu Cui, Shengding Hu, Zhengyan Zhang, Cheng Yang, Zhiyuan Liu, Lifeng Wang, Changcheng Li, and Maosong Sun. 2020. Graph neural networks: A review of methods and applications. AI open 1 (2020), 57--81.
[51]
Yu Zhu, Ziyu Guan, Shulong Tan, Haifeng Liu, Deng Cai, and Xiaofei He. 2016. Heterogeneous hypergraph embedding for document recommendation. Neurocomputing 216 (2016), 150--162.
[52]
Qin Zou, Yanling Wang, Qian Wang, Yi Zhao, and Qingquan Li. 2020. Deep learning-based gait recognition using smartphones in the wild. IEEE Transactions on Information Forensics and Security 15 (2020), 3197--3212.
Index Terms
- Deep Heterogeneous Contrastive Hyper-Graph Learning for In-the-Wild Context-Aware Human Activity Recognition
Recommendations
Hypergraph-Enhanced Self-supervised Heterogeneous Graph Representation Learning
Web and Big DataAbstractHeterogeneous graphs are widely used to model complex systems in the real world, such as social networks, biomedical networks, and citation networks. Learning heterogeneous graph embeddings (i.e., representations) provides a way to perform deep ...
Heterogeneous graph neural networks with denoising for graph embeddings
AbstractWith the increasing popularity of graph structures, Graph embedding, Which aims to project nodes into low dimensional space while preserving the topological structure information of graphs and the information of nodes themselves, Has ...
A multi-sensor deep learning approach for complex daily living activity recognition
DigiBiom '22: Proceedings of the 2022 Workshop on Emerging Devices for Digital BiomarkersWith the ever increasing elderly population, human activity trackers can help monitor the daily physical activities performed by the elderly in order to contribute towards improvements in independent living and quality of life.
Little activity ...
Comments
Information & Contributors
Information
Published In
Copyright © 2024 ACM.
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 12 January 2024
Published in IMWUT Volume 7, Issue 4
Check for updates
Author Tags
Qualifiers
- Research-article
- Research
- Refereed
Funding Sources
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 607Total Downloads
- Downloads (Last 12 months)607
- Downloads (Last 6 weeks)158
Reflects downloads up to 09 Nov 2024
Other Metrics
Citations
View Options
Get Access
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in