research-article

Open access

Semantic-Discriminative Mixup for Generalizable Sensor-based Cross-domain Activity Recognition

Authors:

Sinno Jialin Pan,

Xin QinAuthors Info & Claims

Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies, Volume 6, Issue 2

Article No.: 65, Pages 1 - 19

https://doi.org/10.1145/3534589

Published: 07 July 2022 Publication History

Abstract

It is expensive and time-consuming to collect sufficient labeled data to build human activity recognition (HAR) models. Training on existing data often makes the model biased towards the distribution of the training data, thus the model might perform terribly on test data with different distributions. Although existing efforts on transfer learning and domain adaptation try to solve the above problem, they still need access to unlabeled data on the target domain, which may not be possible in real scenarios. Few works pay attention to training a model that can generalize well to unseen target domains for HAR. In this paper, we propose a novel method called Semantic-Discriminative Mixup (SDMix) for generalizable cross-domain HAR. Firstly, we introduce semantic-aware Mixup that considers the activity semantic ranges to overcome the semantic inconsistency brought by domain differences. Secondly, we introduce the large margin loss to enhance the discrimination of Mixup to prevent misclassification brought by noisy virtual labels. Comprehensive generalization experiments on five public datasets demonstrate that our SDMix substantially outperforms the state-of-the-art approaches with 6% average accuracy improvement on cross-person, cross-dataset, and cross-position HAR.

References

[1]

Davide Anguita, Alessandro Ghio, Luca Oneto, Xavier Parra, and Jorge L Reyes-Ortiz. 2012. Human activity recognition on smartphones using a multiclass hardware-friendly support vector machine. In International workshop on ambient assisted living. Springer, 216--223.

Digital Library

[2]

Yogesh Balaji, Swami Sankaranarayanan, and Rama Chellappa. 2018. Metareg: Towards domain generalization using meta-regularization. Advances in neural information processing systems 31 (2018).

[3]

Billur Barshan and Murat Cihan Yüksek. 2014. Recognizing daily and sports activities in two open source machine learning environments using body-worn sensor units. Comput. J. 57, 11 (2014), 1649--1667.

[4]

Andreas Bulling, Ulf Blanke, and Bernt Schiele. 2014. A tutorial on human activity recognition using body-worn inertial sensors. ACM Computing Surveys (CSUR) 46, 3 (2014), 1--33.

Digital Library

[5]

Youngjae Chang, Akhil Mathur, Anton Isopoussu, Junehwa Song, and Fahim Kawsar. 2020. A systematic study of unsupervised domain adaptation for robust human-activity recognition. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 4, 1 (2020), 1--30.

Digital Library

[6]

Diane Cook, Kyle D Feuz, and Narayanan C Krishnan. 2013. Transfer learning for activity recognition: A survey. Knowledge and information systems 36, 3 (2013), 537--556.

Digital Library

[7]

Gamaleldin Elsayed, Dilip Krishnan, Hossein Mobahi, Kevin Regan, and Samy Bengio. 2018. Large Margin Deep Networks for Classification. In Advances in Neural Information Processing Systems, Vol. 31. 842--852.

[8]

Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario Marchand, and Victor Lempitsky. 2016. Domain-adversarial training of neural networks. The journal of machine learning research 17, 1 (2016), 2096--2030.

[9]

Vikas Garg, Adam Tauman Kalai, Katrina Ligett, and Steven Wu. 2021. Learn to expect the unexpected: Probably approximately correct domain generalization. In International Conference on Artificial Intelligence and Statistics. PMLR, 3574--3582.

[10]

Negar Golestani and Mahta Moghaddam. 2020. Human activity recognition using magnetic induction-based motion signals and deep recurrent neural networks. Nature communications 11, 1 (2020), 1--11.

[11]

Zeyi Huang, Haohan Wang, Eric P Xing, and Dong Huang. 2020. Self-challenging improves cross-domain generalization. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part II 16. Springer, 124--140.

Digital Library

[12]

Maximilian Ilse, Jakub M Tomczak, Christos Louizos, and Max Welling. 2020. Diva: Domain invariant variational autoencoders. In Medical Imaging with Deep Learning. PMLR, 322--348.

[13]

Ya Li, Xinmei Tian, Mingming Gong, Yajing Liu, Tongliang Liu, Kun Zhang, and Dacheng Tao. 2018. Deep domain generalization via conditional invariant adversarial networks. In Proceedings of the European Conference on Computer Vision (ECCV). 624--639.

Digital Library

[14]

Alexander H Liu, Yen-Cheng Liu, Yu-Ying Yeh, and Yu-Chiang Frank Wang. 2018. A unified feature disentangler for multi-domain image translation and manipulation. In Proceedings of the 32nd International Conference on Neural Information Processing Systems. 2595--2604.

[15]

Wang Lu, Yiqiang Chen, Jindong Wang, and Xin Qin. 2021. Cross-domain activity recognition via substructural optimal transport. Neurocomputing 454 (2021), 65--75.

[16]

Massimiliano Mancini, Samuel Rota Bulo, Barbara Caputo, and Elisa Ricci. 2018. Best sources forward: domain generalization through source-specific nets. In 2018 25th IEEE international conference on image processing (ICIP). IEEE, 1353--1357.

[17]

Daniela Micucci, Marco Mobilio, and Paolo Napoletano. 2017. Unimib shar: A dataset for human activity recognition using acceleration data from smartphones. Applied Sciences 7, 10 (2017), 1101.

[18]

Narges Honarvar Nazari and Adriana Kovashka. 2020. Domain generalization using shape representation. In European Conference on Computer Vision. Springer, 666--670.

[19]

Sinno Jialin Pan and Qiang Yang. 2009. A survey on transfer learning. IEEE Transactions on knowledge and data engineering 22, 10 (2009), 1345--1359.

Digital Library

[20]

Giambattista Parascandolo, Alexander Neitz, Antonio Orvieto, Luigi Gresele, and Bernhard Schölkopf. 2021. Learning explanations that are hard to vary. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net. https://openreview.net/forum?id=hb1sDDSLbV

[21]

Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. 2019. Pytorch: An imperative style, high-performance deep learning library. In Advances in neural information processing systems, Vol. 32. 8026--8037.

[22]

Hangwei Qian, Sinno Jialin Pan, and Chunyan Miao. 2021. Latent Independent Excitation for Generalizable Sensor-based Cross-Person Activity Recognition. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 35. 11921--11929.

[23]

Fengchun Qiao, Long Zhao, and Xi Peng. 2020. Learning to learn single domain generalization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 12556--12565.

[24]

Xin Qin, Yiqiang Chen, Jindong Wang, and Chaohui Yu. 2019. Cross-dataset activity recognition via adaptive spatial-temporal transfer learning. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 3, 4 (2019), 1--25.

Digital Library

[25]

Mohammad Mahfujur Rahman, Clinton Fookes, Mahsa Baktashmotlagh, and Sridha Sridharan. 2020. Correlation-aware adversarial domain adaptation and generalization. Pattern Recognition 100 (2020), 107124.

Digital Library

[26]

Attila Reiss and Didier Stricker. 2012. Introducing a new benchmarked dataset for activity monitoring. In 2012 16th international symposium on wearable computers. IEEE, 108--109.

Digital Library

[27]

Shiori Sagawa, Pang Wei Koh, Tatsunori B. Hashimoto, and Percy Liang. 2020. Distributionally Robust Neural Networks. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net. https://openreview.net/forum?id=ryxGuJrFvS

[28]

Yuge Shi, Jeffrey Seely, Philip H. S. Torr, N. Siddharth, Awni Hannun, Nicolas Usunier, and Gabriel Synnaeve. 2021. Gradient Matching for Domain Generalization. CoRR abs/2104.09937 (2021). arXiv:2104.09937 https://arxiv.org/abs/2104.09937

[29]

Baochen Sun and Kate Saenko. 2016. Deep coral: Correlation alignment for deep domain adaptation. In European conference on computer vision. Springer, 443--450.

[30]

Bailin Wang, Mirella Lapata, and Ivan Titov. 2021. Meta-learning for domain generalization in semantic parsing. In NAACL.

[31]

Jindong Wang, Yiqiang Chen, Shuji Hao, Xiaohui Peng, and Lisha Hu. 2019. Deep learning for sensor-based activity recognition: A survey. Pattern Recognition Letters 119 (2019), 3--11.

Digital Library

[32]

Jindong Wang, Yiqiang Chen, Lisha Hu, Xiaohui Peng, and S Yu Philip. 2018. Stratified transfer learning for cross-domain activity recognition. In 2018 IEEE International Conference on Pervasive Computing and Communications (PerCom). IEEE, 1--10.

[33]

Jindong Wang, Cuiling Lan, Chang Liu, Yidong Ouyang, and Tao Qin. 2021. Generalizing to Unseen Domains: A Survey on Domain Generalization. In Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, IJCAI-21. 4627--4635. Survey Track.

[34]

Jindong Wang, Vincent W Zheng, Yiqiang Chen, and Meiyu Huang. 2018. Deep transfer learning for cross-domain activity recognition. In proceedings of the 3rd International Conference on Crowd Science and Engineering. 1--8.

Digital Library

[35]

Yufei Wang, Haoliang Li, and Alex C Kot. 2020. Heterogeneous domain generalization via domain mixup. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 3622--3626.

[36]

Garrett Wilson and Diane J Cook. 2020. A survey of unsupervised deep domain adaptation. ACM Transactions on Intelligent Systems and Technology (TIST) 11, 5 (2020), 1--46.

Digital Library

[37]

Tzu-Tsung Wong. 2015. Performance evaluation of classification algorithms by k-fold and leave-one-out cross validation. Pattern Recognition 48, 9 (2015), 2839--2846.

Digital Library

[38]

Yuan Wu, Diana Inkpen, and Ahmed El-Roby. 2020. Dual mixup regularized learning for adversarial domain adaptation. In European Conference on Computer Vision. Springer, 540--555.

Digital Library

[39]

Minghao Xu, Jian Zhang, Bingbing Ni, Teng Li, Chengjie Wang, Qi Tian, and Wenjun Zhang. 2020. Adversarial domain adaptation with domain mixup. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 34. 6502--6509.

[40]

Qinwei Xu, Ruipeng Zhang, Ya Zhang, Yanfeng Wang, and Qi Tian. 2021. A Fourier-based Framework for Domain Generalization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 14383--14392.

[41]

Zheng Xu, Wen Li, Li Niu, and Dong Xu. 2014. Exploiting low-rank structure from latent domains for domain generalization. In European Conference on Computer Vision. Springer, 628--643.

[42]

Hongyi Zhang, Moustapha Cissé, Yann N. Dauphin, and David Lopez-Paz. 2018. mixup: Beyond Empirical Risk Minimization. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 - May 3, 2018, Conference Track Proceedings. OpenReview.net. https://openreview.net/forum?id=r1Ddp1-Rb

[43]

Mi Zhang and Alexander A Sawchuk. 2012. USC-HAD: a daily activity dataset for ubiquitous activity recognition using wearable sensors. In Proceedings of the 2012 ACM conference on ubiquitous computing. 1036--1043.

Digital Library

Cited By

Hong ZLi ZZhong SLyu WWang HDing YHe TZhang D(2024)CrossHAR: Generalizing Cross-dataset Human Activity Recognition via Hierarchical Self-Supervised PretrainingProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36595978:2(1-26)Online publication date: 15-May-2024
https://dl.acm.org/doi/10.1145/3659597
Wang SWang JXi HZhang BZhang LWei H(2024)Optimization-Free Test-Time Adaptation for Cross-Person Activity RecognitionProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36314507:4(1-27)Online publication date: 12-Jan-2024
https://dl.acm.org/doi/10.1145/3631450
Lu WWang JSun XChen YJi XYang QXie X(2024)Diversify: A General Framework for Time Series Out-of-Distribution Detection and GeneralizationIEEE Transactions on Pattern Analysis and Machine Intelligence10.1109/TPAMI.2024.335521246:6(4534-4550)Online publication date: Jun-2024
https://doi.org/10.1109/TPAMI.2024.3355212
Show More Cited By

Index Terms

Semantic-Discriminative Mixup for Generalizable Sensor-based Cross-domain Activity Recognition
1. Computing methodologies
  1. Machine learning
    1. Learning paradigms
      1. Multi-task learning
        Transfer learning
2. Human-centered computing
  1. Ubiquitous and mobile computing
    1. Ubiquitous and mobile computing theory, concepts and paradigms
      1. Ubiquitous computing

Recommendations

Domain Generalization for Activity Recognition via Adaptive Feature Fusion
Human activity recognition requires the efforts to build a generalizable model using the training datasets with the hope to achieve good performance in test datasets. However, in real applications, the training and testing datasets may have totally ...
Cross-Dataset Activity Recognition via Adaptive Spatial-Temporal Transfer Learning

Human activity recognition (HAR) aims at recognizing activities by training models on the large quantity of sensor data. Since it is time-consuming and expensive to acquire abundant labeled data, transfer learning becomes necessary for HAR by ...
Generalizable Low-Resource Activity Recognition with Diverse and Discriminative Representation Learning
KDD '23: Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining

Human activity recognition (HAR) is a time series classification task that focuses on identifying the motion patterns from human sensor readings. Adequate data is essential but a major bottleneck for training a generalizable HAR model, which assists ...

Comments

Information & Contributors

Information

Published In

cover image Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies

Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies Volume 6, Issue 2

July 2022

1551 pages

EISSN:2474-9567

DOI:10.1145/3547347

Issue’s Table of Contents

Copyright © 2022 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 07 July 2022

Published in IMWUT Volume 6, Issue 2

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed

Funding Sources

Natural Science Foundation of China
Beijing Municipal Science & Technology Commission
National Key Research and Development Plan of China

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

12
Total Citations
View Citations
936
Total Downloads

Downloads (Last 12 months)350
Downloads (Last 6 weeks)31

Reflects downloads up to 01 Sep 2024

Other Metrics

View Author Metrics

Citations

Cited By

Hong ZLi ZZhong SLyu WWang HDing YHe TZhang D(2024)CrossHAR: Generalizing Cross-dataset Human Activity Recognition via Hierarchical Self-Supervised PretrainingProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36595978:2(1-26)Online publication date: 15-May-2024
https://dl.acm.org/doi/10.1145/3659597
Wang SWang JXi HZhang BZhang LWei H(2024)Optimization-Free Test-Time Adaptation for Cross-Person Activity RecognitionProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36314507:4(1-27)Online publication date: 12-Jan-2024
https://dl.acm.org/doi/10.1145/3631450
Lu WWang JSun XChen YJi XYang QXie X(2024)Diversify: A General Framework for Time Series Out-of-Distribution Detection and GeneralizationIEEE Transactions on Pattern Analysis and Machine Intelligence10.1109/TPAMI.2024.335521246:6(4534-4550)Online publication date: Jun-2024
https://doi.org/10.1109/TPAMI.2024.3355212
Gu XDeligianni FHan JLiu XChen WYang GLo B(2024)Beyond Supervised Learning for Pervasive HealthcareIEEE Reviews in Biomedical Engineering10.1109/RBME.2023.329693817(42-62)Online publication date: 2024
https://doi.org/10.1109/RBME.2023.3296938
Wei BYi CZhang QZhu HZhu JJiang F(2024)ActiveSelfHAR: Incorporating Self-Training Into Active Learning to Improve Cross-Subject Human Activity RecognitionIEEE Internet of Things Journal10.1109/JIOT.2023.331415011:4(6833-6847)Online publication date: 15-Feb-2024
https://doi.org/10.1109/JIOT.2023.3314150
Yin YXie LJiang ZXiao FCao JLu S(2024)A Systematic Review of Human Activity Recognition Based on Mobile Devices: Overview, Progress and TrendsIEEE Communications Surveys & Tutorials10.1109/COMST.2024.335759126:2(890-929)Online publication date: 23-Jan-2024
https://dl.acm.org/doi/10.1109/COMST.2024.3357591
Duan HWang SOjha VWang SHuang YLong YRanjan RZheng Y(2024)Wearable-based behaviour interpolation for semi-supervised human activity recognitionInformation Sciences: an International Journal10.1016/j.ins.2024.120393665:COnline publication date: 2-Jul-2024
https://dl.acm.org/doi/10.1016/j.ins.2024.120393
Jianhua TJiangtao GNan GSiwei FShan LChun Y(2023)Human-computer interaction for virtual-real fusionJournal of Image and Graphics10.11834/jig.23002028:6(1513-1542)Online publication date: 2023
https://doi.org/10.11834/jig.230020
Qin XWang JMa SLu WZhu YXie XChen YSingh ASun YAkoglu LGunopulos DYan XKumar ROzcan FYe J(2023)Generalizable Low-Resource Activity Recognition with Diverse and Discriminative Representation LearningProceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining10.1145/3580305.3599360(1943-1953)Online publication date: 6-Aug-2023
https://dl.acm.org/doi/10.1145/3580305.3599360
Li CZhang DHuang WZhang J(2023)Cross Contrasting Feature Perturbation for Domain Generalization2023 IEEE/CVF International Conference on Computer Vision (ICCV)10.1109/ICCV51070.2023.00128(1327-1337)Online publication date: 1-Oct-2023
https://doi.org/10.1109/ICCV51070.2023.00128
Show More Cited By

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

Media

Figures

Other

Tables

View Issue’s Table of Contents