Location via proxy:   [ UP ]  
[Report a bug]   [Manage cookies]                
skip to main content
research-article
Open access

Bootstrapping Human Activity Recognition Systems for Smart Homes from Scratch

Published: 07 September 2022 Publication History
  • Get Citation Alerts
  • Abstract

    Smart Homes have come a long way: From research laboratories in the early days, through (almost) neglect, to their recent revival in real-world environments enabled through the existence of commodity devices and robust, standardized software frameworks. With such availability, human activity recognition (HAR) in smart homes has become attractive for many real-world applications, especially in the domain of Ambient Assisted Living. Yet, getting started with an activity recognition system in specific smart homes, which are highly specialized spaces inhabited by individuals with idiosyncratic behaviors and habits, is a non-trivial endeavor. We present an approach for bootstrapping HAR systems for individual smart homes from scratch. At the beginning of the life cycle of a smart home, our system passively observes activities and derives rich representations for sensor data-action units-which are then aggregated into activity models through motif learning with minimal supervision. The resulting HAR system is then capable of recognizing relevant, most frequent activities in a smart home. We demonstrate the effectiveness of our bootstrapping procedure through experimental evaluations on CASAS datasets that show the practical value of our approach.

    References

    [1]
    2021. https://www.businesswire.com/news/home/20211207005627/en/Global-Smart-Home-Market-Research-Report-2021-to-2026---by-Product-Technologies-Service-Deployment-and-Region---ResearchAndMarkets.com
    [2]
    2021. Smart home - United States: Statista market forecast. https://www.statista.com/outlook/dmo/smart-home/united-states
    [3]
    2021. Smart home market. https://www.marketsandmarkets.com/Market-Reports/smart-homes-and-assisted-living-advanced-technologie-and-global-market-121.html
    [4]
    Hande Alemdar, Halil Ertan, Ozlem Durmaz Incel, and Cem Ersoy. 2013. ARAS human activity datasets in multiple homes with multiple residents. In 2013 7th International Conference on Pervasive Computing Technologies for Healthcare and Workshops. IEEE, 232--235.
    [5]
    Hande Alemdar, van Kasteren, and Cem Ersoy. 2017. Active learning with uncertainty sampling for large scale activity recognition in smart homes. Ambient Intelligence and Smart Environments 9, 2, 209--223.
    [6]
    S. Aminikhanghahi and D. J. Cook. 2017. Using change point detection to automate daily activity segmentation. IEEE international conference on pervasive computing and communications workshops (2017), 262--267.
    [7]
    S. Aminikhanghahi, T. Wang, and D. J. Cook. 2018. Real-time change point detection with application to smart home time series data. IEEE Transactions on Knowledge and Data Engineering 31, 5 (2018), 1010--1023.
    [8]
    Salikh Bagaveyev and Diane J. Cook. 2014. Designing and Evaluating Active Learning Methods for Activity Recognition. In Proceedings of the 2014 ACM International Joint Conference on Pervasive and Ubiquitous Computing: Adjunct Publication, 469--478.
    [9]
    Asma Benmansour, Abdelhamid Bouchachia, and Mohammed Feham. 2015. Multioccupant activity recognition in pervasive smart home environments. ACM Computing Surveys (CSUR) 48, 3 (2015), 1--36.
    [10]
    U. Bermejo, A. Almeida, A. Bilbao-Jayo, and G. Azkune. 2021. Embedding-based real-time change point detection with application to activity segmentation in smart home time series data. Expert Systems with Applications 185, 15 (2021).
    [11]
    Damien Bouchabou, Sao Mai Nguyen, Christophe Lohr, Benoit Leduc, and Ioannis Kanellos. 2021. Fully convolutional network bootstrapped by word encoding and embedding for activity recognition in smart homes. In International Workshop on Deep Learning for Human Activity Recognition. Springer, 111--125.
    [12]
    Damien Bouchabou, Sao Mai Nguyen, Christophe Lohr, Benoit LeDuc, Ioannis Kanellos, et al. 2021. Using Language Model to Bootstrap Human Activity Recognition Ambient Sensors Based in Smart Homes. Electronics 10, 20 (2021), 2498.
    [13]
    Andreas Bulling, Ulf Blanke, and Bernt Schiele. 2014. A tutorial on human activity recognition using body-worn inertial sensors. ACM Computing Surveys (CSUR) 46, 3 (2014), 1--33.
    [14]
    Ricardo Chavarriaga, Hesam Sagha, Alberto Calatroni, Sundara Tejaswi Digumarti, Gerhard Tröster, José del R Millán, and Daniel Roggen. 2013. The Opportunity challenge: A benchmark database for on-body sensor-based activity recognition. Pattern Recognition Letters 34, 15 (2013), 2033--2042.
    [15]
    Bill Chiu, Eamonn Keogh, and Stefano Lonardi. 2003. Probabilistic discovery of time series motifs. In Proceedings of the ninth ACM SIGKDD international conference on Knowledge discovery and data mining. 493--498.
    [16]
    Diane J Cook, Aaron S Crandall, Brian L Thomas, and Narayanan C Krishnan. 2012. CASAS: A smart home in a box. Computer 46, 7 (2012), 62--69.
    [17]
    Diane J Cook, Michael Youngblood, Edwin O Heierman, Karthik Gopalratnam, Sira Rao, Andrey Litvin, and Farhan Khawaja. 2003. MavHome: An agent-based smart home. In Proceedings of the First IEEE International Conference on Pervasive Computing and Communications, 2003.(PerCom 2003). IEEE, 521--524.
    [18]
    Devleena Das, Yasutaka Nishimura, Rajan P Vivek, Naoto Takeda, Sean T Fish, Thomas Ploetz, and Sonia Chernova. 2021. Explainable Activity Recognition for Smart Home Systems. arXiv preprint arXiv:2105.09787 (2021).
    [19]
    Hoang Anh Dau, Anthony Bagnall, Kaveh Kamgar, Chin-Chia Michael Yeh, Yan Zhu, Shaghayegh Gharghabi, Chotirat Ann Ratanamahatana, and Eamonn Keogh. 2019. The UCR time series archive. IEEE/CAA Journal of Automatica Sinica 6, 6 (2019), 1293--1305.
    [20]
    Hongqing Fang and Chen Hu. 2014. Recognizing human activity in smart home using deep learning algorithm. In Proceedings of the 33rd chinese control conference. IEEE, 4716--4720.
    [21]
    Yan Gao, Yang Long, Yu Guan, Anna Basu, Jessica Baggaley, and Thomas Ploetz. 2019. Towards reliable, automated general movement assessment for perinatal stroke screening in infants using wearable accelerometers. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 3, 1 (2019), 1--22.
    [22]
    Hristijan Gjoreski and Daniel Roggen. 2017. Unsupervised online activity discovery using temporal behaviour assumption. In Proceedings of the 2017 ACM International Symposium on Wearable Computers. 42--49.
    [23]
    Yu Guan and Thomas Plötz. 2017. Ensembles of deep lstm learners for activity recognition using wearables. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 1, 2 (2017), 1--28.
    [24]
    Nils Y Hammerla, Shane Halloran, and Thomas Plötz. 2016. Deep, convolutional, and recurrent models for human activity recognition using wearables. arXiv preprint arXiv:1604.08880 (2016).
    [25]
    Shruthi Kashinath Hiremath and Thomas Ploetz. 2021. On the Role of Context Length for Feature Extraction and Sequence Modeling in Human Activity Recognition. In 2021 International Symposium on Wearable Computers. 13--17.
    [26]
    Shruthi K Hiremath and Thomas Plötz. 2020. Deriving effective human activity recognition systems through objective task complexity assessment. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 4, 4 (2020), 1--24.
    [27]
    Clare J Hooper, Anne Preston, Madeline Balaam, Paul Seedhouse, Daniel Jackson, Cuong Pham, Cassim Ladha, Karim Ladha, Thomas Plötz, and Patrick Olivier. 2012. The french kitchen: Task-based learning in an instrumented kitchen. In Proceedings of the 2012 ACM Conference on Ubiquitous Computing. 193--202.
    [28]
    E. Hoque and J. Stankovic. 2012. AALO: Activity recognition in smart homes using Active Learning in the presence of Overlapped activities. Proceedings of the 6th international conference on pervasive computing technologies for healthcare, 139--146.
    [29]
    Michael R Hufford. 2007. Special methodological challenges and opportunities in ecological momentary assessment. The science of real-time data capture: Self-reports in health research (2007), 54--75.
    [30]
    Stephen S Intille, Ling Bao, Emmanuel Munguia Tapia, and John Rondoni. 2004. Acquiring in situ training data for context-aware ubiquitous computing applications. In Proceedings of the SIGCHI conference on Human factors in computing systems. 1--8.
    [31]
    Stephen S Intille, Kent Larson, J Beaudin, E Munguia Tapia, Pallavi Kaushik, Jason Nawyn, and Thomas J McLeish. 2005. The PlaceLab: A live-in laboratory for pervasive computing research (video). Proceedings of PERVASIVE 2005 Video Program (2005).
    [32]
    Stephen S Intille, Emmanuel Munguia Tapia, John Rondoni, Jennifer Beaudin, Chuck Kukla, Sitij Agarwal, Ling Bao, and Kent Larson. 2003. Tools for studying behavior and technology in natural settings. In International Conference on Ubiquitous Computing. Springer, 157--174.
    [33]
    Kenton Lee Jacob Devlin, Ming-Wei Chang and Kristina Toutanova. 2018. Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 (2018).
    [34]
    Saurav Jha, Martin Schiemer, and Juan Ye. 2020. Continual learning in human activity recognition: an empirical analysis of regularization. arXiv preprint arXiv:2007.03032 (2020).
    [35]
    Saurav Jha, Martin Schiemer, Franco Zambonelli, and Juan Ye. 2021. Continual learning in sensor-based human activity recognition: An empirical benchmark analysis. Information Sciences 575 (2021), 1--21.
    [36]
    R. Kavitha and Sumitra Binu. 2019. Performance Evaluation of Area-Based Segmentation Technique on Ambient Sensor Data for Smart Home Assisted Living. International Conference on recent Trends in Advanced Computing 165, 314--321.
    [37]
    Cassim Ladha, Nils Y Hammerla, Patrick Olivier, and Thomas Plötz. 2013. ClimbAX: skill assessment for climbing enthusiasts. In Proceedings of the 2013 ACM international joint conference on Pervasive and ubiquitous computing. 235--244.
    [38]
    Jingyuan Li and Eli Shlizerman. 2020. Iterate & cluster: Iterative semi-supervised action recognition. arXiv preprint arXiv:2006.06911 (2020).
    [39]
    Daniele Liciotti, Michele Bernardini, Luca Romeo, and Emanuele Frontoni. 2020. A sequential deep learning application for recognising human activities in smart homes. Neurocomputing 396 (2020), 501--513.
    [40]
    Jessica Lin, Eamonn Keogh, Li Wei, and Stefano Lonardi. 2007. Experiencing SAX: a novel symbolic representation of time series. Data Mining and knowledge discovery 15, 2 (2007), 107--144.
    [41]
    Yang Liu. 2019. Fine-tune BERT for extractive summarization. arXiv preprint arXiv:1903.10318 (2019).
    [42]
    David Minnen, Thad Starner, Irfan A Essa, and Charles Lee Isbell Jr. 2007. Improving Activity Discovery with Automatic Neighborhood Estimation. In IJCAI, Vol. 7. 2814--2819.
    [43]
    Tudor Miu, Paolo Missier, and Thomas Plötz. 2015. Bootstrapping personalised human activity recognition models using online active learning. In 2015 IEEE International Conference on Computer and Information Technology; Ubiquitous Computing and Communications; Dependable, Autonomic and Secure Computing; Pervasive Intelligence and Computing. IEEE, 1138--1147.
    [44]
    Tudor Miu, Thomas Plötz, Paolo Missier, and Daniel Roggen. 2014. On strategies for budget-based online annotation in human activity recognition. In Proceedings of the 2014 ACM International Joint Conference on Pervasive and Ubiquitous Computing: Adjunct Publication. 767--776.
    [45]
    Mehrab Bin Morshed, Samruddhi Shreeram Kulkarni, Richard Li, Koustuv Saha, Leah Galante Roper, Lama Nachman, Hong Lu, Lucia Mirabella, and Sanjeev Srivastava. 2020. Munmun De Choudhury, Kaya de Barbaro, Thomas Ploetz, and Gregory D Abowd. 2020. A Real-Time Eating Detection System for Capturing Eating Moments and Triggering Ecological Momentary Assessments to Obtain Further Context: System Development and Validation Study. JMIR Mhealth Uhealth 4, 3 (2020), 18.
    [46]
    JT O'Brien, P Gallagher, D Stow, N Hammerla, T Ploetz, M Firbank, C Ladha, K Ladha, D Jackson, Roisin McNaney, et al. 2017. A study of wrist-worn activity measurement as a potential real-world biomarker for late-life depression. Psychological medicine 47, 1 (2017), 93--102.
    [47]
    Patrick Olivier, Guangyou Xu, Andrew Monk, and Jesse Hoey. 2009. Ambient kitchen: designing situated services using a high fidelity prototyping environment. In Proceedings of the 2nd international conference on pervasive technologies related to assistive environments. 1--7.
    [48]
    Francisco Javier Ordóñez and Daniel Roggen. 2016. Deep convolutional and lstm recurrent neural networks for multimodal wearable activity recognition. Sensors 16, 1 (2016), 115.
    [49]
    Alexandros Paraschos, Christian Daniel, Jan R Peters, and Gerhard Neumann. 2013. Probabilistic movement primitives. Advances in neural information processing systems 26 (2013).
    [50]
    Thomas Plotz, Chen Chen, Nils Y Hammerla, and Gregory D Abowd. 2012. Automatic synchronization of wearable sensors and video-cameras for ground truth annotation-A practical approach. In 2012 16th international symposium on wearable computers. IEEE, 100--103.
    [51]
    Thomas Plötz, Paula Moynihan, Cuong Pham, and Patrick Olivier. 2011. Activity recognition and healthier food preparation. In Activity Recognition in Pervasive Intelligent Environments. Springer, 313--329.
    [52]
    Gupta Prankit, Richard McClatchey, and Praminda Caleb-Solly. 2020. Tracking changes in user activity from unlabelled smart home sensor data using unsupervised learning methods. In Neural Computing and Applications, Vol. 32. Springer, 12351--12362.
    [53]
    Bronagh Quigley, Mark Donnelly, George Moore, and Leo Galway. 2018. A comparative analysis of windowing approaches in dense sensing environments. Multidisciplinary Digital Publishing Institute Proceedings 2, 19 (2018), 1245.
    [54]
    Nilam Ram, Miriam Brinberg, Aaron L Pincus, and David E Conroy. 2017. The questionable ecological validity of ecological momentary assessment: Considerations for design and analysis. Research in Human Development 14, 3 (2017), 253--270.
    [55]
    Sylvestre-Alvise Rebuffi, Alexander Kolesnikov, Georg Sperl, and Christoph H Lampert. 2017. icarl: Incremental classifier and representation learning. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition. 2001--2010.
    [56]
    Burr Settles. 2009. Active learning literature survey. (2009).
    [57]
    Joshua M Smyth and Arthur A Stone. 2003. Ecological momentary assessment research in behavioral medicine. Journal of Happiness studies 4, 1 (2003), 35--52.
    [58]
    Jake Snell, Kevin Swersky, and Richard Zemel. 2017. Prototypical networks for few-shot learning. Advances in neural information processing systems 30 (2017).
    [59]
    Emmanuel Munguia Tapia, Stephen S Intille, and Kent Larson. 2004. Activity recognition in the home using simple and ubiquitous sensors. In International conference on pervasive computing. Springer, 158--175.
    [60]
    Y-I Tian, Takeo Kanade, and Jeffrey F Cohn. 2001. Recognizing action units for facial expression analysis. IEEE Transactions on pattern analysis and machine intelligence 23, 2 (2001), 97--115.
    [61]
    Alireza Vahdatpour, Navid Amini, and Majid Sarrafzadeh. 2009. Toward unsupervised activity discovery using multi dimensional motif detection in time series. In Twenty-First International Joint Conference on Artificial Intelligence. Citeseer.
    [62]
    Tim Van Kasteren, Athanasios Noulas, Gwenn Englebienne, and Ben Kröse. 2008. Accurate activity recognition in a home setting. In Proceedings of the 10th international conference on Ubiquitous computing. 1--9.
    [63]
    Karl Weiss, Taghi M Khoshgoftaar, and DingDing Wang. 2016. A survey of transfer learning. Journal of Big data 3, 1 (2016), 1--40.
    [64]
    Garrett Wilson and Diane J Cook. 2020. A survey of unsupervised deep domain adaptation. ACM Transactions on Intelligent Systems and Technology (TIST) 11, 5 (2020), 1--46.
    [65]
    Jason Wu, Chris Harrison, Jeffrey P Bigham, and Gierad Laput. 2020. Automated Class Discovery and One-Shot Interactions for Acoustic Activity Recognition. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems. 1--14.
    [66]
    Renzhi Wu, Nilaksh Das, Sanya Chaba, Sakshi Gandhi, Duen Horng Chau, and Xu Chu. 2022. A Cluster-then-label Approach for Few-shot Learning with Application to Automatic Image Data Labeling. ACM Journal of Data and Information Quality (JDIQ) (March 2022).
    [67]
    Dongkuan Xu and Yingjie Tian. 2015. A comprehensive survey of clustering algorithms. Annals of Data Science 2, 2 (2015), 165--193.

    Cited By

    View all
    • (2024)UbiPhysioProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36435528:1(1-27)Online publication date: 6-Mar-2024
    • (2024)TS2ACTProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36314457:4(1-22)Online publication date: 12-Jan-2024
    • (2024)Reenvisioning Patient Education with Smart Hospital Patient RoomsProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36314197:4(1-23)Online publication date: 12-Jan-2024
    • Show More Cited By

    Recommendations

    Comments

    Information & Contributors

    Information

    Published In

    cover image Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies
    Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies  Volume 6, Issue 3
    September 2022
    1612 pages
    EISSN:2474-9567
    DOI:10.1145/3563014
    Issue’s Table of Contents
    This work is licensed under a Creative Commons Attribution International 4.0 License.

    Publisher

    Association for Computing Machinery

    New York, NY, United States

    Publication History

    Published: 07 September 2022
    Published in IMWUT Volume 6, Issue 3

    Check for updates

    Author Tags

    1. human activity recognition
    2. pattern recognition
    3. smart-home

    Qualifiers

    • Research-article
    • Research
    • Refereed

    Funding Sources

    • KDDI Research

    Contributors

    Other Metrics

    Bibliometrics & Citations

    Bibliometrics

    Article Metrics

    • Downloads (Last 12 months)614
    • Downloads (Last 6 weeks)43
    Reflects downloads up to 11 Aug 2024

    Other Metrics

    Citations

    Cited By

    View all
    • (2024)UbiPhysioProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36435528:1(1-27)Online publication date: 6-Mar-2024
    • (2024)TS2ACTProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36314457:4(1-22)Online publication date: 12-Jan-2024
    • (2024)Reenvisioning Patient Education with Smart Hospital Patient RoomsProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36314197:4(1-23)Online publication date: 12-Jan-2024
    • (2024)Semantic LossProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36314077:4(1-29)Online publication date: 12-Jan-2024
    • (2024)An Improved Masking Strategy for Self- Supervised Masked Reconstruction in Human Activity RecognitionIEEE Sensors Journal10.1109/JSEN.2024.339075524:11(18699-18709)Online publication date: 1-Jun-2024
    • (2024)A Synergistic Large Language Model and Supervised Learning Approach to Zero-Shot and Continual Activity Recognition in Smart Homes2024 9th International Conference on Big Data Analytics (ICBDA)10.1109/ICBDA61153.2024.10607364(113-122)Online publication date: 16-Mar-2024
    • (2024)SelfAct: Personalized Activity Recognition Based on Self-Supervised and Active LearningMobile and Ubiquitous Systems: Computing, Networking and Services10.1007/978-3-031-63989-0_19(375-391)Online publication date: 19-Jul-2024
    • (2023)HARE: Unifying the Human Activity Recognition Engineering WorkflowSensors10.3390/s2323957123:23(9571)Online publication date: 2-Dec-2023
    • (2023)The Lifespan of Human Activity Recognition Systems for Smart HomesSensors10.3390/s2318772923:18(7729)Online publication date: 7-Sep-2023
    • (2023)Review on Human Action Recognition in Smart Living: Sensing Technology, Multimodality, Real-Time Processing, Interoperability, and Resource-Constrained ProcessingSensors10.3390/s2311528123:11(5281)Online publication date: 2-Jun-2023

    View Options

    View options

    PDF

    View or Download as a PDF file.

    PDF

    eReader

    View online with eReader.

    eReader

    Get Access

    Login options

    Full Access

    Media

    Figures

    Other

    Tables

    Share

    Share

    Share this Publication link

    Share on social media