A hybrid and context-aware framework for normal and abnormal human behavior recognition
Authors: 
Roghayeh Mojarad, Abdelghani Chibani, Ferhat Attal, Ghazaleh Khodabandelou, Yacine AmiratAuthors Info & Claims
Roghayeh Mojarad, Abdelghani Chibani, Ferhat Attal, Ghazaleh Khodabandelou, Yacine AmiratAuthors Info & ClaimsAbstract
Human behavior recognition is one of the significant components of Ambient Assisted Living (AAL) systems and personal assistive robots allowing to improve the quality of their lives in terms of safety, autonomy, and well-being. A critical aspect of preventing dangerous situations for users, especially elderlies, is to recognize abnormal human behavior. In spite of the extensive exploration of abnormality recognition in various fields, there remain some challenges in developing effective approaches for recognizing abnormal human behaviors in AAL systems due to the limitations of data-driven and knowledge-driven approaches. In this paper, a context-aware framework combining data-driven and knowledge-driven approaches is proposed to better characterize human behaviors and recognize abnormal behaviors using commonsense reasoning while considering human behavior context. The proposed framework comprises five main modules, which leverage Long Short-Term Memory (LSTM) models and Probabilistic Answer Set Programming (PASP)-based commonsense reasoning to recognize human activities and represent abnormal human behaviors, as well as reason about those behaviors. The proposed framework is evaluated using two datasets, namely Orange4Home and UCI HAR. The obtained results indicate the capability of the proposed framework to characterize human behaviors and recognize abnormal human behaviors with high performance.
References
[1]
Ahmed MA, Zaidan BB, Zaidan AA, et al. Real-time sign language framework based on wearable device: analysis of msl, dataglove, and gesture recognition Soft Comput 2021 25 11101-11122
[2]
Anguita D, Ghio A, Oneto L, Parra X, and Reyes-Ortiz JL A public domain dataset for human activity recognition using smartphones Comput Intell 2013 20 6
[3]
Aran O, Sanchez-Cortes D, Do M-T, Gatica-Perez D (2016) Anomaly detection in elderly daily behavior in ambient sensing environments. In: International workshop on human behavior understanding. Springer, pp 51–67
[4]
Arifoglu D and Bouchachia A Activity recognition and abnormal behaviour detection with recurrent neural networks Proced Comput Sci 2017 110 86-93 14th International Conference on Mobile Systems and Pervasive Computing (MobiSPC 2017)
[5]
Ariza-Colpas PP, Vicario E, Oviedo-Carrascal AI, Butt Aziz S, Piñeres-Melo MA, Quintero-Linero A, and Patara F Human activity recognition data analysis: history, evolutions, and new trends Sensors 2022 22 9 3401
[6]
Artikis A, Makris E, and Paliouras G A probabilistic interval-based event calculus for activity recognition Ann Math Artif Intell 2019 20 1-24
[7]
Artikis A, Paliouras G (2009) Behaviour recognition using the event calculus. In: IFIP international conference on artificial intelligence applications and innovations. Springer, pp 469–478
[8]
Artikis A, Sergot M, Paliouras G (2010) A logic programming approach to activity recognition. In: Proceedings of the 2nd ACM international workshop on events in multimedia, ser. EiMM ’10, pp 3–8
[9]
Banovic N, Buzali T, Chevalier F, Mankoff J, Dey AK (2016) Modeling and understanding human routine behavior. In: Proceedings of the CHI conference on human factors in computing systems, ser. CHI ’16. ACM: New York, pp 248–260
[10]
Batchuluun G, Kim JH, Hong HG, Kang JK, and Park KR Fuzzy system based human behavior recognition by combining behavior prediction and recognition Expert Syst Appl 2017 81 108-133
[11]
Baxter RH, Robertson NM, and Lane DM Human behaviour recognition in data-scarce domains Pattern Recogn 2015 48 8 2377-2393
[12]
Bloch I Fusion d’informations numériques: panorama méthodologique J Natl Rech Robot 2005 2005 79-88
[13]
Chen L and Nugent CD Human activity recognition and behaviour analysis 2019 Berlin Springer
[14]
Chen L, Nugent CD, Mulvenna M, Finlay D, Hong X, and Poland M A logical framework for behaviour reasoning and assistance in a smart home Int J Assist Robot Mechatron 2008 9 4 20-34
[15]
Chen L, Nugent CD, and Wang H A knowledge-driven approach to activity recognition in smart homes IEEE Trans Knowl Data Eng 2012 24 6 961-974
[16]
Chen L, Nugent CD, and Wang H A knowledge-driven approach to activity recognition in smart homes IEEE Trans Knowl Data Eng 2012 24 6 961-974
[17]
Chen L, Hoey J, Nugent CD, Cook DJ, and Yu Z Sensor-based activity recognition IEEE Trans Syst Man Cybern Part C (Appl Rev) 2012 42 6 790-808
[18]
Chen L, Nugent C, and Okeyo G An ontology-based hybrid approach to activity modeling for smart homes IEEE Trans Human Mach Syst 2014 44 1 92-105
[19]
Cumin J, Lefebvre G, Ramparany F, and Crowley JL Ochoa SF, Singh P, and Bravo J A dataset of routine daily activities in an instrumented home Ubiquitous computing and ambient intelligence 2017 Cham Springer International Publishing 413-425
[20]
Cumin J, Ramparany F, Crowley JL et al. (2018) Inferring availability for communication in smart homes using context. In: IEEE international conference on pervasive computing and communications workshops (PerCom Workshops). IEEE, pp 1–6
[21]
Dash SCB, Mishra SR, Srujan Raju K, and Panda G Human action recognition using a hybrid deep learning heuristic Soft Comput 2021 25 18 13079-13092
[22]
Dey AK Understanding and using context Pers Ubiquit Comput 2001 5 1 4-7
[23]
Diamantini C, Freddi A, Longhi S, Potena D, and Storti E A goal-oriented, ontology-based methodology to support the design of AAL environments Expert Syst Appl 2016 64 117-131
[24]
Eppe M (2013) Postdictive reasoning in epistemic action theory. Ph.D. dissertation, Staats-und Universitätsbibliothek Bremen
[25]
Fahad LG, Khan A, and Rajarajan M Activity recognition in smart homes with self verification of assignments Neurocomputing 2015 149 1286-1298
[26]
Forkan ARM, Khalil I, Tari Z, Foufou S, and Bouras A A context-aware approach for long-term behavioural change detection and abnormality prediction in ambient assisted living Pattern Recogn 2015 48 3 628-641
[27]
Garcia-Ceja E, Galván-Tejada CE, and Brena R Multi-view stacking for activity recognition with sound and accelerometer data Inf Fusion 2018 40 45-56
[28]
Gayathri KS, Elias S, Shivashankar S (2015) Composite activity recognition in smart homes using Markov logic network. In: 2015 IEEE 12th international conference on ubiquitous intelligence and computing and 2015 IEEE 12th international conference on autonomic and trusted computing and 2015 IEEE 15th international conference on scalable computing and communications and its associated workshops (UIC-ATC-ScalCom), pp 46–53
[29]
Gayathri K, Elias S, and Ravindran B Hierarchical activity recognition for dementia care using Markov logic network Pers Ubiquit Comput 2015 19 2 271-285
[30]
Gayathri K, Easwarakumar K, and Elias S Probabilistic ontology based activity recognition in smart homes using Markov logic network Knowl-Based Syst 2017 121 173-184
[31]
Gebser M, Kaminski R, Kaufmann B, Schaub T (2014) Clingo = ASP + control: preliminary report. arXiv:1405.3694 [cs]
[32]
Gelfond M and Lifschitz V The stable model semantics for logic programming 1988 New York MIT Press 1070-1080
[33]
Gers FA, Schmidhuber J, Cummins F (1999) Learning to forget: continual prediction with LSTM. In: 1999 ninth international conference on artificial neural networks ICANN 99. (Conf. Publ. No. 470), vol 2, pp 850–855
[34]
Gu J, Wang L, Wang H, and Wang S A novel approach to intrusion detection using SVM ensemble with feature augmentation Comput Secur 2019 86 53-62
[35]
Guarino N, Oberle D, and Staab S What is an ontology? Handbook on ontologies 2009 Berlin Springer 1-17
[36]
Gu T, Wang XH, Pung HK, Zhang DQ (2020) An ontology-based context model in intelligent environments. arXiv:2003.05055 (arXiv preprint)
[37]
Hochreiter S and Schmidhuber J LSTM can solve hard long time lag problems Adv Neural Inf Process Syst 1997 20 473-479
[38]
Hossain HS, Khan MAAH, and Roy N Active learning enabled activity recognition Pervasive Mob Comput 2017 38 312-330
[39]
Ismail WN, Hassan MM, and Alsalamah HA Context-enriched regular human behavioral pattern detection from body sensors data IEEE Access 2019 7 33834-33850
[40]
Jakkula VR, Crandall AS, and Cook DJ Enhancing anomaly detection using temporal pattern discovery Advanced intelligent environments 2009 US Springer 175-194
[41]
Jia H and Chen S Integrated data and knowledge driven methodology for human activity recognition Inf Sci 2020 536 409-430
[42]
Khan IU, Afzal S, and Lee JW Human activity recognition via hybrid deep learning based model Sensors 2022 22 1 323
[43]
Khowaja SA, Prabono AG, Setiawan F, Yahya BN, and Lee S-L Contextual activity based healthcare internet of things, services, and people (HIOTSP): an architectural framework for healthcare monitoring using wearable sensors Comput Netw 2018 145 190-206
[44]
Knox S, Coyle L, Dobson S (2010) Using ontologies in case-based activity recognition. In: Proceedings of the twenty-third international Florida artificial intelligence research society conference (FLAIRS)
[45]
Kordestani H, Mojarad R, Chibani A, Osmani A, Amirat Y, Barkaoui K, Zahran W (2019) Hapicare: a healthcare monitoring system with self-adaptive coaching using probabilistic reasoning. In: 2019 IEEE/ACS 16th international conference on computer systems and applications (AICCSA). IEEE, pp 1–8
[46]
Lago P, Jiménez-Guarín C, and Roncancio C Salah AA, Kröse BJ, and Cook DJ Contextualized behavior patterns for ambient assisted living Human behavior understanding 2015 Cham Springer International Publishing 132-145
[47]
Lee SU, Hofmann A, Williams B (2019) A robust online human activity recognition methodology for human-robot collaboration. In: The AAAI workshop on plan, activity, and intent recognition, p 12
[48]
Lentzas A and Vrakas D Non-intrusive human activity recognition and abnormal behavior detection on elderly people: a review Artif Intell Rev 2019 20 1-47
[49]
Li Q, Huangfu W, Farha F, Zhu T, Yang S, Chen L, and Ning H Multi-resident type recognition based on ambient sensors activity Futur Gener Comput Syst 2020 112 108-115
[50]
Lifschitz V Answer set programming and plan generation Artif Intell 2002 138 1–2 39-54
[51]
Lifschitz V Answer set programming 2019 Berlin Springer
[52]
Liu L, Wang S, Su G, Huang Z-G, and Liu M Towards complex activity recognition using a Bayesian network-based probabilistic generative framework Pattern Recogn 2017 68 295-309
[53]
Liu HC, Chuah JH, Khairuddin ASM, Zhao XM, and Wang XD Campus abnormal behavior recognition with temporal segment transformers IEEE Access 2023 11 38471-38484
[54]
Lühr S, West G, and Venkatesh S Recognition of emergent human behaviour in a smart home: a data mining approach Pervasive Mob Comput 2007 3 2 95-116
[55]
Mabrouk AB and Zagrouba E Abnormal behavior recognition for intelligent video surveillance systems: a review Expert Syst Appl 2018 91 480-491
[56]
Maintaining a healthy lifestyle. https://www.foundationforpn.org/living-well/lifestyle/. Accessed 2020-02-00
[57]
Makantasis K, Doulamis A, Doulamis N, Psychas K (2016) Deep learning based human behavior recognition in industrial workflows. In: IEEE international conference on image processing (ICIP), pp 1609–1613
[58]
McGuinness D, Van Harmelen F et al. (2004) Owl web ontology language overview. W3C recommendation
[59]
Meditskos G, Dasiopoulou S, and Kompatsiaris I Metaq: a knowledge-driven framework for context-aware activity recognition combining sparql and owl 2 activity patterns Pervasive Mob Comput 2016 25 104-124
[60]
Mojarad R, Attal F, Chibani A, and Amirat Y Automatic classification error detection and correction for robust human activity recognition IEEE Robot Autom Lett 2020 5 2 2208-2215
[61]
Mojarad R, Attal F, Chibani A, and Amirat Y Dong Y, Mladenić D, and Saunders C A context-aware approach to detect abnormal human behaviors Machine learning and knowledge discovery in databases: applied data science track 2021 Cham Springer International Publishing 89-104
[62]
Mojarad R, Attal F, Chibani A, Amirat Y (2020) A context-aware hybrid framework for human behavior analysis. In: 2020 IEEE 32nd international conference on tools with artificial intelligence (ICTAI). IEEE, pp 460–465
[63]
Mojarad R, Attal F, Chibani A, Amirat Y (2020) A hybrid context-aware framework to detect abnormal human daily living behavior. In: Conference on neural networks. IEEE, pp 1–8
[64]
Mojarad R, Attal F, Chibani A, Fiorini SR, Amirat Y (2018) Hybrid approach for human activity recognition by ubiquitous robots. In: IEEE/RSJ international conference on intelligent robots and systems (IROS), pp 5660–5665
[65]
Mueller ET (2015) Chapter 15—commonsense reasoning using answer set programming. In: Mueller ET (ed) Commonsense reasoning (Second Edition), Boston, pp 249–269
[66]
Muise C, Wollenstein-Betech S, Booth S, Shah J, Khazaeni Y (2020) Modeling blackbox agent behaviour via knowledge compilation. In: The AAAI workshop on plan, activity, and intent recognition
[67]
Nebel B and Bürckert H-J Reasoning about temporal relations: a maximal tractable subclass of Allen’s interval algebra J ACM 1995 42 1 43-66
[68]
Okeyo G, Chen L, and Wang H Combining ontological and temporal formalisms for composite activity modelling and recognition in smart homes Futur Gener Comput Syst 2014 39 29-43
[69]
Owl 2 web ontology language document overview (second edition). https://www.w3.org/TR/2012/REC-owl2-overview-20121211/. Accessed 2020-02-00
[70]
Patkos T, Plexousakis D, Chibani A, and Amirat Y An event calculus production rule system for reasoning in dynamic and uncertain domains Theory Pract Logic Program 2016 20 325-352
[71]
Phan N, Dou D, Wang H, Kil D, and Piniewski B Ontology-based deep learning for human behavior prediction with explanations in health social networks Inf Sci 2017 384 298-313
[72]
Qu Y, Tang Y, Yang X, Wen Y, and Zhang W Context-aware mutual learning for semi-supervised human activity recognition using wearable sensors Expert Syst Appl 2023 219
[73]
Rafferty J, Nugent CD, Liu J, and Chen L From activity recognition to intention recognition for assisted living within smart homes Trans Human Mach Syst 2017 20 368-379
[74]
Rastogi S and Singh J Human fall detection and activity monitoring: a comparative analysis of vision-based methods for classification and detection techniques Soft Comput 2022 26 3679-3701
[75]
Reiss A, Hendeby G, Stricker D (2013) A competitive approach for human activity recognition on smartphones. In: European symposium on artificial neural networks, computational intelligence and machine learning, 24–26 April. Belgium, Bruges, pp 455–460
[76]
Riboni D and Bettini C Owl 2 modeling and reasoning with complex human activities Pervas Mob Comput 2011 7 3 379-395 Knowledge-Driven Activity Recognition in Intelligent Environments
[77]
Riboni D and Bettini C Cosar: hybrid reasoning for context-aware activity recognition Pers Ubiquit Comput 2011 15 3 271-289
[78]
Riboni D, Bettini C, Civitarese G, Janjua ZH, and Helaoui R Smartfaber: recognizing fine-grained abnormal behaviors for early detection of mild cognitive impairment Artif Intell Med 2016 67 57-74
[79]
Riboni D, Bettini C, Civitarese G, Janjua ZH, Helaoui R (2015) Fine-grained recognition of abnormal behaviors for early detection of mild cognitive impairment. In: IEEE international conference on pervasive computing and communications (PerCom), pp 149–154
[80]
Romera-Paredes B, Aung MS, Bianchi-Berthouze N (2013) A one-vs-one classifier ensemble with majority voting for activity recognition. In: ESANN
[81]
Ronao CA and Cho S-B Human activity recognition with smartphone sensors using deep learning neural networks Expert Syst Appl 2016 59 235-244
[82]
Sánchez VG, Lysaker OM, and Skeie N-O Human behaviour modelling for welfare technology using hidden Markov models Pattern Recogn Lett 2020 137 71-79
[83]
Sfar H (2019) Real time intelligent decision making from heterogeneous and imperfect data. Ph.D. dissertation, Paris Saclay
[84]
Shen Y-D and Eiter T Evaluating epistemic negation in answer set programming Artif Intell 2016 237 115-135
[85]
Shin JH, Lee B, and Park KS Detection of abnormal living patterns for elderly living alone using support vector data description Trans Inf Tech Biomed 2011 15 3 438-448
[86]
Soto-Mendoza V, García-Macías JA, Chávez E, Gomez-Montalvo JR, and Quintana E Detecting abnormal behaviours of institutionalized older adults through a hybrid-inference approach Pervasive Mob Comput 2017 40 708-723
[87]
Speer R, Chin J, Havasi C (2017) Conceptnet 5.5: an open multilingual graph of general knowledge. In: Thirty-first AAAI conference on artificial intelligence
[88]
Springer T and Turhan A-Y Employing description logics in ambient intelligence for modeling and reasoning about complex situations J Ambient Intell Smart Environ 2009 1 3 235-259
[89]
Stavropoulos TG, Meditskos G, Andreadis S, Avgerinakis K, Adam K, and Kompatsiaris I Semantic event fusion of computer vision and ambient sensor data for activity recognition to support dementia care J Ambient Intell Human Comput 2016 20 1-16
[90]
Sun J, Shao J, and He C Abnormal event detection for video surveillance using deep one-class learning Multimed Tools Appl 2019 78 3 3633-3647
[91]
Tay NC, Connie T, Ong TS, Teoh ABJ, and Teh PS A review of abnormal behavior detection in activities of daily living IEEE Access 2023 20 20
[92]
Triboan D, Chen L, Chen F (2019) Fuzzy-based fine-grained human activity recognition within smart environments. In: 2019 IEEE SmartWorld, ubiquitous intelligence and computing, advanced and trusted computing, scalable computing and communications, cloud and big data computing, internet of people and smart city innovation (SmartWorld/SCALCOM/UIC/ATC/CBDCom/IOP/SCI). IEEE 2019, pp 94–101
[93]
Tsai M-F and Li M-H Intelligent attendance monitoring system with spatio-temporal human action recognition Soft Comput 2022
[94]
Van Haaren J, Van den Broeck G, Meert W, and Davis J Lifted generative learning of Markov logic networks Mach Learn 2016 103 1 27-55
[95]
Varshney N, Bakariya B, Kushwaha AKS, et al. Rule-based multi-view human activity recognition system in real time using skeleton data from rgb-d sensor Soft Comput 2023 27 1 405-421
[96]
Vert J-P, Tsuda K, and Schölkopf B A primer on kernel methods Kernel Methods Comput Biol 2004 47 35-70
[97]
Villalonga C, Pomares H, Rojas I, and Banos O Mimu-wear: ontology-based sensor selection for real-world wearable activity recognition Neurocomputing 2017 250 76-100
[98]
Wang Y, Cang S, and Yu H A data fusion-based hybrid sensory system for older people’s daily activity and daily routine recognition IEEE Sens J 2018 18 16 6874-6888
[99]
Wen J and Wang Z Learning general model for activity recognition with limited labelled data Expert Syst Appl 2017 74 19-28
[100]
Wongpatikaseree K, Ikeda M, Buranarach M, Supnithi T, Lim AO, Tan Y (2012) Activity recognition using context-aware infrastructure ontology in smart home domain. In: 2012 seventh international conference on knowledge, information and creativity support systems. IEEE, pp 50–57
[101]
Xiang T and Gong S Video behavior profiling for anomaly detection IEEE Trans Pattern Anal Mach Intell 2008 30 5 893-908
[102]
Ye J, Coyle L, Dobson S, and Nixon P Ontology-based models in pervasive computing systems Knowl Eng Rev 2007 22 4 315-347
[103]
Zambrana C, Palou XR, and Vargiu E Sleeping recognition to assist elderly people at home Artif Intell Res 2016 20 64-69
[104]
Zhang Y, Ding K, Hui J, Lv J, Zhou X, and Zheng P Human-object integrated assembly intention recognition for context-aware human–robot collaborative assembly Adv Eng Inform 2022 54
[105]
Zhao Y, Zhang H, Gao Z, Gao W, Wang M, and Chen S A novel action saliency and context-aware network for weakly-supervised temporal action localization IEEE Trans Multimed 2023 20 1-14
Recommendations
A Context-Aware Approach to Detect Abnormal Human Behaviors
Machine Learning and Knowledge Discovery in Databases: Applied Data Science TrackAbstractAbnormal human behaviors can be signs of a health issue or the occurrence of a hazardous incident. Detecting such behaviors is essential in Ambient Intelligent (AmI) systems to enhance the safety of people. While detecting abnormalities has been ...
Context-aware hybrid reasoning framework for pervasive healthcare
Pervasive computing has emerged as a viable solution capable of providing technology-driven assistive living for elderly. The pervasive healthcare system, Context-Aware Real-time Assistant (CARA), is designed to provide personalized healthcare services ...
Context-aware activity prediction using human behavior pattern in real smart home environments
2016 IEEE International Conference on Automation Science and Engineering (CASE)Nowadays, close monitoring of daily activities of elders is enabled by employment of the advanced wireless sensor networks, whose large quantity data are then analyzed by activity recognition techniques whereby their behavior patterns can be accurately ...
Comments
Information & Contributors
Information
Published In
© The Author(s), under exclusive licence to Springer-Verlag GmbH Germany, part of Springer Nature 2023. Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
Publisher
Springer-Verlag
Berlin, Heidelberg
Publication History
Published: 13 September 2023
Accepted: 29 August 2023
Author Tags
Qualifiers
- Research-article
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 0Total Downloads
- Downloads (Last 12 months)0
- Downloads (Last 6 weeks)0
Reflects downloads up to 25 Jan 2025