Device-Free Human Activity Recognition with Low-Resolution Infrared Array Sensor Using Long Short-Term Memory Neural Network
Abstract
:1. Introduction
- A low-resolution infrared array sensor of 8 × 8 pixels is applied. It is a low-cost, privacy-protective and device-free sensor that establishes a good trade-off between performance and cost.
- A noise reduction method called J-filter is proposed to address the heterogeneity issue of the background temperature for infrared signal preprocessing. The original absolute infrared signals are replaced by the relative values extracted by J-filter to eliminate the influence of the temperature and improve the robustness for HAR.
- A deep learning model based on LSTM is used to extract infrared signal features. The features of low-dimensional infrared signals can be depth mined in the model training process. The well-trained LSTM model can quickly and accurately recognize the human activities through the designed real-time interface.
2. System Framework Overview and Methodology
2.1. System Framework Overview
2.2. Infrared Signal Acquisition
2.3. Signal Segmentation and Preprocessing
2.3.1. Segmentation
2.3.2. Signal Preprocessing
2.4. Model Training
2.5. Real-Time Detection
3. Experimental Evaluation
3.1. Experimental Setup and Data Description
3.2. Results
3.3. Impact of the Frames in the Sample
3.4. Impact of the Number of Hidden Layer Nodes
3.5. Impact of the Timesteps
3.6. Impact of the Filtering Algorithms
3.7. Comparative Analysis
4. Conclusions
Author Contributions
Funding
Institutional Review Board Statement
Informed Consent Statement
Data Availability Statement
Conflicts of Interest
References
- De-La-Hoz-Franco, E.; Ariza-Colpas, P.; Quero, J.M.; Espinilla, M. Sensor-based datasets for Human Activity Recognition—A Systematic Review of Literature. IEEE Access 2018, 6, 59192–59210. [Google Scholar] [CrossRef]
- Zheng, Y.L.; Ding, X.R.; Poon, C.C.Y.; Lo, B.P.L. Unobtrusive Sensing and Wearable Devices for Health Informatics. IEEE Trans. Biomed. Eng. 2014, 61, 1538–1554. [Google Scholar] [CrossRef] [PubMed]
- De Backere, O.; Van den, A.; Nelis, J.; Bonte, P.; Clement, E.; Philpott, M.; Hoebeke, J.; Verstichel, S.; Ackaert, A.; De Turck, F. Towards a social and context-aware multi-sensor fall detection and risk assessment platform. Comput. Biol. Med. 2015, 64, 307–320. [Google Scholar] [CrossRef] [PubMed] [Green Version]
- Kern, N.; Antifakos, S.; Schiele, B.; Schwaninger, A. A model for human interruptability: Experimental evaluation and automatic estimation from wearable sensors. In Proceedings of the Eighth International Symposium on Wearable Computers, Arlington, VA, USA, 31 October–3 November 2004; Volume 1, pp. 158–165. [Google Scholar]
- Khan, A.M.; Lee, Y.; Lee, S.Y.; Kim, T. A Triaxial Accelerometer-Based Physical-Activity Recognition via Augmented-Signal Features and a Hierarchical Recognizer. IEEE Trans. Inf. Technol. Biomed. 2010, 14, 1166–1172. [Google Scholar] [CrossRef]
- Wannenburg, J.; Malekian, R. Physical Activity Recognition from Smartphone Accelerometer Data for User Context Awareness Sensing. IEEE Trans. Syst. Man Cybern. Syst. 2016, 47, 3142–3149. [Google Scholar] [CrossRef]
- Bisio, I.; Delfino, A.; Lavagetto, F.; Sciarrone, A. Enabling IoT for In-Home Rehabilitation: Accelerometer Signals Classification Methods for Activity and Movement Recognition. IEEE Internet Things J. 2017, 4, 135–146. [Google Scholar] [CrossRef]
- Hu, G.; Qiu, X.; Meng, L. RTagCare: Deep human activity recognition powered by passive computational RFID sensors. In Proceedings of the 2016 18th Asia-Pacific Network Operations and Management Symposium (APNOMS), Kanazawa, Japan, 5–7 October 2016; pp. 1–4. [Google Scholar]
- Hu, G.; Qiu, X.; Meng, L. Human activity recognition based on Hidden Markov Models using computational RFID. In Proceedings of the 2017 4th International Conference on Systems and Informatics (ICSAI), Hangzhou, China, 11–13 November 2017; pp. 813–818. [Google Scholar]
- Wu, Y. Research on bank intelligent video image processing and monitoring control system based on OpenCV. In Proceedings of the 2009 3rd International Conference on Anti-Counterfeiting, Security, and Identification in Communication, HongKong, China, 20–23 August 2009; pp. 211–214. [Google Scholar]
- Luo, R.C.; Wu, X. Real-time Gender Recognition Based on 3D Human Body Shape for Human-Robot Interaction. In Proceedings of the 2014 9th ACM/IEEE International Conference on Human-Robot Interaction (HRI), New York, NY, USA, 8–11 March 2014; pp. 236–237. [Google Scholar]
- Ni, B.; Wang, G.; Moulin, P. RGBD-HuDaAct: A color-depth video database for human daily activity recognition. In Proceedings of the 2011 IEEE International Conference on Computer Vision Workshops (ICCV Workshops), Barcelona, Spain, 6–13 November 2011; pp. 1147–1153. [Google Scholar]
- Sanal Kumar, K.P.; Bhavani, R. Human activity recognition in egocentric video using HOG, GiST and color features. Multimed. Tools Appl. 2018, 79, 3543–3559. [Google Scholar] [CrossRef]
- Gupta, A.; Gupta, K.; Gupta, K.; Gupta, K. A Survey on Human Activity Recognition and Classification. In Proceedings of the 2020 International Conference on Communication and Signal Processing (ICCSP), Melmaruvathur, India, 28–30 July 2020; pp. 0915–0919. [Google Scholar]
- Zhao, K.; Xi, W.; Jiang, Z.; Wang, Z.; Zhang, X. Leveraging Topic Model for CSI Based Human Activity Recognition. In Proceedings of the 2016 12th International Conference on Mobile Ad-Hoc and Sensor Networks (MSN), Hefei, China, 16–18 December 2016. [Google Scholar]
- Damodaran, N.; Schäfer, J. Device Free Human Activity Recognition using WiFi Channel State Information. In Proceedings of the 2019 IEEE SmartWorld, Ubiquitous Intelligence Computing, Advanced Trusted Computing, Scalable Computing Communications, Cloud Big Data Computing, Internet of People and Smart City Innovation (SmartWorld/SCALCOM/UIC/ATC/CBDCom/IOP/SCI), Leicester, UK, 19–23 August 2019; pp. 1069–1074. [Google Scholar]
- Wang, W.; Liu, A.X.; Shahzad, M.; Ling, K.; Lu, S. Device-Free Human Activity Recognition Using Commercial WiFi Devices. IEEE J. Sel. Areas Commun. 2017, 35, 1118–1131. [Google Scholar] [CrossRef]
- Gonzalez, L.I.L.; Troost, M.; Amft, O. Using a Thermopile Matrix Sensor to Recognize Energy-related Activities in Offices. Procedia Comput. Sci. 2013, 19, 678–685. [Google Scholar] [CrossRef] [Green Version]
- Chen, W.-H.; Ma, H.-P. A fall detection system based on infrared array sensors with tracking capability for the elderly at home. In Proceedings of the 2015 17th International Conference on E-health Networking, Application Services (HealthCom), Boston, MA, USA, 14–17 October 2015; pp. 428–434. [Google Scholar]
- Yun, J.; Woo, J. A Comparative Analysis of Deep Learning and Machine Learning on Detecting Movement Directions Using PIR Sensors. IEEE Internet Things J. 2020, 7, 2855–2868. [Google Scholar] [CrossRef]
- Guan, Q.; Li, C.; Qin, L.; Wang, G. Daily Activity Recognition Using Pyroelectric Infrared Sensors and Reference Structures. IEEE Sens. J. 2019, 19, 1645–1652. [Google Scholar] [CrossRef]
- Guan, Q.; Yin, X.; Guo, X.; Wang, G. A Novel Infrared Motion Sensing System for Compressive Classification of Physical Activity. IEEE Sens. J. 2016, 16, 2251–2259. [Google Scholar] [CrossRef]
- Torres, A. Adafruit AMG8833 8x8 Thermal Camera Sensor. Available online: https://cdn-learn.adafruit.com/downloads/pdf/adafruit-amg8833-8x8-thermal-camera-sensor.pdf (accessed on 18 May 2021).
- Gochoo, M.; Tan, T.H.; Batjargal, T.; Seredin, O.; Huang, S.C. Device-Free Non-Privacy Invasive Indoor Human Posture Recognition Using Low-Resolution Infrared Sensor-Based Wireless Sensor Networks and DCNN. In Proceedings of the 2018 IEEE International Conference on Systems, Man, and Cybernetics (SMC), Miyazaki, Japan, 7–10 October 2018; pp. 2311–2316. [Google Scholar]
- Shih, C.S.; Wang, Y.T.; Chou, J.J. Multiple-Image Super-Resolution for Networked Extremely Low-Resolution Thermal Sensor Array. In Proceedings of the 2020 IEEE Second Workshop on Machine Learning on Edge in Sensor Systems (SenSys-ML), Sydney, Australia, 21 April 2020; pp. 1–6. [Google Scholar]
- Uddin, M.Z.; Torresen, J. A Deep Learning-Based Human Activity Recognition in Darkness. In Proceedings of the 2018 Colour and Visual Computing Symposium (CVCS), Gjøvik, Norway, 19–20 September 2018; pp. 1–5. [Google Scholar]
- Batchuluun, G.; Nguyen, D.T.; Pham, T.D.; Park, C.; Park, K.R. Action Recognition from Thermal Videos. IEEE Access 2019, 7, 103893–103917. [Google Scholar] [CrossRef]
- Panasonic. Infrared Array Sensor Grid-EYE. 2017. Available online: https://cdn.sparkfun.com/assets/4/1/c/0/1/Grid-EYE_Datasheet.pdf (accessed on 18 May 2021).
- Kodali, R.K.; Mahesh, K.S. A low cost implementation of MQTT using ESP8266. In Proceedings of the 2016 2nd International Conference on Contemporary Computing and Informatics (IC3I), Noida, India, 14–17 December 2016. [Google Scholar]
- Ali, A.S.; Radwan, A.G.; Soliman, A.M. Fractional Order Butterworth Filter: Active and Passive Realizations. IEEE J. Emerg. Sel. Top. Circuits Syst. 2013, 3, 346–354. [Google Scholar] [CrossRef]
- Hochreiter, S.; Schmidhuber, J. Long Short-Term Memory. Neural Comput. 1997, 9, 1735–1780. [Google Scholar] [CrossRef]
- Koutník, J.; Greff, K.; Gomez, F.; Schmidhuber, J. A Clockwork RNN. Comput. Sci. 2014, 32, 1863–1871. [Google Scholar]
- Godfrey, L.B.; Gashler, M.S. Neural decomposition of time-series data. In Proceedings of the 2017 IEEE International Conference on Systems, Man and Cybernetics (SMC), Banff, AB, Canada, 5–8 October 2017. [Google Scholar]
- Mohammed, A.A.; Umaashankar, V. Effectiveness of Hierarchical Softmax in Large Scale Classification Tasks. In Proceedings of the 2018 International Conference on Advances in Computing, Communications and Informatics (ICACCI), Bangalore, India, 19–22 September 2018; pp. 1090–1094. [Google Scholar]
- Lu, Y.; Salem, F.M. Simplified gating in long short-term memory (LSTM) recurrent neural networks. In Proceedings of the 2017 IEEE 60th International Midwest Symposium on Circuits and Systems (MWSCAS), Boston, MA, USA, 6–9 August 2017; pp. 1601–1604. [Google Scholar]
- Zhang, J.R.; Zhang, J.; Lok, T.M.; Lyu, M.R. A hybrid particle swarm optimization–back-propagation algorithm for feedforward neural network training. Appl. Math. Comput. 2007, 185, 1026–1037. [Google Scholar] [CrossRef]
- Clevert, D.A.; Unterthiner, T.; Hochreiter, S. Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs). arXiv 2015, arXiv:1511.07289. [Google Scholar]
- Monner, D.; Reggia, J.A. A generalized LSTM-like training algorithm for second-order recurrent neural networks. Neural Netw. 2012, 25, 70–83. [Google Scholar] [CrossRef] [Green Version]
- Siew, Z.C.K. Extreme learning machine: Theory and applications. Neurocomputing 2006, 70, 489–501. [Google Scholar]
- Zhan, Y.; Shen, D. Design efficient support vector machine for fast classification. Pattern Recognit 2005, 38, 157–161. [Google Scholar] [CrossRef]
- Kramer, O. K-Nearest Neighbors. Intell. Syst. Ref. Libr. 2013, 41, 13–23. [Google Scholar]
- Arandjelovic, R.; Gronat, P.; Torii, A.; Pajdla, T.; Sivic, J. NetVLAD: CNN architecture for weakly supervised place recognition. IEEE Trans. Pattern Anal. Mach. Intell. 2017, 40, 1437–1451. [Google Scholar] [CrossRef] [PubMed] [Green Version]
Raw Data | Background Subtraction | Remove Average Temperature | J-Filter | J-Filter and Butterworh | |
---|---|---|---|---|---|
Accuracy(%) | 84.259 | 90.556 | 92.269 | 96.644 | 98.287 |
Recall(%) | 84.080 | 90.498 | 92.226 | 96.658 | 98.290 |
F1-score(%) | 84.169 | 90.527 | 92.247 | 96.651 | 98.289 |
Method | Number of Sensors | Pixel | Mean Accuracy |
---|---|---|---|
Accelerometer+SVM [7] | 1 | - | 91.00% |
CSI signals+HMM [17] | 2 | - | 96.50% |
PIR sensors+GS-HHMM [22] | 4 | 8 × 16 | 91.00% |
Thermal sensor+SRCNN [25] | 1 | 120 × 160 | 92.31% |
Thermal videos+CNN [27] | 1 | 640 × 480 | 95.97% |
The proposed method | 1 | 8 × 8 | 97.92% |
Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations. |
© 2021 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).
Share and Cite
Yin, C.; Chen, J.; Miao, X.; Jiang, H.; Chen, D. Device-Free Human Activity Recognition with Low-Resolution Infrared Array Sensor Using Long Short-Term Memory Neural Network. Sensors 2021, 21, 3551. https://doi.org/10.3390/s21103551
Yin C, Chen J, Miao X, Jiang H, Chen D. Device-Free Human Activity Recognition with Low-Resolution Infrared Array Sensor Using Long Short-Term Memory Neural Network. Sensors. 2021; 21(10):3551. https://doi.org/10.3390/s21103551
Chicago/Turabian StyleYin, Cunyi, Jing Chen, Xiren Miao, Hao Jiang, and Deying Chen. 2021. "Device-Free Human Activity Recognition with Low-Resolution Infrared Array Sensor Using Long Short-Term Memory Neural Network" Sensors 21, no. 10: 3551. https://doi.org/10.3390/s21103551