Abstract
In recent years, the emergence of the COVID-19 pandemic has led to new viral variants, such as Omicron. These variants are more harmful and impose more restrictions on people’s daily hygiene habits. Therefore, during the COVID-19 pandemic, it is logical to automatically detect epidemic protective behaviors without user intent. In this study, we used multiple sensor data from an off-the-shelf smartwatch to detect several defined behaviors. To increase the utility and generalizability of the research results, we collected audio and inertial measurement unit (IMU) data from eight participants in real environments over a long period. In the model-building process, we first created a binary classification between hand hygiene behaviors(hand washing, disinfection, and face-touching) and daily behavior. Then, we distinguished between specific hand hygiene behaviors based on audio and IMU. Ultimately, our model achieves 93% classification accuracy for three behaviors(Hand washing, face touching, and disinfection). The results prove that the accuracy of the classification of behaviors has improved remarkably, which also emphasizes the feasibility of recognizing hand hygiene behaviors using inertial acoustic data.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Głąbska, D., Skolmowska, D., Guzek, D.: Population-based study of the influence of the COVID-19 pandemic on hand hygiene behaviors-polish adolescents’ COVID-19 experience (place-19) study. Sustainability 12(12), 4930 (2020)
Mondol, M.A.S., Stankovic, J.A.: Hawad: hand washing detection using wrist wearable inertial sensors. In: 2020 16th International Conference on Distributed Computing in Sensor Systems (DCOSS), pp. 11–18. IEEE (2020)
Xu, L., Nishiyama, Y., Sezaki, K.: Enhancing self-protection: what influences human’s epidemic prevention behavior during the COVID-19 pandemic. In: Distributed, A., Interactions, P. (eds.) HCII 2022, Part II. LNCS, vol. 13326, pp. 336–351. Springer, Proceedings (2022). https://doi.org/10.1007/978-3-031-05431-0_23
Bhattacharya, S., Adaimi, R., Thomaz, E.: Leveraging sound and wrist motion to detect activities of daily living with commodity smartwatches. Proc. ACM Interact. Mob. Wearable Ubiquit. Technol. 6(2), 1–28 (2022)
Weiss, G.M., Timko, J.L., Gallagher, C.M., Yoneda, K., Schreiber, A.J.: Smartwatch-based activity recognition: a machine learning approach. In: 2016 IEEE-EMBS International Conference on Biomedical and Health Informatics (BHI), pp. 426–429. IEEE (2016)
Chen, X.: Faceoff: detecting face touching with a wrist-worn accelerometer. arXiv preprint arXiv:2008.01769 (2020)
Ellis, K., Godbole, S., Chen, J., Marshall, S., Lanckriet, G., Kerr, J.: Physical activity recognition in free-living from body-worn sensors. In: Proceedings of the 4th International SenseCam and Pervasive Imaging Conference, pp. 88–89 (2013)
Wang, X., et al.: Social distancing alert with smartwatches. arXiv preprint arXiv:2205.06110 (2022)
Kwok, Y.L.A., Gralton, J., McLaws, M.-L.: Face touching: a frequent habit that has implications for hand hygiene. Am. J. Infect. Control 43(2), 112–114 (2015)
Duan, Y., et al.: Predicting hand washing, mask wearing and social distancing behaviors among older adults during the COVID-19 pandemic: an integrated social cognition model. BMC Geriatr. 22(1), 91 (2022)
Lattanzi, E., Calisti, L., Freschi, V.: Automatic unstructured handwashing recognition using smartwatch to reduce contact transmission of pathogens (2022)
Stork, J.A., Spinello, L., Silva, J., Arras, K.O.: Audio-based human activity recognition using non-Markovian ensemble voting. In: IEEE RO-MAN: The 21st IEEE International Symposium on Robot and Human Interactive Communication, pp. 509–514. IEEE (2012)
Siddiqui, N., Chan, R.H.: Multimodal hand gesture recognition using single IMU and acoustic measurements at wrist. PLoS ONE 15(1), e0227039 (2020)
Dittakavi, B., et al.: Pose tutor: an explainable system for pose correction in the wild. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3540–3549 (2022)
Samyoun, S., Shubha, S.S., Mondol, M.A.S., Stankovic, J.A.: iWash: a smartwatch handwashing quality assessment and reminder system with real-time feedback in the context of infectious disease. Smart Health 19, 100171 (2021)
Mollyn, V., Ahuja, K., Verma, D., Harrison, C., Goel, M.: Samosa: sensing activities with motion and subsampled audio. Proc. ACM Interact. Mob. Wearable Ubiquit. Technol. 6(3), 1–19 (2022)
Becker, V., Fessler, L., Sörös, G.: Gestear: combining audio and motion sensing for gesture recognition on smartwatches. In: Proceedings of the 2019 ACM International Symposium on Wearable Computers, pp. 10–19 (2019)
Nishiyama, Y., et al.: IOS crowd–sensing won’t hurt a bit!: AWARE framework and sustainable study guideline for iOS platform. In: Streitz, N., Konomi, S. (eds.) HCII 2020. LNCS, vol. 12203, pp. 223–243. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-50344-4_17
Gers, F.A., Eck, D., Schmidhuber, J.: Applying LSTM to time series predictable through time-window approaches. In: Dorffner, G., Bischof, H., Hornik, K. (eds.) ICANN 2001. LNCS, vol. 2130, pp. 669–676. Springer, Heidelberg (2001). https://doi.org/10.1007/3-540-44668-0_93
Wen, Q., et al.: Time series data augmentation for deep learning: a survey. arXiv preprint arXiv:2002.12478 (2020)
Um, T.T., et al.: Data augmentation of wearable sensor data for Parkinson’s disease monitoring using convolutional neural networks. In: Proceedings of the 19th ACM International Conference on Multimodal Interaction, pp. 216–220 (2017)
Guan, Y., Plötz, T.: Ensembles of deep LSTM learners for activity recognition using wearables. Proc. ACM Interact. Mob. Wearable Ubiquit. Technol. 1(2), 1–28 (2017)
Desplanques, B., Thienpondt, J., Demuynck, K.: Ecapa-TDNN: emphasized channel attention, propagation and aggregation in TDNN based speaker verification. arXiv preprint arXiv:2005.07143 (2020)
Acknowledgment
This work was partly supported by National Institute of Information and Communications Technology (NICT), Japan.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Zhuang, H., Xu, L., Nishiyama, Y., Sezaki, K. (2023). Detecting Hand Hygienic Behaviors In-the-Wild Using a Microphone and Motion Sensor on a Smartwatch. In: Streitz, N.A., Konomi, S. (eds) Distributed, Ambient and Pervasive Interactions. HCII 2023. Lecture Notes in Computer Science, vol 14037. Springer, Cham. https://doi.org/10.1007/978-3-031-34609-5_34
Download citation
DOI: https://doi.org/10.1007/978-3-031-34609-5_34
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-34608-8
Online ISBN: 978-3-031-34609-5
eBook Packages: Computer ScienceComputer Science (R0)