research-article

Public Access

A User-adaptive Modeling for Eating Action Identification from Wristband Time Series

Authors:

Prajwal Paudyal,

Sandeep K. S. GuptaAuthors Info & Claims

ACM Transactions on Interactive Intelligent Systems (TiiS), Volume 9, Issue 4

Article No.: 22, Pages 1 - 35

https://doi.org/10.1145/3300149

Published: 10 October 2019 Publication History

All formats PDF

Abstract

Eating activity monitoring using wearable sensors can potentially enable interventions based on eating speed to mitigate the risks of critical healthcare problems such as obesity or diabetes. Eating actions are poly-componential gestures composed of sequential arrangements of three distinct components interspersed with gestures that may be unrelated to eating. This makes it extremely challenging to accurately identify eating actions. The primary reasons for the lack of acceptance of state-of-the-art eating action monitoring techniques include the following: (i) the need to install wearable sensors that are cumbersome to wear or limit the mobility of the user, (ii) the need for manual input from the user, and (iii) poor accuracy in the absence of manual inputs. In this work, we propose a novel methodology, IDEA, that performs accurate eating action identification within eating episodes with an average F1 score of 0.92. This is an improvement of 0.11 for precision and 0.15 for recall for the worst-case users as compared to the state of the art. IDEA uses only a single wristband and provides feedback on eating speed every 2 min without obtaining any manual input from the user.

References

[1]

2016. Maps of Trends in Diagnosed Diabetes and Obesity. Retrieved October 1, 2016 from https://www.downtoearth.org/news/2016-09/9171/diet-related-illnesses-cost-us-economy-1-trillion-annually.

[2]

Marios Anthimopoulos, Joachim Dehais, Peter Diem, and Stavroula Mougiakakou. 2013. Segmentation and recognition of multi-food meal images for carbohydrate counting. In Proceedings of the 13th International Conference onBioinformatics and Bioengineering (BIBE’13). IEEE, 1--4.

[3]

Ayan Banerjee, Imane Lamrani, Prajwal Paudyal, and Sandeep Gupta. 2019. Generation of movement explanations for testing gesture based co-operative learning applications. In Proceedings of the 2019 IEEE International Conference On Artificial Intelligence Testing (AITest’19). IEEE, 9--16.

[4]

Abdelkareem Bedri, Richard Li, Malcolm Haynes, Raj Prateek Kosaraju, Ishaan Grover, Temiloluwa Prioleau, Min Yan Beh, Mayank Goel, Thad Starner, and Gregory Abowd. 2017. EarBit: Using wearable sensors to detect eating episodes in unconstrained environments. Proc. ACM Interact. Mob. Wear. Ubiq. Technol. 1, 3 (2017), 37.

Digital Library

[5]

Donald J. Berndt and James Clifford. 1994. Using dynamic time warping to find patterns in time series. In Proceedings of the KDD Workshop, Vol. 10. Seattle, WA, 359--370.

Digital Library

[6]

Steven Cadavid, Mohamed Abdel-Mottaleb, and Abdelsalam Helal. 2012. Exploiting visual quasi-periodicity for real-time chewing event detection using active appearance models and support vector machines. Pers. Ubiq. Comput. 16, 6 (2012), 729--739.

Digital Library

[7]

Andrea Carlson and Elizabeth Frazão. 2014. Food costs, diet quality and energy balance in the United States. Physiol. Behav. 134 (2014), 20--31.

[8]

Jungman Chung, Jungmin Chung, Wonjun Oh, Yongkyu Yoo, Won Gu Lee, and Hyunwoo Bang. 2017. A glasses-type wearable device for monitoring the patterns of food intake and facial activity. Sci. Rep. 7 (2017), 41690.

[9]

Dawei Fan, Jiaqi Gong, and John Lach. 2016. Eating gestures detection by tracking finger motion. In Wireless Health. 1--6.

[10]

Centers for Disease Control et al. 2017. Maps of trends in diagnosed diabetes and obesity. Web site.

[11]

Ye He, Chang Xu, Neha Khanna, Carol J. Boushey, and Edward J. Delp. 2013. Food image analysis: Segmentation, identification and weight estimation. In Proceedings of the 2013 IEEE International Conference on Multimedia and Expo (ICME’13). IEEE, 1--6.

[12]

Sergey Ioffe and Christian Szegedy. 2015. Batch normalization: Accelerating deep network training by reducing internal covariate shift. arXiv preprint arXiv:1502.03167 (2015).

Digital Library

[13]

Azusa Kadomura, Cheng-Yuan Li, Yen-Chang Chen, Koji Tsukada, Itiro Siio, and Hao-hua Chu. 2013. Sensing fork: Eating behavior detection utensil and mobile persuasive game. In Extended Abstracts on Human Factors in Computing Systems (CHI’13). ACM, 1551--1556.

[14]

Azusa Kadomura, Cheng-Yuan Li, Koji Tsukada, Hao-Hua Chu, and Itiro Siio. 2014. Persuasive technology to improve eating behavior using a sensor-embedded fork. In Proceedings of the 2014 ACM International Joint Conference on Pervasive and Ubiquitous Computing. ACM, 319--329.

Digital Library

[15]

Yoshihiro Kawano and Katsuki Yanai. 2013. Real-time mobile food recognition system. In Proceedings of the Computer Vision and Pattern Recognition Workshops (CVPRW’13). IEEE, 1--7.

Digital Library

[16]

Joohee Kim, Kwang-Jae Lee, Mankyung Lee, Nahyeon Lee, Byung-Chull Bae, Genehee Lee, Juhee Cho, Young Mog Shim, and Jun-Dong Cho. 2016. Slowee: A smart eating-speed guide system with light and vibration feedback. In Proceedings of the 2016 CHI Conference Extended Abstracts on Human Factors in Computing Systems. ACM, 2563--2569.

Digital Library

[17]

Diederik Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014).

[18]

Alexander Kokkinos, Carel W. le Roux, Kleopatra Alexiadou, Nicholas Tentolouris, Royce P. Vincent, Despoina Kyriaki, Despoina Perrea, Mohammad A. Ghatei, Stephen R. Bloom, and Nicholas Katsilambros. 2010. Eating slowly increases the postprandial response of the anorexigenic gut hormones, peptide YY and glucagon-like peptide-1. J. Clin. Endocrinol. Metab. 95, 1 (2010), 333--337.

[19]

Junghyo Lee, Ayan Banerjee, and Sandeep K. S. Gupta. 2016. Mt-diet: Automated smartphone based diet assessment with infrared images. In Proceedings of the Annual IEEE International Conference Pervasive Computing and Communication (PerCom’16). IEEE, 1--6.

[20]

Junghyo Lee, Prajwal Paudyal, Ayan Banerjee, and Sandeep K. S. Gupta. 2017. FIT-EVE8ADAM: Estimation of velocity 8 energy for automated diet activity monitoring. In Proceedings of the 16th IEEE International Conference on Machine Learning and Applications (ICMLA’17). IEEE, 1071--1074.

[21]

Junghyo Lee, Prajwal Paudyal, Ayan Banerjee, and Sandeep K. S. Gupta. 2018. IDEA: Instant detection of eating action usingwrist-worn sensors in absence of user-specific model. In Proceedings of the 26th Conference on User Modeling, Adaptation and Personalization. ACM.

[22]

Junghyo Lee, Koosha Sadeghi, Javad Sohankar, Ayan Banerjee, and Sandeep K. S. Gupta. 2017. Enabling real-time internet-of-people mobile applications. In Proceedings of the 2017 IEEE SmartWorld, Ubiquitous Intelligence 8 Computing, Advanced 8 Trusted Computed, Scalable Computing 8 Communications, Cloud 8 Big Data Computing, Internet of People and Smart City Innovation Conference (SmartWorld/SCALCOM/UIC/ATC/CBDCom/IOP/SCI’17). IEEE, 1--6.

[23]

Junghyo Lee and Patrick Seeling. 2013. An overview of mobile device network traffic and network interface usage patterns. In Proceedings of the IEEE International Conference on Electro-Information Technology (EIT’13). IEEE, 1--5.

[24]

Jing Liu, Edward Johns, Louis Atallah, Claire Pettitt, Benny Lo, Gary Frost, and Guang-Zhong Yang. 2012. An intelligent food-intake monitoring system using wearable sensors. In Proceedings of the 9th International Conference on Wearable and Implantable Body Sensor Networks (BSN’12). IEEE, 154--160.

Digital Library

[25]

Rania A. Mekary, Edward Giovannucci, Walter C. Willett, Rob M. van Dam, and Frank B. Hu. 2012. Eating patterns and type 2 diabetes risk in men: Breakfast omission, eating frequency, and snacking. Am. J. Clin. Nutr. 95, 5 (2012), 1182--1189.

[26]

T. Ohkuma, H. Fujii, M. Iwase, Y. Kikuchi, S. Ogata, Y. Idewaki, H. Ide, Y. Doi, Y. Hirakawa, N. Mukai, et al. 2013. Impact of eating rate on obesity and cardiovascular risk factors according to glucose tolerance status: The fukuoka diabetes registry and the hisayama study. Diabetologia 56, 1 (2013), 70--77.

[27]

Temiloluwa Olubanjo and Maysam Ghovanloo. 2014. Real-time swallowing detection based on tracheal acoustics. In Proceedings of the 2014 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP’14). IEEE, 4384--4388.

[28]

Prajwal Paudyal, Ayan Banerjee, and Sandeep K. S. Gupta. 2016. Sceptre: A pervasive, non-invasive, and programmable gesture recognition technology. In Proceedings of the 21st International Conference on Intelligent User Interfaces. ACM, 282--293.

[29]

Prajwal Paudyal, Junghyo Lee, Ayan Banerjee, and Sandeep K. S. Gupta. 2017. Dyfav: Dynamic feature selection and voting for real-time recognition of fingerspelled alphabet using wearables. In Proceedings of the 22nd International Conference on Intelligent User Interfaces. ACM, 457--467.

[30]

Prajwal Paudyal, Junghyo Lee, Ayan Banerjee, and Sandeep K. S. Gupta. 2019. A comparison of techniques for sign language alphabet recognition using armband wearables. ACM Trans. Interact. Intell. Syst. 9, 2--3 (2019), 14.

Digital Library

[31]

Prajwal Paudyal, Junghyo Lee, Azamat Kamzin, Mohamad Soudki, Ayan Banerjee, and Sandeep K. S. Gupta. 2019. Learn2Sign: Explainable AI for sign language learning. In Proceedings of the CEUR Workshop, Vol. 2327.

[32]

Pietro Perona and Jitendra Malik. 1990. Scale-space and edge detection using anisotropic diffusion. IEEE Trans. Pattern Anal. Mach. Intell. 12, 7 (1990), 629--639.

Digital Library

[33]

Parisa Pouladzadeh, Shervin Shirmohammadi, Aslan Bakirov, Ahmet Bulut, and Abdulsalam Yassine. 2014. Cloud-based SVM for food categorization. Multimedia Tools Appl. (2014), 1--18.

[34]

Tauhidur Rahman, Alexander Travis Adams, Mi Zhang, Erin Cherry, Bobby Zhou, Huaishu Peng, and Tanzeem Choudhury. 2014. BodyBeat: A mobile system for sensing non-speech body sounds. In Proceedings of the International Conference on Mobile Systems, Applications, and Services (MobiSys’14), Vol. 14. 2--13.

Digital Library

[35]

Mayuree Rao, Ashkan Afshin, Gitanjali Singh, and Dariush Mozaffarian. 2013. Do healthier foods and diet patterns cost more than less healthy options? A systematic review and meta-analysis. BMJ Open 3, 12 (2013), e004277.

[36]

Frank M. Sacks and Martijn Katan. 2002. Randomized clinical trials on the effects of dietary fat and carbohydrate on plasma lipoproteins and cardiovascular disease. Am. J. Med. 113, 9 (2002), 13--24.

[37]

Koosha Sadeghi, Junghyo Lee, Ayan Banerjee, Javad Sohankar, and Sandeep K. S. Gupta. 2017. Permanency analysis on human electroencephalogram signals for pervasive brain-computer interface systems. In Proceedings of the 2017 39th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC’17). IEEE, 767--770.

[38]

Sougata Sen, Vigneshwaran Subbaraju, Archan Misra, Rajesh Krishna Balan, and Youngki Lee. 2015. The case for smartwatch-based diet monitoring. In Proceedings of the Pervasive Computing and Communication Workshops (PerCom Workshops’15). IEEE, 585--590.

[39]

Sougata Sen, Vigneshwaran Subbaraju, Archan Misra, Rajesh Krishna Balan, and Youngki Lee. 2017. Experiences in building a real-world eating recogniser. In Proceedings of the 4th International on Workshop on Physical Analytics. ACM, 7--12.

Digital Library

[40]

Miroslav Šenk and Laurence Cheze. 2006. Rotation sequence as an important factor in shoulder kinematics. Clin. Biomech. 21 (2006), S3--S8.

[41]

Sangita Sharma, Shelly Vik, Mohammadreza Pakseresht, Lucy Shen, and Laurence N. Kolonel. 2013. Diet impacts mortality from cancer: Results from the multiethnic cohort study. Cancer Causes Contr. 24, 4 (2013), 685--693.

[42]

Mingui Sun, Lora E. Burke, Zhi-Hong Mao, Yiran Chen, Hsin-Chen Chen, Yicheng Bai, Yuecheng Li, Chengliu Li, and Wenyan Jia. 2014. eButton: A wearable computer for health monitoring and personal assistance. In Proceedings of the 51st Annual Design Automation Conference. ACM, 1--6.

Digital Library

[43]

Boyd A. Swinburn, Ian Caterson, Jaap C. Seidell, and W. P. T. James. 2004. Diet, nutrition and the prevention of excess weight gain and obesity. Publ. Health Nutr. 7, 1a (2004), 123--146.

[44]

Shinichi Tanihara, Takuya Imatoh, Motonobu Miyazaki, Akira Babazono, Yoshito Momose, Michie Baba, Yoko Uryu, and Hiroshi Une. 2011. Retrospective longitudinal study on the relationship between 8-year weight change and current eating speed. Appetite 57, 1 (2011), 179--183.

[45]

Edison Thomaz, Irfan Essa, and Gregory D. Abowd. 2015. A practical approach for recognizing eating moments with wrist-mounted inertial sensing. In Proceedings of the 2015 ACM International Joint Conference on Pervasive and Ubiquitous Computing. ACM, 1029--1040.

[46]

Retrieved July 10, 2015 from https://www.myo.com/.

[47]

Samuel Whitehouse, Kristina Yordanova, Adeline Paiement, and Majid Mirmehdi. 2016. Recognition of unscripted kitchen activities and eating behaviour for health monitoring. In Proceedings of the 2nd IET International Conference on Technologies for Active and Assisted Living (TechAAL'16).

[48]

Sen Zhang, Marcelo H. Ang, Wendong Xiao, and Chen Khong Tham. 2009. Detection of activities by wireless sensors for daily life surveillance: Eating and drinking. Sensors 9, 3 (2009), 1499--1517.

[49]

Bo Zhou, Jingyuan Cheng, Mathias Sundholm, Attila Reiss, Wuhuang Huang, Oliver Amft, and Paul Lukowicz. 2015. Smart table surface: A novel approach to pervasive dining monitoring. In Proceedings of the Annual IEEE International Conference Pervasive Computing and Communication (PerCom’15). IEEE, 155--162.

Cited By

Hiraguchi HPerone PToet ACamps GBrouwer A(2023)Technology to Automatically Record Eating Behavior in Real Life: A Systematic ReviewSensors10.3390/s2318775723:18(7757)Online publication date: 8-Sep-2023
https://doi.org/10.3390/s23187757
Al Jlailaty HCelik AMansour MEltawil A(2023)Machine Learning-Based Unobtrusive Intake Gesture Detection via Wearable Inertial SensorsIEEE Transactions on Biomedical Engineering10.1109/TBME.2022.321719670:4(1389-1400)Online publication date: Apr-2023
https://doi.org/10.1109/TBME.2022.3217196
Ghosh TSazonov E(2023)Passive Sensors for Detection of Food IntakeEncyclopedia of Sensors and Biosensors10.1016/B978-0-12-822548-6.00086-8(218-234)Online publication date: 2023
https://doi.org/10.1016/B978-0-12-822548-6.00086-8
Show More Cited By

Index Terms

A User-adaptive Modeling for Eating Action Identification from Wristband Time Series
1. Human-centered computing
  1. Ubiquitous and mobile computing

Recommendations

FitByte: Automatic Diet Monitoring in Unconstrained Situations Using Multimodal Sensing on Eyeglasses
CHI '20: Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems

In an attempt to help users reach their health goals and practitioners understand the relationship between diet and disease, researchers have proposed many wearable systems to automatically monitor food consumption. When a person consumes food, he/she ...
An eating speed guide system using a wristband and tabletop unit
UbiComp '16: Proceedings of the 2016 ACM International Joint Conference on Pervasive and Ubiquitous Computing: Adjunct

In this study, we present a system which can provide users with appropriate eating speed guidance. The purpose of our study is to improve user's eating habit by delivering real-time feedback while eating. We conducted a pilot study with our proposed ...
IDEA: Instant Detection of Eating Action using Wrist-Worn Sensors in Absence of User-Specific Model
UMAP '18: Proceedings of the 26th Conference on User Modeling, Adaptation and Personalization

Eating activity monitoring using wearable sensors can potentially enable interventions based on eating speed for critical healthcare problems such as obesity or diabetes. We propose a novel methodology, IDEA that performs accurate eating action ...

Comments

Information & Contributors

Information

Published In

cover image ACM Transactions on Interactive Intelligent Systems

ACM Transactions on Interactive Intelligent Systems Volume 9, Issue 4

December 2019

187 pages

ISSN:2160-6455

EISSN:2160-6463

DOI:10.1145/3351880

Editor:
Michelle X. Zhou
Juji, Inc., USA

Issue’s Table of Contents

Copyright © 2019 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 10 October 2019

Accepted: 01 May 2019

Revised: 01 January 2019

Received: 01 August 2018

Published in TIIS Volume 9, Issue 4

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed

Funding Sources

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

8
Total Citations
View Citations
908
Total Downloads

Downloads (Last 12 months)347
Downloads (Last 6 weeks)73

Reflects downloads up to 10 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Hiraguchi HPerone PToet ACamps GBrouwer A(2023)Technology to Automatically Record Eating Behavior in Real Life: A Systematic ReviewSensors10.3390/s2318775723:18(7757)Online publication date: 8-Sep-2023
https://doi.org/10.3390/s23187757
Al Jlailaty HCelik AMansour MEltawil A(2023)Machine Learning-Based Unobtrusive Intake Gesture Detection via Wearable Inertial SensorsIEEE Transactions on Biomedical Engineering10.1109/TBME.2022.321719670:4(1389-1400)Online publication date: Apr-2023
https://doi.org/10.1109/TBME.2022.3217196
Ghosh TSazonov E(2023)Passive Sensors for Detection of Food IntakeEncyclopedia of Sensors and Biosensors10.1016/B978-0-12-822548-6.00086-8(218-234)Online publication date: 2023
https://doi.org/10.1016/B978-0-12-822548-6.00086-8
Herath ASharma ANittala A(2022)Utilizing Interactive Technologies to Encourage Healthy Dietary BehaviorAdjunct Publication of the 24th International Conference on Human-Computer Interaction with Mobile Devices and Services10.1145/3528575.3551427(1-5)Online publication date: 28-Sep-2022
https://dl.acm.org/doi/10.1145/3528575.3551427
Shin JLee SGong TYoon HRoh HBianchi ALee S(2022)MyDJ: Sensing Food Intakes with an Attachable on Your Eyeglass FrameProceedings of the 2022 CHI Conference on Human Factors in Computing Systems10.1145/3491102.3502041(1-17)Online publication date: 29-Apr-2022
https://dl.acm.org/doi/10.1145/3491102.3502041
Li YTang HLiu YQiao YXia HZhou J(2022)Oral wearable sensors: Health management based on the oral cavityBiosensors and Bioelectronics: X10.1016/j.biosx.2022.10013510(100135)Online publication date: May-2022
https://doi.org/10.1016/j.biosx.2022.100135
Leon CCasas DLopez J(2021)IoT technological model to improve the control and monitoring of patients with eating disorders (ED): Anorexia and Bulimia in a mental health hospital2021 IEEE Sciences and Humanities International Research Conference (SHIRCON)10.1109/SHIRCON53068.2021.9652296(1-4)Online publication date: 17-Nov-2021
https://doi.org/10.1109/SHIRCON53068.2021.9652296
Banerjee AAmperyani VGupta STran VAshok A(2020)Hand hygiene compliance checking system with explainable feedbackProceedings of the 6th ACM Workshop on Wearable Systems and Applications10.1145/3396870.3400015(34-36)Online publication date: 19-Jun-2020
https://dl.acm.org/doi/10.1145/3396870.3400015

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

HTML Format

View this article in HTML Format.

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

Media

Figures

Other

Tables

View Issue’s Table of Contents