research-article

Detecting Eating Episodes by Tracking Jawbone Movements with a Non-Contact Wearable Sensor

Authors:

Sarnab Bhattacharya,

Edison ThomazAuthors Info & Claims

Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies, Volume 2, Issue 1

Article No.: 4, Pages 1 - 21

https://doi.org/10.1145/3191736

Published: 26 March 2018 Publication History

Abstract

Eating is one of the most fundamental human activities, and because of the important role it plays in our lives, it has been extensively studied. However, an objective and usable method for dietary intake tracking remains unrealized despite numerous efforts by researchers over the last decade. In this work, we present a new wearable computing approach for detecting eating episodes. Using a novel multimodal sensing strategy combining accelerometer and range sensing, the approach centers on a discreet and lightweight instrumented necklace that captures head and jawbone movements without direct contact with the skin. An evaluation of the system with 32 participants comprised of three phases resulted in eating episodes detected with 95.2% precision and 81.9% recall in controlled studies and 78.2% precision and 72.5% recall in the free-living study. This research add technical contributions to the fields of wearable computing, human activity recognition, and mobile health.

References

[1]

2014. Obesity and overweight. Technical Report. World Health Organization. Fact Sheet 311.

[2]

Oliver Amft and Gerhard Tröster. 2008. Recognition of dietary activity events using on-body sensors. Artificial Intelligence in Medicine 42, 2 (Feb. 2008), 121--136.

Digital Library

[3]

Oliver Amft and Gerhard Tröster. 2009. On-Body Sensing Solutions for Automatic Dietary Monitoring. IEEE pervasive computing 8, 2 (April 2009).

Digital Library

[4]

Edward Archer, Gregory A Hand, and Steven N Blair. 2013. Validity of U.S. Nutritional Surveillance: National Health and Nutrition Examination Survey Caloric Energy Intake Data, 1971--2010. PLoS ONE 8, 10 (Oct. 2013), 76632.

[5]

Yicheng Bai, Wenyan Jia, Zhi-Hong Mao, and Mingui Sun. 2014. Automatic eating detection using a proximity sensor. In Bioengineering Conference (NEBEC), 2014 40th Annual Northeast. IEEE, 1--2.

[6]

Ronald J Baken. 1992. Electroglottography. Journal of Voice 6, 2(1992), 98--110.

[7]

Abdelkareem Bedri, Richard Li, Malcolm Haynes, Raj Prateek Kosaraju, Ishaan Grover, Temiloluwa Prioleau, Min Yan Beh, Mayank Goel, Thad Starner, and Gregory Abowd. 2017. EarBit: Using Wearable Sensors to Detect Eating Episodes in Unconstrained Environments. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 1, 3 (2017), 37.

Digital Library

[8]

Abdelkareem Bedri, Apoorva Verlekar, Edison Thomaz, Valerie Avva, and Thad Starner. 2015. A wearable system for detecting eating activities with proximity sensors in the outer ear. In Proceedings of the 2015 ACM International Symposium on Wearable Computers. ACM, 91--92.

Digital Library

[9]

Sheila A Bingham. 1987. The dietary assessment of individuals; methods, accuracy, new techniques and recommendations.

[10]

Hsin-Chen Chen, Wenyan Jia, Yaofeng Yue, Zhaoxin Li, Yung-Nien Sun, John D Fernstrom, and Mingui Sun. 2013. Model-based measurement of food portion size for image-based dietary assessment using 3D/2D registration. Measurement Science and Technology 24, 10 (2013), 105701.

[11]

Jingyuan Cheng, Oliver Amft, and Paul Lukowicz. 2010. Active capacitive sensing: Exploring a new wearable sensing modality for activity recognition. In International Conference on Pervasive Computing. Springer, 319--336.

Digital Library

[12]

Jingyuan Cheng, Bo Zhou, Kai Kunze, Carl Christian Rheinländer, Sebastian Wille, Norbert Wehn, Jens Weppner, and Paul Lukowicz. 2013. Activity recognition and nutrition monitoring in every day situations with a textile capacitive neckband. In Proceedings of the ACM conference on Pervasive and ubiquitous computing adjunct publication. ACM Press, New York, New York, USA, 155.

Digital Library

[13]

N V Dhurandhar, D Schoeller, A W Brown, S B Heymsfield, D Thomas, T I A Sorensen, J R Speakman, M Jeansonne, and D B Allison. 2014. Energy balance measurement: when something is not better than nothing. International Journal of Obesity (Nov. 2014).

[14]

Yujie Dong. 2012. Tracking Wrist Motion to Detect and Measure the Eating Intake of Free-Living Humans. Thesis (Ph. D.) Clemson University (May 2012), 1--106.

[15]

Yujie Dong, Jenna Scisco, Mike Wilson, E Muth, and A Hoover. 2013. Detecting periods of eating during free living by tracking wrist motion. IEEE Journal of Biomedical Health Informatics (Sept. 2013).

[16]

Martin Ester, Hans-Peter Kriegel, Jörg Sander, Xiaowei Xu, and others. 1996. A density-based algorithm for discovering clusters in large spatial databases with noise. In Kdd, Vol. 96. 226--231.

Digital Library

[17]

Muhammad Farooq, Juan M Fontana, and Edward Sazonov. 2014. A novel approach for food intake detection using electroglottography. Physiological measurement 35, 5 (2014), 739.

[18]

Muhammad Farooq and Edward Sazonov. 2015. Comparative testing of piezoelectric and printed strain sensors in characterization of chewing. In Engineering in Medicine and Biology Society (EMBC), 2015 37th Annual International Conference of the IEEE. IEEE, 7538--7541.

[19]

Muhammad Farooq and Edward Sazonov. 2016. Detection of chewing from piezoelectric film sensor signals using ensemble classifiers. In Engineering in Medicine and Biology Society (EMBC), 2016 IEEE 38th Annual International Conference of the. IEEE, 4929--4932.

[20]

Muhammad Farooq and Edward Sazonov. 2016. A novel wearable device for food intake and physical activity recognition. Sensors 16, 7 (2016), 1067.

[21]

Vay Liang W Go, Christine T H Nguyen, Diane M Harris, and Wai-Nang Paul Lee. 2005. Nutrient-gene interaction: metabolic genotype-phenotype relationship. The Journal of nutrition 135, 12 Suppl (Dec. 2005), 3016S--3020S.

[22]

Samuli Hemminki, Petteri Nurmi, and Sasu Tarkoma. 2013. Accelerometer-based Transportation Mode Detection on Smartphones. In Proceedings of the 11th ACM Conference on Embedded Networked Sensor Systems (SenSys '13). ACM, New York, NY, USA, Article 13, 14 pages.

Digital Library

[23]

D R Jacobs. 2012. Challenges in research in nutritional epidemiology. Nutritional Health (2012), 29--42.

[24]

Wenyan Jia, Yaofeng Yue, John D Fernstrom, Zhengnan Zhang, Yongquan Yang, and Mingui Sun. 2012. 3D localization of circular feature in 2D image and application to food volume estimation. In Engineering in Medicine and Biology Society (EMBC), 2012 Annual International Conference of the IEEE. IEEE, 4545--4548.

[25]

Holger Junker, Oliver Amft, Paul Lukowicz, and Gerhard Tröster. 2008. Gesture spotting with body-worn inertial sensors to detect user activities. Pattern Recognition 41, 6 (June 2008), 2010--2024.

Digital Library

[26]

H Kalantarian, N Alshurafa, and M Sarrafzadeh. 2014. A Wearable Nutrition Monitoring System. In Wearable and Implantable Body Sensor Networks (BSN), 2014 11th International Conference on. 75--80.

Digital Library

[27]

Nathaniel Kleitman. 1963. Sleep and wakefulness. The University of Chicago Press, Chicago.

[28]

Flegal KM, Graubard BI, Williamson DF, and Gail MH. 2005. Excess deaths associated with underweight, overweight, and obesity. JAMA 293, 15 (2005), 1861--1867. arXiv:/data/Journals/JAMA/4972/JOC50018.pdf

[29]

N Lifson and Ruth McClintock. 1966. Theory of use of the turnover rates of body water for measuring energy and material balance. Journal of theoretical biology 12, 1 (1966), 46--74.

[30]

Jindong Liu, E Johns, L Atallah, C Pettitt, B Lo, G Frost, and Guang-Zhong Yang. 2012. An Intelligent Food-Intake Monitoring System Using Wearable Sensors. In Wearable and Implantable Body Sensor Networks (BSN), 2012 Ninth International Conference on. IEEE Computer Society, 154--160.

Digital Library

[31]

Oleksandr Makeyev, Paulo Lopez-Meyer, Stephanie Schuckers, Walter Besio, and Edward Sazonov. 2012. Biomedical Signal Processing and Control. Biomedical Signal Processing and Control 7, 6 (Nov. 2012), 649--656.

[32]

Christopher Merck, Christina Maher, Mark Mirtchouk, Min Zheng, Yuxiao Huang, and Samantha Kleinberg. 2016. Multimodality Sensing for Eating Recognition. Proceedings of PervasiveHealth (March 2016), 1--8.

Digital Library

[33]

K B Michels. 2001. A renaissance for measurement error. International journal of epidemiology 30, 3 (June 2001), 421--422.

[34]

Temiloluwa Olubanjo and Maysam Ghovanloo. 2014. Real-time swallowing detection based on tracheal acoustics. In Acoustics, Speech and Signal Processing (ICASSP), 2014 IEEE International Conference on. IEEE, 4384--4388.

[35]

Sebastian Päßler, Matthias Wolff, and Wolf-Joachim Fischer. 2012. Food intake monitoring: an acoustical approach to automated food intake activity detection and classification of consumed food. Physiological Measurement 33, 6 (May 2012), 1073--1093.

[36]

JMC Po, JA Kieser, LM Gallo, AJ Tésenyi, P Herbison, and M Farella. 2011. Time-frequency analysis of chewing activity in the natural environment. Journal of dental research 90, 10 (2011), 1206--1210.

[37]

Tauhidur Rahman, Mary Czerwinski, Ran Gilad-Bachrach, and Paul Johns. 2016. Predicting "About-to-Eat" Moments for Just-in-Time Eating Intervention. In DH '16: Proceedings of the 6th International Conference on Digital Health Conference. Cornell University, ACM.

Digital Library

[38]

Ahmad Safari and E Koray Akdogan. 2008. Piezoelectric and acoustic materials for transducer applications. Springer Science 8 Business Media.

[39]

Edward Sazonov, Stephanie Schuckers, Paulo Lopez-Meyer, Oleksandr Makeyev, Nadezhda Sazonova, Edward L Melanson, and Michael Neuman. 2008. Non-invasive monitoring of chewing and swallowing for objective quantification of ingestive behavior. Physiological Measurement 29, 5 (April 2008), 525--541.

[40]

Schoeller. 1995. Limitations in the assessment of dietary energy intake by self-report. Metabolism 44 (Feb. 1995), 5--5.

[41]

Donna Spruijt-Metz and Wendy Nilsen. 2014. Dynamic models of behavior for just-in-time adaptive interventions. IEEE Pervasive Computing 3, 13 (2014), 13--17.

[42]

E Stellar and E E Shrager. 1985. Chews and swallows and the micro structure of eating. The American journal of clinical nutrition 42, 5 (1985), 973--982.

[43]

Mingui Sun, Lora E Burke, Zhi-Hong Mao, Yiran Chen, Hsin-Chen Chen, Yicheng Bai, Yuecheng Li, Chengliu Li, and Wenyan Jia. 2014. eButton: a wearable computer for health monitoring and personal assistance. In Design Automation Conference (DAC), 2014 51st ACM/EDAC/IEEE.IEEE, 1--6.

Digital Library

[44]

Edison Thomaz, GD Abowd, and Irfan Essa. 2015. A Practical Approach for Recognizing Eating Moments with Wrist-Mounted Inertial Sensing. In UbiComp '15: Proceedings of the 2015 ACM international joint conference on Pervasive and ubiquitous computing. 1--12.

Digital Library

[45]

Edison Thomaz, Irfan Essa, and Gregory D Abowd. 2015. A practical approach for recognizing eating moments with wrist-mounted inertial sensing. In Proceedings of the 2015 ACM International Joint Conference on Pervasive and Ubiquitous Computing. ACM, 1029--1040.

Digital Library

[46]

Edison Thomaz, Cheng Zhang, Irfan Essa, and Gregory D Abowd. 2015. Inferring Meal Eating Activities in Real World Settings from Ambient Sounds. In the 20th Intelligent User Interfaces Conference (IUI). ACM Press, New York, New York, USA, 427--431.

Digital Library

[47]

D Turner and others. 1940. The estimation of the patient's home dietary intake. Journal of the American Dietetic Association 16 (1940), 875--881.

[48]

EM Widdowson. 1936. A stud of English diets by the individual method: Part I. Men. Journal of Hygiene 36, 03 (1936), 269--290.

[49]

EM Widdowson and RA McCance. 1936. A study of English diets by the individual method: Part II. Women. Journal of Hygiene 36, 03 (1936), 293--307.

[50]

Peter Widhalm, Philippe Nitsche, and Norbert Brändie. 2012. Transport mode detection with realistic smartphone sensor data. In Pattern Recognition (ICPR), 2012 21st International Conference on. IEEE, 573--576.

[51]

Dorothy G Wiehl. 1942. Diets of a group of aircraft workers in Southern California. The Milbank Memorial Fund Quarterly (1942), 329--366.

[52]

Dorothy G Wiehl and Robert Reed. 1960. Development of new or improved dietary methods for epidemiological investigations. American Journal of Public Health and the Nations Health 50, 6_Pt_1 (1960), 824--828.

[53]

Koji Yatani and Khai N Truong. 2012. BodyScope: a wearable acoustic sensor for activity recognition. UbiComp '12: Proceedings of the 2012 ACM Conference on Ubiquitous Computing (2012), 341--350.

Digital Library

Cited By

Cherian JRay STaele PKoh JHammond T(2024)Exploring the Impact of the NULL Class on In-the-Wild Human Activity RecognitionSensors10.3390/s2412389824:12(3898)Online publication date: 16-Jun-2024
https://doi.org/10.3390/s24123898
Mahmud SAgarwal DAjit ALiang QViranda TGuimbretiere FZhang CKostakos VKay JHoang T(2024)MunchSonic: Tracking Fine-grained Dietary Actions through Active Acoustic Sensing on EyeglassesProceedings of the 2024 ACM International Symposium on Wearable Computers10.1145/3675095.3676619(96-103)Online publication date: 5-Oct-2024
https://dl.acm.org/doi/10.1145/3675095.3676619
Wang YObie HLi ZSalim FGrundy JMueller F(2024)GustosonicSense: Towards understanding the design of playful gustosonic eating experiencesProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642182(1-12)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3642182
Show More Cited By

Index Terms

Detecting Eating Episodes by Tracking Jawbone Movements with a Non-Contact Wearable Sensor
1. Applied computing
  1. Life and medical sciences
    1. Health informatics
2. Human-centered computing
  1. Ubiquitous and mobile computing
    1. Empirical studies in ubiquitous and mobile computing

Recommendations

Toward a Wearable Sensor for Eating Detection
WearSys '17: Proceedings of the 2017 Workshop on Wearable Systems and Applications

Researchers strive to understand eating behavior as a means to develop diets and interventions that can help people achieve and maintain a healthy weight, recover from eating disorders, or manage their diet and nutrition for personal wellness. A major ...
Auracle: Detecting Eating Episodes with an Ear-mounted Sensor

In this paper, we propose Auracle, a wearable earpiece that can automatically recognize eating behavior. More specifically, in free-living conditions, we can recognize when and for how long a person is eating. Using an off-the-shelf contact microphone ...
MyDJ: Sensing Food Intakes with an Attachable on Your Eyeglass Frame
CHI '22: Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems

Various automated eating detection wearables have been proposed to monitor food intakes. While these systems overcome the forgetfulness of manual user journaling, they typically show low accuracy at outside-the-lab environments or have intrusive form-...

Comments

Information & Contributors

Information

Published In

cover image Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies

Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies Volume 2, Issue 1

March 2018

1370 pages

EISSN:2474-9567

DOI:10.1145/3200905

Issue’s Table of Contents

Copyright © 2018 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 26 March 2018

Accepted: 01 January 2018

Revised: 01 November 2017

Received: 01 August 2017

Published in IMWUT Volume 2, Issue 1

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

54
Total Citations
View Citations
994
Total Downloads

Downloads (Last 12 months)93
Downloads (Last 6 weeks)13

Reflects downloads up to 26 Sep 2024

Other Metrics

View Author Metrics

Citations

Cited By

Cherian JRay STaele PKoh JHammond T(2024)Exploring the Impact of the NULL Class on In-the-Wild Human Activity RecognitionSensors10.3390/s2412389824:12(3898)Online publication date: 16-Jun-2024
https://doi.org/10.3390/s24123898
Mahmud SAgarwal DAjit ALiang QViranda TGuimbretiere FZhang CKostakos VKay JHoang T(2024)MunchSonic: Tracking Fine-grained Dietary Actions through Active Acoustic Sensing on EyeglassesProceedings of the 2024 ACM International Symposium on Wearable Computers10.1145/3675095.3676619(96-103)Online publication date: 5-Oct-2024
https://dl.acm.org/doi/10.1145/3675095.3676619
Wang YObie HLi ZSalim FGrundy JMueller F(2024)GustosonicSense: Towards understanding the design of playful gustosonic eating experiencesProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642182(1-12)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3642182
Hiraguchi HPerone PToet ACamps GBrouwer A(2023)Technology to Automatically Record Eating Behavior in Real Life: A Systematic ReviewSensors10.3390/s2318775723:18(7757)Online publication date: 8-Sep-2023
https://doi.org/10.3390/s23187757
Liang DZhang AThomaz E(2023)Automated Face-To-Face Conversation Detection on a Commodity Smartwatch with Acoustic SensingProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36108827:3(1-29)Online publication date: 27-Sep-2023
https://dl.acm.org/doi/10.1145/3610882
Silva LAnkrah EHuai YEpstein D(2023)Exploring Opportunities for Multimodality and Multiple Devices in Food JournalingProceedings of the ACM on Human-Computer Interaction10.1145/36042567:MHCI(1-27)Online publication date: 13-Sep-2023
https://dl.acm.org/doi/10.1145/3604256
Alsafery WRana OPerera C(2023)Sensing within Smart Buildings: A SurveyACM Computing Surveys10.1145/359660055:13s(1-35)Online publication date: 13-Jul-2023
https://dl.acm.org/doi/10.1145/3596600
Shahi SSen SPedram MAlharbi RGao YKatsaggelos AHester JAlshurafa NMao SWang SFang HGao WZhou G(2023)Detecting Eating, and Social Presence with All Day Wearable RGB-TProceedings of the 8th ACM/IEEE International Conference on Connected Health: Applications, Systems and Engineering Technologies10.1145/3580252.3586974(68-79)Online publication date: 21-Jun-2023
https://dl.acm.org/doi/10.1145/3580252.3586974
Pedram MFernandes GRomano CWei BSen SHester JAlshurafa N(2023)Experience: Barriers and Opportunities of Wearables for Eating ResearchExtended Abstracts of the 2023 CHI Conference on Human Factors in Computing Systems10.1145/3544549.3573841(1-8)Online publication date: 19-Apr-2023
https://dl.acm.org/doi/10.1145/3544549.3573841
Wei BZhang SDiao XXu QGao YAlshurafa N(2023)An End-to-End Energy-Efficient Approach for Intake Detection With Low Inference Time Using Wrist-Worn SensorIEEE Journal of Biomedical and Health Informatics10.1109/JBHI.2023.327662927:8(3878-3888)Online publication date: Aug-2023
https://doi.org/10.1109/JBHI.2023.3276629
Show More Cited By

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Issue’s Table of Contents