research-article

Chewing Detection from Commercial Smart-glasses

Authors:

Vasileios Papapanagiotou,

Anastasia Liapi,

Anastasios DelopoulosAuthors Info & Claims

MADiMa '22: Proceedings of the 7th International Workshop on Multimedia Assisted Dietary Management

Pages 11 - 16

https://doi.org/10.1145/3552484.3555746

Published: 24 October 2022 Publication History

Abstract

Automatic dietary monitoring has progressed significantly during the last years, offering a variety of solutions, both in terms of sensors and algorithms as well as in terms of what aspect or parameters of eating behavior are measured and monitored. Automatic detection of eating based on chewing sounds has been studied extensively, however, it requires a microphone to be mounted on the subject's head for capturing the relevant sounds. In this work, we evaluate the feasibility of using an off-the-shelf commercial device, the Razer Anzu smart-glasses, for automatic chewing detection. The smart-glasses are equipped with stereo speakers and microphones that communicate with smart-phones via Bluetooth. The microphone placement is not optimal for capturing chewing sounds, however, we find that it does not significantly affect the detection effectiveness. We apply an algorithm from literature with some adjustments on a challenging dataset that we have collected in house. Leave-one-subject-out experiments yield promising results, with an F1-score of $0.96$ for the best case of duration-based evaluation of eating time.

References

[1]

Mohammad Aboofazeli and Zahra Moussavi. 2009. Swallowing sound detection using hidden markov modeling of recurrence plot features. Chaos, Solitons & Fractals, 39, 2, 778--783.

[2]

Oliver Amft, Martin Kusserow, and Gerhard Troster. 2009. Bite weight prediction from acoustic recognition of chewing. IEEE Transactions on Biomedical Engineering, 56, 6, 1663--1672.

[3]

Oliver Amft, Mathias Stäger, Paul Lukowicz, and Gerhard Tröster. 2005. Analysis of chewing sounds for dietary monitoring. In UbiComp 2005: Ubiquitous Computing. Michael Beigl, Stephen Intille, Jun Rekimoto, and Hideyuki Tokuda, (Eds.) Springer Berlin Heidelberg, Berlin, Heidelberg, 56--72. isbn: 978--3--540- 31941--2.

[4]

Oliver Amft and Gerhard Troster. 2006. Methods for detection and classification of normal swallowing from muscle activation and sound. In 2006 Pervasive Health Conference and Workshops, 1--10. 4.

[5]

Marios Anthimopoulos, Joachim Dehais, Sergey Shevchik, Botwey H. Ransford, David Duke, Peter Diem, and Stavroula Mougiakakou. 2015. Computer vision-based carbohydrate estimation for type 1 patients with diabetes using smartphones. Journal of Diabetes Science and Technology, 9, 3, 507--515. 25883163. eprint: https://doi.org/10.1777/1932296815580159.

[6]

Marios M. Anthimopoulos, Lauro Gianola, Luca Scarnato, Peter Diem, and Stavroula G. Mougiakakou. 2014. A food recognition system for diabetic patients based on an optimized bag-of-features model. IEEE Journal of Biomedical and Health Informatics, 18, 4, 1261--1271.

[7]

Sinem Aslan, Gianluigi Ciocca, and Raimondo Schettini. 2018. Semantic food segmentation for automatic dietary monitoring. In 2018 IEEE 8th International Conference on Consumer Electronics - Berlin (ICCE-Berlin), 1--6. /ICCE-Berlin.2018.8576231.

[8]

Jungman Chung, Jungmin Chung, Wonjun Oh, Yongkyu Yoo, Won Gu Lee, and Hyunwoo Bang. 2017. A glasses-type wearable device for monitoring the patterns of food intake and facial activity. Scientific Reports, 7, 1, (Jan. 2017), 41690.

[9]

Gianluigi Ciocca, Davide Mazzini, and Raimondo Schettini. 2019. Evaluating cnn-based semantic food segmentation across illuminants. In Computational Color Imaging. Shoji Tominaga, Raimondo Schettini, Alain Trémeau, and Takahiko Horiuchi, (Eds.) Springer International Publishing, Cham, 247--259. isbn: 978--3-030--13940--7.

[10]

Joachim Dehais, Marios Anthimopoulos, Sergey Shevchik, and Stavroula Mougiakakou. 2017. Two-view 3d reconstruction for food volume estimation. IEEE Transactions on Multimedia, 19, 5, 1090--1099.

Digital Library

[11]

Takumi Ege, Wataru Shimoda, and Keiji Yanai. 2019. A new large-scale food image segmentation dataset and its application to food calorie estimation based on grains of rice. In Proceedings of the 5th International Workshop on Multimedia Assisted Dietary Management (MADiMa '19). Association for Computing Machinery, Nice, France, 82-87. isbn: 9781450369169. 57162.

[12]

Hamid Heydarian, Philipp V. Rouast, Marc T. P. Adam, Tracy Burrows, Clare E. Collins, and Megan E. Rollo. 2020. Deep learning for intake gesture detection from wrist-worn inertial sensors: the effects of data preprocessing, sensor modalities, and sensor positions. IEEE Access, 8, 164936--164949. /ACCESS.2020.3022042.

[13]

Qianyi Huang, Wei Wang, and Qian Zhang. 2017. Your glasses know your diet: dietary monitoring using electromyography sensors. IEEE Internet of Things Journal, 4, 3, 705--712.

[14]

Konstantinos Kyritsis, Christos Diou, and Anastasios Delopoulos. 2021. A data driven end-to-end approach for in-the-wild monitoring of eating behavior using smartwatches. IEEE Journal of Biomedical and Health Informatics, 25, 1, 22-34.

[15]

Konstantinos Kyritsis, Petter Fagerberg, Ioannis Ioakimidis, K. Ray Chaudhuri, Heinz Reichmann, Lisa Klingelhoefer, and Anastasios Delopoulos. 2021. Assessment of real life eating difficulties in parkinson's disease patients by measuring plate to mouth movement elongation with inertial sensors. Scientific Reports, 11, 1, (Jan. 2021), 1632.

[16]

Frank P. -W. Lo, Yingnan Sun, Jianing Qiu, and Benny Lo. 2018. Food volume estimation based on deep learning view synthesis from a single depth map. Nutrients, 10, 12.

[17]

Gert Mertes, Hans Hallez, Tom Croonenborghs, and Bart Vanrumste. 2019. Detection of chewing motion using a glasses mounted accelerometer towards monitoring of food intake events in the elderly. In International Conference on Biomedical and Health Informatics. Yuan-Ting Zhang, Paulo Carvalho, and Ratko Magjarevic, (Eds.) Springer Singapore, Singapore, 73--77. isbn: 978--981- 10--4505--9.

[18]

Yue Ming, Xuyang Meng, Chunxiao Fan, and Hui Yu. 2021. Deep learning for monocular depth estimation: a review. Neurocomputing, 438, 14--33.

[19]

Mark Mirtchouk, Christopher Merck, and Samantha Kleinberg. 2016. Automated estimation of food type and amount consumed from body-worn audio and motion sensors. In Proceedings of the 2016 ACM International Joint Conference on Pervasive and Ubiquitous Computing (UbiComp '16). Association for Computing Machinery, Heidelberg, Germany, 451--462. isbn: 9781450344616. https://doi.org/10.1145/2971648.2971677.

Digital Library

[20]

Vasileios Papapanagiotou, Christos Diou, Zhou Lingchuan, Janet van den Boer, Monica Mars, and Anastasios Delopoulos. 2015. Fractal nature of chewing sounds. In New Trends in Image Analysis and Processing - ICIAP 2015 Workshops: ICIAP 2015 International Workshops, BioFor, CTMR, RHEUMA, ISCA, MADiMa, SBMI, and QoEM, Genoa, Italy, September 7--8, 2015, Proceedings. Vittorio Murino, Enrico Puppo, Diego Sona, Marco Cristani, and Carlo Sansone, (Eds.) Springer International Publishing, Cham, 401--408. isbn: 978--3--319--23222--5. 7/978--3--319--23222--5_49.

Digital Library

[21]

Vasileios Papapanagiotou, Christos Diou, Janet van den Boer, Monica Mars, and Anastasios Delopoulos. 2021. Recognition of food-texture attributes using an in-ear microphone. In Pattern Recognition. ICPR International Workshops and Challenges. Alberto Del Bimbo, Rita Cucchiara, Stan Sclaroff, Giovanni Maria Farinella, Tao Mei, Marco Bertini, Hugo Jair Escalante, and Roberto Vezzani(Eds.) Springer International Publishing, Cham, 558--570. isbn: 978--3-030--68821- 9.

[22]

Vasileios Papapanagiotou, Christos Diou, Lingchuan Zhou, Janet van den Boer, Monica Mars, and Anastasios Delopoulos. 2017. A novel chewing detection system based on ppg, audio, and accelerometry. IEEE Journal of Biomedical and Health Informatics, 21, 3, 607--618.

[23]

Vasileios Papapanagiotou, Christos Diou, Lingchuan Zhou, Janet van den Boer, Monica Mars, and Anastasios Delopoulos. 2017. The splendid chewing detection challenge. In 2017 39th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC). (July 2017), 817--820.

[24]

Vasileios Papapanagiotou, Stefanos Ganotakis, and Anastasios Delopoulos. 2021. Bite-weight estimation using commercial ear buds. In 2021 43rd Annual International Conference of the IEEE Engineering in Medicine & Biology Society (EMBC), 7182-7185.

[25]

P. Rayneau, R. Bouteloup, C. Rouf, P. Makris, and S. Moriniere. 2021. Automatic detection and analysis of swallowing sounds in healthy subjects and in patients with pharyngolaryngeal cancer. Dysphagia, 36, 6, (Dec. 2021), 984--992.

[26]

Janet van den Boer, Annemiek van der Lee, Lingchuan Zhou, Vasileios Papapanagiotou, Christos Diou, Anastasios Delopoulos, and Monica Mars. 2018. The splendid eating detection sensor: development and feasibility study. JMIR Mhealth Uhealth, 6, 9, (Sept. 2018), e170.

[27]

Rui Zhang and Oliver Amft. 2016. Bite glasses: measuring chewing using emg and bone vibration in smart eyeglasses. In Proceedings of the 2016 ACM International Symposium on Wearable Computers (ISWC '16). Association for Computing Machinery, Heidelberg, Germany, 50--52. isbn: 9781450344609.

Digital Library

[28]

Rui Zhang and Oliver Amft. 2018. Monitoring chewing and eating in free-living using smart eyeglasses. IEEE Journal of Biomedical and Health Informatics, 22, 1, 23--32.

[29]

Rui Zhang, Severin Bernhart, and Oliver Amft. 2016. Diet eyeglasses: recognising food chewing using emg and smart eyeglasses. In 2016 IEEE 13th International Conference on Wearable and Implantable Body Sensor Networks (BSN), 7--12.

Cited By

Ketmalasiri TWu YButkow KMascolo CLiu Y(2024)IMChew: Chewing Analysis using Earphone Inertial Measurement UnitsProceedings of the Workshop on Body-Centric Computing Systems10.1145/3662009.3662022(29-34)Online publication date: 3-Jun-2024
https://dl.acm.org/doi/10.1145/3662009.3662022
Hiraguchi HPerone PToet ACamps GBrouwer A(2023)Technology to Automatically Record Eating Behavior in Real Life: A Systematic ReviewSensors10.3390/s2318775723:18(7757)Online publication date: 8-Sep-2023
https://doi.org/10.3390/s23187757

Index Terms

Chewing Detection from Commercial Smart-glasses
1. Human-centered computing
  1. Ubiquitous and mobile computing
    1. Ubiquitous and mobile computing systems and tools

Recommendations

Demo: Chewpin: a wearable acoustic device for chewing detection
UbiComp/ISWC '21 Adjunct: Adjunct Proceedings of the 2021 ACM International Joint Conference on Pervasive and Ubiquitous Computing and Proceedings of the 2021 ACM International Symposium on Wearable Computers

Diet intervention has emerged as a promising strategy in obesity prevention and treatment. Existing research predominantly focused on macronutrient intake and food quantity restriction in diet manipulation. Eating habit is difficult to change due to ...
Chewing jockey: augmented food texture by using sound based on the cross-modal effect
ACE '11: Proceedings of the 8th International Conference on Advances in Computer Entertainment Technology

We focus on the dining and show how to improve dining experience. We use sound effects to augment food texture, creating a cross-modal illusion.

Our system is composed of a bone-conduction speaker, a microphone, a photoreflector to measure the motion of ...
Food intake and chewing in women

The frequency and weight of food removed from a plate and placed in the mouth was recorded by computer and synchronized with the pattern of chewing recorded by video filming of the maxillary-mandibular region in women selected for eating with a ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

MADiMa '22: Proceedings of the 7th International Workshop on Multimedia Assisted Dietary Management

October 2022

97 pages

ISBN:9781450395021

DOI:10.1145/3552484

General Chairs:
Stavroula Mougiakakou
University of Bern, Switzerland
,
Giovanni Maria Farinella
University of Catania, Italy
,
Keiji Yanai
The University of Electro-Communications, Tokyo, Japan
,
Dario Allegra
University of Catania, Italy

Copyright © 2022 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGMM: ACM Special Interest Group on Multimedia

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 24 October 2022

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

European Community?s Health, demographic change and well-being Program

Conference

MM '22

Sponsor:

SIGMM

MM '22: The 30th ACM International Conference on Multimedia

October 10, 2022

Lisboa, Portugal

Acceptance Rates

MADiMa '22 Paper Acceptance Rate 9 of 10 submissions, 90%;

Overall Acceptance Rate 16 of 24 submissions, 67%

Upcoming Conference

MM '24

Sponsor:
sigmm

The 32nd ACM International Conference on Multimedia

October 28 - November 1, 2024

Melbourne , VIC , Australia

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

2
Total Citations
View Citations
96
Total Downloads

Downloads (Last 12 months)41
Downloads (Last 6 weeks)4

Reflects downloads up to 15 Oct 2024

Other Metrics

View Author Metrics

Citations

Cited By

Ketmalasiri TWu YButkow KMascolo CLiu Y(2024)IMChew: Chewing Analysis using Earphone Inertial Measurement UnitsProceedings of the Workshop on Body-Centric Computing Systems10.1145/3662009.3662022(29-34)Online publication date: 3-Jun-2024
https://dl.acm.org/doi/10.1145/3662009.3662022
Hiraguchi HPerone PToet ACamps GBrouwer A(2023)Technology to Automatically Record Eating Behavior in Real Life: A Systematic ReviewSensors10.3390/s2318775723:18(7757)Online publication date: 8-Sep-2023
https://doi.org/10.3390/s23187757

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents