research-article

BreathEasy: Assessing Respiratory Diseases Using Mobile Multimodal Sensors

Authors:

Md Mahbubur Rahman,

Mohsin Yusuf Ahmed,

Korosh Vatanparvar,

Ebrahim Nemati,

Daniel McCaffrey,

Jun Alex GaoAuthors Info & Claims

ICMI '20: Proceedings of the 2020 International Conference on Multimodal Interaction

Pages 41 - 49

https://doi.org/10.1145/3382507.3418852

Published: 22 October 2020 Publication History

Abstract

Mobil respiratory assessments using commodity smartphones and smartwatches are unmet needs for patient monitoring at home. In this paper, we show the feasibility of using multimodal sensors embedded in consumer mobile devices for non-invasive, low-effort respiratory assessment. We have conducted studies with 228 chronic respiratory patients and healthy subjects, and show that our model can estimate respiratory rate with mean absolute error (MAE) 0.72$\pm$0.62 breath per minute and differentiate respiratory patients from healthy subjects with 90% recall and 76% precision when the user breathes normally by holding the device on the chest or the abdomen for a minute. Holding the device on the chest or abdomen needs significantly lower effort compared to traditional spirometry which requires a specialized device and forceful vigorous breathing. This paper shows the feasibility of developing a low-effort respiratory assessment towards making it available anywhere, anytime through users' own mobile devices.

Supplementary Material

MP4 File (3382507.3418852.mp4)

Presentation Video.

Download
10.58 MB

References

[1]

U Rajendra Acharya, K Paul Joseph, Natarajan Kannathal, Choo Min Lim, and Jasjit S Suri. 2006. Heart rate variability: a review. Medical and biological engineering and computing 44, 12 (2006), 1031--1051.

[2]

Tao Ai, Zhenlu Yang, Hongyan Hou, Chenao Zhan, Chong Chen, Wenzhi Lv, Qian Tao, Ziyong Sun, and Liming Xia. 2020. Correlation of chest CT and RT-PCR testing in coronavirus disease 2019 (COVID-19) in China: a report of 1014 cases. Radiology (2020), 200642.

[3]

Heba Aly and Moustafa Youssef. 2016. Zephyr: Ubiquitous accurate multi-sensor fusion-based respiratory rate estimation using smartphones. In IEEE INFOCOM.

[4]

Rohan Banerjee and Aniruddha Sinha. 2013. TrigSense: Accelerometer triggered audio sensing for traffic condition monitoring. In IEEE International Conference on Intelligent Sensors, Sensor Networks and Information Processing.

[5]

Rummana Bari, Roy J Adams, Md Mahbubur Rahman, Megan Battles Parsons, Eugene H Buder, and Santosh Kumar. 2018. rConverse: Moment by moment conversation detection using a mobile respiration sensor. Proceedings of the ACM on interactive, mobile, wearable and ubiquitous technologies 2, 1 (2018), 2.

Digital Library

[6]

Joerg Bitzer, Sven Kissner, and Inga Holube. 2016. Privacy-aware acoustic assessments of everyday life. Journal of the Audio Engineering Society 64, 6 (2016), 395--404.

[7]

S Burge and JA Wedzicha. 2003. COPD exacerbations: definitions and classifications. European Respiratory Journal 21, 41 suppl (2003), 46s--53s.

[8]

Soujanya Chatterjee, Md Mahbubur Rahman, Tousif Ahmed, Nazir Saleheen, Ebrahim Nemati, Viswam Nathan, Korosh Vatanparvar, and Jilong Kuang. 2020. Assessing Severity of Pulmonary Obstruction from Respiration Phase-Based Wheeze-Sensing Using Mobile Sensors. In ACM Conference on Human Factors in Computing Systems (CHI).

Digital Library

[9]

IHME COVID, Christopher JL Murray, et al. 2020. Forecasting COVID-19 impact on hospital bed-days, ICU-days, ventilator-days and deaths by US state in the next 4 months. MedRxiv (2020).

[10]

Amy Droitcour, Byung Kwon Park, Charles El Hourani, and Shuhei Yamada. 2010. Systems and methods for measurement of depth of breath and paradoxical breathing. US Patent App. 12/790,769.

[11]

Jérémy Frey, May Grabli, Ronit Slyper, and Jessica R Cauchard. 2018. Breeze: sharing biofeedback through wearable technologies. In ACM Conference on Human Factors in Computing Systems (CHI).

Digital Library

[12]

Lex Fridman, Daniel E Brown, William Angell, Irman Abdi?, Bryan Reimer, and Hae Young Noh. 2016. Automated synchronization of driving data using vibration and steering events. Pattern Recognition Letters 75 (2016), 9--15.

Digital Library

[13]

Stavros Garantziotis and David A Schwartz. 2010. Ecogenomics of respiratory diseases of public health significance. Annual review of public health 31 (2010), 37--51.

[14]

Davide Giavarina. 2015. Understanding bland altman analysis. Biochemia medica: Biochemia medica 25, 2 (2015), 141--151.

[15]

Mayank Goel, Elliot Saba, Maia Stiber, Eric Whitmire, Josh Fromm, Eric C Larson, Gaetano Borriello, and Shwetak N Patel. 2016. Spirocall: Measuring lung function over a phone call. In ACM Conference on Human Factors in Computing Systems (CHI).

Digital Library

[16]

Christian Gollier and Olivier Gossner. 2020. Group testing against Covid-19. Covid Economics 2 (2020).

[17]

Daniel K Goyal, Fatma Mansab, Amir Iqbal, and Sohail Bhatti. 2020. Early intervention likely improves mortality in COVID-19 infection. Clinical Medicine 20, 3 (2020), 248--250.

[18]

Trisha Greenhalgh, Gerald Choon Huat Koh, and Josip Car. 2020. Covid-19: a remote assessment in primary care. bmj 368 (2020).

[19]

Federico Guede-Fernández, Víctor Ferrer-Mileo, Juan Ramos-Castro, Mireya Fernández-Chimeno, and Miguel A García-González. 2015. Real time heart rate variability assessment from Android smartphone camera photoplethysmography: postural and device influences. In IEEE EMBC.

[20]

Gillian R Hayes, Shwetak N Patel, Khai N Truong, Giovanni Iachello, Julie A Kientz, Rob Farmer, and Gregory D Abowd. 2004. The personal audio loop: Designing a ubiquitous audio-based memory aid. In International Conference on Mobile Human-Computer Interaction. Springer, 168--179.

[21]

Javier Hernandez, Daniel McDuff, and Rosalind W Picard. 2015. Biowatch: estimation of heart and breathing rates from wrist motions. In IEEE PervasiveHealth.

[22]

Daniel E Hilleman, Naresh Dewan, Mark Malesker, and Mitchell Friedman. 2000. Pharmacoeconomic evaluation of COPD. Chest 118, 5 (2000), 1278--1285.

[23]

C. Jia and B. L. Evans. 2013. Online calibration and synchronization of cellphone camera and gyroscope. In IEEE Global Conference on Signal and Information Processing.

[24]

Bruce P Krieger and Patti Ershowsky. 1988. Noninvasive detection of respiratory failure in the intensive care unit. Chest (1988).

[25]

Eric C Larson, Mayank Goel, Gaetano Boriello, Sonya Heltshe, Margaret Rosenfeld, and Shwetak N Patel. 2012. SpiroSmart: using a microphone to measure lung function on a mobile phone. In ACM UbiComp.

[26]

Xiao Li, Jessilyn Dunn, Denis Salins, Gao Zhou, Wenyu Zhou, Sophia Miryam Schüssler-Fiorenza Rose, Dalia Perelman, Elizabeth Colbert, Ryan Runge, Shannon Rego, et al. 2017. Digital health: tracking physiomes and activity using wearable biosensors reveals useful health-related information. PLoS biology 15, 1 (2017), e2001402.

[27]

Marc Miravitlles, Cristina Murio, Tina Guerrero, Ramon Gisbert, DAFNE Study Group, et al. 2003. Costs of chronic bronchitis and COPD: a 1-year follow-up study. Chest 123, 3 (2003), 784--791.

[28]

Aravind Natarajan, Hao-Wei Su, and Conor Heneghan. 2020. Assessment of physiological signs associated with COVID-19 measured using wearable devices. medRxiv (2020).

[29]

Ebrahim Nemati, Md Mahbubur Rahman, Viswam Nathan, and Jilong Kuang. 2018. Private audio-based cough sensing for in-home pulmonary assessment using mobile devices. In EAI International Conference on Body Area Networks. Springer, 221--232.

[30]

Panagiotis Pelegris, K Banitsas, T Orbach, and Kostas Marias. 2010. A novel method to detect heart beat rate using a mobile phone. In IEEE EMBC.

[31]

Mayra Caleffi Pereira, Desiderio Cano Porras, Adriana Claudia Lunardi, Cibele Cristine Berto Marques da Silva, Renata Cléia Claudino Barbosa, Letícia Zumpano Cardenas, Renata Pletsch, Jeferson George Ferreira, Isac de Castro, Celso Ricardo Fernandes de Carvalho, et al. 2017. Thoracoabdominal asynchrony: Two methods in healthy, COPD, and interstitial lung disease patients. PloS one (2017).

[32]

Md Juber Rahman, Ebrahim Nemati, Md Mahbubur Rahman, Viswam Nathan, Korosh Vatanparvar, and Jilong Kuang. 2020. Automated assessment of pulmonary patients using heart rate variability from everyday wearables. Smart Health 15 (2020), 100081.

[33]

Md Mahbubur Rahman, Tousif Ahmed, Ebrahim Nemati, Viswam Nathan, Korosh Vatanparvar, Erin Blackstock, and Jilong Kuang. 2020. ExhaleSense: Detecting High Fidelity Forced Exhalations to Estimate Lung Obstruction on Smartphones. In IEEE PerCom.

[34]

Md Mahbubur Rahman, Rummana Bari, Amin Ahsan Ali, Moushumi Sharmin, Andrew Raij, Karen Hovsepian, Syed Monowar Hossain, Emre Ertin, Ashley Kennedy, David H Epstein, et al. 2014. Are we there yet?: Feasibility of continuous stress assessment via wireless physiological sensors. In ACM Conference on Bioinformatics, Computational Biology, and Health Informatics.

Digital Library

[35]

Md Mahbubur Rahman, Ebrahim Nemati, Viswam Nathan, and Jilong Kuang. 2018. Instantrr: Instantaneous respiratory rate estimation on context-aware mobile devices. In EAI International Conference on Body Area Networks. Springer, 267--281.

[36]

Donna Renfrow Rutala, William A Rutala, David J Weber, and Charlotte A Thomann. 1991. Infection risks associated with spirometry. Infection Control & Hospital Epidemiology 12, 2 (1991), 89--92.

[37]

Keum San Chun, Viswam Nathan, Korosh Vatanparvar, Ebrahim Nemati, Md Mahbubur Rahman, Erin Blackstock, and Jilong Kuang. 2020. Towards Passive Assessment of Pulmonary Function from Natural Speech Recorded Using a Mobile Phone. In IEEE PerCom.

[38]

Malay Sarkar, Rajeev Bhardwaz, Irappa Madabhavi, and Mitul Modi. 2019. Physical signs in patients with chronic obstructive pulmonary disease. Lung India: official organ of Indian Chest Society 36, 1 (2019), 38.

[39]

Ronald W Schafer. 2011. What is a Savitzky-Golay filter?[lecture notes]. IEEE Signal processing magazine 28, 4 (2011), 111--117.

[40]

Neil W Schluger and Ram Koppaka. 2014. Lung disease in a global context. A call for public health action. Annals of the American Thoracic Society 11, 3 (2014), 407--416.

[41]

Manish Sharma, Akash Anand, Rupika Srivastava, and Lakshmi Kaligounder. 2018. Wearable Audio and IMU Based Shot Detection in Racquet Sports. arXiv:1805.05456 [cs.LG]

[42]

American Thoracic Society et al. 2000. Guidelines for methacholine and exercise challenge testing 1999. Am. J. Respir. Crit. Care Med. 161 (2000), 309--239.

[43]

A Spinelli and G Pellino. 2020. COVID-19 pandemic: perspectives on an unfolding crisis. The British Journal of Surgery (2020).

[44]

Varun Viswanath, Jake Garrison, and Shwetak Patel. 2018. SpiroConfidence: determining the validity of smartphone based spirometry using machine learning. In IEEE EMBC.

Cited By

Tripathi GSabherwal MSingh P(2024)“I know I have this till my Last Breath”: Unmasking the Gaps in Chronic Obstructive Pulmonary Disease (COPD) Care in IndiaProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642504(1-16)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3642504
Bhalla SLiaqat SWu RGershon Ade Lara EMariakakis A(2023)PulmoListenerProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36108897:3(1-24)Online publication date: 27-Sep-2023
https://dl.acm.org/doi/10.1145/3610889
Adhikary RLodhavia DFrancis CPatil RSrivastava TKhanna PBatra NBreda JPeplinski JPatel S(2023)SpiroMask: Measuring Lung Function Using Consumer-Grade MasksACM Transactions on Computing for Healthcare10.1145/35701674:1(1-34)Online publication date: 27-Feb-2023
https://dl.acm.org/doi/10.1145/3570167
Show More Cited By

Index Terms

BreathEasy: Assessing Respiratory Diseases Using Mobile Multimodal Sensors
1. Human-centered computing
  1. Ubiquitous and mobile computing
    1. Empirical studies in ubiquitous and mobile computing

Recommendations

Assessing Severity of Pulmonary Obstruction from Respiration Phase-Based Wheeze-Sensing Using Mobile Sensors
CHI '20: Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems

Obstructive pulmonary diseases cause limited airflow from the lung and severely affect patients' quality of life. Wheeze is one of the most prominent symptoms for them. High requirements imposed by traditional diagnosis methods make regular monitoring of ...
Assessing respiratory mechanics using pressure reconstruction method in mechanically ventilated spontaneous breathing patient

Mechanically ventilated spontaneous breathing (SB) patient produced entrained wave which is known as M-wave, even though in fully sedated condition, that create problem in estimating the true respiratory mechanics.Airway pressure reconstruction method ...
Real-time robust estimation of breathing rate from PPG using commercial-grade smart devices: demo abstract
SenSys '20: Proceedings of the 18th Conference on Embedded Networked Sensor Systems

In this work, we present a solution for an accurate and real-time monitoring of breathing rate (BR) from photoplethysmogram (PPG) signal on both smartphone and smartwatch. Respiration induces multiple modulations in a PPG signal which are difficult to ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

ICMI '20: Proceedings of the 2020 International Conference on Multimodal Interaction

October 2020

920 pages

ISBN:9781450375818

DOI:10.1145/3382507

General Chairs:
Khiet Truong
University of Twente, the Netherlands
,
Dirk Heylen
University of Twente, the Netherlands
,
Mary Czerwinski
Microsoft Research, USA
,
Program Chairs:
Nadia Berthouze
University College London, United Kingdom
,
Mohamed Chetouani
Sorbonne University, France
,
Mikio Nakano
C4A Research Institute, Japan

Copyright © 2020 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGCHI: ACM Special Interest Group on Computer-Human Interaction

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 22 October 2020

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

ICMI '20

Sponsor:

SIGCHI

ICMI '20: INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION

October 25 - 29, 2020

Virtual Event, Netherlands

Acceptance Rates

Overall Acceptance Rate 453 of 1,080 submissions, 42%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

10
Total Citations
View Citations
407
Total Downloads

Downloads (Last 12 months)72
Downloads (Last 6 weeks)5

Reflects downloads up to 26 Sep 2024

Other Metrics

View Author Metrics

Citations

Cited By

Tripathi GSabherwal MSingh P(2024)“I know I have this till my Last Breath”: Unmasking the Gaps in Chronic Obstructive Pulmonary Disease (COPD) Care in IndiaProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642504(1-16)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3642504
Bhalla SLiaqat SWu RGershon Ade Lara EMariakakis A(2023)PulmoListenerProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36108897:3(1-24)Online publication date: 27-Sep-2023
https://dl.acm.org/doi/10.1145/3610889
Adhikary RLodhavia DFrancis CPatil RSrivastava TKhanna PBatra NBreda JPeplinski JPatel S(2023)SpiroMask: Measuring Lung Function Using Consumer-Grade MasksACM Transactions on Computing for Healthcare10.1145/35701674:1(1-34)Online publication date: 27-Feb-2023
https://dl.acm.org/doi/10.1145/3570167
Ahmed TRahman MNemati EAhmed MKuang JGao A(2023)Remote Breathing Rate Tracking in Stationary Position Using the Motion and Acoustic Sensors of EarablesProceedings of the 2023 CHI Conference on Human Factors in Computing Systems10.1145/3544548.3581265(1-22)Online publication date: 19-Apr-2023
https://dl.acm.org/doi/10.1145/3544548.3581265
Rizos GCalvo RSchuller B(2023)Positive-Pair Redundancy Reduction Regularisation for Speech-Based Asthma Diagnosis PredictionICASSP 2023 - 2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)10.1109/ICASSP49357.2023.10097087(1-5)Online publication date: 4-Jun-2023
https://doi.org/10.1109/ICASSP49357.2023.10097087
Patel NGupta RAli A(2022)Towards Continuous Respiration Monitoring Using MaskProceedings of the 5th Joint International Conference on Data Science & Management of Data (9th ACM IKDD CODS and 27th COMAD)10.1145/3493700.3493753(304-305)Online publication date: 8-Jan-2022
https://dl.acm.org/doi/10.1145/3493700.3493753
Islam BRahman MAhmed TAhmed MHasan MNathan VVatanparvar KNemati EKuang JGao J(2021)BreathTrackProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/34781235:3(1-22)Online publication date: 14-Sep-2021
https://dl.acm.org/doi/10.1145/3478123
Palanque PNavarre D(2021)Dependability and Safety: Two Clouds in the Blue Sky of Multimodal InteractionProceedings of the 2021 International Conference on Multimodal Interaction10.1145/3462244.3479881(781-787)Online publication date: 18-Oct-2021
https://dl.acm.org/doi/10.1145/3462244.3479881
Ahmed MZhu LRahman MAhmed TKuang JGao A(2021)Device Invariant Deep Neural Networks for Pulmonary Audio Event Detection Across Mobile and Wearable Devices2021 43rd Annual International Conference of the IEEE Engineering in Medicine & Biology Society (EMBC)10.1109/EMBC46164.2021.9629853(5631-5637)Online publication date: 1-Nov-2021
https://doi.org/10.1109/EMBC46164.2021.9629853
Alqudaihi KAslam NKhan IAlmuhaideb AAlsunaidi SIbrahim NAlhaidari FShaikh FAlsenbel YAlalharith DAlharthi HAlghamdi WAlshahrani M(2021)Cough Sound Detection and Diagnosis Using Artificial Intelligence Techniques: Challenges and OpportunitiesIEEE Access10.1109/ACCESS.2021.30975599(102327-102344)Online publication date: 2021
https://doi.org/10.1109/ACCESS.2021.3097559

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents