research-article

Automatic Disease Detection and Report Generation for Gastrointestinal Tract Examination

Authors:

Philipp Harzig,

Moritz Einfalt,

Rainer LienhartAuthors Info & Claims

MM '19: Proceedings of the 27th ACM International Conference on Multimedia

Pages 2573 - 2577

https://doi.org/10.1145/3343031.3356066

Published: 15 October 2019 Publication History

Get Access

Abstract

In this paper, we present a method to automatically identify diseases from videos of gastrointestinal (GI) tract examinations using a Deep Convolutional Neural Network (DCNN) that processes images from digital endoscopes. Our goal is to aid domain experts by automatically detecting abnormalities and generating a report that summarizes the main findings. We have implemented a model that uses two different DCNN architectures to generate our predictions, which are also capable of running on a mobile device. Using this architecture, we are able to predict findings on individual images. Combined with class activations maps (CAM), we can also automatically generate a textual report describing a video in detail while giving hints about the spatial location of findings and anatomical landmarks. Our work shows one way to use a multi-disease detection pipeline to also generate video reports that summarize key findings.

References

[1]

Jorge Bernal, Javier Sánchez, and Fernando Vilarino. 2012. Towards automatic polyp detection with a polyp appearance model. Pattern Recognition, Vol. 45, 9 (2012), 3166--3182.

Digital Library

Google Scholar

[2]

Dina Demner-Fushman, Marc D Kohli, Marc B Rosenman, Sonya E Shooshan, Laritza Rodriguez, Sameer Antani, George R Thoma, and Clement J McDonald. 2015. Preparing a collection of radiology examinations for distribution and retrieval. Journal of the American Medical Informatics Association, Vol. 23, 2 (2015), 304--310.

Crossref

Google Scholar

[3]

Philipp Harzig, Yan-Ying Chen, Francine Chen, and Rainer Lienhart. 2019. Addressing Data Bias Problems for Chest X-ray Image Report Generation. In Proceedings of the British Machine Vision Conference .

Google Scholar

[4]

Steven Hicks, Michael Riegler, Pia Smedsrud, Trine B. Haugen, Kristin Ranheim Randel, Konstantin Pogorlov, Håkon Stensland Kvale, Duc-Tien Dang-Nguyen, Mathias Lux, Andreas Petlund, Thomas de Lange, Peter Thelin Schmidt, and Pål Halvorsen. 2019. ACM MM BioMedia 2019 Grand Challenge Overview. In Proceedings of the ACM International Conference on Multimedia (ACM MM'19). ACM.

Digital Library

Google Scholar

[5]

Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. 2017. Densely connected convolutional networks. In Proceedings of the IEEE conference on computer vision and pattern recognition. 4700--4708.

Crossref

Google Scholar

[6]

Baoyu Jing, Pengtao Xie, and Eric Xing. 2018. On the Automatic Generation of Medical Imaging Reports. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) . Association for Computational Linguistics, 2577--2586. http://aclweb.org/anthology/P18--1240

Crossref

Google Scholar

[7]

Diederik P. Kingma and Jimmy Ba. 2015. Adam: A Method for Stochastic Optimization. In 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7--9, 2015, Conference Track Proceedings. http://arxiv.org/abs/1412.6980

Google Scholar

[8]

Jonathan Krause, Justin Johnson, Ranjay Krishna, and Li Fei-Fei. 2017. A hierarchical approach for generating descriptive image paragraphs. In Computer Vision and Pattern Recognition (CVPR), 2017 IEEE Conference on. IEEE, 3337--3345.

Crossref

Google Scholar

[9]

Konstantin Pogorelov, Kristin Ranheim Randel, Thomas de Lange, Sigrun Losada Eskeland, Carsten Griwodz, Dag Johansen, Concetto Spampinato, Mario Taschwer, Mathias Lux, Peter Thelin Schmidt, Michael Riegler, and Pål Halvorsen. 2017a. Nerthus: A Bowel Preparation Quality Video Dataset. In Proceedings of the 8th ACM on Multimedia Systems Conference. ACM, 170--174.

Digital Library

Google Scholar

[10]

Konstantin Pogorelov, Kristin Ranheim Randel, Carsten Griwodz, Sigrun Losada Eskeland, Thomas de Lange, Dag Johansen, Concetto Spampinato, Duc-Tien Dang-Nguyen, Mathias Lux, Peter Thelin Schmidt, Michael Riegler, and Pål Halvorsen. 2017b. Kvasir: A Multi-Class Image Dataset for Computer Aided Gastrointestinal Disease Detection. In Proceedings of the 8th ACM on Multimedia Systems Conference. ACM, 164--169.

Digital Library

Google Scholar

[11]

Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chieh Chen. 2018. Mobilenetv2: Inverted residuals and linear bottlenecks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 4510--4520.

Crossref

Google Scholar

[12]

Nima Tajbakhsh, Suryakanth R Gurudu, and Jianming Liang. 2015. Automated polyp detection in colonoscopy videos using shape and context information. IEEE transactions on medical imaging, Vol. 35, 2 (2015), 630--644.

Google Scholar

[13]

Bolei Zhou, Aditya Khosla, Agata Lapedriza, Aude Oliva, and Antonio Torralba. 2016. Learning deep features for discriminative localization. In Proceedings of the IEEE conference on computer vision and pattern recognition. 2921--2929.

Crossref

Google Scholar

Cited By

View all

Gao DKong MZhao YHuang JHuang ZKuang KWu FZhu Q(2024)Simulating doctors’ thinking logic for chest X-ray report generation via Transformer-based Semantic Query learningMedical Image Analysis10.1016/j.media.2023.10298291(102982)Online publication date: Jan-2024
https://doi.org/10.1016/j.media.2023.102982
Yadav PDube S(2023)Image-Text Correlation Based Remote Sensing Image Retrieval2023 IEEE 11th Region 10 Humanitarian Technology Conference (R10-HTC)10.1109/R10-HTC57504.2023.10461864(1003-1008)Online publication date: 16-Oct-2023
https://doi.org/10.1109/R10-HTC57504.2023.10461864
Ayyoubi Nezhad SKhatibi TSohrabi M(2022)Proposing Novel Data Analytics Method for Anatomical Landmark Identification from Endoscopic Video FramesJournal of Healthcare Engineering10.1155/2022/81511772022(1-14)Online publication date: 23-Feb-2022
https://doi.org/10.1155/2022/8151177
Show More Cited By

Automatic Disease Detection and Report Generation for Gastrointestinal Tract Examination
1. Computing methodologies
  1. Machine learning
    1. Machine learning approaches

Recommendations

A Holistic Multimedia System for Gastrointestinal Tract Disease Detection
MMSys'17: Proceedings of the 8th ACM on Multimedia Systems Conference

Analysis of medical videos for detection of abnormalities and diseases requires both high precision and recall, but also real-time processing for live feedback and scalability for massive screening of entire populations. Existing work on this field does ...
Upernet-Based Deep Learning Method For The Segmentation Of Gastrointestinal Tract Images
ICMIP '23: Proceedings of the 2023 8th International Conference on Multimedia and Image Processing

When giving radiation therapy to patients with gastrointestinal cancers, radiation oncologists must manually outline the locations of the stomach and intestines in order to adjust the direction of the X-ray beam. This process can increase the dose ...
Bleeding Detection in Wireless Capsule Endoscopy Based on Probabilistic Neural Network

Wireless Capsule Endoscopy (WCE), which allows clinicians to inspect the whole gastrointestinal tract (GI) noninvasively, has bloomed into one of the most efficient technologies to diagnose the bleeding in GI tract. However WCE generates large amount of ...

Comments

Information & Contributors

Information

Published In

MM '19: Proceedings of the 27th ACM International Conference on Multimedia

October 2019

2794 pages

ISBN:9781450368896

DOI:10.1145/3343031

General Chairs:
Laurent Amsaleg
CNRS-IRISA, France
,
Benoit Huet
EURECOM, France
,
Martha Larson
Radboud University and TU Delft (Netherlands)
,
Program Chairs:
Guillaume Gravier
CNRS-IRISA, France
,
Hayley Hung
Delft University of Technology Netherlands
,
Chong-Wah Ngo
City University of Hong Kong Hong Kong
,
Wei Tsang Ooi
National University of Singapore Singapore

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 15 October 2019

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

MM '19

Sponsor:

SIGMM

MM '19: The 27th ACM International Conference on Multimedia

October 21 - 25, 2019

Nice, France

Acceptance Rates

MM '19 Paper Acceptance Rate 252 of 936 submissions, 27%;

Overall Acceptance Rate 995 of 4,171 submissions, 24%

Upcoming Conference

MM '24

Sponsor:
sigmm

The 32nd ACM International Conference on Multimedia

October 28 - November 1, 2024

Melbourne , VIC , Australia

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

7
Total Citations
View Citations
326
Total Downloads

Downloads (Last 12 months)31
Downloads (Last 6 weeks)7

Reflects downloads up to 12 Aug 2024

Other Metrics

View Author Metrics

Citations

Cited By

View all

Gao DKong MZhao YHuang JHuang ZKuang KWu FZhu Q(2024)Simulating doctors’ thinking logic for chest X-ray report generation via Transformer-based Semantic Query learningMedical Image Analysis10.1016/j.media.2023.10298291(102982)Online publication date: Jan-2024
https://doi.org/10.1016/j.media.2023.102982
Yadav PDube S(2023)Image-Text Correlation Based Remote Sensing Image Retrieval2023 IEEE 11th Region 10 Humanitarian Technology Conference (R10-HTC)10.1109/R10-HTC57504.2023.10461864(1003-1008)Online publication date: 16-Oct-2023
https://doi.org/10.1109/R10-HTC57504.2023.10461864
Ayyoubi Nezhad SKhatibi TSohrabi M(2022)Proposing Novel Data Analytics Method for Anatomical Landmark Identification from Endoscopic Video FramesJournal of Healthcare Engineering10.1155/2022/81511772022(1-14)Online publication date: 23-Feb-2022
https://doi.org/10.1155/2022/8151177
Messina PPino PParra DSoto ABesa CUribe SAndía MTejos CPrieto CCapurro D(2022)A Survey on Deep Learning and Explainability for Automatic Report Generation from Medical ImagesACM Computing Surveys10.1145/352274754:10s(1-40)Online publication date: 31-Jan-2022
https://doi.org/10.1145/3522747
Tavanapong WOh JRiegler MKhaleel MMittal Bde Groen P(2022)Artificial Intelligence for Colonoscopy: Past, Present, and FutureIEEE Journal of Biomedical and Health Informatics10.1109/JBHI.2022.316009826:8(3950-3965)Online publication date: Aug-2022
https://doi.org/10.1109/JBHI.2022.3160098
Nisal Gunaratna DFernando P(2022)A Systematic Literature Review of Machine Learning based Approaches on Pathology Detection in Gastrointestinal Endoscopy2022 2nd Asian Conference on Innovation in Technology (ASIANCON)10.1109/ASIANCON55314.2022.9909267(1-5)Online publication date: 26-Aug-2022
https://doi.org/10.1109/ASIANCON55314.2022.9909267
Borgli HThambawita VSmedsrud PHicks SJha DEskeland SRandel KPogorelov KLux MNguyen DJohansen DGriwodz CStensland HGarcia-Ceja ESchmidt PHammer HRiegler MHalvorsen Pde Lange T(2020)HyperKvasir, a comprehensive multi-class image and video dataset for gastrointestinal endoscopyScientific Data10.1038/s41597-020-00622-y7:1Online publication date: 28-Aug-2020
https://doi.org/10.1038/s41597-020-00622-y

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Cited By

Recommendations

A Holistic Multimedia System for Gastrointestinal Tract Disease Detection

Upernet-Based Deep Learning Method For The Segmentation Of Gastrointestinal Tract Images

Bleeding Detection in Wireless Capsule Endoscopy Based on Probabilistic Neural Network

Comments

Published In

Sponsors

Publisher

Publication History

Permissions

Check for updates

Author Tags

Qualifiers

Conference

Acceptance Rates

Upcoming Conference

Other Metrics

Article Metrics

Other Metrics

Cited By

Login options

Full Access

PDF

eReader

Abstract

References

Cited By

Recommendations

A Holistic Multimedia System for Gastrointestinal Tract Disease Detection

Upernet-Based Deep Learning Method For The Segmentation Of Gastrointestinal Tract Images

Bleeding Detection in Wireless Capsule Endoscopy Based on Probabilistic Neural Network

Comments

Information

Published In

Sponsors

Publisher

Publication History

Permissions

Check for updates

Author Tags

Qualifiers

Conference

Acceptance Rates

Upcoming Conference

Contributors

Other Metrics

Bibliometrics

Article Metrics

Other Metrics

Citations

Cited By

Get Access

Login options

Full Access

View options

PDF

eReader

Figures

Other

Share

Share this Publication link

Share on social media

Affiliations