research-article

Privacy-preserving Time-series Medical Images Analysis Using a Hybrid Deep Learning Framework

Authors:

Alireza Jolfaei,

Xi ZhengAuthors Info & Claims

ACM Transactions on Internet Technology (TOIT), Volume 21, Issue 3

Article No.: 57, Pages 1 - 21

https://doi.org/10.1145/3383779

Published: 16 June 2021 Publication History

Abstract

Time-series medical images are an important type of medical data that contain rich temporal and spatial information. As a state-of-the-art, computer-aided diagnosis (CAD) algorithms are usually used on these image sequences to improve analysis accuracy. However, such CAD algorithms are often required to upload medical images to honest-but-curious servers, which introduces severe privacy concerns. To preserve privacy, the existing CAD algorithms support analysis on each encrypted image but not on the whole encrypted image sequences, which leads to the loss of important temporal information among frames. To meet this challenge, a convolutional-LSTM network, named HE-CLSTM, is proposed for analyzing time-series medical images encrypted by a fully homomorphic encryption mechanism. Specifically, several convolutional blocks are constructed to extract discriminative spatial features, and LSTM-based sequence analysis layers (HE-LSTM) are leveraged to encode temporal information from the encrypted image sequences. Moreover, a weighted unit and a sequence voting layer are designed to incorporate both spatial and temporal features with different weights to improve performance while reducing the missed diagnosis rate. The experimental results on two challenging benchmarks (a Cervigram dataset and the BreaKHis public dataset) provide strong evidence that our framework can encode visual representations and sequential dynamics from encrypted medical image sequences; our method achieved AUCs above 0.94 both on the Cervigram and BreaKHis datasets, constituting a significant margin of statistical improvement compared with several competing methods.

References

[1]

Ahmed M. Alaa, Kyeong H. Moon, William Hsu, and Mihaela Van Der Schaar. 2016. ConfidentCare: A clinical decision support system for personalized breast cancer screening. IEEE Trans. Multimedia 18, 10 (2016), 1942–1955.

[2]

Shekoofeh Azizi, Sharareh Bayat, Pingkun Yan, Amir Tahmasebi, Jin Tae Kwak, Sheng Xu, Baris Turkbey, Peter Choyke, Peter Pinto, and Bradford Wood. 2018. Deep recurrent neural networks for prostate cancer detection: Analysis of temporal. IEEE Trans. Med. Imag. 37, 12 (2018), 2695–2703.

[3]

Ankan Kumar Bhunia, Aishik Konwer, Ayan Kumar Bhunia, Abir Bhowmick, Partha P. Roy, and Umapada Pal. 2019. Script identification in natural scene image and video frames using an attention based Convolutional-LSTM network. Pattern Recogn. 85 (2019), 172–184.

[4]

Sylvia F. Boj, Chang Il Hwang, Lindsey A. Baker, Iok In Christine Chio, Dannielle D. Engle, Vincenzo Corbo, Myrthe Jager, Mariano Ponz-Sarvise, Hervé Tiriac, Mona S. Spector, Ana Gracanin, Tobiloba Oni, Kenneth H. Yu, Ruben Van Boxtel, Meritxell Huch, Keith D. Rivera, John P. Wilson, Michael E. Feigin, Daniel Öhlund, Abram Handly-Santana, Christine M. Ardito-Abraham, Michael Ludwig, Ela Elyada, Brinda Alagesan, Giulia Biffi, Georgi N. Yordanov, Bethany Delcuze, Brianna Creighton, Kevin Wright, Youngkyu Park, Folkert H. M. Morsink, I. Quintus Molenaar, Inne H. Borel Rinkes, Edwin Cuppen, Yuan Hao, Ying Jin, Isaac J. Nijman, Christine Iacobuzio-Donahue, Steven D. Leach, Darryl J. Pappin, Molly Hammell, David S. Klimstra, Olca Basturk, Ralph H. Hruban, George Johan Offerhaus, Robert G. J. Vries, Hans Clevers and David A. Tuveson. 2015. Organoid models of human and mouse ductal pancreatic cancer. Cell 160, 1–2 (2015), 324–338.

[5]

Joppe W. Bos, Kristin Lauter, Jake Loftus, and Michael Naehrig. 2013. Improved security for a ring-based fully homomorphic encryption scheme. In Proceedings of the IMA International Conference on Cryptography and Coding. 45–64.

Digital Library

[6]

Ümit Budak, Zafer Cömert, Zryan Najat Rashid, Abdulkadir Sengür, and Musa Çibuk. 2019. Computer-aided diagnosis system combining FCN and Bi-LSTM model for efficient breast cancer detection from histopathological images. Appl. Soft Comput. J. 85 (2019), 105765.

Digital Library

[7]

Herve Chabanne, Amaury de Wargny, Jonathan Milgram, Constance Morel, and Emmanuel Prouff. 2017. Privacy-preserving classification on deep neural network. In Proceedings of the International Association for Cryptologic Research (IACR'17), 35.

[8]

Christian Von Der Weth, Ashraf Abdul, Abhinav R. Kashyap, and Mohan S. Kankanhalli. 2019. CloseUp-community-driven live online search engine. ACM Trans. Internet Technol. 19, 3 (2019), 1–21.

Digital Library

[9]

Shuai Ding, Shikang Hu, Jinxin Pan, Xiaojian Li, Gang Li, and Xiao Liu. 2019. A homogeneous ensemble method for predicting gastric cancer based on gastroscopy reports. Expert Syst. 37, 3 (2019), e12499.

[10]

Shuai Ding, Ling Li, Zhenmin Li, Hao Wang, and Yanchun Zhang. 2019. Smart electronic gastroscope system using a cloudâedge collaborative framework. Fut. Gener. Comput. Syst. 100 (2019), 395–407.

[11]

Jeff Donahue, Lisa Anne Hendricks, Marcus Rohrbach, Subhashini Venugopalan, Sergio Guadarrama, Kate Saenko, and Trevor Darrell. 2017. Long-term recurrent convolutional networks for visual recognition and description. IEEE Trans. Pattern Anal. Mach. Intell. 39, 4 (2017), 677–691.

Digital Library

[12]

Nathan Dowlin, Ran Gilad-Bachrach, Kim Laine, Kristin Lauter, Michael Naehrig, and John Wernsing. 2016. CryptoNets: Applying neural networks to encrypted data with high throughput and accuracy. In Proceedings of the International Conference on Machine Learning. 201–210.

Digital Library

[13]

Li Duan, Dongxi Liu, Yang Zhang, Shiping Chen, Ren Ping Liu, Bo Cheng, and Junliang Chen. 2016. Secure data-centric access control for smart grid services based on publish/subscribe systems. ACM Trans. Internet Technol. 16, 4 (2016), 1–17.

Digital Library

[14]

Andre Esteva, Brett Kuprel, Roberto A. Novoa, Justin Ko, Susan M. Swetter, Helen M. Blau, and Sebastian Thrun. 2017. Dermatologist-level classification of skin cancer with deep neural networks. Nature 542, 7639 (2017), 115–118.

[15]

Junfeng Fan and Frederik Vercauteren. 2012. Somewhat practical fully homomorphic encryption. In Proceedings of the International Association for Cryptologic Research (IACR'12). 144.

[16]

Jeremy Farrar. 2014. Medical data: Widen use in research. Nature 506, 7486 (2014), 34.

[17]

Craig Gentry. 2009. A Fully Homomorphic Encryption Scheme. Stanford University (2009).

Digital Library

[18]

Francisco Javier González-Serrano, Ángel Navia-Vázquez, and Adrián Amor-Martín. 2017. Training support vector machines with privacy-protected data. Pattern Recogn. 72 (2017), 93–107.

Digital Library

[19]

Varun Gulshan, Lily Peng, Marc Coram, Martin C. Stumpe, Derek Wu, Arunachalam Narayanaswamy, Subhashini Venugopalan, Kasumi Widner, Tom Madams, Jorge Cuadros, Ramasamy Kim, Rajiv Raman, Philip C. Nelson, Jessica L. Mega, and Dale R. Webster. 2016. Development and validation of a deep learning algorithm for detection of diabetic retinopathy in retinal fundus photographs. J. Am. Med. Assoc. 316, 22 (2016), 2402–2410.

[20]

Zhongyun Hua, Shuang Yi, and Yicong Zhou. 2018. Medical image encryption using high-speed scrambling and pixel adaptive diffusion. Signal Process. 144 (2018), 134–144.

Digital Library

[21]

Multi-view Inference, Yi Zhou, Student Member, Li Liu, Ling Shao, and Senior Member. 2018. Vehicle re-identification by deep hidden multi-view inference. IEEE Trans. Image Process. 27, 7 (2018), 3275–3287.

[22]

Shuiwang Ji, Wei Xu, Ming Yang, and Kai Yu. 2013. 3D convolutional neural networks for human action recognition. IEEE Trans. Pattern Anal. Mach. Intell. 35, 1 (2013), 221–231.

Digital Library

[23]

Jia Deng, Wei Dong, R. Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. 2009. ImageNet: A large-scale hierarchical image database. In Proceedings of the Conference on Computer Vision and Pattern Recognition (CVPR’09). 248–255.

[24]

Yueming Jin, Qi Dou, Hao Chen, Lequan Yu, Jing Qin, Chi Wing Fu, and Pheng Ann Heng. 2017. SV-RCNet: Workflow recognition from surgical videos using recurrent convolutional network. IEEE Trans. Med. Imag. 37, 5 (2017), 1114–1126.

[25]

Daniel S. Kermany, Michael Goldbaum, Wenjia Cai, Carolina C. S. Valentim, Huiying Liang, Sally L. Baxter, Alex McKeown, Ge Yang, Xiaokang Wu, Fangbing Yan, Justin Dong, Made K. Prasadha, Jacqueline Pei, Magdalena Ting, Jie Zhu, Christina Li, Sierra Hewett, Jason Dong, Ian Ziyar, Alexander Shi, Runze Zhang, Lianghong Zheng, Rui Hou, William Shi, Xin Fu, Yaou Duan, Viet A. N. Huu, Cindy Wen, Edward D. Zhang, Charlotte L. Zhang, Oulan Li, Xiaobo Wang, Michael A. Singer, Xiaodong Sun, Jie Xu, Ali Tafreshi, M. Anthony Lewis, Huimin Xia and Kang Zhang. 2018. Identifying medical diagnoses and treatable diseases by image-based deep learning. Cell 172, 5 (2018), 1122–1131.

[26]

Neal Koblitz. 1987. Elliptic curve cryptosystems. Math. Comp. 48, 177 (1987), 203–203.

Digital Library

[27]

Sung-hyun Lee, Joon Hyun Song, and Il Kon Kim. 2016. CDA generation and integration for health information exchange based on cloud computing system. IEEE Trans. Serv. Comput. 9, 2 (2016), 241–249.

Digital Library

[28]

X.-B. Li and Jialun Qin. 2017. Anonymizing and sharing medical text records. Inf. Syst. Res. 28, 2 (2017), 332–352.

Digital Library

[29]

Yang Liu, Shanyu Tang, Ran Liu, Liping Zhang, and Zhao Ma. 2018. Secure and robust digital image watermarking scheme using logistic and RSA encryption. Expert Syst. Appl. 97 (2018), 95–105.

[30]

Chao Ma, Yulan Guo, Jungang Yang, and Wei An. 2019. Learning multi-view representation with LSTM for 3-D shape recognition and retrieval. IEEE Trans. Multimedia 21, 5 (2019), 1169–1182.

Digital Library

[31]

Ahmet Can Mert, Erdinc Ozturk, and Erkay Savas. 2019. Design and implementation of encryption/decryption architectures for bfv homomorphic encryption scheme. IEEE Trans. VLSI Syst. 28, 2 (2019), 353--362.

[32]

Biotecnologia Microbiana, De Hidalgo, Marios Anthimopoilos, Srergios Christodoulidis, Lukas Ebner, Andreas Christe, and Stavroula Mougiakakou. 2016. Lung pattern classification for interstitial lung diseases using a deep convolutional neural network. IEEE Trans. Med. Imag. 35, 5 (2016), 1207–1216.

[33]

Declan O’Loughlin, Barbara L. Oliveira, Adam Santorelli, Emily Porter, Martin Glavin, Edward Jones, Milica Popovic, and Martin O’Halloran. 2019. Sensitivity and specificity estimation using patient-specific microwave imaging in diverse experimental breast phantoms. IEEE Trans. Med. Imag. 38, 1 (2019), 303–311.

[34]

Jane S. Paulsen, Jeff D. Long, Christopher A. Ross, Deborah L. Harrington, Cheryl J. Erwin, Janet K. Williams, Holly James Westervelt, and Hans J. Johnson. 2014. Prediction of manifest Huntington’s disease with clinical and imaging measures: A prospective observational study. Lancet Neurol. 13, 12 (2014), 1193–1201.

[35]

W. Nicholson Price and Glenn Cohen. 2019. Privacy in the age of medical big data. Nat. Med. 25, 1 (2019), 37–43.

[36]

Ahmad Qaisar Ahmad Al Badawi, Yuriy Polyakov, Khin Mi Mi Aung, Bharadwaj Veeravalli, and Kurt Rohloff. 2019. Implementation and performance evaluation of RNS variants of the BFV homomorphic encryption scheme. IEEE Trans. Emerg. Topics Comput. (2019).

[37]

Jean Louis Raisaro, Gwangbae Choi, Sylvain Pradervand, Raphael Colsenet, Nathalie Jacquemont, Nicolas Rosat, Vincent Mooser, and Jean Pierre Hubaux. 2018. Protecting privacy and security of genomic data in i2b2 with homomorphic encryption and differential privacy. IEEE/ACM Trans. Comput. Biol. Bioinf. 15, 5 (2018), 1413–1426.

Digital Library

[38]

Monjoy Saha and Chandan Chakraborty. 2018. Her2Net: A deep framework for semantic segmentation and classification of cell membranes and nuclei in breast cancer evaluation. IEEE Trans. Image Process. 27, 5 (2018), 2189–2200.

[39]

V. M. Silva-García, R. Flores-Carapia, C. Rentería-Márquez, B. Luna-Benoso, and M. Aldape-Pérez. 2018. Substitution box generation using Chaos: An image encryption application. Appl. Math. Comput. 332 (2018), 123–135.

[40]

Wenjuan Tang, Ju Ren, and Yaoxue Zhang. 2019. Enabling trusted and privacy-preserving healthcare services in social media health networks. IEEE Trans. Multimedia 21, 3 (2019), 579–590.

[41]

Hao Wang, Shuai Ding, Desheng Wu, Youtao Zhang, and Shanlin Yang. 2019. Smart connected electronic gastroscope system for gastric cancer screening using multi-column convolutional neural networks. Int. J. Prod. Res. 57, 21 (2019), 6795--6806.

[42]

Hongyu Wang, Jun Feng, Zizhao Zhang, Hai Su, Lei Cui, Hua He, and Li Liu. 2018. Breast mass classification via deeply integrating the contextual information from multi-view data. Pattern Recogn. 80 (2018), 42–52.

[43]

Chenchu Xu, Lei Xu, Zhifan Gao, Shen Zhao, Heye Zhang, and Yanping Zhang. 2018. Direct delineation of myocardial infarction without contrast agents using a joint motion feature learning architecture. Med. Image Anal. 50 (2018), 82–94.

[44]

Tao Xu, Han Zhang, Cheng Xin, Edward Kim, L. Rodney Long, Zhiyun Xue, Sameer Antani, and Xiaolei Huang. 2017. Multi-feature based benchmark for cervical dysplasia classification evaluation. Pattern Recogn. 63 (2017), 468–475.

Digital Library

[45]

Hao Yang, Chunfeng Yuan, Bing Li, Yang Du, Junliang Xing, Weiming Hu, and Stephen J. Maybank. 2019. Asymmetric 3D convolutional neural networks for action recognition. Pattern Recogn. 85 (2019), 1–12.

[46]

Aexander Hauptmann Yi Zhu, Zhenzhong Lan, Shawn Newsam. 2018. Hidden two-stream convolutional networks for action recognition. In Proceedings of the Asian Conference on Computer Vision. Springer International Publishing, 363–378.

[47]

Qingchen Zhang, Laurence T. Yang, and Zhikui Chen. 2016. Privacy preserving deep computation model on cloud for big data feature learning. IEEE Trans. Comput. 65, 5 (2016), 1351–1362.

Digital Library

[48]

Zongjian Zhang, Qiang Wu, Yang Wang, and Fang Chen. 2018. High-quality image captioning with fine-grained and semantic-guided visual attention. IEEE Trans. Multimedia 21, 7 (2018), 1681–1693.

[49]

Chong zhi Gao, Qiong Cheng, Pei He, Willy Susilo, and Jin Li. 2018. Privacy-preserving Naive Bayes classifiers secure against the substitution-then-comparison attack. Inf. Sci. 444 (2018), 72–88.

[50]

Yi Zhuang, Nan Jiang, Qing Li, Lei Chen, and Chunhua Ju. 2015. Progressive batch medical image retrieval processing in mobile wireless networks. ACM Trans. Internet Technol. 15, 3 (2015), 1–27.

Digital Library

Cited By

Guo KTan XChen LZhu XZhang JZhou X(2024)LSDN Empowers the Metaverse Communication to Achieve High-Resolution Visuals with Lower BandwidthACM Transactions on Internet Technology10.1145/3672075Online publication date: 24-Jun-2024
https://dl.acm.org/doi/10.1145/3672075
Nguyen CSaputra YHoang DNguyen DNguyen VXiao YDutkiewicz E(2024)Encrypted Data Caching and Learning Framework for Robust Federated Learning-Based Mobile Edge ComputingIEEE/ACM Transactions on Networking10.1109/TNET.2024.336581532:3(2705-2720)Online publication date: Jun-2024
https://doi.org/10.1109/TNET.2024.3365815
Mujumdar AKar NBiswas PMathur A(2024)Qualitative Analysis of Homomorphic Encryption in Medical Field2024 IEEE International Conference on Information Technology, Electronics and Intelligent Communication Systems (ICITEICS)10.1109/ICITEICS61368.2024.10625392(1-6)Online publication date: 28-Jun-2024
https://doi.org/10.1109/ICITEICS61368.2024.10625392
Show More Cited By

Index Terms

Privacy-preserving Time-series Medical Images Analysis Using a Hybrid Deep Learning Framework

Recommendations

Privacy preserving medical data publishing
Statistical and deep learning models for reference evapotranspiration time series forecasting: A comparison of accuracy, complexity, and data efficiency
Graphical abstract

Display Omitted
Highlights
- Ten statistical, machine learning, and deep learning models forecasted ETo.
- Data from 107 California weather stations were categorized based on their length.
- Most deep learning models suffer from low data efficiency.
- Simpler ...
Abstract
Reference evapotranspiration (ETo) is an essential variable in agricultural water resources management and irrigation scheduling. An accurate and reliable forecast of ETo facilitates effective decision-making in agriculture. Although numerous ...
Privacy Preserving Index for Encrypted Electronic Medical Records

With the development of electronic systems, privacy has become an important security issue in real-life. In medical systems, privacy of patients' electronic medical records (EMRs) must be fully protected. However, to combine the efficiency and privacy, ...

Comments

Information & Contributors

Information

Published In

cover image ACM Transactions on Internet Technology

ACM Transactions on Internet Technology Volume 21, Issue 3

August 2021

522 pages

ISSN:1533-5399

EISSN:1557-6051

DOI:10.1145/3468071

Editor:
Ling Liu
Georgia Institute of Technology, USA

Issue’s Table of Contents

Copyright © 2021 Association for Computing Machinery.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 16 June 2021

Online AM: 07 May 2020

Accepted: 01 February 2020

Revised: 01 January 2020

Received: 01 November 2019

Published in TOIT Volume 21, Issue 3

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Refereed

Funding Sources

National Natural Science Foundation of China
Anhui Provincial Science and Technology Major Project
Fundamental Research Funds for the Central Universities

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

22
Total Citations
View Citations
502
Total Downloads

Downloads (Last 12 months)124
Downloads (Last 6 weeks)8

Reflects downloads up to 02 Sep 2024

Other Metrics

View Author Metrics

Citations

Cited By

Guo KTan XChen LZhu XZhang JZhou X(2024)LSDN Empowers the Metaverse Communication to Achieve High-Resolution Visuals with Lower BandwidthACM Transactions on Internet Technology10.1145/3672075Online publication date: 24-Jun-2024
https://dl.acm.org/doi/10.1145/3672075
Nguyen CSaputra YHoang DNguyen DNguyen VXiao YDutkiewicz E(2024)Encrypted Data Caching and Learning Framework for Robust Federated Learning-Based Mobile Edge ComputingIEEE/ACM Transactions on Networking10.1109/TNET.2024.336581532:3(2705-2720)Online publication date: Jun-2024
https://doi.org/10.1109/TNET.2024.3365815
Mujumdar AKar NBiswas PMathur A(2024)Qualitative Analysis of Homomorphic Encryption in Medical Field2024 IEEE International Conference on Information Technology, Electronics and Intelligent Communication Systems (ICITEICS)10.1109/ICITEICS61368.2024.10625392(1-6)Online publication date: 28-Jun-2024
https://doi.org/10.1109/ICITEICS61368.2024.10625392
Tran ALuong THuynh V(2024)A comprehensive survey and taxonomy on privacy-preserving deep learningNeurocomputing10.1016/j.neucom.2024.127345576:COnline publication date: 25-Jun-2024
https://dl.acm.org/doi/10.1016/j.neucom.2024.127345
Choudhury CGoel TTanveer M(2024)A coupled-GAN architecture to fuse MRI and PET image features for multi-stage classification of Alzheimer’s diseaseInformation Fusion10.1016/j.inffus.2024.102415109(102415)Online publication date: Sep-2024
https://doi.org/10.1016/j.inffus.2024.102415
Zhang YLi P(2023)A Study of Privacy-Preserving Neural Network Prediction Based on Replicated Secret SharingMathematics10.3390/math1104104811:4(1048)Online publication date: 19-Feb-2023
https://doi.org/10.3390/math11041048
Sim SKim DBae H(2023)Correlation Recurrent Units: A Novel Neural Architecture for Improving the Predictive Performance of Time-Series DataIEEE Transactions on Pattern Analysis and Machine Intelligence10.1109/TPAMI.2023.331955745:12(14266-14283)Online publication date: 26-Sep-2023
https://dl.acm.org/doi/10.1109/TPAMI.2023.3319557
Sun DZhao JChen BWu HWu J(2023)Cross-Level Dependability Assessment With a Distributed Split Mechanism for Wireless Communication SystemsIEEE Transactions on Network Science and Engineering10.1109/TNSE.2022.319453510:5(2832-2842)Online publication date: 1-Sep-2023
https://doi.org/10.1109/TNSE.2022.3194535
Saputra YNguyen DHoang DPham QDutkiewicz EHwang W(2023)Federated Learning Framework With Straggling Mitigation and Privacy-Awareness for AI-Based Mobile Application ServicesIEEE Transactions on Mobile Computing10.1109/TMC.2022.317894922:9(5296-5312)Online publication date: 1-Sep-2023
https://dl.acm.org/doi/10.1109/TMC.2022.3178949
Khalid NQayyum ABilal MAl-Fuqaha AQadir J(2023)Privacy-preserving artificial intelligence in healthcareComputers in Biology and Medicine10.1016/j.compbiomed.2023.106848158:COnline publication date: 1-May-2023
https://dl.acm.org/doi/10.1016/j.compbiomed.2023.106848
Show More Cited By

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

HTML Format

View this article in HTML Format.

Media

Figures

Other

Tables

View Issue’s Table of Contents