Exploring Hyperspectral Histopathology Image Segmentation from a Deformable Perspective
Authors: 
Xingran Xie, Ting Jin, Boxiang Yun, Qingli Li, Yan WangAuthors Info & Claims
Xingran Xie, Ting Jin, Boxiang Yun, Qingli Li, Yan WangAuthors Info & ClaimsPages 242 - 251
Abstract
Hyperspectral images (HSIs) offer great potential for computational pathology. However, limited by the spectral redundancy and the lack of spectral prior in popular 2D networks, previous HSI based techniques do not perform well. To address these problems, we propose to segment HSIs from a deformable perspective, which processes different spectral bands independently and fuses spatiospectral features of interest via deformable attention mechanisms. In addition, we propose Deformable Self-Supervised Spectral Regression (DF-S3R), which introduces two self-supervised pre-text tasks based on the low rank prior of HSIs enabling the network learning with spectrum-related features. During pre-training, DF-S3R learns both spectral structures and spatial morphology, and the jointly pre-trained architectures help alleviate the transfer risk to downstream fine-tuning. Compared to previous works, experiments show that our deformable architecture and pre-training method perform much better than other competitive methods on pathological semantic segmentation tasks, and the visualizations indicate that our method can trace the critical spectral characteristics from subtle spectral disparities. Code will be released at https://github.com/Ayakax/DFS3R.
References
[1]
V. Backman, M. B. Wallace, L. T. Perelman, and et al. 2000. Detection of preinvasive cancer cells. Nature, Vol. 406 (2000), 35--36.
[2]
Hangbo Bao, Li Dong, and Furu Wei. 2022. BEiT: BERT Pre-Training of Image Transformers. In ICLR.
[3]
Marcel Bengs, Nils Gessert, Wiebke Laffers, Dennis Eggert, Stephan Westermann, Nina A. Müller, Andreas O. H. Gerstner, Christian Betz, and Alexander Schlaefer. 2020. Spectral-spatial Recurrent-Convolutional Networks for In-Vivo Hyperspectral Tumor Type Classification. In MICCAI.
[4]
Hu Cao, Yueyue Wang, Joy Chen, Dongsheng Jiang, Xiaopeng Zhang, Qi Tian, and Manning Wang. 2022. Swin-Unet: Unet-Like Pure Transformer for Medical Image Segmentation. In ECCV Workshops.
[5]
Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. 2020. End-to-End Object Detection with Transformers. In ECCV, Vol. 12346. 213--229.
[6]
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey E. Hinton. 2020. A Simple Framework for Contrastive Learning of Visual Representations. In ICML.
[7]
Xinlei Chen and Kaiming He. 2021. Exploring Simple Siamese Representation Learning. In CVPR.
[8]
Ö zgü n cC icc ek, Ahmed Abdulkadir, Soeren S. Lienkamp, Thomas Brox, and Olaf Ronneberger. 2016. 3D U-Net: Learning Dense Volumetric Segmentation from Sparse Annotation. In MICCAI.
[9]
Jean-Bastien Grill, Florian Strub, Florent Altché, and et al. 2020. Bootstrap Your Own Latent - A New Approach to Self-Supervised Learning. In NeurIPS.
[10]
Ali Hatamizadeh, Vishwesh Nath, Yucheng Tang, Dong Yang, Holger R. Roth, and Daguang Xu. 2021. Swin UNETR: Swin Transformers for Semantic Segmentation of Brain Tumors in MRI Images. In MICCAI Workshops.
[11]
Ali Hatamizadeh, Yucheng Tang, Vishwesh Nath, Dong Yang, Andriy Myronenko, Bennett A. Landman, Holger R. Roth, and Daguang Xu. 2022. UNETR: Transformers for 3D Medical Image Segmentation. In WACV.
[12]
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross B. Girshick. 2022. Masked Autoencoders Are Scalable Vision Learners. In CVPR.
[13]
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross B. Girshick. 2020. Momentum Contrast for Unsupervised Visual Representation Learning. In CVPR.
[14]
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep Residual Learning for Image Recognition. In CVPR.
[15]
Danfeng Hong, Zhu Han, Jing Yao, Lianru Gao, Bing Zhang, Antonio Plaza, and Jocelyn Chanussot. 2022. SpectralFormer: Rethinking Hyperspectral Image Classification With Transformers. IEEE Trans. Geosci. Remote. Sens., Vol. 60 (2022), 1--15.
[16]
Fabian Isensee, Paul F. Jaeger, Simon A. A. Kohl, Jens Petersen, and Klaus H. Maier-Hein. 2021. nnU-Net: a self-configuring method for deep learning-based biomedical image segmentation. Nature Methods (2021).
[17]
S. Karim, Akeel Q., and et al. 2023. Hyperspectral Imaging: A Review and Trends towards Medical Imaging. Current Medical Imaging, Vol. 19, 5 (2023), 417--427.
[18]
Bing Liu, Anzhu Yu, Xuchu Yu, Ruirui Wang, Kuiliang Gao, and Wenyue Guo. 2021b. Deep Multiview Learning for Hyperspectral Image Classification. IEEE Trans. Geosci. Remote. Sens., Vol. 59, 9 (2021), 7758--7772.
[19]
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. 2021a. Swin Transformer: Hierarchical Vision Transformer using Shifted Windows. In ICCV.
[20]
Ilya Loshchilov and Frank Hutter. 2017. Fixing Weight Decay Regularization in Adam. CoRR, Vol. abs/1711.05101 (2017).
[21]
Zhongtian Ma, Zhiguo Jiang, and Haopeng Zhang. 2022. Hyperspectral Image Classification Using Feature Fusion Hypergraph Convolution Neural Network. IEEE Trans. Geosci. Remote. Sens., Vol. 60 (2022), 1--14.
[22]
Fausto Milletari, Nassir Navab, and Seyed-Ahmad Ahmadi. 2016. V-Net: Fully Convolutional Neural Networks for Volumetric Medical Image Segmentation. In 3DV.
[23]
Lichao Mou, Pedram Ghamisi, and Xiao Xiang Zhu. 2018. Unsupervised Spectral-Spatial Feature Learning via Deep Residual Conv-Deconv Network for Hyperspectral Image Classification. IEEE Trans. Geosci. Remote. Sens., Vol. 56, 1 (2018), 391--406.
[24]
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. 2015. U-Net: Convolutional Networks for Biomedical Image Segmentation. In MICCAI.
[25]
Olga Russakovsky, Jia Deng, Hao Su, and et al. 2015. ImageNet Large Scale Visual Recognition Challenge. Int. J. Comput. Vis., Vol. 115, 3 (2015), 211--252.
[26]
Massimo Salvi, U. Rajendra Acharya, Filippo Molinari, and Kristen M. Meiburger. 2021. The impact of pre- and post-image processing techniques on deep learning frameworks: A comprehensive review for digital pathology image analysis. Comput. Biol. Medicine, Vol. 128 (2021), 104129.
[27]
Haoqing Wang, Yehui Tang, Kai Han, Jianyuan Guo, Zhi-Hong Deng, and Yunhe Wang. 2023. Masked Image Modeling with Local Multi-Scale Reconstruction. In CVPR.
[28]
Qian Wang, Li Sun, Yan Wang, Mei Zhou, Menghan Hu, Jiangang Chen, Ying Wen, and Qingli Li. 2021. Identification of Melanoma From Hyperspectral Pathology Image Using 3D Convolutional Networks. IEEE Trans. Medical Imaging, Vol. 40, 1 (2021), 218--227.
[29]
Yizhou Wang, Shixiang Tang, Feng Zhu, and et al. 2022. Revisiting the Transferability of Supervised Pretraining: an MLP Perspective. In CVPR.
[30]
Enze Xie, Wenhai Wang, Zhiding Yu, Animashree Anandkumar, Jose M. Alvarez, and Ping Luo. 2021. SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformers. In NeurIPS.
[31]
Xingran Xie, Yan Wang, and Qingli Li. 2022a. S3R: Self-supervised Spectral Regression for Hyperspectral Histopathology Image Classification. In MICCAI.
[32]
Zhenda Xie, Zheng Zhang, Yue Cao, and et al. 2022b. SimMIM: a Simple Framework for Masked Image Modeling. In CVPR.
[33]
Zhenda Xie, Zheng Zhang, Yue Cao, Yutong Lin, Jianmin Bao, Zhuliang Yao, Qi Dai, and Han Hu. 2022c. SimMIM: A Simple Framework for Masked Image Modeling. In CVPR.
[34]
Zhang Ying, Wang Yan, Zhang Benyan, and Li Qingli. 2022. A hyperspectral dataset of precancerous lesions in gastric cancer and benchmarks for pathological diagnosis. Journal of Biophotonics (2022), e202200163.
[35]
Boxiang Yun, Yan Wang, Jieneng Chen, Huiyu Wang, Wei Shen, and Qingli Li. 2021. SpecTr: Spectral Transformer for Hyperspectral Pathology Image Segmentation. CoRR, Vol. abs/2103.03604 (2021).
[36]
Jure Zbontar, Li Jing, Ishan Misra, Yann LeCun, and Sté phane Deny. 2021. Barlow Twins: Self-Supervised Learning via Redundancy Reduction. In ICML.
[37]
Qing Zhang, Qingli Li, Guanzhen Yu, Li Sun, Mei Zhou, and Junhao Chu. 2019. A Multidimensional Choledoch Database and Benchmarks for Cholangiocarcinoma Diagnosis. IEEE Access, Vol. 7 (2019), 149414--149421.
[38]
Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang, and Jifeng Dai. 2021. Deformable DETR: Deformable Transformers for End-to-End Object Detection. In ICLR.
[39]
L. Zhuang and M. K. Ng. 2021. FastHyMix: Fast and Parameter-Free Hyperspectral Image Mixed Noise Removal. IEEE Transactions on Neural Networks and Learning Systems, Vol. PP, 99 (2021), 1--15.
Index Terms
- Exploring Hyperspectral Histopathology Image Segmentation from a Deformable Perspective
Recommendations
SR: Self-supervised Spectral Regression for Hyperspectral Histopathology Image Classification
Medical Image Computing and Computer Assisted Intervention – MICCAI 2022AbstractBenefited from the rich and detailed spectral information in hyperspectral images (HSI), HSI offers great potential for a wide variety of medical applications such as computational pathology. But, the lack of adequate annotated data and the high ...
Sparse coding with morphology segmentation and multi-label fusion for hyperspectral image super-resolution
AbstractHyperspectral image (HSI) super-solution to reconstruct high spatial resolution HSIs has attracted increasing interest in recent years. In this paper, we propose a HSI super-resolution framework based on sparse coding with morphology ...
Highlights- Morphology segmentation is used to get unfixed-size 2-D patches of each MSI band.
Hyperspectral image super-resolution through clustering-based sparse representation
AbstractPromoting the spatial resolution of hyperspectral sensors is expected to improve computer vision tasks. However, due to the physical limitations of imaging sensors, the hyperspectral image is often of low spatial resolution. In this paper, we ...
Comments
Information & Contributors
Information
Published In
October 2023
9913 pages
ISBN:9798400701085
DOI:10.1145/3581783
- General Chairs:
- Abdulmotaleb El Saddik,
- Tao Mei,
- Rita Cucchiara,
- Program Chairs:
- Marco Bertini,
- Diana Patricia Tobon Vallejo,
- Pradeep K. Atrey,
- M. Shamim Hossain
Copyright © 2023 ACM.
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].
Sponsors
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 27 October 2023
Check for updates
Author Tags
Qualifiers
- Research-article
Funding Sources
- Shanghai Natural Science Foundation
- National Natural Science Foundation of China
- Science and Technology Commission of Shanghai Municipality
Conference
MM '23
Sponsor:
MM '23: The 31st ACM International Conference on Multimedia
October 29 - November 3, 2023
Ottawa ON, Canada
Acceptance Rates
Overall Acceptance Rate 995 of 4,171 submissions, 24%
Upcoming Conference
MM '24
- Sponsor:
- sigmm
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 155Total Downloads
- Downloads (Last 12 months)155
- Downloads (Last 6 weeks)10
Reflects downloads up to 30 Aug 2024
Other Metrics
Citations
View Options
Get Access
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in