Interactive Spectral-Spatial Transformer for Hyperspectral Image Classification
Pages 8589 - 8601
Abstract
The abundant spectral signatures and spatial contexts are effectively utilized as the key to hyperspectral image (HSI) classification. Existing convolutional neural networks (CNNs), only focus on locally spatial context information and lack the ability to learn global spectral sequence representations, whereas the transformer performs well in learning the global dependence of sequential data. To solve this issue, inspired by the transformer, we propose an interactive global spectral and local spatial feature fusion transformer called ISSFormer. Specifically, we achieve an elegant integration of self-attention and convolution in a parallel design, i.e., the multi-head self-attention mechanism (MHSA) and the local spatial perception mechanism (LSP). ISSFormer can learn both local spatial feature representation and global spectral feature representation simultaneously. More significantly, we propose a bi-directional interaction mechanism (BIM) of features across the parallel branch to provide complementary clues. The local spatial features and the global spectral features interact through the BIM which could emphasize the local spatial details and add spatial constraints to overcome spectral variability, and can further improve classification performance. With extensive experiments on three benchmark datasets, including Indian Pines, Pavia University, and WHU-Hi-HanChuan, ISSFormer can accomplish superior classification accuracy and visualization performance.
References
[1]
Y. Zhang, Y. Wang, X. Chen, X. Jiang, and Y. Zhou, “Spectral–spatial feature extraction with dual graph autoencoder for hyperspectral image clustering,” IEEE Trans. Circuits Syst. Video Technol., vol. 32, no. 12, pp. 8500–8511, Dec. 2022.
[2]
Z. Feng, S. Tong, S. Yang, X. Zhang, and L. Jiao, “Pseudo-label-assisted subdomain adaptation for hyperspectral image classification,” IEEE Trans. Circuits Syst. Video Technol., early access, Nov. 6, 2024. 10.1109/TCSVT.2023.3330193.
[3]
M. Dalponte, H. O. Ørka, T. Gobakken, D. Gianelle, and E. Næsset, “Tree species classification in boreal forests with hyperspectral data,” IEEE Trans. Geosci. Remote Sens., vol. 51, no. 5, pp. 2632–2645, May 2013.
[4]
A. Ibrahimet al., “Atmospheric correction for hyperspectral ocean color retrieval with application to the hyperspectral imager for the coastal ocean (HICO),” Remote Sens. Environ., vol. 204, pp. 60–75, Jan. 2018.
[5]
F. Fogliniet al., “Underwater hyperspectral imaging for seafloor and benthic habitat mapping,” in Proc. IEEE Int. Workshop Metrology Sea, Learn. Measure Sea Health Parameters (MetroSea), Oct. 2018, pp. 201–205.
[6]
F. Melgani and L. Bruzzone, “Classification of hyperspectral remote sensing images with support vector machines,” IEEE Trans. Geosci. Remote Sens., vol. 42, no. 8, pp. 1778–1790, Aug. 2004.
[7]
Q. Ye, P. Huang, Z. Zhang, Y. Zheng, L. Fu, and W. Yang, “Multiview learning with robust double-sided twin SVM,” IEEE Trans. Cybern., vol. 52, no. 12, pp. 12745–12758, Dec. 2022.
[8]
Q. Yeet al., “L1-norm distance minimization-based fast robust twin support vector k-plane clustering,” IEEE Trans. Neural Netw. Learn. Syst., vol. 29, no. 9, pp. 4494–4503, Sep. 2018.
[9]
J. Fan, T. Chen, and S. Lu, “Superpixel guided deep-sparse-representation learning for hyperspectral image classification,” IEEE Trans. Circuits Syst. Video Technol., vol. 28, no. 11, pp. 3163–3173, Nov. 2018.
[10]
C. Cariou and K. Chehdi, “A new k-nearest neighbor density-based clustering method and its application to hyperspectral images,” in Proc. IEEE Int. Geosci. Remote Sens. Symp. (IGARSS), Jul. 2016, pp. 6161–6164.
[11]
A. Villa, J. A. Benediktsson, J. Chanussot, and C. Jutten, “Hyperspectral image classification with independent component discriminant analysis,” IEEE Trans. Geosci. Remote Sens., vol. 49, no. 12, pp. 4865–4876, Dec. 2011.
[12]
T. V. Bandos, L. Bruzzone, and G. Camps-Valls, “Classification of hyperspectral images with regularized linear discriminant analysis,” IEEE Trans. Geosci. Remote Sens., vol. 47, no. 3, pp. 862–873, Mar. 2009.
[13]
P. Ghamisiet al., “New frontiers in spectral–spatial hyperspectral image classification: The latest advances based on mathematical morphology, Markov random fields, segmentation, sparse representation, and deep learning,” IEEE Geosci. Remote Sens. Mag., vol. 6, no. 3, pp. 10–43, Sep. 2018.
[14]
X. Zhang, Y. Wei, W. Cao, H. Yao, J. Peng, and Y. Zhou, “Local correntropy matrix representation for hyperspectral image classification,” IEEE Trans. Geosci. Remote Sens., vol. 60, 2022, Art. no.
[15]
X. Cao, L. Xu, D. Meng, Q. Zhao, and Z. Xu, “Integration of 3-dimensional discrete wavelet transform and Markov random field for hyperspectral image classification,” Neurocomputing, vol. 226, pp. 90–100, Feb. 2017.
[16]
P. Ghamisi, J. A. Benediktsson, and M. O. Ulfarsson, “Spectral–spatial classification of hyperspectral images based on hidden Markov random fields,” IEEE Trans. Geosci. Remote Sens., vol. 52, no. 5, pp. 2565–2574, May 2014.
[17]
X. Cao, F. Zhou, L. Xu, D. Meng, Z. Xu, and J. Paisley, “Hyperspectral image classification with Markov random fields and a convolutional neural network,” IEEE Trans. Image Process., vol. 27, no. 5, pp. 2354–2367, May 2018.
[18]
M. Fauvel, J. A. Benediktsson, J. Chanussot, and J. R. Sveinsson, “Spectral and spatial classification of hyperspectral data using SVMs and morphological profiles,” IEEE Trans. Geosci. Remote Sens., vol. 46, no. 11, pp. 3804–3814, Nov. 2008.
[19]
J. A. Benediktsson, J. A. Palmason, and J. R. Sveinsson, “Classification of hyperspectral data from urban areas based on extended morphological profiles,” IEEE Trans. Geosci. Remote Sens., vol. 43, no. 3, pp. 480–491, Mar. 2005.
[20]
M. Dalla Mura, A. Villa, J. A. Benediktsson, J. Chanussot, and L. Bruzzone, “Classification of hyperspectral images by using extended morphological attribute profiles and independent component analysis,” IEEE Geosci. Remote Sens. Lett., vol. 8, no. 3, pp. 542–546, May 2011.
[21]
M. E. Paoletti, J. M. Haut, J. Plaza, and A. Plaza, “Deep learning classifiers for hyperspectral imaging: A review,” ISPRS J. Photogramm. Remote Sens., vol. 158, pp. 279–317, Dec. 2019.
[22]
W. Zhao and S. Du, “Spectral–spatial feature extraction for hyperspectral image classification: A dimension reduction and deep learning approach,” IEEE Trans. Geosci. Remote Sens., vol. 54, no. 8, pp. 4544–4554, Aug. 2016.
[23]
Y. Chen, H. Jiang, C. Li, X. Jia, and P. Ghamisi, “Deep feature extraction and classification of hyperspectral images based on convolutional neural networks,” IEEE Trans. Geosci. Remote Sens., vol. 54, no. 10, pp. 6232–6251, Oct. 2016.
[24]
C. Shi, D. Liao, T. Zhang, and L. Wang, “Hyperspectral image classification based on expansion convolution network,” IEEE Trans. Geosci. Remote Sens., vol. 60, 2022, Art. no.
[25]
Z. Zhong, J. Li, Z. Luo, and M. Chapman, “Spectral–spatial residual network for hyperspectral image classification: A 3-D deep learning framework,” IEEE Trans. Geosci. Remote Sens., vol. 56, no. 2, pp. 847–858, Feb. 2017.
[26]
L. Song, Z. Feng, S. Yang, X. Zhang, and L. Jiao, “Self-supervised assisted semi-supervised residual network for hyperspectral image classification,” Remote Sens., vol. 14, no. 13, p. 2997, Jun. 2022.
[27]
S. K. Roy, S. Manna, T. Song, and L. Bruzzone, “Attention-based adaptive spectral–spatial kernel ResNet for hyperspectral image classification,” IEEE Trans. Geosci. Remote Sens., vol. 59, no. 9, pp. 7831–7843, Sep. 2021.
[28]
R. Li, S. Zheng, C. Duan, Y. Yang, and X. Wang, “Classification of hyperspectral image based on double-branch dual-attention mechanism network,” Remote Sens., vol. 12, no. 3, p. 582, Feb. 2020.
[29]
W. Wang, S. Dou, Z. Jiang, and L. Sun, “A fast dense spectral–spatial convolution network framework for hyperspectral images classification,” Remote Sens., vol. 10, no. 7, p. 1068, Jul. 2018.
[30]
Z. Feng, L. Song, S. Yang, X. Zhang, and L. Jiao, “Cross-modal contrastive learning for remote sensing image classification,” IEEE Trans. Geosci. Remote Sens., vol. 61, 2023, Art. no.
[31]
X. Yang, W. Cao, Y. Lu, and Y. Zhou, “QTN: Quaternion transformer network for hyperspectral image classification,” IEEE Trans. Circuits Syst. Video Technol., vol. 33, no. 12, pp. 7370–7384, 2023. 10.1109/TCSVT.2023.3283289.
[32]
B. Yunet al., “SpecTr: Spectral transformer for microscopic hyperspectral pathology image segmentation,” IEEE Trans. Circuits Syst. Video Technol., early access, Oct. 20, 2024. 10.1109/TCSVT.2023.3326196.
[33]
J. Wang and X. Tan, “Mutually beneficial transformer for multimodal data fusion,” IEEE Trans. Circuits Syst. Video Technol., vol. 33, no. 12, pp. 7466–7479, Dec. 2023.
[34]
D. Honget al., “SpectralFormer: Rethinking hyperspectral image classification with transformers,” IEEE Trans. Geosci. Remote Sens., vol. 60, pp. 1–15, 2021.
[35]
X. He, Y. Chen, and Z. Lin, “Spatial–spectral transformer for hyperspectral image classification,” Remote Sens., vol. 13, no. 3, p. 498, Jan. 2021.
[36]
A. Vaswaniet al., “Attention is all you need,” in Proc. Int. Conf. Neural Inf. Process. Syst., 2017, pp. 6000–6010.
[37]
A. Dosovitskiyet al., “An image is worth 16 × 16 words: Transformers for image recognition at scale,” 2020, arXiv:2010.11929.
[38]
H. Touvron, M. Cord, M. Douze, F. Massa, A. Sablayrolles, and H. Jégou, “Training data-efficient image transformers & distillation through attention,” in Proc. Int. Conf. Mach. Learn., 2021, pp. 10347–10357.
[39]
J. Liang, Y. Cui, Q. Wang, T. Geng, W. Wang, and D. Liu, “ClusterFomer: Clustering as a universal visual learner,” in Proc. 37th Conf. Neural Inf. Process. Syst. (NeurIPS), vol. 36, 2024, pp. 64029–64042.
[40]
Y. Luet al., “TransFlow: Transformer as flow learner,” in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit., Jun. 2023, pp. 18063–18073.
[41]
Q. Wanget al., “MUSTIE: Multimodal structural transformer for web information extraction,” in Proc. 61st Annu. Meeting Assoc. Comput. Linguistics, 2023, pp. 2405–2420.
[42]
K. Simonyan and A. Zisserman, “Very deep convolutional networks for large-scale image recognition,” 2014, arXiv:1409.1556.
[43]
Y. Qing, W. Liu, L. Feng, and W. Gao, “Improved transformer net for hyperspectral image classification,” Remote Sens., vol. 13, no. 11, p. 2216, Jun. 2021.
[44]
Z. Xue, X. Tan, X. Yu, B. Liu, A. Yu, and P. Zhang, “Deep hierarchical vision transformer for hyperspectral and LiDAR data classification,” IEEE Trans. Image Process., vol. 31, pp. 3095–3110, 2022.
[45]
X. Yang, W. Cao, Y. Lu, and Y. Zhou, “Hyperspectral image transformer classification networks,” IEEE Trans. Geosci. Remote Sens., vol. 60, 2022, Art. no.
[46]
D. Liu, Y. Cui, L. Yan, C. Mousas, B. Yang, and Y. Chen, “DenserNet: Weakly supervised visual localization using multi-scale feature aggregation,” in Proc. AAAI Conf. Artif. Intell., May 2021, vol. 35, no. 7, pp. 6101–6109.
[47]
D. Liu, Y. Cui, W. Tan, and Y. Chen, “SG-Net: Spatial granularity network for one-stage video instance segmentation,” in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR), Jun. 2021, pp. 9816–9825.
[48]
D. Liu, J. Liang, T. Geng, A. Loui, and T. Zhou, “Tripartite feature enhanced pyramid network for dense prediction,” IEEE Trans. Image Process., vol. 32, pp. 2678–2692, 2023.
[49]
V. Sharma, A. Diba, T. Tuytelaars, and L. Van Gool, “Hyperspectral CNN for image classification & band selection, with application to face recognition,” ESAT, Leuven, Belgium, Tech. Rep. KUL/ESAT/PSI/1604, 2016.
[50]
L. Ran, Y. Zhang, W. Wei, and T. Yang, “Bands sensitive convolutional network for hyperspectral image classification,” in Proc. Int. Conf. Internet Multimedia Comput. Service, Aug. 2016, pp. 268–272.
[51]
W. Hu, Y. Huang, L. Wei, F. Zhang, and H. Li, “Deep convolutional neural networks for hyperspectral image classification,” J. Sensors, vol. 2015, pp. 1–12, Nov. 2015.
[52]
S. K. Roy, G. Krishna, S. R. Dubey, and B. B. Chaudhuri, “HybridSN: Exploring 3-D–2-D CNN feature hierarchy for hyperspectral image classification,” IEEE Geosci. Remote Sens. Lett., vol. 17, no. 2, pp. 277–281, Feb. 2019.
[53]
K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR), Jun. 2016, pp. 770–778.
[54]
J. Hu, L. Shen, and G. Sun, “Squeeze-and-excitation networks,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., Jun. 2018, pp. 7132–7141.
[55]
R. Hang, Q. Liu, D. Hong, and P. Ghamisi, “Cascaded recurrent neural networks for hyperspectral image classification,” IEEE Trans. Geosci. Remote Sens., vol. 57, no. 8, pp. 5384–5394, Aug. 2019.
[56]
K. Makantasis, K. Karantzalos, A. Doulamis, and N. Doulamis, “Deep supervised learning for hyperspectral data classification through convolutional neural networks,” in Proc. IEEE Int. Geosci. Remote Sens. Symp. (IGARSS), Jul. 2015, pp. 4959–4962.
[57]
Y. Li, H. Zhang, and Q. Shen, “Spectral–spatial classification of hyperspectral imagery with 3D convolutional neural network,” Remote Sens., vol. 9, no. 1, p. 67, Jan. 2017.
[58]
H. Lee and H. Kwon, “Going deeper with contextual CNN for hyperspectral image classification,” IEEE Trans. Image Process., vol. 26, no. 10, pp. 4843–4855, Oct. 2017.
[59]
L. Sun, G. Zhao, Y. Zheng, and Z. Wu, “Spectral–spatial feature tokenization transformer for hyperspectral image classification,” IEEE Trans. Geosci. Remote Sens., vol. 60, 2022, Art. no.
[60]
C.-C. Chang and C.-J. Lin, “LIBSVM: A library for support vector machines,” ACM Trans. Intell. Syst. Technol., vol. 2, no. 3, pp. 1–27, Apr. 2011.
Index Terms
- Interactive Spectral-Spatial Transformer for Hyperspectral Image Classification
Index terms have been assigned to the content through auto-classification.
Recommendations
Multiscale spatial–spectral transformer network for hyperspectral and multispectral image fusion
AbstractFusing hyperspectral images (HSIs) and multispectral images (MSIs) is an economic and feasible way to obtain images with both high spectral resolution and spatial resolution. Due to the limited receptive field of convolution kernels, ...
Graphical abstractDisplay Omitted
Highlights- A multiscale spatial–spectral Transformer network is proposed.
- Spectral multi-...
Spatial-spectral transformer for hyperspectral image denoising
AAAI'23/IAAI'23/EAAI'23: Proceedings of the Thirty-Seventh AAAI Conference on Artificial Intelligence and Thirty-Fifth Conference on Innovative Applications of Artificial Intelligence and Thirteenth Symposium on Educational Advances in Artificial IntelligenceHyperspectral image (HSI) denoising is a crucial preprocessing procedure for the subsequent HSI applications. Unfortunately, though witnessing the development of deep learning in HSI denoising area, existing convolution-based methods face the trade-off ...
Dual attention transformer network for hyperspectral image classification
AbstractHyperspectral image classification (HSIC) has been a significant topic in the field of remote sensing in the past few years. Convolutional neural networks have shown promising performance in HSIC applications due to their strong local feature ...
Comments
Information & Contributors
Information
Published In
1051-8215 © 2024 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See https://www.ieee.org/publications/rights/index.html for more information.
Publisher
IEEE Press
Publication History
Published: 01 September 2024
Qualifiers
- Research-article
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 0Total Downloads
- Downloads (Last 12 months)0
- Downloads (Last 6 weeks)0
Reflects downloads up to 11 Feb 2025