research-article

QTN: Quaternion Transformer Network for Hyperspectral Image Classification

Authors:

Yicong ZhouAuthors Info & Claims

IEEE Transactions on Circuits and Systems for Video Technology, Volume 33, Issue 12

Pages 7370 - 7384

https://doi.org/10.1109/TCSVT.2023.3283289

Published: 01 December 2023 Publication History

Abstract

Numerous state-of-the-art transformer-based techniques with self-attention mechanisms have recently been demonstrated to be quite effective in the classification of hyperspectral images (HSIs). However, traditional transformer-based methods severely suffer from the following problems when processing HSIs with three dimensions: (1) processing the HSIs using 1D sequences misses the 3D structure information; (2) too expensive numerous parameters for hyperspectral image classification tasks; (3) only capturing spatial information while lacking the spectral information. To solve these problems, we propose a novel Quaternion Transformer Network (QTN) for recovering self-adaptive and long-range correlations in HSIs. Specially, we first develop a band adaptive selection module (BASM) for producing Quaternion data from HSIs. And then, we propose a new and novel quaternion self-attention (QSA) mechanism to capture the local and global representations. Finally, we propose a new and novel transformer method, i.e., QTN by stacking a series of QSA for hyperspectral classification. The proposed QTN could exploit computation using Quaternion algebra in hypercomplex spaces. Extensive experiments on three public datasets demonstrate that the QTN outperforms the state-of-the-art vision transformers and convolution neural networks.

References

[1]

J. Liu, Y. Feng, W. Liu, D. Orlando, and H. Li, “Training data assisted anomaly detection of multi-pixel targets in hyperspectral imagery,” IEEE Trans. Signal Process., vol. 68, pp. 3022–3032, 2020.

[2]

J. Peng, Y. Zhou, W. Sun, Q. Du, and L. Xia, “Self-paced nonnegative matrix factorization for hyperspectral unmixing,” IEEE Trans. Geosci. Remote Sens., vol. 59, no. 2, pp. 1501–1515, Feb. 2020.

[3]

J. M. Bioucas-Dias, A. Plaza, G. Camps-Valls, P. Scheunders, N. Nasrabadi, and J. Chanussot, “Hyperspectral remote sensing data analysis and future challenges,” IEEE Geosci. Remote Sens. Mag., vol. 1, no. 2, pp. 6–36, Jun. 2013.

[4]

V. E. Brando and A. G. Dekker, “Satellite hyperspectral remote sensing for estimating estuarine and coastal water quality,” IEEE Trans. Geosci. Remote Sens., vol. 41, no. 6, pp. 1378–1387, Jun. 2003.

[5]

F. Hu, G.-S. Xia, J. Hu, and L. Zhang, “Transferring deep convolutional neural networks for the scene classification of high-resolution remote sensing imagery,” Remote Sens., vol. 7, no. 11, pp. 14680–14707, Nov. 2015.

[6]

G. Hughes, “On the mean accuracy of statistical pattern recognizers,” IEEE Trans. Inf. Theory, vol. IT-14, no. 1, pp. 55–63, Jan. 1968.

Digital Library

[7]

B. Scholkopf and A. J. Smola, Learning With Kernels: Support Vector Machines, Regularization, Optimization, and Beyond. Cambridge, MA, USA: MIT Press, 2001.

[8]

L. Ma, M. M. Crawford, and J. Tian, “Local manifold learning-based k-nearest-neighbor for hyperspectral image classification,” IEEE Trans. Geosci. Remote Sens., vol. 48, no. 11, pp. 4099–4109, Nov. 2010.

[9]

L. Sunet al., “Low rank component induced spatial–spectral kernel method for hyperspectral image classification,” IEEE Trans. Circuits Syst. Video Technol., vol. 30, no. 10, pp. 3829–3842, Oct. 2020.

[10]

H. Liu, Y. Jia, J. Hou, and Q. Zhang, “Global-local balanced low-rank approximation of hyperspectral images for classification,” IEEE Trans. Circuits Syst. Video Technol., vol. 32, no. 4, pp. 2013–2024, Apr. 2022.

[11]

M. Fauvel, J. A. Benediktsson, J. Chanussot, and J. R. Sveinsson, “Spectral and spatial classification of hyperspectral data using SVMs and morphological profiles,” IEEE Trans. Geosci. Remote Sens., vol. 46, no. 11, pp. 3804–3814, Nov. 2008.

[12]

Q. Wang, Z. Meng, and X. Li, “Locality adaptive discriminant analysis for spectral–spatial classification of hyperspectral images,” IEEE Geosci. Remote Sens. Lett., vol. 14, no. 11, pp. 2077–2081, Nov. 2017.

[13]

C. Tao, W. Lu, J. Qi, and H. Wang, “Spatial information considered network for scene classification,” IEEE Geosci. Remote Sens. Lett., vol. 18, no. 6, pp. 984–988, Jun. 2021.

[14]

X. Yanget al., “Synergistic 2D/3D convolutional neural network for hyperspectral image classification,” Remote Sens., vol. 12, no. 12, p. 2033, Jun. 2020.

[15]

S. Li, W. Song, L. Fang, Y. Chen, P. Ghamisi, and J. A. Benediktsson, “Deep learning for hyperspectral image classification: An overview,” IEEE Trans. Geosci. Remote Sens., vol. 57, no. 9, pp. 6690–6709, Sep. 2019.

[16]

B. Liu, X. Yu, P. Zhang, X. Tan, A. Yu, and Z. Xue, “A semi-supervised convolutional neural network for hyperspectral image classification,” Remote Sens. Lett., vol. 8, no. 9, pp. 839–848, Sep. 2017.

[17]

B. Xiet al., “DGSSC: A deep generative spectral–spatial classifier for imbalanced hyperspectral imagery,” IEEE Trans. Circuits Syst. Video Technol., vol. 33, no. 4, pp. 1535–1548, Apr. 2022.

[18]

B. Rastiet al., “Feature extraction for hyperspectral imagery: The evolution from shallow to deep: Overview and toolbox,” IEEE Geosci. Remote Sens. Mag., vol. 8, no. 4, pp. 60–88, Dec. 2020.

[19]

H. Lee and H. Kwon, “Going deeper with contextual CNN for hyperspectral image classification,” IEEE Trans. Image Process., vol. 26, no. 10, pp. 4843–4855, Oct. 2017.

Digital Library

[20]

X. Cao, F. Zhou, L. Xu, D. Meng, Z. Xu, and J. Paisley, “Hyperspectral image classification with Markov random fields and a convolutional neural network,” IEEE Trans. Image Process., vol. 27, no. 5, pp. 2354–2367, May 2018.

Digital Library

[21]

W. Li, G. Wu, F. Zhang, and Q. Du, “Hyperspectral image classification using deep pixel-pair features,” IEEE Trans. Geosci. Remote Sens., vol. 55, no. 2, pp. 844–853, Feb. 2017.

[22]

A. B. Hamida, A. Benoit, P. Lambert, and C. B. Amar, “3-D deep learning approach for remote sensing image classification,” IEEE Trans. Geosci. Remote Sens., vol. 56, no. 8, pp. 4420–4434, Aug. 2018.

[23]

X. Yang, Y. Ye, X. Li, R. Y. K. Lau, X. Zhang, and X. Huang, “Hyperspectral image classification with deep learning models,” IEEE Trans. Geosci. Remote Sens., vol. 56, no. 9, pp. 5408–5423, Sep. 2018.

[24]

H. Lee and H. Kwon, “Contextual deep CNN based hyperspectral classification,” in Proc. IEEE Int. Geosci. Remote Sens. Symp. (IGARSS), Jul. 2016, pp. 3322–3325.

[25]

Y. Li, H. Zhang, and Q. Shen, “Spectral–spatial classification of hyperspectral imagery with 3D convolutional neural network,” Remote Sens., vol. 9, no. 1, p. 67, Jan. 2017.

[26]

Y. Chen, H. Jiang, C. Li, X. Jia, and P. Ghamisi, “Deep feature extraction and classification of hyperspectral images based on convolutional neural networks,” IEEE Trans. Geosci. Remote Sens., vol. 54, no. 10, pp. 6232–6251, Oct. 2016.

[27]

V. Sharma, A. Diba, T. Tuytelaars, and L. Van Gool, “Hyperspectral CNN for image classification & band selection, with application to face recognition,” 2016.

[28]

P. R. Lorenzo, L. Tulczyjew, M. Marcinkiewicz, and J. Nalepa, “Band selection from hyperspectral images using attention-based convolutional neural networks,” 2018, arXiv:1811.02667.

[29]

X. Zheng, H. Sun, X. Lu, and W. Xie, “Rotation-invariant attention network for hyperspectral image classification,” IEEE Trans. Image Process., vol. 31, pp. 4251–4265, 2022.

Digital Library

[30]

H. Sun, X. Zheng, and X. Lu, “A supervised segmentation network for hyperspectral image classification,” IEEE Trans. Image Process., vol. 30, pp. 2810–2825, 2021.

Digital Library

[31]

A. Dosovitskiyet al., “An image is worth 16×16 words: Transformers for image recognition at scale,” in Proc. Int. Conf. Learn. Represent., 2020, pp. 1–11.

[32]

Z. Liuet al., “Swin transformer: Hierarchical vision transformer using shifted windows,” in Proc. IEEE/CVF Int. Conf. Comput. Vis. (ICCV), Oct. 2021, pp. 9992–10002.

[33]

D. Zhouet al., “DeepViT: Towards deeper vision transformer,” 2021, arXiv:2103.11886.

[34]

L. Yuanet al., “Tokens-to-Token ViT: Training vision transformers from scratch on ImageNet,” in Proc. IEEE/CVF Int. Conf. Comput. Vis. (ICCV), Oct. 2021, pp. 538–547.

[35]

E. Xie, W. Wang, Z. Yu, A. Anandkumar, J. M. Alvarez, and P. Luo, “SegFormer: Simple and efficient design for semantic segmentation with transformers,” in Proc. Adv. Neural Inf. Process. Syst., vol. 34, 2021, pp. 12077–12090.

[36]

H. Touvron, M. Cord, A. Sablayrolles, G. Synnaeve, and H. Jégou, “Going deeper with image transformers,” in Proc. IEEE/CVF Int. Conf. Comput. Vis. (ICCV), Oct. 2021, pp. 32–42.

[37]

B. Grahamet al., “LeViT: A vision transformer in ConvNet’s clothing for faster inference,” in Proc. IEEE/CVF Int. Conf. Comput. Vis. (ICCV), Oct. 2021, pp. 12239–12249.

[38]

C. R. Chen, Q. Fan, and R. Panda, “CrossViT: Cross-attention multi-scale vision transformer for image classification,” in Proc. IEEE/CVF Int. Conf. Comput. Vis. (ICCV), Oct. 2021, pp. 347–356.

[39]

X. Chuet al., “Twins: Revisiting the design of spatial attention in vision transformers,” in Proc. Adv. Neural Inf. Process. Syst., vol. 34, 2021, pp. 9355–9366.

[40]

M. Caronet al., “Emerging properties in self-supervised vision transformers,” in Proc. IEEE/CVF Int. Conf. Comput. Vis. (ICCV), Oct. 2021, pp. 9630–9640.

[41]

B. Heo, S. Yun, D. Han, S. Chun, J. Choe, and S. J. Oh, “Rethinking spatial dimensions of vision transformers,” in Proc. IEEE/CVF Int. Conf. Comput. Vis. (ICCV), Oct. 2021, pp. 11916–11925.

[42]

H. Wuet al., “CvT: Introducing convolutions to vision transformers,” in Proc. IEEE/CVF Int. Conf. Comput. Vis. (ICCV), Oct. 2021, pp. 22–31.

[43]

J. He, L. Zhao, H. Yang, M. Zhang, and W. Li, “HSI-BERT: Hyperspectral image classification using the bidirectional encoder representation from transformers,” IEEE Trans. Geosci. Remote Sens., vol. 58, no. 1, pp. 165–178, Jan. 2020.

[44]

X. He, Y. Chen, and Z. Lin, “Spatial–spectral transformer for hyperspectral image classification,” Remote Sens., vol. 13, no. 3, p. 498, Jan. 2021.

[45]

M. Xiang, B. S. Dees, and D. P. Mandic, “Multiple-model adaptive estimation for 3-D and 4-D signals: A widely linear quaternion approach,” IEEE Trans. Neural Netw. Learn. Syst., vol. 30, no. 1, pp. 72–84, Jan. 2019.

[46]

B. Chen, M. Yu, Q. Su, and L. Li, “Fractional quaternion cosine transform and its application in color image copy-move forgery detection,” Multimedia Tools Appl., vol. 78, no. 7, pp. 8057–8073, Apr. 2019.

Digital Library

[47]

Y. Liu, Y. Zheng, J. Lu, J. Cao, and L. Rutkowski, “Constrained quaternion-variable convex optimization: A quaternion-valued recurrent neural network approach,” IEEE Trans. Neural Netw. Learn. Syst., vol. 31, no. 3, pp. 1022–1035, Mar. 2020.

[48]

E. C. Mengüç, N. Acir, and D. P. Mandic, “Widely linear quaternion-valued least-mean kurtosis algorithm,” IEEE Trans. Signal Process., vol. 68, pp. 5914–5922, 2020.

[49]

J. Flamant, S. Miron, and D. Brie, “Quaternion non-negative matrix factorization: Definition, uniqueness, and algorithm,” IEEE Trans. Signal Process., vol. 68, pp. 1870–1883, 2020.

[50]

M. Xiang, Y. Xia, and D. P. Mandic, “Performance analysis of deficient length quaternion least mean square adaptive filters,” IEEE Trans. Signal Process., vol. 68, pp. 65–80, 2020.

[51]

H. Li, H. Li, and L. Zhang, “Quaternion-based multiscale analysis for feature extraction of hyperspectral images,” IEEE Trans. Signal Process., vol. 67, no. 6, pp. 1418–1430, Mar. 2019.

Digital Library

[52]

H. Li, H. Huang, Z. Ye, and H. Li, “Hyperspectral image classification using adaptive weighted quaternion Zernike moments,” IEEE Trans. Signal Process., vol. 70, pp. 701–713, 2022.

[53]

R. Rajabi and H. Ghassemian, “Multilayer structured NMF for spectral unmixing of hyperspectral images,” in Proc. 6th Workshop Hyperspectral Image Signal Processing: Evol. Remote Sens. (WHISPERS), Jun. 2014, pp. 1–4.

[54]

Q. Tian, T. Arbel, and J. J. Clark, “Task dependent deep LDA pruning of neural networks,” Comput. Vis. Image Understand., vol. 203, Feb. 2021, Art. no.

[55]

Q. Wang, F. Zhang, and X. Li, “Hyperspectral band selection via optimal neighborhood reconstruction,” IEEE Trans. Geosci. Remote Sens., vol. 58, no. 12, pp. 8465–8476, Dec. 2020.

[56]

S. L. Al-Khafaji, J. Zhou, X. Bai, Y. Qian, and A. W. Liew, “Spectral–spatial boundary detection in hyperspectral images,” IEEE Trans. Image Process., vol. 31, pp. 499–512, 2022.

Digital Library

[57]

J. Xie, N. He, L. Fang, and P. Ghamisi, “Multiscale densely-connected fusion networks for hyperspectral images classification,” IEEE Trans. Circuits Syst. Video Technol., vol. 31, no. 1, pp. 246–259, Jan. 2021.

Digital Library

[58]

S. Huang, H. Zhang, and A. Pižurica, “Subspace clustering for hyperspectral images via dictionary learning with adaptive regularization,” IEEE Trans. Geosci. Remote Sens., vol. 60, 2022, Art. no.

[59]

W. Yao, C. Lian, and L. Bruzzone, “ClusterCNN: Clustering-based feature learning for hyperspectral image classification,” IEEE Geosci. Remote Sens. Lett., vol. 18, no. 11, pp. 1991–1995, Nov. 2021.

[60]

S. K. Roy, G. Krishna, S. R. Dubey, and B. B. Chaudhuri, “HybridSN: Exploring 3-D–2-D CNN feature hierarchy for hyperspectral image classification,” IEEE Geosci. Remote Sens. Lett., vol. 17, no. 2, pp. 277–281, Feb. 2020.

[61]

D. Hong, L. Gao, J. Yao, B. Zhang, A. Plaza, and J. Chanussot, “Graph convolutional networks for hyperspectral image classification,” IEEE Trans. Geosci. Remote Sens., vol. 59, no. 7, pp. 5966–5978, Jul. 2021.

[62]

W. Song, S. Li, L. Fang, and T. Lu, “Hyperspectral image classification with deep feature fusion network,” IEEE Trans. Geosci. Remote Sens., vol. 56, no. 6, pp. 3173–3184, Jun. 2018.

[63]

J. D. M.-W. C. Kenton and L. K. Toutanova, “BERT: Pre-training of deep bidirectional transformers for language understanding,” in Proc. NAACL-HLT, 2019, pp. 4171–4186.

[64]

D. Honget al., “SpectralFormer: Rethinking hyperspectral image classification with transformers,” IEEE Trans. Geosci. Remote Sens., vol. 60, 2022, Art. no.

[65]

X. Yang, W. Cao, Y. Lu, and Y. Zhou, “Hyperspectral image transformer classification networks,” IEEE Trans. Geosci. Remote Sens., vol. 60, 2022, Art. no.

[66]

Q. Yin, J. Wang, X. Luo, J. Zhai, S. Kr. Jha, and Y. Shi, “Quaternion convolutional neural network for color image classification and forensics,” IEEE Access, vol. 7, pp. 20293–20301, 2019.

[67]

C. J. Gaudet and A. S. Maida, “Deep quaternion networks,” in Proc. Int. Joint Conf. Neural Netw. (IJCNN), Jul. 2018, pp. 1–8.

[68]

X. Zhu, Y. Xu, H. Xu, and C. Chen, “Quaternion convolutional neural networks,” in Proc. Eur. Conf. Comput. Vis. (ECCV), 2018, pp. 631–647.

[69]

S. Zhang, L. Yao, L. V. Tran, A. Zhang, and Y. Tay, “Quaternion collaborative filtering for recommendation,” in Proc. 28th Int. Joint Conf. Artif. Intell., Aug. 2019, pp. 4313–4319.

[70]

A. Mackiewicz and W. Ratajczak, “Principal components analysis (PCA),” Comput. Geosci., vol. 19, pp. 303–342, Mar. 1993.

Digital Library

[71]

D. Hendrycks and K. Gimpel, “Gaussian error linear units (GELUs),” 2016, arXiv:1606.08415.

[72]

H. Abdi and L. J. Williams, “Principal component analysis,” Wiley Interdiscipl. Rev., Comput. Statist., vol. 2, no. 4, pp. 433–459, 2010.

Digital Library

[73]

Q. Wang, Q. Li, and X. Li, “A fast neighborhood grouping method for hyperspectral band selection,” IEEE Trans. Geosci. Remote Sens., vol. 59, no. 6, pp. 5028–5039, Jun. 2021.

[74]

Q. Wang, F. Zhang, and X. Li, “Optimal clustering framework for hyperspectral band selection,” IEEE Trans. Geosci. Remote Sens., vol. 56, no. 10, pp. 5910–5922, Oct. 2018.

Cited By

Wang NMei SWang YZhang YZhan D(2025)WHANet:Wavelet-Based Hybrid Asymmetric Network for Spectral Super-Resolution From RGB InputsIEEE Transactions on Multimedia10.1109/TMM.2024.352171327(414-428)Online publication date: 1-Jan-2025
https://dl.acm.org/doi/10.1109/TMM.2024.3521713
Zhang YQi JWang XCai ZPeng JZhou Y(2024)Tensorial Global-Local Graph Self-Representation for Hyperspectral Band SelectionIEEE Transactions on Circuits and Systems for Video Technology10.1109/TCSVT.2024.343936934:12_Part_2(13213-13225)Online publication date: 1-Dec-2024
https://dl.acm.org/doi/10.1109/TCSVT.2024.3439369
Guo KTian DHu YLin CSun YZhou JDuan XGao JYin B(2024)CFMMC-Align: Coarse-Fine Multi-Modal Contrastive Alignment Network for Traffic Event Video Question AnsweringIEEE Transactions on Circuits and Systems for Video Technology10.1109/TCSVT.2024.340945334:11_Part_1(10538-10550)Online publication date: 4-Jun-2024
https://dl.acm.org/doi/10.1109/TCSVT.2024.3409453
Show More Cited By

Index Terms

QTN: Quaternion Transformer Network for Hyperspectral Image Classification
1. Computing methodologies
  1. Artificial intelligence
    1. Computer vision
  2. Machine learning
    1. Learning paradigms
      1. Supervised learning
    2. Machine learning approaches
      1. Neural networks

Index terms have been assigned to the content through auto-classification.

Recommendations

Rotation-Invariant Attention Network for Hyperspectral Image Classification
<italic>Hyperspectral image</italic> (HSI) classification refers to identifying land-cover categories of pixels based on spectral signatures and spatial information of HSIs. In recent deep learning-based methods, to explore the spatial information of HSIs,...
Dual attention transformer network for hyperspectral image classification
Abstract
Hyperspectral image classification (HSIC) has been a significant topic in the field of remote sensing in the past few years. Convolutional neural networks have shown promising performance in HSIC applications due to their strong local feature ...
Multiscale spatial–spectral transformer network for hyperspectral and multispectral image fusion
Abstract
Fusing hyperspectral images (HSIs) and multispectral images (MSIs) is an economic and feasible way to obtain images with both high spectral resolution and spatial resolution. Due to the limited receptive field of convolution kernels, ...
Graphical abstract

Display Omitted
Highlights
- A multiscale spatial–spectral Transformer network is proposed.
- Spectral multi-...

Comments

Information & Contributors

Information

Published In

cover image IEEE Transactions on Circuits and Systems for Video Technology

IEEE Transactions on Circuits and Systems for Video Technology Volume 33, Issue 12

Dec. 2023

875 pages

ISSN:1051-8215

Issue’s Table of Contents

1051-8215 © 2023 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See https://www.ieee.org/publications/rights/index.html for more information.

Publisher

IEEE Press

Publication History

Published: 01 December 2023

Qualifiers

Research-article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

5
Total Citations
View Citations
0
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 12 Feb 2025

Other Metrics

View Author Metrics

Citations

Cited By

Wang NMei SWang YZhang YZhan D(2025)WHANet:Wavelet-Based Hybrid Asymmetric Network for Spectral Super-Resolution From RGB InputsIEEE Transactions on Multimedia10.1109/TMM.2024.352171327(414-428)Online publication date: 1-Jan-2025
https://dl.acm.org/doi/10.1109/TMM.2024.3521713
Zhang YQi JWang XCai ZPeng JZhou Y(2024)Tensorial Global-Local Graph Self-Representation for Hyperspectral Band SelectionIEEE Transactions on Circuits and Systems for Video Technology10.1109/TCSVT.2024.343936934:12_Part_2(13213-13225)Online publication date: 1-Dec-2024
https://dl.acm.org/doi/10.1109/TCSVT.2024.3439369
Guo KTian DHu YLin CSun YZhou JDuan XGao JYin B(2024)CFMMC-Align: Coarse-Fine Multi-Modal Contrastive Alignment Network for Traffic Event Video Question AnsweringIEEE Transactions on Circuits and Systems for Video Technology10.1109/TCSVT.2024.340945334:11_Part_1(10538-10550)Online publication date: 4-Jun-2024
https://dl.acm.org/doi/10.1109/TCSVT.2024.3409453
Zhang XChen BZou WLiu SZhang YXiong RZhang J(2024)Progressive Content-Aware Coded Hyperspectral Snapshot Compressive ImagingIEEE Transactions on Circuits and Systems for Video Technology10.1109/TCSVT.2024.340942134:11_Part_1(10817-10830)Online publication date: 6-Jun-2024
https://dl.acm.org/doi/10.1109/TCSVT.2024.3409421
Song LFeng ZYang SZhang XJiao L(2024)Interactive Spectral-Spatial Transformer for Hyperspectral Image ClassificationIEEE Transactions on Circuits and Systems for Video Technology10.1109/TCSVT.2024.338657834:9(8589-8601)Online publication date: 1-Sep-2024
https://dl.acm.org/doi/10.1109/TCSVT.2024.3386578

View Options

View options

Figures

Tables

Media

View Issue’s Table of Contents