1. Introduction
Hyperspectral (HS) imaging is an advanced technology that enables the acquisition of a high-dimensional image dataset encompassing numerous diverse bands. Due to their rich spectral information, HS images (HSIs) have become extensively utilized for numerous earth observation tasks, such as image classification [
1,
2], ecosystem monitoring [
3], and change detection [
4]. Nevertheless, HS imaging sensors struggle to balance their spatial and spectral resolutions due to finite sun irradiation [
5,
6,
7]. As a result, HSIs often prioritize spectral resolution over spatial resolution, restricting their applications in the study of land surface dynamics occurring in small areas [
8,
9,
10]. Fortunately, multispectral (MS) imaging sensors generally outperform HS imaging sensors in terms of spatial resolution. This provides an opportunity to perform HSI super-resolution by integrating complementary information of the high spatial-resolution MS images (HR-MSIs) and the low spatial-resolution HSIs (LR-HSIs) captured from the same scene, resulting in the generation of the desired HR-HSIs. Current HSI super-resolution approaches are primarily categorized into three groups [
11]: deep learning (DL)-based methods [
12,
13,
14], matrix factorization-based methods [
15,
16], tensor-based methods [
17,
18,
19].
The remarkable learning performance and effective computing capabilities of DL contributed to the development of DL-based HSI super-resolution techniques [
13,
20]. For example, the work in [
21] designed a deep residual learning network for HSI super-resolution, which effectively learns the spectral prior by using the deep convolution neural network (CNN). The work in [
14] presented a simple and efficient CNN architecture that can maintain the spatial and spectral specifics of the observations. The main benefit of these techniques lies in their ability to achieve satisfactory results by leveraging the powerful feature capturing capabilities of CNNs. However, it is worth noting that these methods are supervised and rely on a substantial amount of labeled samples to pretrain the CNN. In order to tackle this matter, numerous scholars have put forth unsupervised deep learning-based approaches [
22,
23]. For example, in work [
22], an unsupervised and non-pretraining encoder–decoder architecture for HSI super-resolution was designed. However, these unsupervised methods may produce slightly worse results than the supervised methods.
Matrix factorization-based methods approximate the latent HR-HSI using a dictionary and its corresponding coefficients, assuming that every spectral component is able to be expressed as a composition of a small set of distinct spectral patterns. Within this theoretical framework, the primary focus centers around the precise estimation of both the dictionary and its accompanying coefficients. Several methods [
24,
25,
26] estimated the dictionary and its corresponding coefficients using certain predetermined priors. For instance, Xue et al. [
26] exploited the subspace low-rank priors in both spatial and spectral dimensions to accomplish HSI super-resolution. In addition, previous studies [
27,
28] successfully addressed the HSI super-resolution problem through the utilization of spatial structures inherent in HR-HSI. For example, Simoes et al. [
27] attempted to capture the spatial self-similarity features through the total variation regularization. However, fusion approaches based on matrix factorization encounter difficulties in fully leveraging the spatial and spectral correlations of the HR-HSI due to their approach of unfolding the high-dimensional HSI into matrices without considering its underlying data structure. As a result, those methods often yield unsatisfactory fusion outcomes [
29,
30].
Given that HSI is a three-order tensor, methods based on tensor theory have gained significant attention in addressing the HSI super-resolution problem [
31]. These methods address the HSI super-resolution problem using various tensor decompositions, including tensor ring decomposition [
17,
18,
19,
31], tensor train decomposition [
9], Canonical polyadic decomposition [
32,
33,
34], and Tucker decomposition [
8,
17,
29,
30,
35,
36,
37], while incorporating specific priors. For example, Dian et al. [
35] developed an HSI super-resolution approach via Tucker decomposition, which reconstructs the latent HR-HSI by utilizing dictionary learning and sparse coding. He et al. [
18] developed a coupled tensor ring factorization-based approach that can simultaneously learn tensor ring core tensors from the observations for reconstructing the latent HR-HSI. Although these tensor decomposition-based methods have yielded promising performances, they still encounter certain challenges and limitations. One of the challenges is the high computational cost associated with representing the strong spectral and spatial correlations in the original HSI domain through tensor decompositions, particularly when dealing with a large number of bands in HSI [
6].
The spectral bands in the HR-HSI exhibit a strong correlation and typically reside inside a subspace of reduced dimensionality [
38]. This implies that HSI super-resolution is able to be modeled by projecting the high-dimensional HR-HSI into a lower-dimensional subspace. This strategy can largely decrease the computing complexity and enhance the spectral correlation property [
6]. Recently, many HSI super-resolution methods [
6,
38,
39,
40] have been proposed through subspace representation. Xu et al. [
6] exploited a tensor subspace representation to effectively leverage the nonlocal and global spectral–spatial low-rank priors to accomplish HSI super-resolution. Xing et al. [
39] presented a fusion approach based on variational tensor subspace decomposition. This method effectively captures the distinctions and interrelationships among the three modes of the HR-HSI. To exploit both the global and nonlocal priors, Xu et al. [
40] developed a tensor ring decomposition-based approach that blends spectral subspace learning, nonlocal similarity learning, and the tensor nuclear norm. Compared with matrix/tensor decomposition-based methods, subspace representation-based methods project the high-dimensional HR-HSI into the low-dimensional spectral subspace, consuming less computational complexity and generating better results. However, the current subspace representation-based methods often encounter challenges in adaptively selecting the appropriate subspace dimension, making them highly sensitive to changes in subspace dimension and affecting their overall performance.
To tackle the aforementioned concerns, this study develops a factor group sparsity regularized subspace representation-based method for HSI super-resolution. Specifically, we first propose a factor group sparsity regularized subspace representation (FGSSR) model, which automatically selects the appropriate subspace dimension, resulting in a precise depiction of the high spectral correlations in the latent HR-HSI. The proposed FGSSR model has several advantages that can be summarized as follows: (1) FGSSR is established based on subspace representation framework, resulting in an optimization process with low computational cost. (2) FGSSR is rank-revealing by seeking to minimize the number of nonzero frontal slices of the coefficient tensor. Thus, it provides a solution to the challenge of adaptively selecting the appropriate subspace dimension faced by previous subspace representation methods. (3) FGSSR corresponds to the Schatten-
p norm, allowing it the selection of a subspace dimension that is closer to the true rank of the latent HR-HSI compared to the nuclear norm. Under the FGSSR framework, the depiction of the spatial structure prior in the high-dimensional HR-HSI domain can be translated into an exploration of the low-dimensional coefficient domain. Driven by the efficacy of the tensor nuclear norm (TNN) [
41], we impose it on the low-dimensional coefficient tensor to capture and retain the spatial information inherent in the latent HR-HSI. Furthermore, we design an effective optimization algorithm under the proximal alternating minimization (PAM) framework to address our FGSSR-based HSI super-resolution model. This study makes several notable contributions in the following areas:
We propose an FGSSR-based HSI super-resolution method that overcomes the sensibility of the subspace dimension determination of earlier subspace representation-based methods. Specifically, by incorporating factor group sparsity regularization into subspace representation, FGSSR becomes rank-revealing, enabling it to adaptively and accurately approximate the subspace dimension to the rank of the latent HR-HSI.
The FGSSR-based method utilizes the FGSSR model to explore spectral correlation and employs TNN regularization to capture spatial self-similarity in the latent HR-HSI. FGSSR can adaptively and effectively capture the high spectral correlation by fully leveraging the advantages of subspace representation and the Schatten-p norm while mitigating their limitations. TNN regularization on the low-dimensional coefficients can effectively capture the spatial self-similarity and facilitate efficient computations through dimensionality reduction.
An effective PAM-based algorithm is developed with the aim of addressing the FGSSR-based model. Extensive studies carried out with simulated and real-world datasets reveal that our FGSSR-based method outperforms the state-of-the-art (SOTA) HSI super-resolution methods in both quantitative and visual judgements.
The ensuing section of this paper is structured as follows:
Section 2 introduces the related work.
Section 3 presents the FGSSR-based method, outlining its key features and methodology.
Section 4 and
Section 5 present the experimental results and model discussions, respectively. Finally,
Section 6 concludes this paper and discusses the potential future research directions.
3. Proposed FGSSR-Based Method
In this section, we present an FGSSR-based method for fusing the LR-HSI and HR-MSI to generate the target HR-HSI. In this method, we first propose a FGSSR model that combines the subspace learning of a low-dimensional spectral subspace with the imposition of group sparsity regularization on a low-dimensional coefficient tensor to depict the high spectral correlations of HS-HSI. The proposed FGSSR model offers several advantages, including enhanced rank minimization, automatic subspace dimension selection, and low computational cost. These advantages allow us adaptive and effective capture of the high spectral correlations. Furthermore, since each frontal slice of the coefficient tensor can be regarded as the eigen-image [
38], we also introduce TNN regularization on the low-dimensional coefficient tensor to depict the spatial self-similarity prior, which offers high efficiency while successfully maintaining the spatial features. The flowchart of the proposed FGSSR-based method is demonstrated in
Figure 1.
3.1. Subspace Learning
For the latent HR-HSI, there is typically a significant correlation among spectral vectors. This indicates that spectral vectors in the latent HR-HSI often reside within a low-dimensional subspace [
38,
40,
49]. Hence, the latent HR-HSI can be approximated by adopting a subspace representation strategy, that is,
where
and
represent the spectral subspace and its coefficients, respectively. Notably, the subspace dimension
is significantly smaller than
, which indicates that the spectral bands reside inside a low-dimensional subspace. This reduction in dimensionality drastically improves computational efficiency. However, the effectiveness of Subspace representation model (
12) heavily relies on accurately determining the appropriate subspace dimension
, namely the number of the frontal slices of
. Therefore, the subspace dimension
should be precisely chosen to approximate the rank of
(i.e., the unfolding matrix of
along the 3rd dimension) as closely as possible. Motivated by the FGS regularizer (
8), we define a factor group sparsity regularized subspace representation (FGSSR) model (see
Figure 2) for HSI super-resolution, that is,
Based on Theorem 3, FGSSR is equivalent to the Schatten-
p norm
when
and
, where
is the unfolding matrix of
along the third dimension,
,
, and
come from the SVD of
, i.e.,
. From Observation model (
9), the upsampled LR-HSI consists of three components: the latent HR-HSI, the difference image, and noise. This suggests that a significant portion of spectral information of the latent HR-HSI is preserved in the upsampled LR-HSI. Consequently, it can be inferred that both LR-HSI and HR-HSI are situated inside the identical spectral subspace. As a result, the low-dimensional spectral subspace
can be learned by SVD and Theorem 3 from the upsampled LR-HSI. We represent the upsampled LR-HSI using its SVD as follows:
where
and
are semiunitary, and
is a diagonal matrix comprising the singular values sorted in a non-increasing manner. By saving the
largest singular values and deleting the remaining (
) smallest ones from
, a low-rank approximation of
can be given by
Based on Theorem 3, the spectral subspace
can be achieved by
Compared to Direct subspace representation model (
12), the FGSSR model applies the group sparsity regularization to the coefficients
, resulting in certain frontal slices in
being quickly forced to zero during the iterative process. This allows the FGSSR model automatic reduction in the number of non-zero frontal slices and dynamic approximation of the rank
of
. Regarding to the spectral subspace
, we make two remarks: (1) The spectral subspace
is dynamic since its subspace dimension
reduces with the reduction in the number of nonzero frontal slices of
in the iterative optimization process. (2) As the iteration progresses, the FGSSR model dynamically reduces the number of columns in
, allowing it the consideration as the spectral basis matrix of
. This leads to
inheriting the potential property of the latent HR-HSI
. Therefore, the spatial property in the HR-HSI can be captured and modeled in its low-dimensional coefficient domain, which can reduce the computational complexity.
To sum up, the FGSSR model has the following advantages:
Enhanced Rank Minimization: The FGSSR model corresponds to the Schatten-p norm, which enables it to provide a more precise approximation to rank minimization compared to the nuclear norm.
Reduced Computational Complexity: Unlike other rank minimization problems that require SVD calculation at each iteration, the FGSSR model can be effectively solved through the soft-threshold shrinkage operator or tiny linear equations, resulting in enhanced computational efficiency. In addition, the FGSSR model employs the subspace learning strategy to learn the spectral subspace , which can also reduce the computational complexity.
Automatic Subspace Dimension Selection: Compared to the direct subspace representation model (
12), the FGSSR model (
13) imposes the group sparsity regularization on the coefficients
. This leads to the rapid elimination of specific frontal slices in
during the iterative optimization process, automatically reducing the number of non-zero frontal slices (i.e., subspace dimension
d), and dynamically approximating the rank
r of
.
3.2. Adaptive FGSSR-Based HSI Super-Resolution Model
By incorporating FGSSR model (
13) into Model (
11), we have
where
is the FGSSR model, which serves to depict the high spectral correlations of the target HR-HSI
.
is a regularization term, which serves to capture the self-similarity property of the target HR-HSI
. The idea of applying the TNN-based regularization to the low-dimensional coefficient tensor
to capture the nonlocal self-similarity has been successfully used for HSI super-resolution [
16,
38,
39], but often demands significant computational resources [
50]. In this paper, the TNN is employed to capture the global spatial self-similarity property of the latent HR-HSI by directly performing on the low-dimensional coefficient tensor
. This strategy offers a reduction in computational complexity while effectively preserving the image details. Moreover, through the subspace learning strategy, the spectral subspace
is essentially known. Therefore, the proposed FGSSR-based HSI super-resolution model can ultimately be expressed as follows:
where
, defined in (
4), is the TNN regularization term.
In our proposed method, the spectral correlation property is adaptively and effectively depicted by the FGSSR model, and the spatial self-similarity property is effectively captured by imposing the TNN regularization on the low-dimensional coefficients . Specifically, we first propose the FGSSR model by combining the subspace learning of subspace with the imposition of group sparsity regularization on coefficients . The proposed FGSSR model offers several advantages, including enhanced rank minimization, automatic subspace dimension selection, and low computational cost. These advantages allow us adaptive and effective capture of the high spectral correlations. Furthermore, we introduce TNN regularization on the low-dimensional coefficients to depict the spatial self-similarity prior, which offers high efficiency while successfully maintaining the spatial features. Next, we design an efficient PAM-based algorithm for the proposed FGSSR-based HSI super-resolution model.
3.3. Optimization Algorithm
In this section, we employ the PAM framework [
51] to address our Fusion model (
18). We choose the PAM framework for two reasons. Firstly, the PAM framework has a proven ability that can guarantee convergence to a critical point given specific criteria. Secondly, the PAM framework provides us with an opportunity to update subspace dimension
d at each PAM iteration. It should be noted that
is initialized using the subspace learning strategy and subsequently updated only according to the subspace dimension
d, which is determined by applying the group sparsity regularization to the coefficients
during the iterative optimization process. In other words,
is known with the subspace dimension
d determined in each PAM iteration. As a result, solving Model (
18) can be changed into an alternative optimization of the following subproblems:
where
is the objective function implicitly defined in (
18),
is the variable predicted in the preceding iteration, and
is a proximal parameter. Next, we address the two subproblems by using a two-step iterative scheme:
(1) Step 1: Optimization with respect to
From (
19), we have
where
is the coefficients determined in the preceding PAM iteration. To effectively deal with Problem (
20), the alternating direction method of the multiplier (ADMM) framework [
52] is adopted. By introducing the auxiliary valuables
and
, we obtain the augmented Lagrangian function as follows:
where
and
are Lagrange multipliers, and
is a penalty parameter. The ADMM iteration process is given by
For the
subproblem, we have
Problem (
27) is quadratic, and it can be addressed via the Sylvester matrix equation
where
.
For the
subproblem, we have
The efficient solution to Problem (
29) can be achieved through the utilization of the group sparse soft-threshold shrinkage operator. Subsequently, the closed-form solution for each frontal slice of
can be derived as follows:
where
, and
.
The
subproblem can be updated by
Based on the Definition 4 and Theorem 2, Problem (
31) yields closed-form solution
where
is the SVT operator.
To enhance comprehension, the process of optimizing
subproblem (
20) is succinctly outlined in Algorithm 1.
Algorithm 1: ADMM Algorithm for Subproblem |
Input: , , , , , , , , , , and |
Initialization: Let , , . |
1: while not converged and do |
2: |
3: Update via (28). |
4: Update via (30). |
5: Update via (32). |
6: Update and via (25) and (26), respectively. |
7: Check the convergence condition: . |
8: end while |
Output: Low-dimensional coefficient tensor |
(2) Step 2: Optimization with respect to
Updating the difference image
is solved by
where
represents the difference image determined in the preceding PAM iteration. Similar to the
subproblem, we also employ ADMM to solve
subproblem (
33). By introducing the auxiliary valuable
, we have
where
is the Lagrangian multipliers. ADMM iterations follow this procedure:
The
subproblem can be updated by
Problem (
38) can be effectively solved by FFT strategy:
where
is a tensor with all elements equal to 1.
For the
subproblem, we have
Problem (
40) that involves the
norm is nonconvex. Here, we employ the generalized shrinkage/thresholding (GST) operation [
53] to address Problem (
40) due to its efficiency and low computational cost. Consequently, Nonconvex problem (
40) yields the following solution:
where
is the GST operator.
The process of optimizing the
subproblem is succinctly outlined in Algorithm 2. Combining the solutions for the aforementioned two block subproblems, we derive the PAM algorithm (summarized in Algorithm 3) to solve the FGSSR-based model (
18). Algorithm 3 has a strong capability to converge towards a critical point of the objective function, as proven in [
51].
Algorithm 2: ADMM Algorithm for Subproblem |
Input: , , , , , , , , and |
Initialization: Let , ,, .
|
1: while not converged and do |
2: . |
3: Update via (39). |
4: Update , , via solving (41) with the GST operator. |
5: Update , , via (37). |
6: Check the convergence condition: . |
7: end while |
Output: Difference image |
Algorithm 3: PAM Algorithm for FGSSR-based Model |
Input: , , , , , , w, , , , and |
Initialize: , , ,
, where , , and come from the
SVD of .
|
1: while not converged and do |
2: .
|
3: Update via solving (20) with Algorithm 1.
|
4: Update via solving (33) with Algorithm 2.
|
5: Update , where is the number of the nonzero frontal slices of |
6: Remove the zero frontal slices of .
|
7: Update via (16).
|
8: Check the convergence condition: .
|
9: end while |
Output: Target HR-HSI: |
3.4. Computational Complexity
Let us denote the acquired LR-HSI as and the acquired HR-MSI as . We let represent the subspace dimension. In Algorithm 3, the most computationally intensive parts involve solving the and subproblems. In Algorithm 1, updating involves a group soft threshold shrinkage operation, which costs . Updating costs , and updating costs . In Algorithm 2, updating involves an FFT operation, which costs . The computational complexity of updating through a soft threshold shrinkage operation is . To summarize, the overall computational complexity of Algorithm 3 at every iteration can be expressed as . It is noteworthy that the subspace dimension d reduces as the iteration progresses during the execution of Algorithm 3.
6. Conclusions
In this study, we present a novel FGSSR-based method for HSI super-resolution. In our FGSSR-based method, the high spectral correlation property in HR-HSI is effectively modeled by the proposed FGSSR model, which offers several advantages, including enhanced rank minimization, adaptive subspace dimension determination, and low computational cost. Moreover, we employ TNN regularization on the low-dimensional coefficients to depict the spatial self-similarity property in HR-HSI, which offers high efficiency while effectively preserving the spatial features of HR-HSI. An efficient PAM-based algorithm is developed to solve the FGSSR-based model. The FGSSR-based method is evaluated against the SOTA HSI super-resolution methods using the simulated and real-world datasets. The comparison is conducted through visual and quantitative analysis, which reveals the superior performance of our FGSSR-based method.
Despite these promising results, there are several areas that require further investigation. For example, our proposed method currently concentrates solely on the global spatial self-similarity of HR-HSI, which aids in reducing computational complexity but may limit its ability to reconstruct spatial structure and preserve spatial details. Therefore, our future work will focus on employing the nonlocal self-similarity property of HR-HSI under the FGSSR framework to enhance the performance of HSI super-resolution. In addition, we plan to further extend our proposed method to a broader range of hyperspectral datasets for comparative analysis, aiming to provide a more comprehensive stability assessment of our proposed method. Furthermore, work [
62] provides a novel sight to improve the quality of remote sensing images. Thus, our other future work is focused on enhancing the spatial resolution of HSI from the perspective of dynamic differential evolution theory.