A Frequency Domain Kernel Function-Based Manifold Dimensionality Reduction and Its Application for Graph-Based Semi-Supervised Classification

Abstract

With the increasing demand for high-resolution images, handling high-dimensional image data has become a key aspect of intelligence algorithms. One effective approach is to preserve the high-dimensional manifold structure of the data and find the accurate mappings in a lower-dimensional space. However, various non-sparse, high-energy occlusions in real-world images can lead to erroneous calculations of sample relationships, invalidating the existing distance-based manifold dimensionality reduction techniques. Many types of noise are difficult to capture and filter in the original domain but can be effectively separated in the frequency domain. Inspired by this idea, a novel approach is proposed in this paper, which obtains the high-dimensional manifold structure according to the correlationships between data points in the frequency domain and accurately maps it to a lower-dimensional space, named Frequency domain-based Manifold Dimensionality Reduction (FMDR). In FMDR, samples are first transformed into frequency domains. Then, interference is filtered based on the distribution in the frequency domain, thereby emphasizing discriminative features. Subsequently, an innovative kernel function is proposed for measuring the similarities between samples according to the correlationships in the frequency domain. With the assistance of these correlationships, a graph structure can be constructed and utilized to find the mapping in a low-dimensional space. To further demonstrate the effectiveness of the proposed algorithm, FMDR is employed for the semi-supervised classification problems in this paper. Experiments using public image datasets indicate that, compared to baseline algorithms and state-of-the-art methods, our approach achieves superior recognition performance. Even with very few labeled data, the advantages of FMDR are still maintained. The effectiveness of FMDR in dimensionality reduction and feature extraction of images makes it widely applicable in fields such as image processing and image recognition.

1. Introduction

In modern intelligent image processing and application scenarios, one of the most tricky problems is dealing with high-dimensional data [1]. With the advancements in camera lenses and remote sensing technologies, obtaining high-quality images of a target object has gradually achieved breakthroughs [2]. However, high-quality images imply high-dimensional data, which not only contain a large amount of redundant information but also pose challenges for image processing algorithms due to the heavy computational resources they demand, so called “the curse of dimensionality” [3]. On the other hand, imprudently reducing the dimensionality of data may lead to the loss of valuable discriminative information, leading to the invalidation of recognition [4]. Therefore, a key points is to delicately reduce the dimensions while preserving the major details and relationships between samples.

In the fields of pattern recognition and data mining, dimensionality reduction techniques have been developed for many years and continue to achieve new advancements in various applications [5]. Jose Luis Vieira Sobrinho et al. designed a two-stage dimensionality reduction technique for social media engagement classification [6]. Mahmood et al. put forward a dimensionality reduction-based machine learning technique for tumor classification of RNA data [7]. Barkalov et al. proposed a method for constructing multiple Peano curves to preserve the object proximity information in the course of dimensionality reduction [8]. Jie Li et al. proposes a novel non-negative matrix factorization method with dual-graph regularization constrained for dimensionality reduction and clustering [9]. An algorithm that computes all rough sets constructed for dimensionality reduction was proposed by Yanir et al. [10]. Ramin Heidarian Dehkordi et al. retrieved the wheat crop traits of hyperspectral data using machine learning and dimensionality reduction algorithms [11]. Chao Yao et al. proposed a framework named ColAE for hyperspectral image feature extraction and clustering with auto-encoders and dimension reduction techniques [12]. Md Rashedul Islam et al. designed a dimension reduction method that combines feature extraction and selection for effective image classification [13].

Generally speaking, the goal of dimensionality reduction is to transform high-dimensional data from the original space into a new low-dimensional space by utilizing linear or nonlinear transformation [1]. The most widely known techniques include linear discriminant analysis (LDA) [14], principle component analysis (PCA) [15], robust principle component analysis (RPCA) [16] and manifold learning-based algorithms [17]. LDA is a supervised technique that aims to reduce the dimensionality of high-dimensional pattern samples by projecting them onto an optimal discriminant vector space [18]. This process facilitates the extraction of discriminative information for classification purposes while simultaneously reducing the dimensionality of the feature space through compression. PCA is a linear transformation that selects multiple eigenvectors for feature extraction and dimension reduction [19]. RPCA assumes that observed data are composed of low-rank clean data and sparse noise and it achieves feature extraction and dimensionality reduction by solving an optimization problem [20]. Manifold learning aims to reveal the geometric relationship between samples in a high-dimensional manifold space. ISOMAP computes the geometric distance between two data points by constructing a neighborhood graph and searching for the shortest paths [21]. Locally linear embedding (LLE)  [22] reconstructs the linear weights of two neighbors and maps it to the embedded coordinates. Laplacian eigenmaps (LE) [23] define a heart-kernel function for measuring the distance between two points and finds the best low-dimensional mapping by solving a trace minimization problem.

However, the existing dimensionality reduction techniques basically treat data in the original domain. In other words, the relationships between high-dimensional data are measured based on the distances in the original domain. As is widely known, there are various types of interference that are difficult to capture and eliminate in the original domain, but that are much easier to address in the frequency domain. For instance, in the case of facial images affected by extensive occlusions, such as shadows and obstructions, the interference is not sparse and possesses high energy. This makes it challenging to separate in the original domain. PCA and RPCA might recognize the occlusions as the principle components due to their high energy. Distance-based algorithms may erroneously classify different objects that are corrupted by similar occlusion into the same category.

On the other hand, by transforming the corrupted data into the frequency domain, the key features and interferences can be effectively separated according to their coordinated frequency band, offering valuable guidance for the subsequent feature extraction and dimensionality reduction processes. Inspired by this idea, in this paper, a novel dimensionality reduction algorithm via frequency domain correlationship is proposed, named Frequency domain-based Manifold Dimensionality Reduction (FMDR), which measures the high-dimensional manifold similarities between data points by the frequency domain information. The overall workflow and processing phases are illustrated in Figure 1. Specifically, FMDR includes the following major steps. Firstly, a transformation is proposed, including transforming data into the frequency domain, extracting the discriminative details, reducing the interference and reconstructing datasets. Secondly, a novel kernel function is proposed for measuring the high-dimensional manifold relationship between samples based on frequency domain information. Finally, according to the relationship measured by the new kernel function, a graph structure is constructed by calculating the Laplacian matrix, which can be effectively utilized for finding low-dimensional mapping for the original data. Once the graph structure is obtained, the manifold relationship of the original data can be preserved, which can be conveniently adopted to solve various problems such as classification, clustering and semi-supervised classification.

Integrating frequency domain information to design machine learning models has become a new trend. FreMixer [24] is a lightweight machine learning architecture for short-term waves forecasting, which utilizes frequency domain information for data training. Stuchi, José A et al. integrated a new frequency extractor layer in the deep neural network for feature extraction [25]. However, most existing methods only treat frequency domain information as a feature or input, without effectively incorporating it into the model design and kernel function design.

In recent years, there have been several relevant studies about manifold learning and dimensionality reduction. Wenhui Song et al. introduced spectral information divergence into a neighbor manifold graph for hyperspectral image dimensionality reduction [26]. But, the distance of pairwise data points was calculated by Euclidean distance in the original domain. FMDR conducts a novel kernel function for measuring the similarity between samples based on the frequency domain correlationship, which is more precise, especially when samples are corrupted by interference. Jinghao Situ proposed a manifold learning-based contrastive framework, which applies ISOMAP to extract nonlinear structures and neural networks for comparative learning training [27]. Compared to this framework, our FMDR does not require additional neural networks or training processes and can achieve excellent dimensionality reduction results with minimal computational resource requirements.

2. Materials and Methods

In this section, several related background concepts are briefly introduced, including two-dimensional Discrete Fourier Transformation (2D DFT), high-frequency texture components and Laplacian eigenmaps.

2.1. Related Background

2.1.1. Discrete Fourier Transformation for Two-Dimensional Image

In the field of signal processing, there is a fundamental principle that every signal can be decomposed into the sum of several sine waves, a concept known as the Fourier series. In the field of image processing, a common approach involves treating data as two-dimensional discrete signals, referred to as the spatial domain signals. Consider an object image with dimensions denoted by H and W for its height and width, respectively. The pixels can be organized into a matrix $\mathbf{Pix}$ $\in {\mathbb{R}}^{H\times W}$. The frequency domain signal $\mathbf{Fre}$ of the $\mathbf{Pix}$ can be obtained using the two-dimensional Discrete Fourier Transform (2D DFT):

$$\mathbf{Fre}(u,v)=\sum _{m=0}^{H-1}\sum _{n=0}^{W-1}\mathbf{Pix}(m,n)e^{-j2\pi (\frac{mu}{H}+\frac{nv}{W})},$$

(1)

where $(u,v)$ and $(m,n)$ denote the coordinates in frequency domain and original domain, respectively. Equation (1) can be further expand into a matrix multiplication form, written as:

$$\begin{array}{cc}\hfill \mathbf{Fre}& ={\mathbf{B}}_1\mathbf{Pix}{\mathbf{B}}_2=\hfill \\ \hfill & \left[\begin{array}{cccc}{e}^{-j2\pi \frac{0\times 0}{H}}& e^{-j2\pi \frac{0\times 1}{H}}& \cdots & e^{-j2\pi \frac{0\times (H-1)}{H}}\\ ⋮& ⋮& ⋮& ⋮\\ e^{-j2\pi \frac{(H-1)\times 0}{H}}& e^{-j2\pi \frac{(H-1)\times 1}{H}}& \cdots & e^{-j2\pi \frac{(H-1)\times (H-1)}{H}}\end{array}\right]\hfill \\ \hfill & \times \mathbf{Pix}\times \hfill \\ & \left[\begin{array}{cccc}{e}^{-j2\pi \frac{0\times 0}{W}}& e^{-j2\pi \frac{0\times 1}{W}}& \cdots & e^{-j2\pi \frac{0\times (W-1)}{W}}\\ ⋮& ⋮& ⋮& ⋮\\ e^{-j2\pi \frac{(W-1)\times 0}{W}}& e^{-j2\pi \frac{(W-1)\times 1}{W}}& \cdots & e^{-j2\pi \frac{(W-1)\times (W-1)}{W}}\end{array}\right].\hfill \end{array}$$

(2)

${\mathbf{B}}_1$ and ${\mathbf{B}}_2$ are the base matrices, which only related to the height and width of the image. The transformation indicates that the frequency domain signal $\mathbf{Fre}$ contains all the characteristics of the original signal while ${\mathbf{B}}_1$ and ${\mathbf{B}}_2$ are irrelevant.

2.1.2. High-Frequency Texture Component

Images are often described as a combination of structure and texture components, with each playing a vital role in the overall visual representation. The structure components primarily focus on defining the fundamental characteristics and prominent features present in an image, providing a framework for understanding its basic layout and composition. On the other hand, texture components delve into the finer details and nuances within the image, capturing the subtle variations and complex patterns that contribute to its overall appearance and aesthetic quality (see Figure 2).

By analyzing and interpreting the structure and texture components of an image, researchers and practitioners can gain deeper insights into its content, allowing for more advanced image processing tasks such as segmentation, object recognition and content-based retrieval. Understanding how these components interact and contribute to overall visual perception is essential in various fields such as computer vision, medical imaging, remote sensing and the digital arts.

High-frequency texture components are particularly valuable for identifying subjects within images, like human faces. Various techniques exist for extracting or enhancing these texture components from an image. Leveraging high-frequency texture components helps preserve essential image characteristics, facilitating image recognition and reconstruction [28].

2.2. The Proposed Method

In this section, the detailed design and implementation of FMDR are introduced, including its mathematical expressions, theoretical derivations and its application in semi-supervised classification problems.

In practice, images are usually assembled as a data matrix, where each column (or row) represents a vectorized sample. At first, a transformation will be adopted for extracting the discriminative information and reducing the interference in frequency domain. The transformation consists of several steps. Firstly, the vectorized samples are reshaped as images according to their height and width. Then, each image in the original spatial domain is converted into the frequency domain using 2D DFT. Based on the distribution of crucial morphological features, a designed filter is employed to extract the high-frequency texture components. Lastly, in order to maintain consistency in the data structure, the extracted features are then vectorized and assembled into a new matrix.

For the ith sample in the data matrix $\mathbf{X}$, the high-frequency texture components ${\mathbf{HF}}_i$ of pixel matrix ${\mathbf{Pix}}_i$ can be written in the matrix multiplication form according to (2):

$${\mathbf{HF}}_i={\mathbf{B}}_1\mathbf{K}{\mathbf{B}}_2\odot {\mathbf{B}}_1{\mathbf{Pix}}_i{\mathbf{B}}_2.$$

(3)

Base matrices ${\mathbf{B}}_1$ and ${\mathbf{B}}_2$ are introduced in 2, which are the irrelevant bases. $\mathbf{K}$ refers to the convolution kernel of the filter in the original domain. ⊙ is a Hadamard product operator. For each pixel point, the transformation can be rewritten in a more detailed form:

$${\mathbf{HF}}_i(u,v)=\sum _{m=0}^{H-1}\sum _{n=0}^{W-1}\mathcal{F}{\mathbf{Pix}}_i(m,n)e^{-j2\pi (\frac{mu}{H}+\frac{nv}{W})},$$

(4)

where $\mathcal{F}$ represents the designed filter. In practical applications, the distribution of valuable information within images is different across every dataset. There is no universally applicable filter design that can suit all scenarios. In this study, we employed the Butterworth filter, which is a high-pass filter and primarily involves two adjustable parameters: the order and the cut-off frequency. Based on experimental findings, the characteristics of high-frequency texture components are correlated with the filter’s parameters. Specifically, the quantity of features is linked to the cut-off frequency, while the texture thickness depends on the order of the filter. With lower parameters, a significant amount of irrelevant information and interference is retained. As the parameters increase, more low-frequency interference is filtered out, making the texture components more pronounced. However, excessively large parameters result in the abandonment of a substantial portion of the information. Samples from diverse datasets contain noise with varying distributions and structures. To achieve optimal performance, comprehensive consideration of filter parameter tuning is essential in practical applications. After extracting the high-frequency texture components corresponding to n samples, we can finally reassemble the frequency information into a data matrix by vectorization:

$$\mathbf{HF}=\left[vec\left({\mathbf{HF}}_1\right),vec\left({\mathbf{HF}}_2\right),\cdots ,vec\left({\mathbf{HF}}_n\right)\right].$$

(5)

Subsequently, a novel kernel function is designed for measuring the relationship between samples based on the frequency domain information. The key to manifold learning algorithms is to determine the distance measurement of samples in the high-dimensional manifold space. A more precise distance measure leads to better preservation of sample relationships after dimensionality reduction. With conventional graph-based methods, the distance between samples is calculated by the absolute distance. The heat kernel, proposed in Laplacian eigenmaps [23], is widely adopted for estimating manifold similarities, which are defined as:

$$s({\mathbf{x}}_i,{\mathbf{x}}_l)=\mathbf{exp}(-\frac{{\left|\left|{\mathbf{x}}_i-{\mathbf{x}}_l\right|\right|}_F^2}{t}),$$

(6)

where $\mathbf{exp}$$\left(x\right)$ refers to $e^{\left(x\right)}$ and e is the natural logarithm. t is known as the heat kernel parameter and ${\left|\right|.\left|\right|}_F$ is the Frobenius norm. As can be seen, the similarity between samples ${\mathbf{x}}_i$ and ${\mathbf{x}}_l$ is calculated based on the absolute distance ${\left|\left|{\mathbf{x}}_i-{\mathbf{x}}_l\right|\right|}_F^2$. Approaches that rely on absolute distances face a common challenge in manifold space: samples may have close absolute distances while exhibiting significant actual differences, posing difficulties for accurate recognition. For instance, when considering two different faces both wearing the same scarf, distance-based algorithms may mistakenly recognize them as belonging to the same category.

In order to leverage the discriminative information provided by the frequency domain, we innovatively proposed a kernel function for measuring the relationships between samples. Firstly, the similarity degree between frequency domain signal ${\mathbf{HF}}_i$ and ${\mathbf{HF}}_l$ corresponding to ${\mathbf{x}}_i$ and ${\mathbf{x}}_l$ is calculated according to their correlationship, denoted as:   

$$Cor({\mathbf{HF}}_i,{\mathbf{HF}}_l)=\frac{vec{\left({\mathbf{HF}}_i\right)}^{T}vec\left({\mathbf{HF}}_l\right)}{\sqrt{\parallel {\mathbf{HF}}_i{\parallel }_F^2{\parallel {\mathbf{HF}}_l\parallel }_F^2}}.$$

(7)

Then, the similarity ${\mathbf{S}}_{il}$ is calculated by:

$${\mathbf{S}}_{il}=exp(-\frac{t}{Cor({\mathbf{HF}}_i,{\mathbf{HF}}_l)}).$$

(8)

It is worth noting that, different from the absolute distance, the correlation coefficient between two signals will be larger when their similarity is higher. Hence, if ${\mathbf{x}}_i$ and ${\mathbf{x}}_l$ are more similar in the high-dimensional space, their discriminative frequency domain signals ${\mathbf{HF}}_i$ and ${\mathbf{HF}}_l$ should be closer to each other and the correlation $Cor({\mathbf{HF}}_i,{\mathbf{HF}}_l)$ should be larger, resulting in a greater similarity ${\mathbf{S}}_{il}$. Conversely, when ${\mathbf{x}}_i$ and ${\mathbf{x}}_l$ are dissimilar, their similarity ${\mathbf{S}}_{il}$ will be smaller.

By substituting Equation (7) into (8), we can finally get the complete definition of the similarity measurement, named the high-frequency kernel function (HFK):

$${\mathbf{S}}_{il}=HFK({\mathbf{HF}}_i,{\mathbf{HF}}_l)=exp(-\frac{t\sqrt{\parallel {\mathbf{HF}}_i{\parallel }_F^2{\parallel {\mathbf{HF}}_l\parallel }_F^2}}{vec{\left({\mathbf{HF}}_i\right)}^{T}vec\left({\mathbf{HF}}_l\right)}).$$

(9)

The high-frequency kernel function satisfies the definition and requirements of kernel functions, guaranteed by the $\mathit{Mercer}$ theorem and Proof 1.

Proof. 

HFK satisfies the definition of the kernel function. based on the $\mathit{Mercer}$ theorem. The following two conditions need to be verified:

The symmetry: for arbitrary pairwise input ${\mathbf{X}}_i$ and ${\mathbf{X}}_j$, HFK calculates the distance according to the correlation (7), which satisfies $Cor({\mathbf{X}}_i,{\mathbf{X}}_j)=Cor({\mathbf{X}}_j,{\mathbf{X}}_i)$. Therefore, $HKF({\mathbf{X}}_i,{\mathbf{X}}_j)=HKF({\mathbf{X}}_j,{\mathbf{X}}_i)$, proving that HFK is a symmetric function.

The positive semi-definiteness: for any input ${\mathbf{X}}_i$ and ${\mathbf{X}}_j$, the Gram matrix of $HFK({\mathbf{X}}_i,{\mathbf{X}}_j)$ is a positive semi-definite matrix. Suppose matrix $\widehat{\mathbf{H}}$ is the Gram matrix of $HFK({\mathbf{X}}_i,{\mathbf{X}}_j)$:

$$\widehat{\mathbf{H}}=\left[\begin{array}{cccc}HFK({\mathbf{X}}_1,{\mathbf{X}}_1)& HFK({\mathbf{X}}_1,{\mathbf{X}}_2)& \cdots & HFK({\mathbf{X}}_1,{\mathbf{X}}_n)\\ HFK({\mathbf{X}}_2,{\mathbf{X}}_1)& HFK({\mathbf{X}}_2,{\mathbf{X}}_2)& \cdots & HFK({\mathbf{X}}_2,{\mathbf{X}}_n)\\ ⋮& ⋮& ⋮& ⋮\\ HFK({\mathbf{X}}_n,{\mathbf{X}}_1)& HFK({\mathbf{X}}_n,{\mathbf{X}}_2)& \cdots & HFK({\mathbf{X}}_n,{\mathbf{X}}_n)\end{array}\right].$$

(10)

For any n dimensional column vector $\mathbf{a}$, it should satisfy:

$${\mathbf{a}}^T\widehat{\mathbf{H}}\mathbf{a}\ge 0.$$

(11)

Expanding the above equation for each element, we have:

$$\sum _{i=1}^n\sum _{j=1}^n{a}_i{a}_{j}HFK({\mathbf{X}}_i,{\mathbf{X}}_j)\ge 0.$$

(12)

By substituting the definition of HFK (9), we can get:

$$\sum _{i=1}^n\sum _{j=1}^n{a}_i{a}_{j}exp(-\frac{t\sqrt{\parallel {\mathbf{HF}}_i{\parallel }_F^2{\parallel {\mathbf{HF}}_l\parallel }_F^2}}{vec{\left({\mathbf{HF}}_i\right)}^{T}vec\left({\mathbf{HF}}_l\right)}\ge 0$$

(13)

Based on the property of the natural logarithm exponential function, the above inequality is clearly satisfied.

Hence, with fulfilling the above two conditions, according to the $\mathit{Mercer}$ theorem, HFK satisfies the definition of the kernel function.   □

After computing the similarity between all pairs of samples based on HFK function (9), the similarity matrix (also known as the adjacency matrix) S can be obtained. According to the matrix S, the topological graph structure of the data in manifold space can also be simultaneously constructed.

Following the guiding ideology of Laplacian eigenmaps, we need to find out the best low-dimensional mapping of the original data which maintains the manifold relationships in high-dimensional space. Given n samples in the original space $\mathbf{X}=\left\{{\mathbf{x}}_i\cdots {\mathbf{x}}_n\right\}\in {\mathbb{R}}^{d_o\times n}$. Our goal is to find out the best mapping $\mathbf{F}=\left\{{\mathbf{f}}_i\cdots {\mathbf{f}}_n\right\}\in {\mathbb{R}}^{d_t\times n}$ in the target low-dimensional space, where $d_t\ll d_o$. While reducing the dimensions, it is essential to preserve the relationships between data points in the original high-dimensional manifold space, therefore, an objective minimization function is constructed with the constraint of matrix S:

$$Obj=min\sum _{i,j}^n{\Vert {\mathbf{f}}_i-{\mathbf{f}}_j\Vert }_2^2{\mathbf{S}}_{ij},$$

(14)

where $\left|\right|·{\left|\right|}_2$ is the vector norm for calculating the Euclidean distance of two vectors. The target is to minimize the value of $Obj$. Constrained by the similarity matrix S, those two sample points that are close in the original high-dimensional space must also be close in the mapping of the low-dimensional space, otherwise this will result in a larger penalty. Given the definition of Laplacian matrix $\mathbf{L}=\mathbf{D}-\mathbf{S}$, where D is a diagonal matrix and its elements are defined as the sum of rows (or columns) of S, named the degree matrix. Then, the objective function (14) can be rewritten in the matrix form:

$$Obj=minTr\left({\mathbf{F}}^T\mathbf{LF}\right),s.t.{\mathbf{F}}^T\mathbf{DF}=\mathbf{I}.$$

(15)

I is the identity matrix. The equivalence of (14) and (15) is guaranteed by the following Proof 2.

Proof. 

The equivalence of (14) and (15). Equation (14) can be further derived:

$$\begin{array}{cc}\hfill Obj& =min\sum _{i,j}^n{\Vert {\mathbf{f}}_i-{\mathbf{f}}_j\Vert }_2^2{\mathbf{S}}_{ij}\hfill \\ \hfill & =min\sum _{i,j}^n\left({({\mathbf{f}}_i-{\mathbf{f}}_j)}^T({\mathbf{f}}_i-{\mathbf{f}}_j)\right){\mathbf{S}}_{ij}\hfill \\ \hfill & =min\sum _{i,j}^n({\mathbf{f}}_i^T{\mathbf{f}}_i-2{\mathbf{f}}_i^T{\mathbf{f}}_j+{\mathbf{f}}_j^T{\mathbf{f}}_j){\mathbf{S}}_{ij}\hfill \\ \hfill & =min\sum _i^n{\mathbf{f}}_i^T{\mathbf{f}}_i{\mathbf{D}}_{ii}+\sum _j^n{\mathbf{f}}_j^T{\mathbf{f}}_j{\mathbf{D}}_{jj}-2\sum _{ij}^n{\mathbf{f}}_i^T{\mathbf{f}}_j{\mathbf{S}}_{ij}\hfill \\ \hfill & =min2\sum _{ij}^n{\mathbf{f}}_i^T{\mathbf{f}}_j({\mathbf{D}}_{ij}-{\mathbf{S}}_{ij})\hfill \\ \hfill & =min2\sum _{ij}^n{\mathbf{f}}_i^T{\mathbf{f}}_j{\mathbf{L}}_{ij}\hfill \end{array}$$

(16)

Noting that ${\mathbf{f}}_i$ and ${\mathbf{f}}_j$ are column vectors of matrix F, expressing the above conclusion in a matrix form yields:

$$Obj=minTr\left({\mathbf{F}}^T\mathbf{LF}\right),$$

(17)

which is equivalent to Equation (15).    □

It is worth noting that, in order to enhance the robustness against outliers and improve computational efficiency, non fully connected graphs are commonly constructed in practice. Generally speaking, the range of neighborhoods can be defined based on the k-nearest or $ϵ$-distance rules. The similarities of vertexes beyond the neighborhood’s range will be assigned as zero.

Once the minimization of $Obj$ is achieved, the optimal low-dimensional mapping F is obtained, with a significantly reduced dimensionality while preserving the manifold relationships. The problem (15) can be converted into the eigenvalue decomposition problem and be solved effectively. The low-dimensional data F can be conveniently applied to solve various problems, such as feature selection, clustering and semi-supervised classification.

In order to further demonstrate effectiveness, FMDR is adopted for a semi-supervised classification problem. Given n data points with l labeled samples and u unlabeled samples ($1\le l\le n$), where ${\mathbf{Y}}_l$ represents the label matrix such that each row ${\mathbf{y}}_i$ denotes the label vector corresponding to the ith sample. The matrix $\mathbf{F}$ is composed of two parts, $\mathbf{F}=[{\mathbf{F}}_l;{\mathbf{F}}_{n-l}]$. The semi-supervised graph-based learning problem can be formulated as follows:

$$\underset{\mathbf{F}}{min}Tr\left({\mathbf{F}}^T\mathbf{L}\mathbf{F}\right)s.t.{\mathbf{F}}_l={\mathbf{Y}}_l,$$

(18)

where the Laplacian matrix $\mathbf{L}$ consists of four parts:

$$\mathbf{L}=\left[\begin{array}{cc}{\mathbf{L}}_{l,l}& {\mathbf{L}}_{l,n-l}\\ {\mathbf{L}}_{n-l,l}& {\mathbf{L}}_{n-l,n-l}\end{array}\right].$$

(19)

Based on the conclusion from [29], there is a harmonic solution to problem (18), which is defined as:

$${\mathbf{F}}_{n-l}=-{\left({\mathbf{L}}_{n-l,n-l}\right)}^{-1}{\mathbf{L}}_{n-l,l}{\mathbf{Y}}_l.$$

(20)

Therefore, the labels of unknown samples can be directly indicated by ${\mathbf{F}}_{n-l}$. The algorithm of FMDR for semi-supervised classification is summarized in Algorithm 1.

Algorithm 1: FMDR for semi-supervised classification

One of the prominent advantages of FMDR is its minimal use of computing resources. The high-frequency texture components are extracted by 2D DFT and filtering, with a computational complexity of $O\left(n\right)$. The similarity is calculated by the HFK, which adopts matrix multiplication instead of a loop algorithm, only with the computational complexity $O\left(1\right)$. The complexities of dimensionality reduction and classification are $O\left(n\right)$ and $O\left(3n\right)$, respectively. Finally, the total computational complexity of FMDR is $O(n+1+n+3n)$, which is simplified to $O\left(n\right)$ with the assumption that n will be large.

3. Experiments and Discussions

A convenient method for evaluating the effectiveness of dimensionality reduction techniques is to apply them to address specific recognition problems and compare the predictive accuracy. A higher accuracy implies more precise and effective dimensionality reduction and feature extraction. In this paper, FMDR is utilized for solving the image semi-supervised classification problem. To better demonstrate the advantages and superiority of our approach, several public facial image datasets are selected as the experimental objects. Facial images not only contain abundant morphological texture information, but also commonly suffer from occlusions such as shadows, sunglasses and scarfs, which are challenging for conventional techniques.

In this section, experiments are designed from several aspects to demonstrate the mechanism and evaluate the performance of FMDR. At first, the preparations of implementation are introduced, including the details of objective datasets, the performance indicators and their definitions and the comparison baselines and the state-of-the-art methods. Subsequently, the implementations and results of the experiments are introduced.

3.1. Preparations

This section presents the preparations for the experiments, detailing the datasets selected, the indicators for evaluating performance and the comparison algorithms.

3.1.1. Datasets

Several real-world image datasets are selected to evaluate the effectiveness of our methods. The public facial datasets are widely adopted for machine learning recognition tasks, including AT&T, ORL, AR, Yale and YaleB. AT&T [30], which is a commonly used dataset for face recognition, consisting of 400 grayscale images of 40 individuals. Each person has 10 images captured under different conditions (varying angles, lighting, facial expressions, etc.). ORL [31] is another widely used dataset for face recognition, comprising 400 grayscale images of 40 individuals. Each person has 10 images taken at different times and under different states (with glasses, without glasses, etc.). The AR (Augmented Reality) Face Database [32] contains 2600 images of 126 individuals. Each person has 20 images, with 10 unoccluded and 10 occluded (with obstructions such as glasses, scarfs, etc.). Here, we conduct a mini-batch of AR with 260 samples of 10 individuals. Yale [33] consists of 165 grayscale images of 15 individuals. Each person has 11 images captured under varying lighting conditions. YaleB [34] is an extension of the Yale Face Database, containing 2540 grayscale images of 38 individuals. Each person has 64 images covering different lighting conditions and facial expressions. The partial samples are presented in Figure 3.

3.1.2. The Filter and Parameters

In FMDR, the primary concept revolves around describing the relationships between samples by leveraging the advantage of separating features and noise in the frequency domain. The selection and design of filters offer considerable flexibility, allowing for customization based on specific applications and research objectives. The Butterworth filter is selected in this study mainly because of its suitability for extracting morphological features from facial images. Its adjustable cut-off frequency and order parameters allow for tailored adjustments, making it particularly suited for this classification task.

In practice, the distribution of valuable information in facial images varies across different datasets. There is no universally applicable filter design that can be suitable for all situations. Based on the experimental experience, the features of high-frequency texture components are relevant to the parameters of the filter. Specifically, the quantity of features is related to the cut-off frequency and the texture thickness is dependent on the order of the filter. When parameters are small, most of the useless information and interference are retained. As parameters increase, more low-frequency interference is filtered out and the texture components become more obvious. However, when parameters are set to be too large, most of the information is abandoned. Therefore, in practical applications, for optimal recognition performance, if there is significant interference in the image, it is advisable to increase the parameters appropriately to filter out more noise. Conversely, reducing the parameters would be preferable to retain more information. The different results extracted by different parameters setting are presented in Figure 4.

Samples from diverse datasets contain noise with different distributions and structures. In order to achieve the best clustering performance, the tuning of the filter’s parameters should be comprehensively considered in practice.

3.1.3. Comparison Methods and Performance Indicators

To comprehensively evaluate the effectiveness of the proposed method, different types of algorithms are selected for comparison. Firstly, the conventional techniques are selected as the baselines, including the semi-supervised k-nearest neighbors (${\mathrm{KNN}}_{\mathit{semi}}$) [35] and the semi-supervised k-means (${\mathrm{KMeans}}_{\mathit{semi}}$) [36], both based on the absolute distance. Subsequently, the dimensionality reduction technique is utilized for semi-supervised problem in the same experimental condition. Robust principal component analysis (RPCA) [37] and non-negative matrix factorization (NMF) [38] are two widely adopted techniques for dimension reduction. RPCA assumes that the observed data consists of the clean low-rank component and a sparse noise part and finds the low-rank component of the data by solving a rank-minimization objective function. NMF model decomposes the data matrix into two non-negative matrices to search for the low-dimensional representation. Laplacian eigenmaps (LE) [23] search for the best low-dimensional mapping according to the manifold Gaussian kernel in the data. At last, two state-of-the-art algorithms are also employed. ABNMTF is a semi-supervised non-negative matrix tri-factorization model that learns the similarity graph via adaptive k-nearest neighbors [39]. Deep Autoencoder-like Nonnegative Matrix Factorization (DANMF) consists of an encoder and decoder to learn the hierarchical mapping of the original data [40].

There are several evaluation indicators designed for evaluating the performance of semi-supervised classification algorithms. Four common indicators, namely accuracy (Acc), precision (Pre), recall (Rec) and F1 score, are chosen in this paper. In the pattern recognition task, the prediction of a single sample can fall into one of four situations: true positive ($TP$), false negative ($FN$), false positive ($FP$) and true negative ($TN$). Accuracy is calculated as the ratio of correctly classified samples to the total number of samples, defined as follows:

$$\begin{array}{c}\hfill Accuracy=\frac{TP+TN}{TP+FN+FP+TN}.\end{array}$$

(21)

Precision indicates the proportion of correctly classified positive samples to the total number of samples determined as positive by the algorithm. It is defined as follows:

$$\begin{array}{c}\hfill Precision=\frac{TP}{TP+FP}.\end{array}$$

(22)

Recall refers to the proportion of correctly classified positive samples to the total number of positive samples in the dataset. It is defined as follows:

$$\begin{array}{c}\hfill Recall=\frac{TP}{TP+FN}.\end{array}$$

(23)

The F1 score is the harmonic mean of precision and recall. It is calculated using the following formula:

$$\begin{array}{c}\hfill F1=\frac{2\times Precision\times Recall}{Precision+Recall}.\end{array}$$

(24)

The values of these indicators are recorded as percentages. In general, a higher value indicates a better performance in clustering tasks.

3.2. Visualization of Dimensionality Reduction

One direct way to evaluate the effectiveness of the dimensionality reduction technique is to visualize the low-dimensional data. As is widely known, the original data points are distributed in a high-dimensional manifold space, which is difficult to visualize and observe. One of the primary goals of dimensionality reduction algorithms is to identify the most significant feature dimensions within the data and map the samples according to these dimensions, so called principle components. Once the dimensionality of the data points is reduced, the retained features represent the principal components of the dataset. By examining the distribution of samples along these principal component dimensions, we can clearly observe the variations in the key discriminated features of the samples.

Therefore, FMDR is employed to reduce the dimensionality of the image dataset, compressing it into two dimensions and visualizing the results. As shown in Figure 5, when the samples of the ORL dataset are compressed to two dimensions by FMDR, “facial orientation angle” and “eye closure degree” are recognized as the two major feature dimensions. Samples can be effectively distinguished and categorized along these two dimensions.

Similarly, we utilize FMDR to compress the dimensions of the YaleB dataset to two dimensions as illustrated in Figure 6. It can be observed that, “size of shadow coverage area” and “angle of the shadow” are considered to be the most discriminative major components that effectively identify every sample.

In summary, FMDR effectively identifies the principal component features of data while reducing the dimensionality of samples, thus providing an accurate low-dimensional representation.

3.3. Semi-Supervised Classification for Facial Images

In this section, we utilize the FMDR for facial image semi-supervised classification. A conventional way to evaluate the effectiveness of the dimensionality reduction is to classify the data in the low-dimensional space. As the effectiveness of dimensionality reduction improves, the discriminability of the resulting low-dimensional data becomes stronger, thereby leading to higher classification accuracy. In this experiment, to smooth the bias, we conducted cross-validation that contains 100 repeated trials, randomly partitioning 20% of the data as labeled samples. The labeled and unlabeled samples are assembled into a unified data matrix, serving as input for all semi-supervised algorithms. The performances of semi-supervised classification for facial images is presented in

Table 1. The performances of semi-supervised classification for facial images.

Dataset	Acc
	FMDR	${\mathbf{KNN}}_{\mathit{semi}}$	${\mathbf{KMeans}}_{\mathit{semi}}$	RPCA	NMF	LE	ABNMTF	DANMF
AT&T	79.81	73.89	57.00	33.02	50.31	34.35	78.35	51.95
ORL	64.44	45.59	42.25	32.33	31.85	31.85	41.30	33.60
AR	97.14	77.00	38.46	50.23	52.23	50.23	67.62	61.27
Yale	64.44	45.59	28.48	32.85	31.85	31.85	41.30	33.60
YaleB	76.18	43.30	32.14	31.38	32.63	28.07	45.56	29.05
Dataset	Pre
	FMDR	${\mathrm{KNN}}_{\mathit{semi}}$	${\mathrm{KMeans}}_{\mathit{semi}}$	RPCA	NMF	LE	ABNMTF	DANMF
AT&T	82.81	75.80	71.23	28.31	45.00	30.92	81.56	88.49
ORL	62.76	58.24	60.05	46.71	31.87	40.99	55.79	46.54
AR	97.87	75.54	42.62	48.58	50.34	53.58	70.15	63.11
Yale	62.76	58.24	35.09	46.71	31.87	40.99	55.79	46.54
YaleB	77.32	41.46	48.30	32.96	31.24	31.99	57.47	29.77
Dataset	Rec
	FMDR	${\mathrm{KNN}}_{\mathit{semi}}$	${\mathrm{KMeans}}_{\mathit{semi}}$	RPCA	NMF	LE	ABNMTF	DANMF
AT&T	79.12	79.12	62.64	31.58	48.17	35.97	78.47	55.78
ORL	62.11	52.20	46.63	34.36	29.61	38.33	47.61	31.84
AR	97.29	78.43	42.76	49.63	48.41	51.04	67.77	63.40
Yale	62.11	52.20	30.82	34.36	29.61	38.33	47.61	31.84
YaleB	78.29	43.41	33.12	29.84	31.45	28.84	46.04	29.77
Dataset	F1 score
	FMDR	${\mathrm{KNN}}_{\mathit{semi}}$	${\mathrm{KMeans}}_{\mathit{semi}}$	RPCA	NMF	LE	ABNMTF	DANMF
AT&T	40.19	39.35	34.01	14.54	23.13	16.73	41.23	35.77
ORL	32.37	29.28	24.10	22.05	15.99	22.34	25.23	18.45
AR	53.91	43.07	20.59	26.71	26.15	28.84	47.39	42.81
Yale	32.37	29.28	17.86	22.05	15.99	22.34	25.23	18.45
YaleB	39.68	21.50	19.23	15.68	16.42	16.06	34.01	23.38

The highest values are emphasized in bold font.

. The highest values are indicated in bold font.

The experimental results indicate that FMDR achieves the best performance in the majority of cases. Specifically, FMDR gains the highest values in AR, with 97.14% Acc, 97.87% Pre, 97.29% Rec and 53.91% F1 score. Also, for YaleB, FMDR achieves 76.18% Acc, 77.32% Pre, 78.29% Rec and 39.68% F1 score. The samples of those two datasets exhibit significant large area occlusions, such as masks, sunglasses, scarves and shadows, which are difficult to handle in the original domain and invalidate the existing algorithms. The superiority of our algorithm on these datasets demonstrates its ability to accurately extract key information from samples in the frequency domain, thereby mitigating the interference of occlusions and improving recognition.

Samples from the AT&T and ORL datasets exhibit variations in multiple angles, leading to challenges in accurate categorization. As can be seen, the conventional dimensionality reduction techniques fail to recognize the same object at different angles, resulting in very low accuracy.

On the other hand, compared to RPCA and LE, FMDR achieves superiority of 46.79% and 45.46% in Acc, 54.5% and 51.89% in Pre, respectively. For the ORL dataset, FMDR gains 64.44% Acc and 62.76% Pre, which are 32.11% and 16.05% higher than RPCA, respectively. This phenomenon indicates that FMDR can accurately describe the manifold relationship between samples according to the high-frequency kernel function, which leads to increased similarity between different angle samples of the same object, enabling correct classification.

Furthermore, the standard deviations of 100 repeated trials in th AR dataset are recorded in

Table 2. Standard deviations of 100 repeated trials in the AR dataset.

Algorithm	Acc	Pre	Rec	F1
FMDR	2.34	3.27	3.43	2.87
KNN	4.24	5.63	6.02	3.50
Kmeans	3.62	3.53	3.69	3.77
ATNMTF	5.22	5.89	7.34	5.35
LE	3.55	4.17	4.53	2.95
NMF	4.32	5.37	5.13	3.06
RPCA	3.94	4.55	4.58	3.26
DANMF	3.02	4.21	4.88	3.23

The highest values are emphasized in bold font.

. The units are the same as the indicators, presented as percentages, with bold highlighting indicating optimal performance. As the results indicate, compared to the baseline method and SOTA method, the FMDR algorithm achieves the lowest standard deviation of results across multiple iterations, indicating the higher stability of the algorithm and better performance obtained in repeated experiments.

3.4. Algorithm Performance with Changes in Labeled Data Proportion

For semi-supervised algorithms, one of the best evaluation perspectives is the ability of the algorithm to maintain its effectiveness as the proportion of labeled data varies. According to our analysis, the proposed approach effectively captures the intrinsic manifold relationship within the original data in the frequency domain, thereby achieving accurate mapping in low-dimensional space. Different from the supervised classification techniques, despite relying on only a small amount of labeled data, the algorithm can still accurately classify samples based on precise low-dimensional mapping relationships. In this section, experiments are conducted to compare the performance variation of our FMDR with comparison algorithms under varying proportions of labeled data. Specifically, six levels of labeled data proportions are set from small to large: 5%, 10%, 15%, 20%, 25% and 30%. In the semi-supervised classification experiments conducted on four datasets, the accuracy variations of FMDR and the comparative algorithms are recorded in

Table 3. The accuracy with different proportions of labeled data.

Dataset	Ratio	Acc
		FMDR	${\mathbf{KNN}}_{\mathit{semi}}$	${\mathbf{KMeans}}_{\mathit{semi}}$	RPCA	NMF	LE	ABNMTF	DANMF
ORL	5%	52.50	48.89	22.50	35.83	45.56	35.28	51.39	52.22
	10%	55.83	46.11	33.00	35.83	46.11	36.11	52.22	53.61
	15%	63.86	53.87	32.75	46.69	60.76	49.07	58.33	58.77
	20%	71.91	66.77	42.25	46.46	51.23	30.88	57.99	59.63
	25%	68.55	68.55	45.50	59.45	64.60	60.48	61.19	62.66
	30%	70.44	64.56	45.75	58.48	64.97	60.63	62.77	63.84
AR	5%	74.80	53.20	22.31	30.00	30.40	30.40	47.60	44.00
	10%	92.61	74.03	25.77	46.98	41.30	43.53	53.04	49.00
	15%	94.57	75.11	27.31	49.33	48.87	50.45	62.53	51.67
	20%	97.14	77.00	38.46	50.23	52.80	50.23	67.62	61.27
	25%	96.94	81.63	38.85	57.25	57.95	57.00	73.85	56.36
	30%	96.26	81.05	40.38	60.21	64.25	58.51	72.13	75.44
Yale	5%	52.00	41.33	16.67	25.33	25.33	25.33	32.00	20.67
	10%	56.00	45.33	22.03	24.00	24.67	24.67	35.33	25.67
	15%	58.52	45.26	25.39	38.52	33.58	35.04	39.71	28.53
	20%	64.44	45.26	28.48	32.85	31.85	31.85	41.30	33.60
	25%	57.02	48.36	28.27	38.52	45.60	40.32	44.63	41.20
	30%	62.50	51.22	30.60	38.21	37.90	38.02	41.94	42.80
YaleB	5%	53.93	24.29	13.67	22.70	27.59	22.02	19.95	22.53
	10%	60.09	32.42	21.13	26.72	29.17	26.16	26.35	25.76
	15%	68.45	41.03	24.36	31.91	34.47	33.11	42.40	28.08
	20%	76.18	43.30	32.14	31.38	32.63	28.07	45.56	29.05
	25%	77.08	50.11	37.30	37.38	41.03	37.90	49.02	31.50
	30%	77.32	52.86	40.36	40.16	43.19	40.20	53.72	32.70

The highest values are emphasized in bold font.

.

Achieving high accuracy with very few labeled datasets presents a challenge for all semi-supervised algorithms. All accuracies decrease due to the insufficient information, with some algorithms even becoming ineffective. However, FMDR demonstrates an advantage in accuracy even with a smaller proportion of labeled data. Despite having only 5% of labeled data, FMDR achieves the highest accuracy in four datasets, with values of 52.50%, 74.80%, 52.00% and 53.93%, respectively. As the proportion of labeled data increases, the accuracy of FMDR also improves. In the AR dataset, with labeled data comprising 10%, FMDR achieved an accuracy of 92.61%, marking a significant increase of 17.84% compared to when only 5% of the data were labeled.

The performances of all algorithms are improved when the labeled data increase. FMDR still maintains its superiority with a higher improvement effect. To visually observe the trends of each algorithm’s changes, a set of line charts are plotted in Figure 7. The red solid line represents FMDR, while other algorithms are represented by different colors of dashed lines. As the figure shows, FMDR consistently maintains its leading advantage at all four indicators as the proportion of labeled data varies.

In summary, FMDR accurately describes the manifold relationships between samples through frequency domain correlationships, enabling precise recognition and classification of samples. Moreover, FMDR demonstrates robustness to labeled data, achieving optimal classification accuracy both when labeled data are scarce and when they are sufficient.

4. Conclusions

In this paper, a novel dimensionality reduction approach is proposed, named FMDR. The existing manifold learning dimensionality reduction techniques mainly treat data in the original space and estimate the similarity between samples by distance such as absolute distance and Gaussian distance. When samples are corrupted by high-energy or non-sparse interference, such as shadows and occlusions, the distance calculated in the original space cannot accurately estimate the manifold relationships between samples. Although such noise is challenging to handle in the original space, it can be efficiently captured and separated in the frequency domain. Inspired by this idea, FMDR transforms the data into the frequency domain for noise filtering and calculates the manifold relationships between samples via a novel kernel function. Then, the graph structure is constructed based on the correlationship in the frequency domain and can be conveniently utilized to find the accurate low-dimensional representation and solve recognition tasks.

In order to evaluate the effectiveness of FMDR comprehensively, several experiments are conducted in the public image datasets. Firstly, dimensions of samples from ORL and YaleB are compressed into two dimensions by FMDR and visualized by the feature dimensions. The illustrations indicate that FMDR is able to recognize the major components for distinguishing samples and dimensionality reduction. Even with the interference such as shadows, angles and various expression, samples can be effectively distinguished in the low-dimensional space. Further, FMDR is employed to solve the semi-supervised classification problems in five datasets. According to four indicators, our approach achieves the best classification performances in the most cases compared to the baselines and the state-of-the-art algorithms. At last, by setting up labeled datasets with varying proportions, FMDR demonstrates its superiority and stability of the recognition performance as the proportion of labeled data increases from low to high.

In conclusion, FMDR leverages the correlationship in the frequency domain of samples to obtain the intrinsic high-dimensional manifold relationship, which enables FMDR to find an optimal low-dimensional representation of the original data. According to its outstanding dimensionality reduction performance and operational efficiency, FMDR can be conveniently applied across multiple domains. It can be utilized for preprocessing tasks such as feature extraction and compressed sensing for high-dimensional data, as well as for recognition and clustering of multi-type data. This facilitates the development of subsequent intelligent algorithms and applications.