DSCEH: Dual-Stream Correlation-Enhanced Deep Hashing for Image Retrieval

Abstract

Deep Hashing is widely used for large-scale image-retrieval tasks to speed up the retrieval process. Current deep hashing methods are mainly based on the Convolutional Neural Network (CNN) or Vision Transformer (VIT). They only use the local or global features for low-dimensional mapping and only use the similarity loss function to optimize the correlation between pairwise or triplet images. Therefore, the effectiveness of deep hashing methods is limited. In this paper, we propose a dual-stream correlation-enhanced deep hashing framework (DSCEH), which uses the local and global features of the image for low-dimensional mapping and optimizes the correlation of images from the model architecture. DSCEH consists of two main steps: model training and deep-hash-based retrieval. During the training phase, a dual-network structure comprising CNN and VIT is employed for feature extraction. Subsequently, feature fusion is achieved through a concatenation operation, followed by similarity evaluation based on the class token acquired from VIT to establish edge relationships. The Graph Convolutional Network is then utilized to enhance correlation optimization between images, resulting in the generation of high-quality hash codes. This stage facilitates the development of an optimized hash model for image retrieval. In the retrieval stage, all images within the database and the to-be-retrieved images are initially mapped to hash codes using the aforementioned hash model. The retrieval results are subsequently determined based on the Hamming distance between the hash codes. We conduct experiments on three datasets: CIFAR-10, MSCOCO, and NUSWIDE. Experimental results show the superior performance of DSCEH, which helps with fast and accurate image retrieval.

1. Introduction

In the era of big data, the explosive growth of image data shared on social networks has generated significant interest in large-scale image retrieval [1]. To address the challenge of accurate retrieval while ensuring efficient computation, there has been a surge in interest towards approximate nearest neighbor (ANN) search [2]. By leveraging the clustered distribution of large-scale data in space, ANN aims to identify potential neighboring data items instead of focusing solely on returning the most probable matches. This approach provides enhanced retrieval efficiency, albeit with a tolerable loss in accuracy. Among various ANN methods, hash learning [3,4,5] has emerged as a prominent research area due to its ability to achieve low storage requirements and facilitate high-speed retrieval.

The idea of hash learning revolves around the process of mapping high-dimensional image data to low-dimensional hash codes while preserving both the semantic information of the image itself and the similarity between images. Existing hashing methods can be divided into shallow methods and deep-learning-based methods. Shallow methods [6,7,8] learn hash functions by manually extracting image features. However, these methods are constrained by the limitations of the semantic description capability inherent in the features themselves, hindering their practical application effectiveness. In contrast, deep-learning-based methods have demonstrated notable advancements in image hashing. Deep-learning-based methods [3,9] leverage Convolutional Neural Networks (CNNs) to extract image features, followed by low-dimensional mapping using Fully Connected  (FC) layers and the Sigmoid function. The overall model is then optimized through a loss function. Deep-learning-based methods can learn more discriminative hash codes through end-to-end training, thereby enhancing the performance of image retrieval. Figure 1 shows the workflow of deep-learning-based methods. As shown in Figure 1, deep hashing involves the conversion of all images into hash codes. The retrieved items’ hash codes are compared with the hash codes stored in the database and subsequently sorted based on the Hamming distance. The retrieval results are then returned accordingly.

Currently, deep hashing methods predominantly rely on Convolutional Neural Network (CNN) as the image feature extractor, employing CNN to extract local features for low-dimensional mapping. With the emergence of Vision Transformer (VIT) [10,11,12], VIT outperforms CNN in various computer vision tasks (image classification, object detection, etc.), and some studies [1,13,14,15] try to use VIT to replace CNN for feature extraction and achieve better results. However, existing approaches typically focus on utilizing a single type of feature (local or global) of the image for low-dimensional mapping. When using CNN to extract local features for low-dimensional mapping, the size of the receptive field limits the global representation ability of the generated hash codes. On the other hand, ViT models global dependencies through a self-attention mechanism and can effectively capture the global features of images. ViT has significant advantages in processing complex image structures and long-distance dependencies compared to CNN. However, this approach overlooks the local details of the image, diminishing the distinguishability between the foreground and background. Consequently, the sole use of any form of image feature negatively impacts the quality of the generated hash codes. Furthermore, existing models [16,17] primarily focus on correlation optimization through manually designed loss functions, without considering the reinforcement of correlation optimization at the model architecture level, which may not fully exploit the potential of model architecture-level enhancements. Due to the inherent limitations in considering all possible scenarios during the design of the loss function, certain situations are inevitably overlooked. And the loss function can only constrain the image similarity of pairs or triples and can only optimize the local data distribution relationship. Consequently, the distribution of the generated hash codes may significantly deviate from the distribution of data in the high-dimensional image space, leading to a decline in retrieval performance.

Regarding these issues, we propose a novel deep hashing framework, Dual-Stream Correlation-Enhanced Deep Hashing (DSCEH), for large-scale image retrieval tasks. The core idea of this framework is to leverage both local and global features of images through a dual-network structure, followed by correlation optimization using a Graph Convolutional Network (GCN) [18]. Specifically, our approach involves constructing a parallel dual-network architecture that integrates both CNN and VIT as complementary image feature extractors. CNN excels in capturing local details, while VIT is proficient in modeling global dependencies across images. First, we extract local features using CNN and global features using VIT separately from the input images. These features are then fused through a concatenation operation, which enhances the representation capability by leveraging both local discriminative details and global context. This integration enriches the semantic information encoded in the low-dimensional binary hash codes. Subsequently, GCN is employed to optimize the correlation among images. Here, the fused features are treated as nodes, and edge relationships are established based on similarity evaluations using global representations derived from VIT. This process enhances feature propagation and facilitates effective information transfer between images, aiming to maximize the alignment of data distributions between the low-dimensional hash space and the high-dimensional image space. By combining the strengths of CNN and VIT in feature extraction and leveraging GCN for correlation optimization, our method not only enhances the discriminative power of generated hash codes but also effectively addresses the challenge of aligning data distributions across different dimensional spaces.

The rest of this paper is organized as follows. We summarize previous research on deep hashing and review the application of Graph Convolutional Networks in the field of data retrieval in Section 2. In Section 3, we give an overview of our framework through formal definitions. Section 4 reports the datasets, experiment settings, compared methods, evaluation criteria, results, and discussions. Finally, we conclude this paper in Section 5.

2. Related Work

In this section, we introduce the related work of image retrieval based on deep hashing, and the application of GCN in the field of data retrieval.

2.1. Deep Hashing for Image Retrieval

As Convolution Neural Network (CNN) continues to advance, and with the rise of Vision Transformer (VIT), most deep hashing methods [1,3,19,20,21] consider one of two backbones for feature extraction. On the one hand, recent works [9,22,23,24,25] have extensively shown that CNN obtains a compact vector representation of local image neighborhoods utilizing hierarchical convolution operations. Song et al. [22] designed a deep adversarial network with CNN as the backbone to embed images into binary hash codes in an unsupervised manner. Zhang et al. [23] combined CNN and KL divergence (KLD) to improve feature robustness by gradually reducing intra-class KLD. On the other hand, VIT aggregates compressed image segmentation patches and obtains global representation through the self-attention module. Qiao et al. [9] deployed an end-to-end CNN-based Siamese network for a specific application of face image retrieval to related videos. Guerin et al. [24] proposed a multi-input neural network architecture, which uses different CNN architectures to extract image features to enrich the information contained in image features. Ng et al. [25] used Vgg19 [26] as the backbone and enhanced the complementarity between features by training hash codes at different levels of Vgg19. A number of studies [1,21] have explored the feature extraction model with VIT as the backbone network and achieved an impressive retrieval effect compared to the CNN-based network. As mentioned above, both CNN-based models and VIT-based models typically utilize a single form of image feature for low-dimensional mapping. While these models have achieved promising experimental results, there is a need to enhance image retrieval performance further. To address this, we introduce a novel approach that leverages both image feature forms to produce hash codes with enhanced semantic representation.

2.2. Graph Convolutional Network for Data Retrieval

The Graph Convolutional Network (GCN) utilizes the adjacency matrix of nodes in a graph to update the features of similar nodes. By leveraging the relationships of edges, GCN iteratively propagates information among nodes to preserve the similarity between input data instances. When combined with hash learning [27], GCN effectively integrates similar information into image features, enabling the generation of hash codes that preserve the inherent relationships present in the original images. This integration enhances the performance of hash retrieval, contributing to improved retrieval accuracy and efficiency.

At present, GCN has been widely used in the field of multi-modal data retrieval. Graph Convolution Hashing (GCH) [28] uses GCN to mine the structural similarity and semantic similarity between text data and image data and designs a semantic encoder to guide the feature encoding process, to obtain better hash codes and achieve better retrieval performance. Graph Convolution Network Hashing (GCNH) [29] improves on GCH by introducing asymmetric graph convolution (AGC) and simultaneously convolving input data, anchor graph, and convolutional filters to solve the scalability problem of the hash of affinity graph. Flexible Graph Convolutional Multi-modal Hashing (FGCMH) [30] uses the structural similarity of intra-modality and modality-fused to learn the fusion features, aggregates the feature information of each modality, uses graphs to optimize the correlation in an adaptive weighted manner, alleviates the gap between modalities, and solves the problem of losing some modal features when querying single modal data.

Benefiting from the influence of the above multi-modal hash retrieval methods, we propose the incorporation of GCN into our framework. In contrast to prior studies [17,31,32] that primarily rely on loss functions to constrain hash code similarity, our approach aims to enhance correlation optimization at the model architecture level. This innovative strategy leverages GCN’s capabilities in facilitating information propagation and reinforcing the relationships among image features. By doing so, we strive to ensure that the low-dimensional hash space maintains a data distribution consistent with the high-dimensional image space.

In this section, we give the framework overview and the details of the proposed DSCEH framework.

We first give a formal description of deep-hash-based image retrieval. Given the image training dataset $\mathcal{X}=[{\mathit{x}}_1,{\mathit{x}}_2,\dots ,{\mathit{x}}_n]$ containing n data samples, where ${\mathit{x}}_i\in {\mathcal{R}}^{W\times H\times C}$ is the i-th sample with W width, H height, and C channels, we denote the corresponding label set as $\mathcal{Y}=\{{\mathit{y}}_1,{\mathit{y}}_2,\dots ,{\mathit{y}}_n\}$, where ${\mathit{y}}_i\in {\mathcal{R}}^c$ and c is the number of categories. For any two images in the dataset, we can generate similarity matrix S, where ${\mathit{s}}_{ij}=1$ if ${\mathit{x}}_i$ and ${\mathit{x}}_j$ are similar and where ${\mathit{s}}_{ij}=0$ if ${\mathit{x}}_i$ and ${\mathit{x}}_j$ are dissimilar. The framework aims to learn a hash function that maps images from high-dimensional space to low-dimensional space to generate k-bit hash codes $\mathcal{H}=[{\mathit{h}}_1,{\mathit{h}}_2,\dots ,{\mathit{h}}_n]\in {\{-1,1\}}^{n\times k}$. We formulate the process as:

$$\mathit{h}=\mathtt{sgn}\left(\mathcal{F}\right(\mathit{x}\left)\right)$$

(1)

where $\mathcal{F}$ is the hash function that needs to be optimized by the model and $\mathtt{sgn}(·)$ represents the activation function, which is used to discretize the continuous hash code. The important notations and definitions are summarized in

Table 1. Symbols and Definition.

Symbol	Definition
n	The number of the input image.
W, H, C	The width, height, and channels of image.
$\mathcal{X}$	The matrix of input data, ${\mathcal{R}}^{n\times W\times H\times C}$.
c	The number of categories.
$\mathcal{Y}$	The label of input data, ${\mathcal{R}}^{n\times c}$.
$\mathcal{L}$	The predicted label vector, ${\mathcal{R}}^{n\times c}$.
S	The similarity matrix, ${\mathcal{R}}^{n\times n}$.
$\mathcal{F}$	The hash function.
k	The length of hash bits.
$\mathcal{H}$	The hash codes, ${\mathcal{R}}^{n\times k}$.
P	The length of the patch.
b	The number of the patch.
${\mathit{x}}_p$	The patch vector, ${\mathcal{R}}^{P\times P\times C}$.
${\mathit{x}}_{cls}$	The class token, ${\mathcal{R}}^{P\times P\times C}$.
${\mathit{x}}_{emd}$	Patch embedding vector, ${\mathcal{R}}^{(b+1)\times (P\times P\times C)}$.
$\mathtt{Pos}$	Image’s position information, ${\mathcal{R}}^{(b+1)\times (P\times P\times C)}$.
${\mathit{x}}_{pos}$	Position embedding vector, ${\mathcal{R}}^{(b+1)\times (P\times P\times C)}$.
${\mathcal{S}}_{adj}$	The similarity adjacency matrix, ${\mathcal{R}}^{n\times n}$.
${\mathcal{F}}_{local}$	The local feature of image.
${\mathcal{F}}_{global}$	The global representation of image.
${\mathcal{F}}_{fused}$	The fused feature of image.
${\mathcal{L}}_{sim}$	The Similarity loss.
${\mathcal{L}}_{sem}$	The Semantic loss.
${\mathit{D}}_{ij}$	The Hamming distance.

.

2.3. Overview

Figure 2 shows the overview of our proposed DSCEH consisting of two steps: training and retrieval. In the training step, we first employ the parallel CNN and VIT to extract the input image’s local and global features simultaneously and finally realize feature fusion through the concatenation operation. We use the class token from VIT for similarity estimation and construct a similarity adjacency matrix. Then, we introduce GCN to mine the similarity of the image by using the fusion feature and the similarity adjacency matrix. Finally, we obtain a hash function that can generate efficient and compact hash codes through model optimization. In the retrieval step, we use the model optimized in the training stage to quantify all the images in the test set and the database, retrieve the test set according to the Hamming distance, and finally give feedback on the retrieval performance. In the following, we provide a detailed overview of our proposed framework, including feature extraction, graph-based correlation optimization, and learning to preserve the similarity hash.

3. DSCEH Framework

3.1. Feature Extraction

In this stage, we focus on learning an end-to-end network that effectively produces compact representations of image features. The local features and global representations we consider both have been extensively studied and aim to achieve fusion to enhance the representation learning of the network. This paper designs the parallel dual-network structure, which consists of a CNN branch and a transformer branch, which are respectively constructed by the Alexnet [33] and VTS16 [11]. Finally, we fuse the feature of the two branches through the feature fused unit, and this operation dramatically enhances the global perception ability and local expression details of the network.

3.1.1. CNN Branch

It has the pyramidal model structure and achieves local feature extraction by reducing the resolution of the layer feature mapping through the depth of the network. We deploy the classic model Alexnet [33], with its first five layers and well-trained parameters for local feature extraction. As defined in Alexnet, the network flow can be referred to as Convolution, Relu, and Maxpool. The kernel size and stride of the convolution kernel of each layer are [(11,4), (5,1), (3,1)],  the kernel size of Maxpool is 3, and the stride is 2. Finally, we can obtain the local features ${\mathcal{F}}_{local}\in {\mathcal{R}}^{256\times 6\times 6}$ through the feature mapping formed by such stacked convolution kernels and Maxpool.

3.1.2. Transformer Branch

This branch extracts the global feature of the input image, which consists of the following modules: patch embedding, position embedding, and self-attention encoder. Specifically, given the input image $\mathit{x}$, which is then divided into b patches, i.e., $b=(H\times W)/P^2$, where P is the length of the patch, in this experiment, we divided the 224 × 224 image into 14 × 14 patches, such that the length of each patch is 16. Each image patch ${\mathit{x}}_p\in {\mathcal{R}}^{P\times P\times C}$ is flattened into the 1D vector, and then the class token ${\mathit{x}}_{cls}$ is added to obtain the embedding image patch representation. The embedding vector ${\mathit{x}}_{emd}$ can be formulated as

$${\mathit{x}}_{emd}={\mathit{x}}_{cls}+[{\mathit{x}}_{p1},\dots ,x_{pb}]$$

(2)

where ${\mathit{x}}_{emd}\in {\mathcal{R}}^{(b+1)\times (P\times P\times C)}$. We follow the standards of VIT by adding position embedding Pos with consistent dimensions for each embedding vector to obtain the output vector ${\mathit{x}}_{pos}$ and formulate it as

$${\mathit{x}}_{pos}={\mathit{x}}_{emd}+\mathtt{Pos}$$

(3)

We notice that the transformer encoder generally consists of M blocks, which sequentially include the layer norm (LN), multi-headed self-attention layer (MSA), and multilayer perceptron (MLP). The computation of the m-th transformer block can be formulated as

$$\begin{array}{cc}\hfill & {\mathit{z}}_m^{\prime }=\mathtt{MSA}\left(\mathtt{LN}\left({\mathit{z}}_{m-1}\right)\right)+{\mathit{z}}_{m-1}\hfill \\ \hfill & {\mathit{z}}_m=\mathtt{MLP}\left(\mathtt{LN}\left({\mathit{z}}_m^{\prime }\right)\right)+{\mathit{z}}_m^{\prime }\hfill \end{array}$$

(4)

where $m=[1,2,\dots ,M]$. Finally, we can obtain the class token ${\mathit{x}}_{cls}$ and the global representation ${\mathcal{F}}_{global}$ in this branch.

3.1.3. Feature Fusion Unit

This unit fuses ${\mathcal{F}}_{local}$ and ${\mathcal{F}}_{global}$ into ${\mathcal{F}}_{fused}$ by the concatenation function and finally obtains the fusion feature ${\mathcal{F}}_{fused}$ as follows

$${\mathcal{F}}_{fused}=[{\mathcal{F}}_{global},{\mathcal{F}}_{local}]$$

(5)

In this way, for any image ${\mathit{x}}_i$, we can extract the corresponding fusion data feature ${\mathcal{F}}_{fused}^i$ through the designed dual network and the corresponding transformer class token ${\mathit{x}}_{cls}^i$, which is  used for the subsequent correlation optimization.

The fused feature ${\mathcal{F}}_{fused}$ enhances the retrieval process by integrating both fine-grained local information and broad global context, which improves the discriminative power of the feature representation. This is particularly beneficial in complex retrieval tasks where local and global information is crucial for accurate matching.

3.2. Graph-Based Correlation Optimization

To make full use of the obtained fused features ${\mathcal{F}}_{fused}$ and improve the hash-retrieval accuracy under supervised learning, we exploit GCN to mine correlation information of the image data through the class token ${\mathit{x}}_{cls}$ and the fused feature ${\mathcal{F}}_{fused}$ introduced above. Hence, given the image pair (${\mathit{x}}_i$, ${\mathit{x}}_j$), we can easily obtain the corresponding class token pair (${\mathit{x}}_{cls}^i$, ${\mathit{x}}_{cls}^j$) via the feature extraction process above.

Following the general procedure introduced in [28], we use class tokens ${\mathit{x}}_{cls}$ to calculate the cosine distance between image pairs to construct the similarity adjacency matrix ${\mathcal{S}}_{adj}$ to generate edge relationships between nodes. The correlation constraints between nodes are strengthened based on edge relationships. The specific calculation formula is as follows:

$${\mathcal{S}}_{adj}={\displaystyle \frac{{\mathit{x}}_{cls}^i·{\mathit{x}}_{cls}^j}{\left|\right|{\mathit{x}}_{cls}^i\left|\right|\times \left|\right|{\mathit{x}}_{cls}^j\left|\right|}}$$

(6)

where ${\mathcal{S}}_{adj}\in {\mathcal{R}}^{n\times n}$. With the result of similarity adjacency matrix ${\mathcal{S}}_{adj}$ computation, we adopt a multi-layer GCN structure to utilize fused features with strong semantic correlation to update and generate the interactions to improve the semantic-preservation performance. Specifically, we regard each image as a node, the fused features ${\mathcal{F}}_{fused}$ as node features, and similarity adjacency matrix ${\mathcal{S}}_{adj}$ as edge relationships. We continuously optimize node features based on edge relationships to promote feature flow between nodes. As introduced in [28], we formulate the layer-wise propagation rule as follows:

$${\mathcal{H}}^{(l+1)}=\sigma ({\tilde{\mathcal{D}}}^{-{\displaystyle \frac{1}{2}}}{\tilde{\mathcal{S}}}_{adj}{\tilde{\mathcal{D}}}^{-{\displaystyle \frac{1}{2}}}{\mathcal{H}}^{\left(l\right)}{\mathcal{W}}^{\left(l\right)})$$

(7)

where ${\mathcal{H}}^{\left(l\right)}$ denotes the l-th GCN layer. $\sigma (·)$ represents the activation function, e.g., the Relu function. ${\tilde{\mathcal{S}}}_{adj}$ = ${\mathcal{S}}_{adj}$ + ${\mathcal{I}}_n$ and ${\mathcal{I}}_n$ refers as the unit matrix. $\tilde{\mathcal{D}}$ is the degree matrix of ${\tilde{S}}_{adj}$. ${\mathcal{W}}^{\left(l\right)}$ represents the weight parameter of l-th GCN layer. We dynamically update the network parameters of GCN during training for similarity enhancement.

3.3. Learning to Preserve the Similarity Hash

To further ensure that the hash codes generated by the hash model are used for efficient image retrieval, we set two optimization goals as follows:

• Similarity Loss ${\mathcal{L}}_{sim}$ aims to have the low-dimensional hash space keep similar information consistent with the high-dimensional image space.
• Semantic Loss ${\mathcal{L}}_{sem}$ aims to make the hash code finally generated by the model still effectively maintain the semantic information of the original image.

We jointly optimize the above two loss functions through the Adam optimizer [34] to obtain the best network parameters, and the final loss function ${\mathcal{L}}_{CoDe}$ can be obtained as follows:

$${\mathcal{L}}_{CoDe}=\underset{\mathcal{W}}{argmin}{\mathcal{L}}_{sim}+\sigma {\mathcal{L}}_{sem}+{\alpha \left|\right|\mathcal{W}\left|\right|}_2$$

(8)

where $\sigma$,$\alpha$ is the balance constant, $\mathcal{W}$ is the model parameter, and $\left|\right|·{\left|\right|}_2$ is the $L_2$ norm. We describe the definition of the loss function in detail below.

3.3.1. Similarity Loss ${\mathcal{L}}_{sim}$

As described in [21,35], the goal of hash learning is to maximize the similarity between the high-dimensional feature distribution of the original image (denoted as $\mathcal{A}$) and the Hamming space distribution of the hash code (denoted as $\mathcal{D}$). ${\mathit{a}}_{ij}$ denotes the similarity of original image pair (${\mathit{x}}_i$, ${\mathit{x}}_j$) in $\mathcal{A}$, and ${\mathit{d}}_{ij}$ denotes the similarity in $\mathcal{D}$. Here, we use ${\mathit{s}}_{ij}$ instead of ${\mathit{a}}_{ij}$ and calculate $d_{ij}$ based on the cosine similarity between binary hash codes $h_i$ and $h_j$.

$${\mathit{d}}_{ij}={\displaystyle \frac{cos({\mathit{h}}_i,{\mathit{h}}_j)+1}{2}}$$

(9)

Similar to previous work [36,37], we adopt the JS divergence-based similarity loss function ${\mathcal{L}}_{sim}$ as follows:

$$\begin{array}{cc}\hfill {\mathcal{L}}_{sim}=& \sum _{i=1}^n\sum _{j=1}^n{\mathit{w}}_{ij}{\mathit{s}}_{ij}log{\displaystyle \frac{2{\mathit{s}}_{ij}}{{\mathit{s}}_{ij}+{\mathit{d}}_{ij}}}+\sum _{i=1}^n\sum _{j=1}^n{\mathit{w}}_{ij}{\mathit{d}}_{ij}log{\displaystyle \frac{2{\mathit{d}}_{ij}}{{\mathit{s}}_{ij}+{\mathit{d}}_{ij}}}\hfill \end{array}$$

(10)

where ${\mathit{w}}_{ij}$ refers to the weight of every image pair (${\mathit{x}}_i$, ${\mathit{x}}_j$), and we use the weighted likelihood method to calculate it.

$${\mathit{w}}_{ij}=\left\{\begin{array}{c}\left|\mathcal{S}\right|/|{\mathcal{S}}_1|,{\mathit{s}}_{ij}=1\hfill \\ \left|\mathcal{S}\right|/|{\mathcal{S}}_0|,{\mathit{s}}_{ij}=0\hfill \end{array}\right.$$

(11)

where ${\mathcal{S}}_1=\{{\mathit{s}}_{ij}\in \mathcal{S}:{\mathit{s}}_{ij}=1\}$ denotes the similar items in the dataset; otherwise, ${\mathcal{S}}_0=\{{\mathit{s}}_{ij}\in \mathcal{S}:{\mathit{s}}_{ij}=0\}$ denotes the dissimilar items in the data set.

3.3.2. Semantic Loss ${\mathcal{L}}_{sem}$

We make full use of the given label pair information (${\mathit{y}}_i$, ${\mathit{y}}_j$) to preserve the semantic information of the image. We encode the labels in one-hot form and classify the hash code to obtain classification vectors $\mathcal{L}={\left\{l_i\right\}}_{i=1}^n$, where each $l_i$ is represented by a c-dimensional vector, $l_i=\{l_{i1},l_{i2},...,l_{ic}\}$. Then, we deploy the cross-entropy function to construct the loss function ${\mathcal{L}}_{sem}$:

$${\mathcal{L}}_{sem}=-\sum _{i=1}^n{\mathit{p}}_i\sum _{j=1}^c{\mathit{y}}_{ij}log{\mathit{l}}_{ij}$$

(12)

where ${\mathit{p}}_i$ is the weight parameter to enhance the penalty for misclassified pairs of data, and we use the precision calculation method to calculate it.

$${\mathit{p}}_i={\displaystyle \frac{{\mathit{c}}_t-{\mathit{c}}_{tp}}{{\mathit{c}}_t}}$$

(13)

where $c_t$ is the number of images predicted to be the category of the ith image, and where ${\mathit{c}}_{tp}$ is the number of images in the training batch that are correctly classified into the category of the ith image. After the well-trained process, for any input image ${\mathit{x}}_i$, we can obtain the k-bits hash code ${\mathit{h}}_i$.

3.3.3. Model Optimization

In our optimization process, our model utilizes a feedforward neural network to generate low-dimensional binary hash codes. We compute the discrepancy between the output data and labels using the aforementioned two loss functions. Subsequently, we employ the Adam optimizer to adjust model parameters in reverse. Once the variation of the loss function stabilizes, we consider the optimization process complete. To enhance optimization effectiveness, we employ the hyperparameter optimization library Optuna to optimize hyperparameters $\sigma$ and $\alpha$ within the loss function.

3.4. Hash Model Application for Image Retrieval

Through the training phase, we can obtain the well-trained hash model. We first binarize all images in the database by the hash model. Next, we input the images in the test set to evaluate the performance of the hash model and report the retrieval performance based on the top item of the return list result. It returns the top-K items of similar images based on the Hamming distance. Hamming distance is the different number of bits corresponding to two hash codes with the same length. Here, we use the inner product of the hash code to measure the Hamming distance ${\mathit{D}}_{ij}$, which can be calculated as follows.

$${\mathit{D}}_{ij}=\mathit{k}-{\mathit{h}}_i·{\mathit{h}}_j$$

(14)

4. Experimentation

This section describes the experiment details, including the dataset description, experiment settings, compared methods, and evaluation criteria. Finally, we report the experimental results and employ the discussions.

4.1. Datasets

We conduct our experiments on three popular dataset: CIFAR-10 [16], MSCOCO [38], and NUSWIDE [39]. CIFAR-10 has 60,000 images with 10 categories. MSCOCO contains 132,218 with 80 categories. NUSWIDE has 206,334 images with 21 categories. For experimental comparison, we follow [16] to split the dataset into training dataset, query dataset, and retrieval dataset. We show the details of the three dataset settings in

Table 2. Data statistics of three datasets.

Datasets	#Label	#Train	#Query	#Retrieval
CIFAR-10	10	5000	1000	54,000
MSCOCO	80	10,000	5000	117,218
NUSWIDE	21	10,500	2100	197,374

. In the training phase, we exploit the training dataset and obtain the image data pairs (i.e., ${\mathit{x}}_i$, ${\mathit{x}}_j$) for model training. In the evaluation phase, we generate the database of the retrieval dataset and obtain the performance of the hashing model by querying the database with the query dataset. The specific image examples are shown in the Figure 3.

4.2. Experiment Settings

We implement the proposed framework and other compared methods on the Pytorch platform. In our experiment, we set the random cropping function for data augmentation, and the input images are cropped to (224, 224). For the feature extraction module, we set the Alexnet [33] as the model backbone for the CNN branch. Moreover, we exploit the VTS16 [11] for the VIT branch, and the detailed model setting is as follows: partition patch size is 16, the hidden size of the encoder is 768, the number of heads for multi-head attention is 12, and the number of the transformer blocks is 6. The GCN we use consists of only two layers. The first layer of GCN is to map the 4096 × 2 fused feature of image ${\mathcal{F}}_{fused}$ to 2048 dimensions and the second layer is to map 2048 dimensions to 512 dimensions. For all experiments, we use Adam as the optimizer, the learning rate is set to $1\times {10}^{-5}$, and the batch size is 100.

4.3. Compared Methods

We compare our DSCEH framework with the following deep hashing methods: DSH [40], HashNet [17], DCH [31], IDHN [20], and QSMIH [41]. In these comparative experiments, Alexnet is used as the feature extractor. DSH, HashNet, and DCH utilize the similarity loss and quantization loss to optimize the hash model. IDHN also optimizes its model using the corresponding loss function, but it has been specifically designed for multi-label scenarios. QSMIH employs Quadratic Mutual Information (QMI) to learn a compact code. The comparative experiments conducted primarily focus on optimizing image similarity by employing suitable loss functions, without explicitly considering the impact of network architecture on retrieval performance. To demonstrate the effectiveness of our framework, we compare it with the open-source code of the aforementioned methods.

4.4. Evaluation Criteria

We exploit the popular criteria, mean Average Precision (mAP) as introduced in [42]. Specifically, mAP is the mean of the Average Precision (AP), which can be calculated as follows:

$$\mathtt{AP}={{\displaystyle \frac{1}{R}}}_{rel}\times \sum _{i=1}^K{\displaystyle \frac{R_{reli}}{i}}rel\left(i\right)$$

(15)

where K represents the top K items of the return retrieval list, $R_{rel}$ denotes the number of similar images in the topp K items, $R_{reli}$ is the number of similar images in the first i item and $rel\left(i\right)$ indicates whether the ith image is similar. Moreover, we use the Precision–Recall curve (PR curve) [43] for experimental comparison.

4.5. Experimental Results and Discussion

Table 3. Mean Average Precision (mAP) of deep-hash-based retrieval on three datasets. We take the best results with bold for the mark. Our method performs with the best value of mAP on three datasets.

Datasets	CIFAR-10@54000			MSCOCO@5000			NUSWIDE@5000
Methods	32 bits	48 bits	64 bits	32 bits	48 bits	64 bits	32 bits	48 bits	64 bits
HashNet	0.7686	0.8031	0.7914	0.6801	0.7108	0.7093	0.8237	0.8453	0.8466
DCH	0.7558	0.7455	0.7781	0.6536	0.6540	0.6583	0.7931	07942	0.7865
DSH	0.4255	0.4049	0.4469	0.4314	0.4383	0.4501	0.5627	0.5885	0.5755
IDHN	0.7920	0.7851	0.7909	0.6733	0.6798	0.6802	0.8097	0.8146	0.8189
QSMIH	0.7834	0.8127	0.8157	0.6670	0.6772	0.6877	0.8256	0.8348	0.8410
DSCEH	0.9824	0.9863	0.9946	0.7851	0.7974	0.8007	0.8642	0.8678	0.8761

presents the results of mean Average Precision (mAP) for each hash bit on the three datasets. Additionally, Figure 4 displays the PR curves for each hash bit on these datasets. Based on the experimental results, we make the following observations. Table 3 confirms that DSCEH consistently achieves better retrieval performance across all experimental settings compared to the other methods. Moreover, Figure 4 demonstrates that the curve of DSCEH consistently outperforms the curves of other methods. This indicates that our proposed framework achieves superior performance in terms of precision and recall compared to the alternative methods.

For example, DSCEH achieves the highest mAP value of 0.9824 on CIFAR-10 with 32-bit hash code compared to other hash methods, where the improvement in the proposed method is 0.1990 compared with QSMIH, 0.1904 compared with IDHN, 0.2266 compared with DCH, 0.2138 compared with HashNet, and 0.5569 compared with DSH. On MSCOCO, DSCEH with a 64-bit hash code has an absolute improvement of 0.1130 compared with QSMIH, 0.1205 compared with IDHN, 0.1424 compared with DCH, 0.0914 compared with HashNet, and 0.3506 compared with DSH. On NUSWIDE, DSCEH with a 64-bit hash code has an improvement of 0.0351 compared with QSMIH, 0.0572 compared with IDHN, 0.0896 compared with DCH, 0.0295 compared with HashNet, and 0.3006 compared with DSH.

The better performance of the proposed DSCEH framework can be attributed to the following reasons. Unlike the state-of-the-art methods, DSCEH performs hash mapping considering the simultaneous extraction of local features and global representations of images. More specifically, we design a dual-stream network feature extractor based on VIT and CNN for image feature extraction, which can consider both the global and local features of an image when conducting image retrieval tasks. As a result, it helps to preserve richer semantic information of the original image and generate higher-quality hash codes than methods that just use a single CNN or VIT as a feature extractor, such as DSH and HashNet. In addition, we use the GCN model to explore the potential relationships between images, especially when the relationship is complex. We introduced GCN in the correlation optimization module to enhance the correlation between similar image features. Compared with other methods, such as QSMIH, IDHN, etc., which only optimize by a similarity loss function, our correlation optimization module can better preserve relevant information in high-dimensional space.

Moreover, we derive the following two observations. First, with the increase of hash bits, the retrieval effect of DSCEH will be improved. This proves that more hash bits can better preserve the semantic information of the original high-dimensional image. Second, the mAP of DSCEH on MSCOCO and NUSWIDE is not as high as CIFAR-10 because the properties of the three datasets are different, and they have much higher data volumes and data categories than CIFAR-10. Specifically, MSCOCO and NUSWIDE contain a higher degree of intra-class variability and a more complex set of inter-class relationships, which can challenge the model’s ability to learn discriminative features effectively. Additionally, the increased data volume requires more robust computational resources and optimization techniques to manage the higher dimensionality and variability, which might not be fully leveraged in our current model configuration.

4.6. Ablation Study

4.6.1. On Feature Extraction

To verify the effectiveness of the feature extractor in the DSCEH, we conduct comparative experiments on CIFAR-10 and only keep the CNN branch (denoted as CNN+GCN) or VIT branch (denoted as VIT+GCN) to extract image features. When we only use CNN as the feature extractor, the features extracted by CNN are regarded as the point information of GCN. The similarity adjacency matrix obtained by similarity evaluation of image features is regarded as edge information. When we only use VIT as the feature extractor, the features extracted by VIT are regarded as point information of GCN. The similarity adjacency matrix obtained by similarity evaluation of class token is regarded as edge information.

We report the mAP and the corresponding PR curve of DSCEH with different feature extraction settings in

Table 4. The mAP comparison with different feature extraction and correlation optimization on CIFAR-10.

Discussion	Settings	32 bits	48 bits	64 bits
Origin	VIT+CNN+GCN	0.9824	0.9863	0.9946
Feature	VIT+GCN	0.9147	0.9334	0.9628
	CNN+GCN	0.8185	0.9262	0.8736
Correlation	CNN+VIT	0.8341	0.7121	0.7331

and Figure 5, respectively. From the experimental results, we have the following observation. First, for the model without CNN setting on 32 bits hash code, the value of mAP is reduced from 0.9824 to 0.9147 compared to the original setting, while in the no-VIT setting, the value of mAP is reduced from 0.9824 to 0.8185, and the difference is larger than in the model without the CNN-based feature. Second, for the model without VIT setting, the mAP of 48 bits is 0.0526 higher than the mAP of 64 bits, which shows the instability of the setting. The above experimental results demonstrate the effectiveness of our dual-network feature extractor.

4.6.2. On Correlation Optimization

In addition, in order to test the effectiveness of GCN in the correlation optimization process, we conducted an ablation study without the GCN module (denoted as CNN+VIT) on CIFAR-10 for a comparative test. We report the experimental results with different correlation optimization setting in Table 4. Similarly, the value of mAP in each hash bit length setting significantly decreases. It is obvious that the mAP of 32 bits is much higher than the other lengths. The shortest length but the best effect proves that the model is very unstable without GCN for correlation optimization. We also present the corresponding PR-curve in Figure 6. Hence, we can demonstrate the effectiveness of our designed network.

4.7. Impact on the Number of Vit Encoders

More importantly, since the running time of VIT is much longer than that of CNN. To test our feature extractor’s efficiency, we also explore the effect of the number of encoders in VIT on the performance of our model. As shown in

Table 5. The mAP comparison with different number of VIT Encoder on CIFAR-10.

#VIT Encoder	32 bits	48 bits	64 bits
6	0.9824	0.9863	0.9946
9	0.9892	0.9953	0.9971
12	0.9889	0.9962	0.9981

, on CIFAR-10, when the number of Encoders is 6, the retrieval performance is already excellent, and the value of mAP reaches 0.9877. Even if the number of encoders is increased, the retrieval effect cannot be significantly improved but will increase the running time of the model. This demonstrates that our method can deliver excellent retrieval results without the need for an excessive number of encoders. Furthermore, it operates efficiently during practical usage, avoiding lengthy running times.

5. Conclusions

This paper proposes a novel dual-stream correlation-enhanced deep hashing framework (DSCEH) for large-scale image retrieval tasks. Specifically, we use CNN and VIT to extract local features and global representations of images and perform feature fusion so that the generated hash codes contain richer semantic information. In addition, we introduce GCN to enhance the similarity check of images so that the hash codes can better maintain the similarity relationship of the original space. The entire framework is optimized for training in an end-to-end manner. We conducted extensive experiments on three datasets, CIFAR-10, MSCOCO, and NUSWIDE. The experimental results demonstrate that the retrieval performance of our framework is remarkable compared to other deep hashing methods. Looking ahead, future research will concentrate on refining our model architecture to reduce computational complexity while enhancing retrieval effectiveness. Specifically, we aim to explore adaptive learning techniques for the better handling of intra-class variability in large datasets and investigate ways to incorporate domain-specific priors or constraints to further refine retrieval accuracy.