Robust Hashing via Global and Local Invariant Features for Image Copy Detection

Some typical schemes employ the moments, histogram, Tucker decomposition, Multi-dimensional Scaling (MDS) or Global Neighborhood Structure (GNS) to extract global features for hash computation. For instance, Reference [2] uses the Quaternion Zernike Moments (QZMs) to produce a hashing scheme. This scheme has good discrimination, but the robustness ought to be improved. To enhance the robustness, Reference [16] extracts a hash from a third-order tensor by Tucker decomposition. This scheme is robust to common digital operations, but it can only resist small angle rotation. To resist large angle rotation, Reference [17] computes Color Vector Angles (CVAs) of color image, employs the Histogram of CVAs (HCVAs) as image feature and compresses it with DCT. This scheme has excellent rotation robustness, but the discrimination ought to be improved. In another work [18], the polar coordinate transformation and DFT are combined to construct the rotation-invariant secondary image and then the MDS is employed to create a hash from the secondary image. This scheme has excellent robustness to rotation, but the discrimination ought to be enhanced. Recently, Reference [19] exploited Dominant Neighborhood Structure (DNS) and GNS to derive global features for hash generation. This hashing scheme has good application in Tor domain identification, but the time efficiency is too low.

These hashing schemes utilize local features to generate a hash. Common-used local features include local feature points, block features, and ring features. For example, Reference [20] uses the endpoints of linear targets to design a hashing scheme. This scheme has desirable robustness against gamma correction and geometric attack. In Reference [21], the Random Walk (RW) theory and the statistical features of blocks are employed to form a hash. This scheme is effective in security. The reference [22] proposes an adaptive local feature extraction technology to construct a multi-scale hashing scheme. However, the robustness of this scheme is not satisfactory. To enhance robustness, Reference [23] employs the important DCT coefficients from each image block to make a hash. This scheme reaches the expected robustness of brightness adjustment and contrast adjustment. In another work [24], the Canny operator and DWT are employed to calculate a hash. This scheme performs well in image quality assessment. In Reference [25], the DCT coefficients and LLE technology are utilized to form a hash. This scheme has promising robustness. In Reference [26], a new scheme by combining the Center-Symmetric Local Binary Pattern (CSLBP) and SVD is introduced. But the discrimination of this scheme needs to be enhanced. To enhance discrimination, Reference [27] uses the singular values of Quaternion SVD (QSVD) to construct a hash. This scheme has excellent discrimination, but the robustness is not optimal yet. To take full account of the robustness of geometric distortions, Reference [28] extracts mean, variance, kurtosis and skewness of image rings to form a hash. This scheme is robust to geometric distortions. In another work [29], Ring Partition (RP) theory and Non-negative Matrix Factorization (NMF) are employed to compute a hash. This scheme has good robustness. Recently, in Reference [30], the Saliency Map (SM) and Isometric Mapping (Isomap) are jointly used to generate a robust hash. This scheme obtains a relatively preferable balance between discrimination and robustness. In another work [31], a novel scheme with the Feature Map (FM) and Two-Dimensional Principal Component Analysis (2D PCA) is introduced. This scheme has good detection performance for single distortion, but it does not consider the issue of multiple distortions.

These image hashing schemes combine local and global features to produce a hash. For example, Reference [15] calculates a hash by combining Zernike moment, texture features and position features of the salient region. This scheme has a good application for tampered image detection. To improve robustness, Reference [32] proposes a Fourier Mellin Transform (FMT)-based hashing scheme. This scheme has favorable robustness, but there is still room for improving discrimination. In another work [33], the FMT and Fractal Coding (FC) are utilized to generate a hash. This scheme has excellent robustness, but the discrimination ought to be improved. To enhance discrimination, Reference [34] employs local energy features and three-dimensional global statistical features to generate a hash. This scheme has high spatio-temporal efficiency, but security issue is not considered. To ensure security, Reference [35] utilizes the local features of Three-dimensional Space Angle (3DSA) and the global features of Cool and Warm Hue opposite Color Quaternion (CWHCQ) to form a hash. This scheme has a good key dependency, but its classification performance is far from adequate. To improve classification performance, Reference [36] designs a novel hashing scheme based on two-dimensional DCT and GLCM theory. This scheme obtains a relatively preferable classification performance. In another work [1], the Hu invariant moment and two-dimensional DCT are employed to form a hash. This scheme performs well in copy detection. Recently, Reference [37] combines the KAZE feature and Singular Value Decomposition (SVD) to generate a hash. This scheme has desirable robustness.

The categories and core techniques of some famous schemes are presented in Table 1. In summary, there is still much scope for getting a preferable classification performance and making good performance in copy detection, especially for detecting image copies with multiple distortions. In this work, we present a new hashing scheme with global and local invariant features for copy detection.

The input image is firstly converted to the size of \(M_{0} \times M_{0}\) . Subsequently, the global features are calculated by the PQFT and GLCM. These features can enhance discrimination without reducing robustness. Calculations of PQFT and GLCM are explained below.

The computation of the local invariant features includes three phases. The first phase is to select the \(\mathbf {V}\) component with the size of \(M_{0} \times M_{0}\) in HSV color space for extracting stable image features. The second phase is to learn discriminative local invariant features via KPCA. The third phase is to calculate the vector distances in the KPCA domain for producing compact local features. The KPCA and vector distance calculation are explained in detail below.

The global feature sequence and the local invariant feature sequence are concatenated to produce the feature sequence \(\mathbf {g} = \lbrack g(1), g(2),\ldots , g(L)\rbrack\) , where the length is \(L = 16 + N\) . To guarantee security, the pseudo-random generator is exploited to permute the elements of \(\mathbf {g}\) . This can be achieved below. First, a secret key is used as a seed of the random generator to produce \(L\) pseudo-random numbers. Second, these pseudo-random numbers are sorted to form an ordered sequence. Third, the original positions of the sorted numbers are recorded in a sequence \(\mathbf {r}_{1}\) . Finally, the permutation can be done by the following equation:in which \(r_{1}(j)\) and \(r_{2}(j)\) are the \(j\) th elements of \(\mathbf {r}_{1}\) and \(\mathbf {r}_{2}\) , respectively. Clearly, there are \(L!\) possibilities of random permutation. Without the correct key, it is difficult to generate the correct encrypted feature. Therefore, the encrypted feature sequence \(\mathbf {r}_{2}\) can be formally defined as follows:

Ordinal measures [49] are useful compact features for many robust image applications [50], such as iris recognition and copy detection. To make a short and robust hash, the ordinal measures are exploited to quantize the encrypted feature vector \(\mathbf {r}_{2}\) . To do so, the elements in the encrypted feature sequence are firstly sorted in ascending order. Next, their positions in the sorted sequence are selected as the compact features.

To make easy understanding of the ordinal measures, an example is presented in Table 2, in which the first row is the original sequence, the second row is the ascending sorted sequence, and the last row is the ordinal measures of the sequence in the first row. Obviously, the first element of the original sequence is 7, which is at the third position of the sorted sequence. Therefore, its ordinal number is 3. Similarly, the second element of the original sequence is 3, which is at the first position of the sorted sequence. Thus, its ordinal number is 1.

The final hash h can be written as follows:where \(h(j)(1 \le j \le L)\) is the position of \(r_{2}(j)\) in the sorted feature sequence. Therefore, our hash contains L integers.

Our scheme consists of four phases. They are global feature calculation, local invariant feature computation, encryption, and ordinal measures. To improve readability, the pseudo-code of our scheme is described in Algorithm 1.

The L₁ norm is a well-known distance for measuring similarity and its inputs are generally the sequences of integers or decimals. As the L₁ norm is simple and effective and the hash sequence of our scheme consists of integers, we select the L₁ norm as the metric for determining the similarity of two hash sequences. Suppose that \(\mathbf {h}_{1}\) and \(\mathbf {h}_{2}\) are two hash sequences. Thus, the L₁ norm \(\zeta \left(\mathbf {h}_{\mathbf {1}},\mathbf {h}_{\mathbf {2}} \right)\) is defined as follows:in which \(h_{1}(j)\) and \(h_{2}(j)\) are the jth elements of \(\mathbf {h}_{1}\) and \(\mathbf {h}_{2}\) , respectively. The smaller the L₁ norm is, the more similar their corresponding images are. A threshold \(T\) can be employed to judge whether two images are similar or not.

The Kodak database¹ is exploited to evaluate the robustness of our scheme. This database has 24 color images. For each image, there are 100 similar versions, which are formed by 16 multiple distortion types, including Image Deformation with Rotation (IDR) (angle: 30 \(°\) )+Salt and Pepper Noise (SPN) (density: 0.01)+BA, IDR+SPN+Contrast Adjustment (CA), IDR+SPN+Gamma Correction (GC), IDR+SPN+Gaussian Low-pass Filtering (GLF), IDR+SPN+JC, IDR+SPN+Image Scaling (IS), IDR+SPN+Speckle Noise (SN), IDR+SPN+AWGN, IDR+AWGN (variance: 0.01)+BA, IDR+AWGN+CA, IDR+AWGN+GC, IDR+AWGN+GLF, IDR+AWGN+SPN, IDR+AWGN+JC, IDR+AWGN+IS, IDR+AWGN+SN. In the experiment, the total number of similar images is 2,400 pairs. In summary, 2,400 + 24 = 2,424 images are employed. Table 3 lists the specific distortions and their parameter settings.

Table 4 shows the Mean of Median Values (MMV), Mean of Lower Quartiles (MLQ), and Mean of Upper Quartiles (MUQ) of L₁ norms under multiple distortions with different parameters to further validate the robustness of our hashing. Clearly, the MUQs of multiple distortions are all less than 600. The results show that when the threshold is 600, the proposed scheme can correctly identify 95.75% similar images. When the threshold is chosen as 700, our scheme can obtain a correct detection rate of 98.38%. Therefore, our scheme has good robustness according to the high correct detection rate of the multiple distortions.

The VOC2007 database² is used to check the discrimination of our scheme. There are 5,011 color images in the VOC2007 database. The L₁ norms between each pair of hash codes in this database are produced. Hence, the total number of the distances can be established by the combinatorial number \(C(5,011, 2)= 5,011(5,011-1)/2=12,552,555\) . Figure 5 is the distribution of these distances. The calculation results show that the median value, the lower quartile and the upper quartile of the L₁ norms are 1,342.0, 1,236.0, and 1,444.0, respectively, which are all larger than the biggest MUQ (562.5) of the similar images. This illustrates that our scheme is discriminative.

To view our performances under different block sizes, the Receiver Operating Characteristics (ROC) graph [51] is employed. The ordinate axis and the abscissae axis are defined by \(P_1\) and \(P_2\) as follows:

A small \(P_2\) value means a good discrimination, while a large \(P_1\) value means a good robustness. A set of points with coordinates \((P_{1}, P_{2})\) are employed to plot a curve of ROC graph. If there are two curves in a graph, then the curve near the upper-left corner has better performance than that away from it. In addition, the Area Under the ROC Curve (AUC) is computed for performance comparison, where the scope of AUC is [0, 1]. The curve with a big AUC is better than the curve with a small one.

In this section, the used datasets are the same with the image databases of robustness in Section 4.1 and discrimination in Section 4.2. Note that block size is varied and the settings of other parameters remain unchanged. Figure 6 shows the curves under different block sizes. The curve of 64 \(\times\) 64 is nearer the upper left corner than those of other sizes. Furthermore, the AUCs of different block sizes are calculated. The AUC values of 256 \(\times\) 256, 128 \(\times\) 128, 64 \(\times\) 64, 32 \(\times\) 32, and 16 \(\times\) 16 are 0.15438, 0.77170, 0.99999, 0.99985, and 0.99977, respectively. It is found that the AUC of 64 \(\times\) 64 is larger than the values of other block sizes. This means that the classification performance of the block size 64 \(\times\) 64 is superior to those of 256 \(\times\) 256, 128 \(\times\) 128, 32 \(\times\) 32, and 16 \(\times\) 16. The computational time of 256 \(\times\) 256, 128 \(\times\) 128, 64 \(\times\) 64, 32 \(\times\) 32, and 16 \(\times\) 16 is 0.1005, 0.1163, 0.1453, 0.2601, and 1.0620 s, respectively. This means that the selection of 64 \(\times\) 64 is the optimal choice. The hash lengths of the selections of 256 \(\times\) 256, 128 \(\times\) 128, 64 \(\times\) 64, 32 \(\times\) 32, and 16 \(\times\) 16 are 20, 32, 80, 272, and 1040 integers, respectively. Table 5 lists the performance comparison under different block sizes. Hence, the block size 64 \(\times\) 64 can provide the best performance of the proposed scheme.

A key parameter of our scheme is the dimension selection in the KPCA, i.e., the parameter \(d\) . In the experiments, the \(d\) value is chosen from the set \(\left\lbrace 2, 3, 4, 5, 6 \right\rbrace\) , and other parameter settings remain the same. Figure 7 shows the ROC curves under different dimensions. The quantitative results show that the AUC values of the dimension values of 2, 3, 4, 5, and 6 are 0.99905, 0.99995, 0.99999, 0.99994, and 0.99996, respectively. The AUC value of \(d\) =4 is greater than those of other dimensions. The computational time of the dimension values of 2, 3, 4, 5, and 6 is 0.1453, 0.1597, 0.1625, 0.1686, and 0.1694 s, respectively. As to the computational time, there is little difference between different dimensions. Performance comparisons are summarized in Table 6. Our scheme achieves better performances in both classification performance and computational time when the dimension value is 4.

To ensure the security of our scheme, the similarity between different hashes created by different keys should be small. In our scheme, the key dependence means that the \(L_{1}\) norms between the hash codes of different keys are very large. The six test images named “Airplane,” “Baboon,” “House,” “Peppers,” “Splash,” and “Tree” are employed to compute the \(L_{1}\) norms between hash codes generated by different keys. First, the hash of each image is generated by the correct key. Second, 100 wrong keys are used to compute 100 different hash codes of each image. Finally, the \(L_{1}\) norms between hashes of the correct key and the incorrect keys are computed. Figure 8 shows the \(L_{1}\) norms between the hashes generated by different keys. In Figure 8, the \(y\) -axis describes the \(L_{1}\) norm and the \(x\) -axis describes the index of the wrong key. The minimum \(L_{1}\) norm is greater than 1,700, which shows that our scheme has good key dependence.

The image databases described in Sections 4.1 and 4.2 are used to evaluate the classification performance of our scheme. Specifically, 2,424 images are used for robustness and 5,011 images are taken for discrimination. The ROC graph is still chosen to conduct visual comparison. In Figure 9, all evaluated schemes’ curves are drawn in the same graph for easy comparison. For a better view of these curves, their local details are enlarged and drawn in the same graph. Clearly, the curve of our scheme is nearer to the upper-left corner than those of the compared schemes. The visual comparison shows that the classification performance of our scheme outperforms those of the compared schemes. For further analysis, the quantitative metric called AUC is also computed.

The experimental results show that the AUC of our scheme is 0.99999. However, the AUCs of the RW scheme, SVD-CSLBP scheme, HCVA-DCT scheme, MDS scheme, FM-2D PCA scheme, DCT-GLCM scheme, our scheme only with global features, and our scheme only with local features are 0.98298, 0.87868, 0.93143, 0.96125, 0.997642, 0.99913, 0.15394, and 0.99981, respectively. Obviously, the AUC of our scheme is bigger than those of the compared schemes. The AUC comparison also illustrates that our scheme is superior to all compared schemes. Our scheme obtains the competitive advantage of classification performance because of the following reasons. The global feature calculation with PQFT and GLCM can enhance discrimination without reducing robustness. Since the KPCA can maintain the geometric relationships within image, the local invariant features learned with KPCA can guarantee discrimination.

Time performance is measured by the computational time of calculating a hash code. The results show that the computational time of our scheme is 0.1453 s. The computational times of the RW scheme, SVD-CSLBP scheme, HCVA-DCT scheme, MDS scheme, FM-2D PCA scheme, DCT-GLCM scheme, our scheme only with global features, and our scheme only with local features are 0.0587, 0.1213, 0.0328, 0.4285, 0.0847, 0.1211, 0.0671, and 0.1024 s, respectively. Obviously, our scheme is quicker than some compared schemes. Compared with some dimension reduction-based schemes, such as the MDS scheme, our scheme is much quicker.

Storage performance is determined by the required bits of a hash code. In the experiment, 5,011 different images in VOC2007 are utilized to analyze the needed bits of storing our scheme. As each hash includes 80 integers, there are \(\text{5,011} \times 80 = \text{400,880}\) integers in total. According to the ordinal measures, the maximum hash value is 80. As \(80 \lt 128 = 2^{7}\) , only 7 bits are used to store a hash element. Hence, there are \(80 \times 7 = 560\) bits for our hash. As for our scheme only with global features, only 4 bits are used to store a hash element because of \(16 = 2^{4}\) . Thus, there are \(16 \times 4 = 64\) bits for a hash. As for our scheme only with local features, only 6 bits are used to store a hash element because of \(64 = 2^{6}\) . As a result, there are \(64 \times 6 = 384\) bits for a hash. For the SVD-CSLBP scheme and the HCVA-DCT scheme, their lengths are 64 and 20 floating-point numbers. Note that a floating-point number requires 32 bits according to IEEE standard. Thus, the SVD-CSLBP scheme and the HCVA-DCT scheme need 2048 bits and 640 bits for saving a hash code, respectively. The lengths of the RW scheme, SVD-CSLBP scheme, HCVA-DCT scheme, MDS scheme, FM-2D PCA scheme, DCT-GLCM scheme, our scheme only with global features, our scheme only with local features, and our scheme are 144 bits, 2048 bits, 640 bits, 720 bits, 832 bits, 720 bits, 64 bits, 384 bits, and 560 bits, respectively. Obviously, the storage complexity of our scheme is higher than those of RW scheme and DCT-LLE scheme, but it is lower than those of the SVD-CSLBP scheme, HCVA-DCT scheme, MDS scheme, FM-2D PCA scheme, and GLCM-DCT scheme. The complexity summary is presented in Table 7.

To build an image database for copy detection, the UCID [52] is employed. In the experiment, 16 images are randomly chosen as the query images from 1338 color images in the UCID. Figure 10 shows the thumbnails of these query images. To simulate copy detection, 10 digital operations are utilized to produce 10 copies of each query image. These 10 operations are listed below: IDR+JC+BA (angle: 30 \(^{\circ }\) , quality factor: 30, parameter: 20), IDR+JC+CA (angle: 30 \(^{\circ }\) , quality factor: 30, parameter: 20), IDR+JC+GC (angle: 30 \(^{\circ }\) , quality factor: 30, \(\gamma\) : 0.9), IDR+JC+GLF (angle: 30 \(^{\circ }\) , quality factor: 30, standard deviation: 0.2), IDR+JC+SPN (angle: 30 \(^{\circ }\) , quality factor: 30, density: 0.02), IDR+JC+IS (angle: 30 \(^{\circ }\) , quality factor: 30, ratio: 0.75), IDR+JC+SN (angle: 30 \(^{\circ }\) , quality factor: 30, variance: 0.02), IDR+JC with Text Adding (IDR+JC+TA) (text content: Copyright 2022), IDR+JC with Logo Embedding (IDR+JC+LE) (size of logo: \(66\times 70\) , weight of logo: 0.2), and IDR+JC+SN (angle: 30 \(^{\circ }\) , quality factor: 30, variance: 0.02). So there are 160 image copies. These image copies and the images of UCID excluding the above chosen 16 images are employed to form the copy image database. Therefore, the total image number in the database is \(160+1,338-16=1482\) . For every query image, there are 1,472 different images and 10 image copies.

To validate the copy detection performance of different schemes, the Mean Average Precision (MAP) is used to test. The MAP is computed by the Average Precision (AP). The calculation of AP is related to the order of the returned images of different schemes. The equation of AP is as follows:in which \(f_{i}=1\) when the \(i\) th returned image is an image copy. Otherwise, \(f_{i}=0\) . The MAP is acquired by computing the average of the APs of all query images. The scope of MAP is [0, 1]. In general, a bigger MAP means a superior copy detection performance.

To show advantage, the copy detection performance of our scheme is also compared with the RW scheme [21], SVD-CSLBP scheme [26], HCVA-DCT scheme [17], MDS scheme [18], FM-2D PCA scheme [31], and DCT-GLCM scheme [36]. Note that the SVD-CSLBP scheme, the MDS scheme, and the FM-2D PCA scheme are also dimension reduction-based schemes. The MAPs of these schemes are calculated and shown in Figure 11. It can be seen that the MAP of our scheme is 0.99820. The MAPs of the RW scheme, SVD-CSLBP scheme, HCVA-DCT scheme, MDS scheme, FM-2D PCA scheme, and DCT-GLCM scheme are 0.79779, 0.35234, 0.47503, 0.72384, 0.96875, and 0.98748, respectively. Clearly, the MAP of our scheme is greater than the MAPs of the compared schemes. Our scheme performs better in copy detection than the compared schemes. This is because our scheme has better classification performance, which can reduce classification error during copy detection.