Fusion Algorithm of Multi-focus Images with Weighted Ratios and Weighted Gradient Based on Wavelet Transform

1 Introduction

As a crucial image processing technology, image fusion technology has been widely noticed in recent years. The new image is reconstructed by image fusion, which provides richer visual information than the original images. The main purpose of image fusion is that all of the important salient visual information of input images should be well preserved in the fused image. Multi-focus image fusion is a process during which the blurred part of an image becomes clearer with appropriate image processing technology. At present, it is very difficult for optical imaging systems to make the image of the target with different distances in the same scene. Two or more images with the same scene but different focusing targets become multi-focus images. In order to eliminate the defocus of the same scene, the multi-focus image fusion arises at the historic moment. Multi-focus image fusion means to fuse the images of different focus objects in the same scene according to a specific algorithm to obtain the clear image of the whole scene. For multi-focus image fusion, two or more images of the same scene taken with different focus settings are combined into a single all-in-focus image with an extended depth of field [17]. As a research field of image fusion technology, multi-focus image fusion technology can be widely applied in other fields like machine vision, feature recognition, medical image processing, and so on.

Currently, the popular methods that multi-focus image fusion uses include the weighted average algorithm, principal components analysis transform, Hue saturation intensity transform, contourlet transform, and wavelet transform [13, 16, 21, 22]. The existing multi-focus fusion methods can be mainly divided into two groups: spatial domain and transform domain techniques [1]. The spatial domain methods directly fuse source images by a linear combination. Moreover, the spatial domain methods are often subject to noise and misregistration [6]. With good localization performance both in time domain and frequency domain, wavelet transform can analyze the signal in details and multiple scales by using calculation functions like scaling and translation [4]. Additionally, wavelet analysis can spread the image layer according to wavelet basis, spread the image information to a certain level according to the given requirement of processing, and effectively control the calculation amount [5, 8, 9, 10, 19, 20]. Therefore, the research and exploration of multi-focus fusion based on wavelet transform are most popular.

Based on wavelet transform, the paper points out a multi-focus fusion algorithm with the ratio weighted method and the gradient weighted method. The feature of multi-focus images and the difference between the coefficients are analyzed. The fusion rules of low-frequency coefficients and high-frequency coefficients are improved.

2 Realization of Two-Dimensional Wavelet Transform

Wavelet transform is the decomposition of an image from a high scale to a low scale. The result of wavelet decomposition divides an image into a collection of sub-images. On the first level of wavelet decomposition, the original image is divided into a collection of a low-frequency sub-image and three high-frequency sub-images. On the second level of wavelet decomposition, the low-frequency image produced on the first level will be divided into a collection of a low-frequency sub-image and three high-frequency sub-images, while the three high-frequency sub-images produced on the first level remain the same. The process of wavelet decomposition can go on according to the set levels and produce more collections of sub-images.

Any given image f(x, y) with the size M*N will be divided into four images with the same size of 1/4 of the original image after going through each level of wavelet transform. The four images result from the double interval sampling of both row and line after the scalar product of the original image and a wavelet basis image. The first-level wavelet decomposition can be expressed as the following, and further decompositions can be made by such analogy [3, 11, 23].

Here, ϕ is a scaling function, ψ is a wavelet function, and ϕ and ψ follow the equational relations below:

Here, j≥0, l=1, 2, 3, j, l, m, n are integers. By studying and analyzing the wavelet-based function, the paper chooses biorNr.Nd bi-orthogonal spline wavelet-based function. This kind of wavelet includes two filters: reconstruction scale filter and decomposition scale filter. When a filter has linear phase characteristics, the output signal will not produce phase distortion. This feature is rather important to the analysis and processing of image signal because visual information is especially sensitive to phase distortion. Figure 1 shows the low-frequency and high-frequency images produced in the first- and second-level decomposition after the original image has gone through the transform of biorNr.Nd wavelet-based function [14].

Figure 1:

Wavelet Transformation Map.

3 Design of Fusion Rules

The method of image fusion-based wavelet transform is shown in Figure 2. First of all, process the original image A and original image B with wavelet decomposition and transform. Secondly, process the low-frequency coefficient and high-frequency coefficient resulting from decomposition separately with different fusion rules, and produce a new low-frequency coefficient and high-frequency coefficient. In the end, re-construct the low-frequency coefficient and high-frequency coefficient that have been processed, and produce a new image, which is the fusion image.

Figure 2:

Framework of Image Fusion Based on Wavelet Transform.

3.2 Fusion Algorithm of High-Frequency Coefficient Based on the Weighted Analysis of Improved Edge Detection Operator Gradient

The high-frequency coefficient resulting from wavelet transform reflects the edge features of an image and the gray mutation of the original image, consisting of lots of details, which include the edge, boundary information of an image, and mutant parts of pixel luminance. Regarding the fusion algorithm of high-frequency coefficient, it may follow the rule that the greater value shall be selected. However, this kind of method does not consider the impact of surrounding pixels on the central pixel and can easily commit the mistake by selecting the greater noise value as the high-frequency coefficient, which will decrease the fusion quality of images. In fact, the human visual system is highly sensitive to features such as boundaries and directions most of the time, and is insensitive to brightness at each location. Therefore, this paper chooses the weighted analysis of the improved edge detection operator gradient as the fusion rule of high-frequency coefficient.

To extract the useful information about the impact of surrounding pixels on the central pixel, a pixel area of 3*3 is selected from the fusion rule of the high-frequency part of the image and gradient operation on this pixel field is performed. The selected gradient operator is the improved Sobel edge detection operator. The classic Sobel operator only conducts gradient operation on the difference of the neighboring pixels in horizontal and vertical directions of images, while it does not consider the relations of neighboring pixels in other directions, which easily lose some edge information of images [2, 18]. Therefore, it is considered to add the templates at the 45° and 135° on the basis of the Sobel operator [4, 12] and conduct gradient operation. The templates of 45° and 135° are shown in Figure 3. The gradient magnitude can be expressed in the norm of vector [G_x, G_y, G_45°, G_135°]^T. The paper chooses the gradient with the norm of 1 as the gradient magnitude, which is G=|G_x|+|G_y|+|G_45°|+|G_135°|. G_x and G_y are the gradient magnitudes of the image in horizontal and vertical directions.

Figure 3:

Templates of 45° and 135° of Sobel Operator.

Suppose that the gradient value of the two images are G_A(i, j) and G_B(i, j), when the difference between the two values is larger than T [7] (T is the value of empirical constant). It will be considered that the image with a larger gradient value has more luminance mutant information; therefore, the gray value of the high-frequency image with the larger gradient value will be retained. When the difference between the two values is smaller than T, it means that the two images have similar details. Suppose that the high-frequency coefficient of the two images are F_A(i, j) and F_B(i, j), using weighted gradient fusion method for the two images, the fusion image F_F(i, j) is shown as

(9)FF(i, j)={max[FA(i, j), FB(i, j)], abs[GA(i, j)−GB(i, j)]≥Tmax[GA(i, j), GB(i, j)]GA(i, j)+GB(i, j)[FA(i, j)+FB(i, j)], abs[GA(i, j)−GB(i, j)]<T.

4 Analysis of the Experiment Outcomes and Evaluation of Indicators

To test the validity of the algorithm in the paper, the paper chooses three groups of multi-focus images for a simulation experiment. Besides, it mainly compares and tests different common fusion rules based on wavelet transform. Figure 4 shows the multi-focus image fusion effect resulting from the improved fusion algorithm mentioned in the paper. The fusion rule selected in Figure 5 is the average algorithm for the low-frequency coefficient and takes the greatest absolute value for the high-frequency coefficient. The fusion rule selected in Figure 6 takes the greatest absolute value both for the low-frequency coefficient and high-frequency coefficient. Figure 7 shows the multi-focus image fusion effect based on the contourlet transform method. Figure 8 shows the multi-focus image fusion effect based on the non-sub-sampled contourlet (NSCT) transform method. All the experiments are carried out in the Matlab (R2015b) (US. Mathworks company) environment running on a PC with Intel(R) Core(TM) i7-5820K CPU 3.30 GHz and a Titan X GPU (US. State of California Nvidia corporation).

Figure 4:

Simulation Results of the Proposed Algorithm.

(A) Fusion algorithm based on weighted ratios and the weighted gradient (T=4) (source image A, source image B, fusion image). (B) Fusion algorithm based on weighted ratios and the weighted gradient (T=4) (source image A, source image B, fusion image). (C) Fusion algorithm based on weighted ratios and the weighted gradient (T=5) (source image A, source image B, fusion image).

Figure 5:

Fusion Results of Average and Absolute Maximum Fusion Rules.

Figure 6:

Absolute Maximum Fusion Rules of Low-Frequency Coefficient and High-Frequency Coefficient.

Figure 7:

Fusion Results Based on Contourlet Transform Method.

Figure 8:

Fusion Results Based on NSCT Transform Method.

The paper chooses entropy, mutual information, cross entropy, average gradient as objective evaluation, and running time indicators to objectively evaluate the fusion algorithm [12, 15]. Information entropy is defined as

where p_i is the probability of character number i showing up in a stream of characters of the given image and n is the total number of grayscale.

The mutual information of t fusion image F and original image A can be defined as

where γ_i,j is the joint probability distribution function of images F and A. The gray distribution of image F is p={p₁, p₂, …, p_i, …, p_n}. The gray distribution of image A is q={q₁, q₂, …, q_i, …, q_n}.

The mutual information of t fusion image F and original images A, B can be defined as

The cross entropy for fusion image F and original image A over a given set is defined as follows:

The gray distribution of image F is p={p₁, p₂, …, p_i, …, p_n}. The gray distribution of image A is q={q₁, q₂, …, q_i, …, q_n}. The total cross entropy is defined as

The average gradient is defined as

Among these objective indicators, the larger the entropy of fusion is, the more fusion image information will be added. The larger the mutual information is, the more information the fusion image extracting from the original image is, which means the result of fusion is better. The smaller the cross entropy is, the less the difference between images will be, and the average gradient can reflect minor differences of details and the features of textual changes. The larger the average gradient is, the clearer the image is. Table 1 evaluates the fusion image of multi-focus images resulting from different fusion rules. (Take the second group of images of each algorithm, for example.)

It can be seen that through comparing and analyzing the experiment outcomes, the fusion image resulting from the fusion algorithm of this paper has a better visual effect than the fusion images resulting from the other two fusion algorithms. The traditional algorithm, taking the greater absolute value, can lose many information of the original image that is related to visual features. According to the analysis of the result of the objective evaluation indicator, the five objective evaluation indicators of the algorithm that take the greater and average values of low-frequency coefficient and greater value of high-frequency coefficient do not perform as well as the fusion effect resulting from the ratio weighted analysis of low-frequency coefficient and weighted analysis of high-frequency coefficient based on the improved edge detection operator gradient. For image fusion by taking the greater value (in which the value of n-levels decides the type of decomposition), the type of decomposition is wavelet decomposition if it is zero; otherwise, the type is pyramid decomposition.

5 Conclusion

The multi-resolution of wavelet transform can decompose images on different scales. Outline and detailed information on different levels of the target image can be obtained by decomposing. Moreover, during the wavelet transform, the decomposition equation and filter coefficient keep constant compared to any neighboring scale. Through wavelet transform, a multi-focus image can obtain the details and edge information in different directions of the image, providing more information for multi-focus image fusion to refer to. In addition, wavelet transform features concentrated signal energy, which helps decrease the complexity of wavelet transform both in time and space, enjoying a comparative advantage in the application field of multi-focus image fusion. The paper applies wavelet transform to multi-focus image fusion and points out weighted analysis algorithm based on weighted ratio and improved edge detection operator gradient. This algorithm is applied to the multi-focus image fusion experiment. By evaluating through objective evaluation indicators and comparing with traditional fusion algorithm, it can be shown that this algorithm can effectively fuse multi-focus image information. After being evaluated by using entropy, peak signal-to-noise ratio, and average gradient, it can be seen that the fusion rule pointed out in the paper can provide more image information and, at the same time, produces clearer fusion images. As the fusion algorithms used by multi-focus image fusion are different, how to establish a rational and optimized multi-focus image fusion system and evaluation system is one of the key research fields in the future.