An Automatic Surface Defect Inspection System for Automobiles Using Machine Vision Methods

3.1. AIS

The overall architecture of vehicle automatic inspection system is depicted in Figure 3. The automatic inspection system of appearance defect for automobile mainly includes two parts, namely, image acquisition subsystem (hardware) and image processing subsystem (software). The hardware system mainly includes light sources, industrial cameras, camera lens, slide rails, servo controller, servo driver and servo motors. The slide rails, servo controller, servo driver and servo motors constitute the motion control system. Software system mainly includes image acquisition, vehicle model database, image processing algorithms, damage detection results display and motor control program module.

3.2. Cameral Resolution

The key component of the proposed system is a GCP4241 GigE Vision plane-array CCD camera (SMARTEK Vision, Cakovec, Croatia) with 12 megapixels (4240 × 2824) at a frame rate of 9 fps which is the perfect for moving object inspection, low-light and outdoor applications or in high-speed automation with very short exposure times. Full Gigabit Ethernet bandwidth based on the proven GigE Vision standard data interface is utilized to support high resolution image transmission. With a C-10MP-4/3-12mm lens (Foshan HuaGuo Optical Co., Ltd., Guangdong, China) with a shooting distance of 1.5 m, the resolution of the camera can reach 72 dpi (0.35 mm/pixel) which meets the accuracy requirements for detecting defects above 0.5 mm in size.

3.3. Framework of Image Processing Subsystem

After a vehicle body image is generated by the IAS, an image processing subsystem is proposed to detect possible defects in the image. This subsystem includes three main modules: image preprocessing, defect binarization and pseudo-defect remove and classification. Figure 4 shows the diagram of the subsystem:

• 1)

  Image preprocessing: Defects are easily hidden or confused in vehicle images because of illumination inequality and the variation of reflection property of vehicle surfaces, so image preprocessing is a necessary procedure to highlight defects area from the image background. Effective image preprocessing process can improve the efficiency and classification accuracy of subsequent algorithms.

• 2)

  Defect binarization: Defect binarization is the process of extracting candidate defect areas from a vehicle body. The second-order information of defect region can be extracted by DBHM method, and the detection rate of micro-defects can be improved by binarizing the possible defect regions.

• 3)

  Feature extration and classification: A defect binarization image comprises three types of pixels: defects, background and pseudo-defect generated by noise of the preprocessed image. Due to the influence of noise and image quality, there are cases of pseudo-defects after binarization of defects. Through feature extraction of candidate feature regions, we can classify defects while excluding some pseudo-authorities, thereby improving the detection accuracy of AIS.

4.1. Image Preprocessing

The image formation process consists of two steps: optical reflection and photoelectric conversion. In the first step, the illumination model converts the physical properties such as the reflectivity of the vehicle body surface into the reflected light intensity. The second step converts the reflected light intensity into a digital image of the vehicle body through a camera lens and an analog-to-digital converter (ADC or A/D). Based on the understanding of the physical properties of light, the Phong illumination model (PIM) is a widely used illumination physics model. PIM was first proposed by Phong [26] and improved by Whitted [27]. Its main structure consists of ambient term, diffuse term and specular term:

where

• R_p reflected intensity at position p;

• $I_p^{amb}$ reflection intensity of ambient light at position p;

• m serial number of light sources

• $I_{m,d}^l$ Light intensity of the m-(th) light source

• $k_d^{dif}$ diffuse reflection coefficient;

• $k_d^{spe}$ specular reflection coefficient;

• $\overrightarrow{N}$ normal direction of position p;

• $\overrightarrow{V}$ direction of view (usually the direction of the camera);

• ${\overrightarrow{L}}_m$ direction of the m-(th) light source;

• ${\overrightarrow{R}}_m$ perfect mirror direction.

For multiple point light sources, the PIM sums up diffuse term and specular term simultaneously for each individual light source [26]. In the second step, the photoelectric process converts the reflected light intensity R_p into a digital image I_p of the vehicle body through a camera lens and an analog-to-digital converter (ADC or A/D). This transformation can be expressed by a linear formula:

where α and β are constant parameters determined by camera setup [27].

The quality of automobile body image acquisition determines the results of defect detection in subsequent image processing. The two formulas of the body imaging model provide guidance for automotive appearance defect detection experiments:

• 1)

  Using indirect diffused light pattern and increasing the incident angle of the light source can reduce or remove the specular reflection item in the body image to obtain a uniform image of the vehicle body;

• 2)

  The method of image superposition can effectively remove the Gaussian noise in the initial image (this experiment uses the output image after superimposing six consecutive image frames as the actual image to be detected).

4.2. Candidate Defect Binarization

The Hessian matrix is a common method for extracting image features by high-order differential [28]. The Hessian method regards the direction of the second-order directional derivative with the largest module as the direction perpendicular to the image feature, and the direction perpendicular to it as the direction parallel to the image feature. For a linear model with Gauss function, it can be expressed by the second derivative with larger absolute value orthogonal to the straight line and the second derivative with smaller absolute value along the line. This is exactly the geometric meaning of the two eigenvalues of the two-dimensional Hessian matrix. In the process of binarization of candidate defects, we conceive the binarization of defects as a filtering process that searches for geometric structures which can be regarded as point or linear defects. Because the size of defects is different, it is important to introduce measurement scales that vary within a certain range.

The local behavior of an image I can be approximated to its second-order Taylor expansion [29]. Thus, image I in the neighborhood of point x_o can be expressed as:

This expansion approximates the structure of the image up to second order. ∇_o,s and H_o,s are respectively the gradient vector and Hessian matrix of the image in position x_o at scale s. According to the concept of linear scale space theory, differentiation is defined as a convolution with derivatives of Gaussians:

where the two-dimensional Gaussian is defined as:

Assume that the Hessian matrix is expressed as $H_{o,s}=\left[\begin{array}{cc}{H}_{xx}& H_{xy}\\ H_{yx}& H_{yy}\end{array}\right], (H_{xy}=H_{yx}$). Eigenvalues of the Hessian matrix can be calculated as:

The eigenvalue of the Hessian matrix satisfies the case of $|{\lambda }_1|\gg 0,|{\lambda }_2|\gg 0$ or $|{\lambda }_1|\gg |{\lambda }_2|,|{\lambda }_2|\approx 0$ at the location where a body defect exists. Thus, a two-dimensional candidate defect enhanced image is defined as:

where th is a constant threshold value, which is used to enhance the possible defective regions. The background noise in the vehicle body diagram is directly removed by f(x,y) < f_L and f(x,y) > f_H, and the subsequent mask to background interference operation is reduced.

For body defect structural elements, the output of defect region in enhanced image is maximized only when the scale factor s best matches the actual width of the defect, so we introduce a three level scales detection method as shown in Figure 5. By performing a global thresholding of enhanced images ^s1e, ^s2e, ^s3e in different scales s₁, s₂, s₃ (usually s₁ = 1), the middle result images can be obtained as ^s1r, ^s2r, ^s3r, the presentations of which are defined as ^s1r ∈ {0,1}^X, ^s2r ∈ {0,1}^Y, ^s3r ∈ {0,1}^Z. $X={\mathbb{Z}}_m\times {\mathbb{Z}}_n=\left\{x:=\left(x,y\right)\in {\mathbb{Z}}^2|0\le x\le m-1,0\le y\le n-1\right\}$, where m and n are the number of columns and rows of the image, $Y={\mathbb{Z}}_{[m\times \frac{s2}{s1}]}\times {\mathbb{Z}}_{[n\times \frac{s3}{s1}]}$, and $Z={\mathbb{Z}}_{[m\times \frac{s3}{s1}]}\times {\mathbb{Z}}_{[n\times \frac{s3}{s1}]}$. The binary pattern images can be obtained as follows:

where $\eta =\{\begin{array}{c}1 \mathrm{if} e>0\\ 0 \mathrm{if} e<0\end{array}$. By iterating the scale factor, the images at different scales are obtained, and the union of the defect regions is taken as the final candidate defect image:

where ${\chi }_1:{\mathbb{R}}^Y\to {\mathbb{R}}^X, {\chi }_2:{\mathbb{R}}^Z\to {\mathbb{R}}^X$ are the up-scaling operators with Z ⊂ Y ⊂ X. ‖ is the bitwise or operator.

4.3. Feature Extraction of Candidate Defects

Since the candidate defects filtered by the Hessian matrix are easily affected by image noise, features of the defective region like shape, size and divergence are introduced to generate the feature vectors for training and remove pseudo-defects produced by image noise. In order to quantify the shape and size of the defect region, five common image features is utilized: area, major length, minor length, aspect ratio of minimum bounding rectangle, thinness ratio, and to distinguish the real defects from the pseudo defect regions, five statistical characteristics of the candidate region are utilized: mean pixel value, standard deviation, mean value of regional divergence, standard deviation of regional divergence and maximum value of regional divergence. The details of the defect feature are described as follows.

Area is the actual number of pixels in the candidate defect region, which is proportional to the actual area of the defect on the vehicle body.

The major length and minor length of minimum-area-rectangle (MAR) specify the major and minor side length (in pixels) of the minimum rotational circumscribed rectangle that contains all the point sets in the defective region. The aspect ratio of MAR is the ratio of the major length and minor length of the smallest circumscribed rectangle in the defect region.

Thinness ratio [30] describes the similarity between the defect area and the circle which can be calculated from the perimeter and area. The equation for thinness ratio is defined as follows:

where A is the area and P is the perimeter of the defect which is defined as the total pixels that constitute the edge of the defect. As the defect becomes thinner and thinner, the perimeter becomes larger and larger relative to the area and the ratio decreases so the roundness decreases. As the shape of defect becomes closer to a circle, this ratio become closer to 1.

Mean pixel value and standard deviation of candidate defect regions are utilized to eliminate pseudo-defect areas. In addition to mean and standard deviation, we also propose the concept of divergence in feature extraction. From the point of view of energy diffusion, the divergence of gradient image (two-dimensional vector field) is the extent to which the vector field flow behaves like a source at a given point. It is a local measure of its “outgoingness”—the extent to which there is more of some quantity exiting an infinitesimal region of space than entering it [31]. If the divergence is positive, it means that the vector fields are scattered outward. If it is negative, it means that the vector fields are concentrated inward. That is to say, divergence indicates the direction of the flow of the vector, and the greater the degree of the flow, the faster the divergence, the greater the corresponding divergence value. The divergence of a vector field is defined as follows:

In the 2-dimensional Descartes coordinate system, the divergence of image gradient field grad(I) = Ui + Vj is defined as the scalar-valued function:

where i, j are the corresponding basis of the unit vectors, grad(I) is the gradient of image I. Although expressed in terms of coordinates, the result is invariant under rotations, as the physical interpretation suggests. Thus, the mean value, standard deviation and maximum value of regional divergence also have the properties of translation and rotation invariance.

4.4. Defect Classification

After the feature of candidate defects been extracted, we employ SVM [32] to classify the candidate defects into three types: pseudo-defect, dents and scratches. SVM first project the data points into a feature space by kernel function, and then perform linear classification in the feature space. A typical mathematical formulation is to solve the following convex optimization problem:

where x_i are the data samples, y_i are the labels, φ(·) is the kernel function, and C is the penalty coefficient. Common kernel functions include linear kernels: $\varphi {\left(x_i\right)}^T\varphi \left(x_j\right)=x_i^T{x}_j$, and radial basis function (RBF) kernels: $\varphi {\left(x_i\right)}^T\varphi \left(x_j\right)=\exp \left(-\gamma {\Vert x_i-x_j\Vert }_2^2\right)$, where γ is a positive hyperparameter. The candidate Defects are divided into training and test sets using LIBSVM [33] to perform training and testing.

5.1. Preprocessing Step: Image Superposition and Region of Interest Extration

In Figure 6, the red one-dimensional signal is the result of the average value of multiple images. By comparing the results of image Figure 6a,b, we can draw the conclusion that the superposition results of six images are similar to those of 20 images, which can effectively reduce the Gauss noise in the image. At the same time, the computational complexity of six images superposition is less than that of 20 images superposition. Therefore, the follow-up experimental pictures all use six image superimposed pictures.

Although the photoelectric distance sensor installed on the boom can accurately determine the position of the camera, the manual parking way used may cause a certain deviation in the spatial coordinate position of the vehicle image. Therefore, before the candidate defect is extracted, the region of interest (ROI) region is extracted using image registration. Image registration is obtained by feature point extraction, feature point description and feature point matching, and then the spatial transformation matrix estimation of two similar images is obtained. Compared with the traditional SIFT method [34], the SURF method [35] uses a small amount of data and has a short calculation time, therefore, the SURF registration method is adopted in this paper.

When a new body is ready to be introduced into the inspection system, the checker manually selects the test vehicle type on the PC software detection interface and the database provides all the necessary body-related data (e.g., model, color, template, mask, etc.) The complete body defect detection process is illustrated in Figure 7. Firstly, the spatial transformation matrix H is obtained from the experimental image and template image by the registration principle (steps ① and ②). Secondly, the template mask corresponding to the template image is read by the database, and the mask of the image to be detected is obtained by interpolating the space transformation matrix H, that is, the area to be detected (ROI) of the experimental image (steps ③ and ④). Finally, the actual defect area is obtained by the proposed defect detection algorithm and marked with green box on the experimental map (steps ⑤ and ⑥).

In order to improve the accuracy of image registration, we introduce three steps before registration which are camera calibration (Figure 8), edge detection and area opening. This means that the registration process of ROI extraction is not performed directly on gray scale vehicle images, but on the edge images after area opening operations (excluding noise pixels). The lens distortion of the camera has a great influence on the accuracy of image registration. Through experiments, we find that the edge pixel error of the calibrated camera is decreased from more than a dozen pixels to 2–5 pixels compared with the uncalibrated camera, which meets the requirement of body detection area extraction.

5.2. Candidate Defect Binarization: Case Study and Multi-Scale Fusion Method

Figure 9 shows an example of defect detection for a dent with a diameter of 1.4 mm (only occupying nine pixels) near the style line on the fuel tank cap. From the 3D representation of the detection region in Figure 9b,c, it can be seen that the location of the defect is very close to the style line and the edge of the fuel tank cap.

With a global threshold (T = 16) set by the binarization filter in Equations (8) and (9), we can distinguish between noise and defect and thus identify the defect on the surface as a result which is shown in Figure 9d.

In Figure 10, the picture used for experiments is acquired from an automated vision-based quality control inspection system named QEyeTunnel in Molina’s paper [2]. Their inspection system consists of two parts: an external fixed structure where a determined number of cameras are optimally placed in order to see the entire surface of the body to be analyzed, as well as a moving internal structure similar to a scanning machine which houses curved screens known as ‘sectors’ which act as the light sources that project the illumination patterns over the body surface. Figure 10a shows nine defects of automobile surface after spraying paint on automobile production line. This vehicle body image contains punctiform pits, linear defects and uneven paint spray that are difficult for the human eye to distinguish.

In our method, the most obvious line and point scratches are extracted in the first scale level detection (s₁ = 1). Defects of uneven paint spray which are difficult for human eye to distinguish are detected in second and third scale level detection (s₂ = 0.5, s₃ = 0.25). After the second and third level images are rescaled to the first level scale of detection result, the final image result detects all types of defects which is merged by the bitwise or operator using the three level rescaled results.

By comparing the graphs of Figure 10b,e, we can conclude that, for the spot-like, linear and unevenly sprayed surface defects that are difficult to detect in the human eye, our proposed multi-scale Hessian eigenvalue method can still detect these defects.

5.3. Feature Extration and Classification of Candidate Defect Region

We extract 96 images from the image acquisition system. Then we search the connected components form the fused binary pattern of candidate defect regions and label the defect types manually. We then split it into a training subset and a test subset, with the training subset containing 70% of the data and the test subset containing the remainder. Grid search with fivefold cross-validation is performed on the training subset, and report generalization performance on the test subset. Specifically, for linear kernel, we try C with C = 2⁻⁵, 2⁻⁴, …, 2¹⁵. The best hyperparameters for the linear kernel are shown in Table 1.

From Table 1, we can see that the classification accuracy of scratches is high which is up to 97.1%, but the relative accuracy of pits, 95.6%, is low. A possible reason is that the average pixel area of dents usually occupies only a single-digit number of pixels. The occurrence of these misclassified pit defects may be due to dust falling on the surface of the vehicle body during transportation and false defects caused by image noise.

To demonstrate the performance of the defect detection algorithm presented in this paper, a selection of results obtained on the production line with different parts and car bodies is shown in Figure 11. In some of these, detections are observed close to or on style lines and surface edges, as in Figure 11a–c. In addition, it is possible to see detections carried out in concave areas such as the handle, Figure 11d.