Support Vector Regression-based Reduced-Reference Perceptual Quality Model for Compressed Point Clouds

1). Compression Features:

So far, MPEG has standardized two point cloud compression schemes: V-PCC(focusing on immersive communication) and G-PCC(focusing on autonomous driving) [45]. In the V-PCC encoder, the point cloud sequence is converted into a set of 2D images along six projection planes. Then state-of-the-art video codecs are used to compress the geometry and color information separately. During the compression process, the main distortion is generated by quantization which is controlled by a geometry quantization parameter (QP) and a color QP. Therefore, we use the geometry QP ($Q{P}_g$) and color QP ($Q{P}_c$) as features reflecting compression distortion.

2). Geometry Features:

The geometry features are based on the coordinate information of the points [35].

Consider a distorted point cloud $\mathbb{P}$ formed of $N$ points ${\mathit{p}}_i=\left[x_i,y_i,z_i,R_i,G_i,B_i\right]$, $i=1,\dots ,N$, where $x_i$, $y_i$, $z_i$ are the geometry coordinates and $R_i$, $G_i$, $B_i$ are the intensity values of the red, green, and blue components, respectively. We use the K-nearest neighbor (KNN) search method [46] to determine the set ${\mathbb{S}}_i$ of the $K$ nearest neighbors of the point ${\mathit{p}}_i$. Then, the variance ($Va{r}^i$), median ($Me{d}^i$), mean absolute deviation ($mA{D}^i$), median absolute deviation ($MA{D}^i$), and coefficient of variation ($Co{V}^i$) for every point ${\mathit{p}}_i$ are obtained as

where $A_n^i$ is the Euclidean distance between ${\mathit{p}}_i$ and its $n$-th nearest neighbor in ${\mathbb{S}}_i$, ${\mu }_i$ is the mean value of the Euclidean distances between ${\mathit{p}}_i$ and its $K$ nearest neighbors, and $Med$ is the median operator. For the point cloud $\mathbb{\text{P}}$, we build five geometry features $Va{r}_{geo}$, $Me{d}_{geo}$, $mA{D}_{geo}$, $MA{D}_{geo}$, and $Co{V}_{geo}$ defined as the mean values of $Va{r}^i$, $Me{d}^i$, $mA{D}^i$, $MA{D}^i$, and $Co{V}^i$, $i=1,\dots ,N$.

3). Normal Features:

The normal features are built to reflect the uniformity of the shape of the local surface. We first obtain the normal of each point ${\mathit{p}}_i$ in $\mathbb{P}$ from the coordinate information [35]. Then, we compute the angular similarity between the normal vector of ${\mathit{p}}_i$ and its $K$ nearest neighbors [47]. Next, we compute the variance, median, mean absolute deviation, median absolute deviation, and coefficient of variation of the angular similarity for every point ${\mathit{p}}_i$ as in (1). Finally, as in 2), we obtain five normal features ($Va{r}_{nor}$, $Me{d}_{nor}$, $mA{D}_{nor}$, $MA{D}_{nor}$, and $Co{V}_{nor}$) by computing the mean values over all points in the distorted point cloud.

4). Curvature Features:

The curvature can be calculated by the methods in [13] [35] [48]. We use a curvature prediction algorithm from the point cloud library [48] to calculate the curvature of each point ${\mathit{p}}_i$. To better consider the structural relationship between the current point and its neighbors, we calculate the standard deviations

where $Cu{r}_i$ is the set of curvature values of the points in ${\mathbb{S}}_i$. Then the maximum ($Max$), average of maximum and minimum values ($med$), and standard deviation ($TwiStd$) of $St{d}_i$ are calculated as

For consistency of notation, we call these curvature features $Ma{x}_{cur}$, $me{d}_{cur}$, and $TwiSt{d}_{cur}$, respectively.

5). Luminance Features:

When extracting color features, we work in the luminance channel as it shows a better correlation with the human perception of colors [49]. We convert the color attributes R, G, B at each point ${\mathit{p}}_i$ into Y, ${\text{C}}_B$, ${\text{C}}_R$ components using the matrix defined in ITU-R Recommendation BT.709 [50]. Then, for each point ${\mathit{p}}_i(i=1,\dots ,N)$ in the distorted point cloud, we compute the standard deviation $St{d}_i^{lum}$ of the luminance of its $K$ nearest neighbors. Finally, we take as luminance features $Ma{x}_{lum}=max\left(St{d}_i^{lum}\right)$, ${med}_{lum}=med\left(St{d}_i^{lum}\right)$, $meanSt{d}_{lum}=mean\left(St{d}_i^{lum}\right)$, and $TwiSt{d}_{lum}=std\left(St{d}_i^{lum}\right)$, that is, the maximum, the average of maximum and minimum values, mean, and standard deviation of $St{d}_i^{lum}$, $i=1,\dots ,N$.

To evaluate how well the features in the FCP are related to the perceptual quality, the MOSs computed from the ratings of subjects participating in an experiment are used as the ground truth. The correlation between every independent feature and MOS is typically benchmarked with the application of a four-parameter regression model [51]

where $x$ is the value of a feature, $f(x)$ is the fitted objective score using the feature value, and the parameters ${\beta }_j(j=1,2,3,4)$ are chosen to minimize the least-squares error between the subjective score and the fitted objective score. For an ideal match between the independent features and the subjective quality scores, both the Pearson linear correlation coefficient (PLCC) [52] and the Spearman rank order correlation coefficient (SRCC) [53] should be equal to 1. Fig. 2 shows that all the features are correlated with MOS to some extent. However, considering the time and hardware cost of the quality metric, it is necessary to select as few and as effective features as possible.

1). Feature Selection:

Fig. 3 shows the importance of the 19 features for MOS on the WPC2.0 dataset. We can see that the four features, $Q{P}_g$, $Q{P}_c$, $MeanSt{d}_{lum}$, and $Ma{x}_{lum}$ were more important than the remaining 15 features, and that their importance exceeded the threshold 10⁻⁵. Therefore, we only retained these four features as input features to the SVR regression module. It is worth noting that the feature importance of $Q{P}_g$ and $Q{P}_c$ is significantly better than that of other features. This result can be explained as follows. PCQAML is designed for point clouds compressed with V-PCC. V-PCC converts the input point cloud sequence into two video sequences and encodes each video with H.265. The point cloud sequence is then reconstructed from the decoded videos. As the reconstruction accuracy of the two videos decreases with increasing quantization error, the two quantization parameters $Q{P}_g$ and $Q{P}_c$ play the most important role in the point cloud distortion.

2). Accuracy of the Proposed PCQAML:

The common evaluation criteria PLCC, SRCC, Kendall’s rank-order correlation coefficient (KRCC) [66], mean absolute error (MAE), and root mean squared error (RMSE) were adopted to compare the accuracy of the objective evaluation methods. PLCC represents the linear correlation between the predicted objective scores and the subjective scores. SRCC can evaluate the prediction monotonicity. MAE represents the average value of the difference between the predicted objective scores and the subjective scores. RMSE is a measure of deviation between the predicted scores and the subjective scores. In general, for an accurate visual quality assessment method the PLCC and SRCC should be close to 1, and the MAE and RMSE should be close to 0. To evaluate the accuracy and reliability of the proposed PCQAML with different kernel functions, we calculated the PLCC, SRCC, KRCC, MAE, and RMSE between the ground truth MOS and predicted MOS 1000 times. Each time, 50% of the distorted point clouds are randomly selected from the dataset, while the remaining 50% point clouds in the dataset are treated as the testing set. There is no overlap between the training set and the testing set. Fig. 4 shows the results for the WPC2.0 dataset. The results confirm the accuracy of the proposed model with different kernel functions. Although there is not a huge difference between the performance of the different kernels, the polynomial and RBF kernels were relatively more effective. This is because they can handle non-linear problems, while the linear kernel can handle only linear ones.

To further compare the performance of different kernels and verify the reliability of the proposed PCQAML, we computed various statistics such as the mean, median, maximum, minimum, mode, and variance, of 1000 repeated experimental results. The results in Table II show that the proposed PCQAML was stable for all kernels as the variance was low. The results also show that the polynomial and RBF kernels gave high-quality results with the best performance achieved with the polynomial kernel followed by the RBF kernel. Since the RBF kernel requires fewer hyper-parameters than the polynomial kernel, we adopted it along with the polynomial kernel although its performance was lower.

We also repeated the experiment 1000 times with different kernel functions to verify the reliability of the proposed quality model on M-PCCD. The average PLCC, SRCC, and KRCC are shown in Table III. We can see that the average PLCC of the proposed quality model with different kernel functions was larger than 0.82, confirming its effectiveness on a different dataset. Note how the PLCC was similar on the two datasets, indicating the reliability of the proposed quality model.

3). Results without Feature Selection:

To illustrate the effectiveness of the feature selection module, we give the results with feature selection (PCQAML) and without(PCQAML_W/O_FS). The RBF kernel was used in this experiment. For a fair comparison, the same training and test sets were used. The results are presented in Table IV. They show that in all cases PCQAML gave the same or better results than PCQAML W/O FS. We explain this as follows. PCQAML_W/O_FS performed less effectively than PCQAML due to its use of a large number of features, leading to overfitting. Note that PCQAML significantly reduced the time complexity as it uses four features instead of 19.

some of the comparison results of these two methods are the same. In some cases, the results show that PCQAML is slightly more accurate than PCQAML_W/O_FS. The main reason is that PCQAML_W/O_FS used 19 features, which led to overfitting; that is, the prediction of the machine learning model on the training set is much more accurate than on the test set. More importantly, PCQAML significantly reduces the time complexity as it uses four features instead of 19.

4). Comparison with FR and RR Methods:

In this section, we compare our method with 14 traditional and state-of-the-art FR and RR point cloud quality metrics on WPC2.0 (Table V). In our method, we used the polynomial kernel. In GraphSIM, we used the same parameters as in [11]. The FR metrics were chosen to cover diverse design strategies, including point-based, plane-based, projection-based and feature-based ones. As RR point cloud quality metric benchmarks, we used $PCQ{A}_{RR}$ [14] and $PC{M}_{RR}$ [15].

We chose the eight point clouds (“bag”, “cake”, “cauliflower”, “mushroom”, “ping − pong_bat”, “puer_tea”, “ship”, and “statue”) for testing and the remaining eight for training. The SRCC, PLCC, KRCC, MAE, and RMSE results are given in Table V. Since the quality ratings predicted by different point cloud quality metrics may have different ranges, we used a four-parameter mapping function [51] to map them to a common space. Note that the quality metrics can either directly evaluate the quality or indirectly predict the quality by evaluating the distortion. Therefore, the criteria (e.g., PLCC, SRCC, and KRCC) can be positive or negative.

As shown in Table V, the methods that only consider the geometry structure or color(i.e., $PSN{R}_{M,o}$, $PSN{R}_{M,l}$, $PSN{R}_{H,o}$, $PSN{R}_{H,l}$, $PSN{R}_Y$, and $PSN{R}_p$) did not perform well. The major reason is that point clouds contain both geometric and color information, and it is insufficient to consider only one of them. The performance of the projection-based and feature-based methods that comprehensively consider geometry and color information was significantly better, especially $VIF{P}_p$ whose PLCC was 0.80 and SRCC was 0.79. Note that the PLCC and SRCC of $PC{M}_{RR}$ were only 0.43 with the optimal feature weights provided in [67]. We can explain the poor performance of $PC{M}_{RR}$ by the fact that it uses a very large number of features, which limits its generalization ability. In the proposed PCQAML, we overcome this shortcoming by selecting effective features. In addition, $PC{M}_{RR}$ does not use compression features, which play an important role in the reconstruction quality of the point cloud. On the other hand, $PCQ{A}_{RR}$ uses the compression features to predict the point cloud quality, resulting in a PLCC rise to 0.90. In contrast, the proposed method considers multiple features, such as compression features, geometry features, normal features, curvature features and luminance features, and selects the most effective ones. As a result, both the PLCC and SRCC of the proposed PCQAML reached 0.91, outperforming the existing point cloud quality methods.

To further verify the performance of the proposed PCQAML, we tested it on M-PCCD. We trained our model on the four points clouds amphoriskos, biplane, loot, and the20smaria, and tested it on the four point clouds head, longdress, romanoillamp and soldier. We used the polynomial kernel in SVR. Table VI shows the PLCC, SRCC, KRCC, MAE, and RMSE. In addition to our method, we provide results for all other methods using their published models. We can see that the proposed metric showed an excellent performance and was more accurate than the state-of-the-art ones.

In general, the performance of $PCQ{A}_{RR}$ was much higher on WPC2.0 than on M-PCCD, while the converse was true for $PC{M}_{RR}$. This is because the transformation base of the model parameters of $PCQ{A}_{RR}$ was obtained by training on WPC2.0, and the model parameters of $PC{M}_{RR}$ were obtained by training on M-PCCD. In both cases, the parameters were determined from the entire sample of the training dataset, which inevitably introduces different degrees of overfitting due to the different sizes of the datasets. In contrast, the parameters of the proposed quality metric are determined from the support vector under relaxed conditions, which overcomes the overfitting problem to a certain extent. This explains why our method was more robust than these two previous methods, achieving a PLCC of more than 0.91 on both datasets.

5). Cross-dataset Validation:

To further verify the robustness of our method, we trained our model on the WPC2.0 dataset with different kernel functions and tested it on the M-PCCD dataset. The results in Table VII show that the proposed approach was still able to deliver competitive performance in a cross dataset setting. The experimental results in this section show that the proposed PCQAML has a strong generalization capability.

6). Time Complexity:

An effective point cloud quality assessment metric should both provide an accurate point cloud quality prediction and be computationally efficient. In this subsection, we compare the time complexity of the proposed PCQAML to that of state-of-the-art RR point cloud quality metrics. All competing methods were evaluated in terms of their execution time for eight representative point cloudswith different content complexity levels. Experimental results are shown in Table VIII. The source codes were written in MATLAB R2018a and run on a Windows 7 Pro 64-bit desktop computer with a 8G RAM and 2.8 GHz CPU processor. The metrics. All competing methods were evaluated in terms of proposed PCQAML was not only much faster than $PC{M}_{RR}$, their execution time for eight representative point clouds it also has a relatively low time complexity.