Point Cloud Completion Guided by Prior Knowledge via Causal Inference

$K_{i}$ is extracted through the pre-trained complete shape encoder from $V_{i}$, which can be denoted as $F^{V_{i}}$. It is worth noting that the updating strategy only works at training because we take the training set as our external knowledge base, which can not be available during testing.

We compute the cosine similarity between $F^{\mathcal{I}}$ and $F^{V_{i}}$ to match a “key-value” pair as follows:

To measure whether it is a valid match, we adopt the Chamfer distance [12] as the similarity measurement between the corresponding ground truth $\mathcal{V}$ and the value $V_{i}$ in 3D space. If the Chamfer Distance $Sim_{value}$ is lower than a threshold $\delta$ (discussed in Section 4.6), it is a positive match and vice versa. For a positive match, the value $V_{n_{0}}$ stays unchanged, while the key $F^{V_{n_{0}}}$ is updated as below:

where $n_{0}=\arg\max_{i}Sim_{key}\left(F^{\mathcal{I}},F^{V_{i}}\right)$. In the meantime, except for the corresponding age $A_{n_{0}}$ to be set to zero, all the other ages should be increased by one. For a negative match, $\mathcal{V}$ is read into the memory and should overwrite the oldest slot as follows:

where $n_{1}$ depends on $n_{1}=\arg\max_{i}\left(A_{i}\right)$. The ages here are updated in the same way as mentioned above. In this way, the memory network reinforces its reception ability with similar shapes, saves the unknown shapes, and refreshes the oldest memory slot.

We propose a query strategy for obtaining shape priors that are rich in geometric information for completion and very similar to the partial input. These shape priors are the values in the memory, which are complete point clouds. To fix the number of shape priors fed forward, we retrieve $\hat{k}$ shapes through top-$\hat{k}$ keys with the largest similarity for convenience, which can be formulated as:

For a more organized elaboration of the update and query process, we describe the simplified procedure of the memory network in Algorithm 1.

Suppose that $\mathcal{I}_{i,n_{1}}$ and $\mathcal{I}_{i,n_{2}}$ are fed into the partial shape encoder $E_{K}$ to extract features $F_{i,n_{1}}^{K},F_{i,n_{2}}^{K}\in\mathbb{R}^{1\times C}$, where $C$ is the feature dimension. Following the NT-Xent loss in SimCLR [42], given a positive pair ($F_{i,n_{1}}^{K}$, $F_{i,n_{2}}^{K}$), we treat the other $2(N-1)$ examples within a minibatch as negative examples, where $N$ is the size of the minibatch. The intra-modality contrastive loss $\mathcal{L}_{intra}$ can be formulated as:

where $Sim_{pos}(i;n_{1},n_{2})$ and $Sim_{neg}(i;n_{1},n_{2})$ represent the positive and negative cosine similarity between the same partial inputs but with a different incomplete pattern. The cosine similarity function is defined as follows:

where $\mathbb{I}_{[j\neq i]}\in\{0,1\}$ is an indicator function evaluating to 1 if $j\neq i$ and $\tau$ is the temperature parameter which we set to 0.1.

Considering that the partial shape features should remain consistent with the complete shape features, for each ${\mathcal{S}_{i}}$, we extract features $F_{i}^{V}\in\mathbb{R}^{1\times C}$ by the complete shape encoder $E_{V}$. Together with the partial shape features $F_{i}^{K}$, the cross-modality contrastive loss $\mathcal{L}_{cross}$ is indicated as follows:

where $Sim_{pos}(i;K,V)$ and $Sim_{neg}(i;K,V)$ represent the positive and negative cosine similarity between the partial and complete shape features. The overall pre-training loss function $\mathcal{L}_{pre}$ is the sum of the intra-modality and cross-modality loss $\mathcal{L}_{pre}=\mathcal{L}_{intra}+\mathcal{L}_{cross}$.

Instead of modeling the confounded $P(Y|I)$ in Figure 4, we need to eliminate the backdoor path. According to causal theory, we utilize the do-calculus technique on variable $M$ to eliminate the backdoor path by estimating $P_{B}(Y|I)=P(Y|do(I))$ which stratifies the confounder $M$. We then obtain the following derivations:

• •

  The features extracted from memory will not be affected by cutting off the backdoor path. Thus, $P(m)=P_{B}(m)$.

• •

  $C$ has nothing to do with the causal effect between the variable $M$ and $I$, which we can get $P_{B}(C|I,m)=P(C|I,m)$.

• •

  After the causal intervention, the variable $m$ is independent from $I$, for which we have $P_{B}(m)=P_{B}(m|I)$.

$B$ refers to the case when the backdoor path is cut off, and $m\in M$ denotes the confounder sets. Driven by the derivations above, the backdoor adjustment for Figure 4 can be written as:

where $P(Y|I,m)$ represents the conditional probability given the partial shape feature and confounder; $P(m)$ is the prior probability of the confounder.

Driven by Eq 10, we design the causal feature selection module to alleviate the confounding effect in shape priors. Our implementation idea is stratifying the confounder and pairing the partial shape feature with every stratification. To achieve this objective, we perform an implicit intervention on feature-wise stratification. Let’s consider $\mathcal{H}$ as the index set comprising dimensions of the concatenated shape prior feature obtained from the final layer of the shape prior encoder. We divide $\mathcal{H}$ into n disjoint subsets of equal size. For instance, if the output feature dimension of the shape prior encoder is 384 and we choose the top-3 shape priors with $n=6$, each subset would consist of a feature dimension index set of size 192. In other words, $\mathcal{H}_{m}$ can be represented as ${192(m-1)+1,...,192m}$.

• •

  $P(Y|I,m)=P_{\phi}(Y|cat(F_{I},[F_{V}]_{c}))$, where $F_{I}$ and $F_{V}$ are the partial shape feature and the concatenated shape prior feature, respectively. $[F_{V}]_{c}$ refers to a feature selector that extracts dimensions of $F_{V}$ based on the index set c. Note that c is defined as $c=\{k|k\in\mathcal{H}_{m}\cap\mathcal{S}_{t}\}$, where $\mathcal{S}_{t}$ represents an index set containing absolute values in $F_{V}$ exceeding the threshold $t$. Additionally, $\phi$ denotes the parameters of the shape decoder.

• •

  $P(m)=1/n$, where We assume that each adjusted feature has an equal prior probability, which is calculated as the reciprocal of the number of confounder sets, represented by $n$.

Thus, the overall feature-wise adjustment is:

To optimize the $\phi$ in the above Eq 11, we propose a slightly modified L1 Chamfer Distance loss guided by the backdoor adjustment. Let $\mathcal{G}$ be the notation of high-resolution ground truth, and $\mathcal{P}$ be the notation of the completed prediction. The $\mathcal{L}_{caus}$ can be written as:

where $\Phi$ represents the shape decoder, and $cat(\cdot,\cdot)$ denotes the concatenate operation. The Eq 13 encourages the model to generate predictions for the intervened partial-complete probability that remain consistent and stable across various stratification groups, owing to the shared causal features.

We follow the existing works [17] to use the L1 Chamfer Distance [44] as a quantitative measurement for the quality of output. Apart from generating $\mathcal{P}$, Point-PC also predicts local centers $\mathcal{C}$ of the completed point cloud. For each prediction, the L1 Chamfer Distance loss function between the central point set and the ground truth $\mathcal{G}$ is calculated as:

The final objective function can be defined as the sum of the losses: $\mathcal{L}=\lambda\mathcal{L}_{caus}+(1-\lambda)\mathcal{L}_{recon}$, where $\lambda$ is a hype-parameter used to control the contribution of different losses in the optimization process.

ShapeNet-55 contains 55 categories of synthetic objects, derived from the ShapeNet dataset. This dataset was first released in PoinTr [17], to improve sample diversity and breadth. They randomly sample 8,192 points to obtain the complete point cloud, with a total of 41,952 point cloud models for training and the rest 10,518 models for testing. The partial point clouds are generated by cutting off certain farthest points from a pre-fixed viewpoint and Keeping 2048 points through the furthest point sample(FPS).

Following the evaluation setting in [17], 8 specific viewpoints have been chosen, and the partial point cloud is configured to contain 2,048, 4,096, or 6,144 points, which correspond to 25%, 50%, and 75% of the total points in the entire point cloud, respectively. In this way, we divide the testing stage into three difficulty degrees simple, moderate, and hard(denoted as CD-S, CD-M, and CD-H). As shown in Table. I, Point-PC achieves 0.83 average CD-$\ell_{2}$ (multiplied by 1000) and 0.479 F-Score@1% on ShapeNet-55, which demonstrates that Point-PC outperforms the SOTA methods encountering. Compared with previous methods, Point-PC achieves 0.45, 0.72, and 1.32 CD-$\ell_{2}$ under the three difficulty settings, respectively. The increment of CD-$\ell_{2}$ under CD-M(+0.27) and CD-H(+0.87) strategy also shows that Point-PC better deals with diverse incompleteness levels and diverse incomplete patterns. Furthermore, we report the results on categories with sufficient(first 5 columns) and insufficient(following 5 columns) training samples. Point-PC performs better than all previous SOTA methods on diverse categories of objects despite the training sample imbalance. Quantitative results on ShapeNet-55 clearly demonstrate that Point-PC is capable of generating complete point clouds under diverse settings.

The qualitative comparison results are shown in Figure. 4. The proposed Point-PC performs better with fine details than the other methods. For example, in the bottle category, Point-PC can predict a smoother and more regular structure of bottle edges compared with the other methods. Moreover, Point-PC retains the original details of the partial shapes. In the fifth column of Figure. 4, Point-PC not only generates the incomplete lamp bracket with a clear structure but also keeps the texture of the lamp shade, which makes it a more plausible completion than the other methods. Consequently, Point-PC can effectively learn the geometric information based on the existing partial shape, retrieve similar shape priors based on the learned information and reconstruct complete shapes with more regular arrangements and surface smoothness.

ShapeNet-34 is designed to train on a subset of 34 categories and reserves an additional 21 categories for testing. This setup is to assess how well Point-PC can generalize its performance to new objects from categories that were not encountered during the training stage. ShapeNet-34 uses a subset of 34 categories for training and leaves 21 unseen categories for testing. We utilize ShapeNet-34 to evaluate the performance of Point-PC on novel objects from categories that do not appear in the training phase.

In Table. II, we present the metrics for 34 familiar categories and 21 unfamiliar categories under three levels of difficulty evaluated as CD-$\ell_{2}$(multiplied by 1000) and F-Score@1%. For Point-PC, we observe fewer gaps between the results of 34 seen categories and 21 unseen categories under each difficulty setting, which demonstrates the superiority of shape priors offered by the memory network. We also provide the visual comparison with GRNet in Figure. 6 to show the effectiveness of Point-PC on the unseen categories.

The PCN dataset, a prominent benchmark for point cloud completion, encompasses shapes from 8 different categories. Each shape’s ground truth consists of 16,384 points that are uniformly sampled from the mesh surface, while the partial input is formed from 2,048 points derived from 8 randomly chosen viewpoints. To evaluate our method against established benchmarks and to conduct a comparative analysis with leading-edge techniques, we performed experiments on the PCN dataset. We adhered to the standard procedures and evaluation criteria as outlined in [45, 46, 15] for our model analysis and ablation studies.

We apply the PCN dataset on Point-PC together with several SOTA methods. The CD-$\ell_{1}$ metric between the completed shapes and ground truth is reported in Table. III. Our proposed method stands out and has the best results in all the categories. In terms of average CD-$\ell_{1}$, Point-PC achieves the best score of 6.62, which illustrates that Point-PC outperforms the SOTA competitors.

The KITTI dataset [13] consists of a series of scans from real-world environments, from which 2,401 partial car models have been extracted using 3D bounding boxes. It should be noted that these partial point clouds from the KITTI dataset do not possess corresponding complete point clouds to serve as ground truth. We follow the standard protocol to finetune our model on ShapeNetCars [12] and evaluate it on KITTI with metrics of Fidelity Distance and MMD(Minimal Matching Distance). We exploit KITTI to mimic a real-world situation where point clouds are sparse and irregular.

In Table. IV, we present the results for both Fidelity and MMD metrics. Fidelity quantifies the mean distance from points in the input to their closest counterparts in the output, reflecting the degree to which the input has been maintained in the completed model. Meanwhile, MMD, which stands for the Chamfer Distance between the completed output and the nearest ground truth within the ShapeNetCars dataset, measures the similarity of the reconstructed model to a prototypical car shape. Observed in Table. IV, Point-PC shows better generalization ability compared with previous methods achieving a Fidelity of 0.136 and MMD of 0.509. Qualitative results are shown in Figure. 7, which illustrates that Point-PC predicts general structures even if the input is severely sparse and proves the necessity of prior knowledge for guiding the point cloud completion in the realistic scenario.