Similarity Graph-Based Camera Tracking for Effective 3D Geometry Reconstruction with Mobile RGB-D Camera

Abstract

In this paper, we present a novel approach for reconstructing 3D geometry from a stream of images captured by a consumer-grade mobile RGB-D sensor. In contrast to previous real-time online approaches that process each incoming image in acquisition order, we show that applying a carefully selected order of (possibly a subset of) frames for pose estimation enables the performance of robust 3D reconstruction while automatically filtering out error-prone images. Our algorithm first organizes the input frames into a weighted graph called the similarity graph. A maximum spanning tree is then found in the graph, and its traversal determines the frames and their processing order. The basic algorithm is then extended by locally repairing the original spanning tree and merging disconnected tree components, if they exist, as much as possible, enhancing the result of 3D reconstruction. The capability of our method to generate a less error-prone stream from an input RGB-D stream may also be effectively combined with more sophisticated state-of-the-art techniques, which further increases their effectiveness in 3D reconstruction.

1. Introduction

The reconstruction of 3D worlds from 2D images has been a fundamental challenge in computer graphics and vision for decades. Its importance is equally apparent in the fields of virtual, augmented, and mixed reality where real geometry must be naturally mingled with virtual geometry. Since the Microsoft Kinect sensor became available in 2010, the direct dense methods of visual odometry and simultaneous localization and mapping (SLAM) have shown their potential for extracting 3D geometry models from RGB-D image streams captured by consumer-grade cameras (refer to the recent survey article [1] for notable results). The traditional visual odometry and SLAM approaches used for 3D reconstruction have mostly focused on real-time online computation. When a current frame is taken to estimate its camera pose, it is generally registered against the previous frame (frame-to-frame) or the geometry accumulated thus far in the form of signed distance fields or surfel maps (frame-to-model). The global relationships between captured frames may be considered through either bundle adjustment [2] or pose-graph optimization [3,4], optimizing the pose estimation errors across frames or distributing them across the graph.

In this work, we basically assume the use of a color-depth sensor with low spatial resolution and limited depth precision, such as the mobile sensor on the Google Tango-enabled smartphone, which suffers easily from sensor errors, noises, blurs, and distortions. Under this situation, if high-quality geometry reconstruction is of more concern, it is quite desirable for a user to first scan along areas of interest rather densely enough to acquire sufficient RGB-D data, and carefully select a set of (not necessarily all) image frames from the scan that would allow faithful reproduction of 3D objects. In particular, if the real-time online camera tracking is not mandatory, there is no need to process the RGB-D stream in its given order or for all frames to participate in reproducing the 3D geometry despite any possible defects in the images.

In this paper, we experiment with a different approach for 3D geometry reconstruction that is free from these restrictions. Given an input RGB-D stream, in an aim to extract the best-possible sequence of (possibly a subset of) input frames for stable camera tracking and 3D reconstruction, we build a graph, called the similarity graph, each of whose edges indicates the similarity of the pair of incident frames with respect to pose estimation. Then, a set of pairs of frames that enable a robust 3D reconstruction are carefully selected from the graph along with the pose estimation order by traversing a maximum spanning tree in the graph.

In addition to the generation of an effective set of input frames and their registration order, the presented similarity graph scheme also automatically removes from consideration those frames that may introduce intolerable errors in the results of pose estimation and 3D reconstruction. This may disconnect the similarity graph, resulting in multiple separate components. Therefore, we perform an additional process, called the component-to-component camera tracking, to connect them as much as possible.

To evaluate the effect of the similarity graph clearly, our method currently employs a simple frame-to-frame tracking model for the pose estimation, which can be easily modified to include advanced mechanisms, such as the frame-to-model tracking, bundle adjustment, and pose-graph optimization. This does not mean that the presented similarity graph technique is orthogonal to these sophisticated mechanisms because our method may also be easily combined with them to enhance their effectiveness by providing them with fewer error-inducing input streams.

2. Previous Work

The introduction of the Microsoft Kinect camera led to the possibility of using commodity RGB-D cameras for effective 3D reconstruction. In their seminal work, Newcombe et al. [5] presented a frame-to-model camera tracking system that successfully manipulated the captured depth information to reproduce 3D geometry in a volumetric signed distance field. Since then, several follow-up studies have scanned large-scale areas [6,7] and devised an efficient representation of the memory-intensive volume data structure [8,9]. Unlike the Kinect system, different pose estimation models were proposed that use photometric information [10,11] and both photometric and geometric information [12] to formulate the cost function for direct dense visual odometry. Since then, variants of these approaches were applied to build the RGB-D camera-based tracking and reconstruction systems.

In order to cope with the pose estimation errors that are inevitably accumulated significantly over time, more sophisticated global optimization mechanisms such as bundle adjustment [2] and pose-graph optimization [3,4] were applied to maintain the global consistency between camera pose estimates. The spatial relation between a selection of keyframes was estimated using the image features in the input RGB-D images (e.g., using randomized ferns [13]), which allowed a pose graph to be incrementally optimized for a consistent pose estimation [14,15]. Statistical methods, such as the surfel-based matching likelihood measure [16] and the entropy-based likelihood estimate [17], were also explored to evaluate the spatial relations between the input frames.

The geometric relationships between submaps respectively constructed from chunks of temporarily adjacent frames in the input stream were explored for more accurate large-scale reconstruction [18,19,20,21,22,23]. Several surfel-based, frame-to-model tracking methods were also proposed (e.g., [16,24]) to efficiently represent the 3D models incrementally accumulated during the camera tracking. This approach was then combined with various nonrigid dense map deformation techniques for maintaining globally consistent maps [25,26,27]. All these research efforts have culminated into the recent development of several open-source camera tracking systems (e.g., [22,27,28]).

Note that, similarly to our method that builds a maximum spanning tree from the similarity graph, some previous works also constructed a minimum spanning tree over the pose graph but in the context of global optimization or loop closures for selecting appropriate keyframes [28,29,30]. On the other hand, the maximum spanning tree is used in the presented method to reorder frames in the input sequence for stable pose estimations.

3. Preliminaries: 6-DOF Camera Pose Estimation

As input data, we assume an RGB-D sequence such that each frame at time i $(i=0,1,2,\dots )$ provides a pair of aligned images ${\mathcal{F}}_i=({\mathcal{I}}_i,{\mathcal{D}}_i)$, consisting of an intensity image ${\mathcal{I}}_i\left(\mathbf{u}\right)$ and a depth map ${\mathcal{D}}_i\left(\mathbf{u}\right)$, respectively seen through every pixel $\mathbf{u}\in U\subseteq {\mathbb{R}}^2$. For a set of valid pixels ${\mathbf{u}}_m$ ($m=1,2,\dots ,n_p)$ of the ith frame that are used in the 6-DOF camera pose estimation, let ${\mathbf{p}}_m\in {\mathbb{R}}^3$ ($m=1,2,\dots ,n_p$) be the nearest 3D point in the ith frame’s camera space that is visible through ${\mathbf{u}}_m$.

Given the $3\times 3$ intrinsic matrix K of the depth sensor, each point can be described as ${\mathbf{p}}_m={\mathcal{D}}_i\left({\mathbf{u}}_m\right)K^{-1}{\left({\mathbf{u}}_m^{\top }1\right)}^{\top }$. Reversely, the perspective projection from an arbitrary 3D point $\mathbf{p}$ in the camera space to a pixel $\mathbf{u}$ becomes $\mathbf{u}=\mu \left(\mathbf{p}\right)=\pi \left(K\mathbf{p}\right)$, where $\pi (x,y,w)={(\frac{x}{w},\frac{y}{w})}^{\top }$ is the dehomogenization operator in ${\mathbb{R}}^2$. Let $\xi ={\left({\omega }^{\top }{\upsilon }^{\top }\right)}^{\top }$ denote the 6-vector of coordinates in the Lie algebra $\mathrm{se}\left(3\right)$ with $\omega \in {\mathbb{R}}^3$ and $\upsilon \in {\mathbb{R}}^3$ respectively determining the rotation and the translation. Then, the motion parameter $\xi$ generates a rigid transformation matrix $T\left(\xi \right)\in \mathrm{SE}\left(3\right)$ through the exponential map $T\left(\xi \right)=exp\left(\widehat{\xi }\right)$, where, for the skew-symmetric matrix operator ${[·]}_{\times }$, $\widehat{\xi }=\left[\begin{array}{cc}{\left[\omega \right]}_{\times }& \upsilon \\ {\mathbf{0}}^{\top }& 0\end{array}\right].$

The goal of the frame-to-frame pose estimation is to estimate the rigid transformation $T_{ij}\in \mathrm{SE}\left(3\right)$ from the camera space of the ith frame to that of the jth frame based upon ${\mathcal{F}}_i$ and ${\mathcal{F}}_j$. This is done by, from an initial guess $T_{ij}^{\left(0\right)}$, iteratively improving the current estimate $T_{ij}^{\left(k\right)}$ until convergence such that $T_{ij}^{(k+1)}\leftarrow T\left({\xi }_{min}\right)T_{ij}^{\left(k\right)}$, where ${\xi }_{min}$ is obtained by solving a minimization problem that is defined shortly. Consider the image warping function that maps the mth pixel ${\mathbf{u}}_m$ of the ith frame to a pixel ${\mathbf{v}}_m$ of the jth frame via a given rigid-body motion $T\left(\xi \right)T_{ij}^{\left(k\right)}$. Then, it can be expressed as ${\mathbf{v}}_m=(\mu \circ h_m)\left(\xi \right)$, where $h_m\left(\xi \right)=T\left(\xi \right)T_{ij}^{\left(k\right)}{\left({\mathbf{p}}_m^{\top }1\right)}^{\top }$. (Note that the warping process, defined with respect to each pixel ${\mathbf{u}}_m$, is a function of $\xi$.) If we define the error vector function $\mathbf{g}\left(\xi \right)={(\dots g_m\left(\xi \right)\dots g_{m+n_p}\left(\xi \right)\dots )}^{\top }\in {\mathbb{R}}^{2n_p}$, where

$$g_m\left(\xi \right)=\sqrt{w_{\mathcal{I}}\left({\mathbf{u}}_m\right)}\left\{({\mathcal{I}}_j\circ \mu \circ h_m)\left(\xi \right)-({\mathcal{I}}_i\circ \mu )\left({\mathbf{p}}_m\right)\right\}$$

(1)

and

$$g_{m+n_p}\left(\xi \right)=\lambda \sqrt{w_{\mathcal{D}}\left({\mathbf{u}}_m\right)}\left\{({\mathcal{D}}_j\circ \mu \circ h_m)\left(\xi \right)-{\left[h_m\left(\xi \right)\right]}_z\right\}$$

(2)

for $m=1,2,\dots ,n_p$, then the new estimate $T_{ij}^{(k+1)}$ can be obtained by optimizing the cost function $\mathbf{g}{\left(\xi \right)}^{\top }\mathbf{g}\left(\xi \right)$:

$${\xi }_{min}=arg\underset{\xi }{min}\mathbf{g}{\left(\xi \right)}^{\top }\mathbf{g}\left(\xi \right).$$

(3)

In this objective function, $w_{\mathcal{I}}(·)$ and $w_{\mathcal{D}}(·)$ are the respective weights obtained to reduce the negative effect of outliers, $\lambda$ is the relative gain for the depth component, and the operator ${[·]}_z$ returns the z coordinate of a 3D point. While any feasible method (such as the one presented in, for instance, [5,10,11]) may be applied to implement the motion estimation process, we use a slight variant of the method by Tykkälä et al. [12], in which the weights in the iteratively re-weighted least squares formulation are computed using the t-distribution as proposed by Kerl et al. [17].

4. Similarity Graph-Based Camera Pose Estimation

The proposed algorithm proceeds in four steps: (i) estimation of similarity measure between frames, (ii) construction of similarity graph, (iii) extraction of maximum spanning tree, and (iv) pose estimation via tree traversal. In this section, we first explain each step in detail, and show how this basic algorithm can be extended to improve its performance further in the next section.

4.1. Similarity Measure between Two Frames

Given two images ${\mathcal{F}}_i$ and ${\mathcal{F}}_j$ captured at two arbitrary times i and j, we are to estimate the relative rigid-body motion $T_{ij}$, which maps the camera space of the ith frame (source frame) to that of the jth frame (target frame), by optimizing the cost function described in Section 3. As is well known, the stability and accuracy of the solution process are greatly influenced by how close the initial guess $T_{ij}^{\left(0\right)}$ is to the solution $T_{ij}$. In general, the identity matrix $I\in \mathrm{SE}\left(3\right)$ is selected as an initial value because there is no obvious better alternative. When a frame-to-frame tracking method is applied between two temporarily consecutive frames, however, such a choice may lead to a poor estimation when the camera has moved fast at a high angular velocity between the frames, resulting in abrupt view changes and annoying motion blurs in the captured images. This is mainly because, in such case, the unknown rigid-body motion $T_{ij}$ is located far from the null motion I in the search space $\mathrm{SE}\left(3\right)$.

If we are free to choose the pairs of frames to be registered to each other, it is very advisable to select the pairs whose images are as similar as possible. For such pairs, whose relative rigid transformation would be close to the initial guess (which is set to the null motion), we observe that even the simple, drift-prone, frame-to-frame tracking tends to produce quite satisfactory pose estimations. Under this observation, we define the similarity between the two frames ${\mathcal{F}}_i$ and ${\mathcal{F}}_j$ over a rigid-body motion T as the ratio between the cardinalities of two pixel sets $P_{valid}\left({\mathcal{F}}_i\right)$ and $P_{matched}({\mathcal{F}}_i,{\mathcal{F}}_j,T)$:

$$\sigma ({\mathcal{F}}_i,{\mathcal{F}}_j,T)=\frac{|P_{matched}({\mathcal{F}}_i,{\mathcal{F}}_j,T)|}{|P_{valid}\left({\mathcal{F}}_i\right)|},$$

(4)

where $\sigma ({\mathcal{F}}_i,{\mathcal{F}}_j,T)$ ranges from 0 to 1 as explained immediately below. First, $P_{valid}\left({\mathcal{F}}_i\right)$ indicates the set of pixels of ${\mathcal{F}}_i$ that are found valid and thus may participate in the process of camera pose estimation. Note that problematic pixels often appear in various forms in the captured RGB-D images. Above all, depth sensors in general generate pixels of invalid values for out-of-range readings, which should be eliminated from consideration. In addition, we get rid of pixels further that may cause numerical instability during the pose estimation computation. In particular, to remove those pixels with possibly faulty depth values, the normal and contour filtering techniques, proposed by An et al. [31], are applied after the usual bilateral filtering.

Second, $P_{matched}({\mathcal{F}}_i,{\mathcal{F}}_j,T)$, which is a subset of $P_{valid}\left({\mathcal{F}}_i\right)$, represents the set of valid pixels of ${\mathcal{F}}_i$ that after warped from ${\mathcal{F}}_i$ to ${\mathcal{F}}_j$ through T, have both similar intensity and depth values in ${\mathcal{F}}_j$. Ideally, the pixel values of $\mathbf{u}\in P_{valid}\left({\mathcal{F}}_i\right)$ must be the same as those of the corresponding pixel ${\mathbf{v}}_{\mathbf{u}}$ of ${\mathcal{F}}_j$ if $T=T_{ij}$, where ${\mathbf{v}}_{\mathbf{u}}=\mu \left(T{\left({\mathbf{p}}_{\mathbf{u}}^{\top }1\right)}^{\top }\right)$ and ${\mathbf{p}}_{\mathbf{u}}={\mathcal{D}}_i\left(\mathbf{u}\right)K^{-1}{\left({\mathbf{u}}^{\top }1\right)}^{\top }$. In reality, however, since the captured RGB-D images often contain noisy and faulty pixel values, the rigid transformation T is only an approximate for $T_{ij}$, and the 3D objects in the scene are not ideally diffusive, they usually do not coincide exactly. If T is a rather good estimate, however, there is a good chance that there exists a pixel in the neighborhood of ${\mathbf{v}}_{\mathbf{u}}$ in the pixel space of ${\mathcal{F}}_j$ whose intensity and depth values are very similar to those of $\mathbf{u}$.

To reflect this uncertainty, we regard that a pixel $\mathbf{u}$ in ${\mathcal{F}}_i$matches some pixel in ${\mathcal{F}}_j$ with respect to T and an image kernel of fixed odd size $n_{ker}$, that is, $\mathbf{u}\in P_{matched}({\mathcal{F}}_i,{\mathcal{F}}_j,T)$, if, for ${\mathbf{v}}_{\mathbf{u}}=\mathbf{v}(x,y)$, there exists at least one neighboring pixel $\mathbf{v}(x+k,y+l)$ in ${\mathcal{F}}_j$ with $-\frac{n_{ker}-1}{2}\le k,l\le \frac{n_{ker}-1}{2}$ such that

• $\mathbf{v}(x+k,y+l)\in P_{valid}\left({\mathcal{F}}_j\right)$, i.e., $\mathbf{v}(x+k,y+l)$ is valid in ${\mathcal{F}}_j$,
• $|{\mathcal{I}}_j\left(\mathbf{v}(x+k,y+l)\right)-{\mathcal{I}}_i\left(\mathbf{u}\right)|<{\tau }_I$ for some threshold ${\tau }_I$, and
• $|{\mathcal{D}}_j\left(\mathbf{v}(x+k,y+l)\right)-{\left[T{\left({\mathbf{p}}_{\mathbf{u}}^{\top }1\right)}^{\top }\right]}_z|<{\tau }_D$ for some threshold ${\tau }_D$.

It is known that the convergence of the Gauss–Newton method, which is used to implement our optimizer, can suffer when the optimal objective value is relatively large [32]. Note the resemblance between the residual errors in the cost function of Section 3 and the similarity criteria. When a pair of frames with high similarity are chosen, the iteration will tend to start with a relatively small objective value $\mathbf{g}{\left({\xi }_0\right)}^{\top }\mathbf{g}\left({\xi }_0\right)$, ensuring numerically more stable optimization. See Figure 1 for some examples of the similarity measure.

4.2. Construction of Similarity Graph

Now, for a given stream of $n_{fr}$ frames ${\mathcal{F}}_i=({\mathcal{I}}_i,{\mathcal{D}}_i)$$(i=0,1,\dots ,n_{fr}-1)$, we build a weighted undirected similarity graph$G=\left(V\right(G),E(G\left)\right)$, where $V\left(G\right)$ is the set of vertices respectively representing the $n_{fr}$ frames, $E\left(G\right)$ is the set of edges connecting each pair of frames, and the weight of an edge $({\mathcal{F}}_i,{\mathcal{F}}_j)(i>j)$ is the similarity value $\sigma ({\mathcal{F}}_i,{\mathcal{F}}_j,T)$ for a given rigid transformation T. Two things need to be noted about the similarity graph. First, strictly speaking, the similarity measure on a pair of frames is not commutative in general, i.e., $\sigma ({\mathcal{F}}_i,{\mathcal{F}}_j,T)\ne \sigma ({\mathcal{F}}_j,{\mathcal{F}}_i,T^{-1})$, implying the graph must be directed. However, in order to reduce the computational burden of constructing the similarity graph, we confine ourselves to registering frames to the temporarily precedent frames only, which still produces sufficiently good results.

Second, whereas evaluating the weight function requires the matrix T, which must ideally be $T_{ij}$, there is no information at all on the rigid-body motion in particular when the image sequence has just been inputted. When the similarity graph is constructed for the first time, we use the null transformation I as T. In this case, the pairs of frames whose corresponding cameras have closer positions and orientations in the global space tend to have higher similarity measures. Therefore, when edges with larger weights are selected for pose estimation, as will be explained shortly, they usually produce less error-prone estimates for the wanted rigid transformations.

4.3. Extraction of Maximum Spanning Tree

Recall that when the camera pose is tracked along the usual, temporarily linear trajectory, a single substantial error involved in registering a specific pair of consequent frames may cause severe drifts for all subsequent frames (frame-to-frame tracking) or accumulate incorrect geometry information into the model being built (frame-to-model tracking). Under the intuition that selecting higher weighted edges will result in more reliable registrations for the corresponding frame pairs, we are naturally led to the greedy strategy in which the edges are selected from the similarity graph on the order of non-increasing weights until all vertices (frames) are included without forming a cycle. Such a choice can be made by constructing a maximum spanning tree in the graph, which is done by negating the edge weights and finding a minimum spanning tree. In this work, we implemented the priority-queue-based version of Kruskal’s algorithm that finds a desirable tree in $O\left(\right|E\left(G\right)|+\alpha log|V\left(G\right)\left|\right)$ time, where $\alpha$ is the number of graph edges not longer than the longest edge in the minimum spanning tree [33].

After a maximum spanning tree is obtained, we select the root frame whose camera space becomes the global space common to all frames. It is highly probable that the longer the distance from a frame to the root frame is, the more numerical errors are accumulated while transforming the frame’s camera space to the global space. Therefore, we find a center ${\mathcal{F}}_c$ of the tree whose greatest distance to other frames in the maximum spanning tree is minimal: $c=arg{min}_i{max}_{j\ne i}{d}_{T_G}({\mathcal{F}}_i,{\mathcal{F}}_j)$, where $d_{T_G}(·,·)$ is the distance (i.e., the number of edges) between two frames in the tree T. Refer to Figure 2 to see an example of maximum spanning tree that was built from an RGB-D sequence.

4.4. Pose Estimation through Tree Traversal

Finally, we traverse the maximum spanning tree in depth-first search manner, performing the frame-to-frame pose estimation for each visited edge which represents a pair of source-target frames. During the tree traversal, the camera space of each frame is converted to the global space through the transformation obtained by accumulating the relative camera motions from the frame to the root. Here, since the pairs of frames for which the camera poses are sought are known a priori, the camera tracking computation can easily be parallelized.

Note that the edges with low similarity measures may introduce intolerable pose estimation errors although they may happen to be selected during the tree construction, depending on how the input stream was captured. In that case, we often witness poor pose estimations between the corresponding pairs of frames, resulting in visually annoying artifacts in the resulting 3D reconstruction. Therefore, when the tree is built, our method allows for limiting the edges in the similarity graph to have edge weights greater than a minimum threshold value ${\tau }_{good}$ (e.g., ${\tau }_{good}=0.8$). Depending on the input stream, this restriction may disconnect the similarity graph, producing a forest of trees, each of which is itself a maximum spanning tree. We discuss how to handle the multiple connected components of the similarity graph in Section 5.2. See Figure 3 for a branch of an example tree generated by our method.

5. Extending the Idea of a Similarity Graph

5.1. Local Repair of a Maximum Spanning Tree

Despite the selection of frame pairs having as high similarity as possible during the construction of the maximum spanning tree, it is often that the rigid transformations estimated for some edges are not sufficiently accurate. One reason for this finding is that some edges connecting irrelevant frames were wrongly chosen because of the inaccurate approximations of the similarity measure. Recall that, when the similarity is evaluated for a given pair of frames, the identity matrix is initially used as the rigid transformation between them because there is no better choice. However, this may cause a serious problem when there was a nontrivial translational and/or rotational motion between the two frames, reducing the reliability of the similarity measure. On the other hand, the rigid transformation estimated for each tree edge during the tree traversal is usually a better approximation for the wanted transformation than the identity matrix.

In our method, we (optionally) traverse the tree again and repair it, if possible, using the more reliable estimates of the rigid transformations. Given a maximum spanning tree, let ${\mathcal{F}}_i$ and ${\mathcal{F}}_j$ be the child (source) and parent (target) frames, respectively, of an edge currently being visited. To estimate the appropriateness of using this pair of frames in the pose estimation, we evaluate the similarity measure $\sigma ({\mathcal{F}}_i,{\mathcal{F}}_j,{\widehat{T}}_{ij})$ again using the rigid transformation ${\widehat{T}}_{ij}$ estimated in the first round. If the new similarity measure is less than a given threshold ${\tau }_{repair}$, we examine the proper ancestors ${\mathcal{F}}_k$ of ${\mathcal{F}}_j$ up to, say, three levels, evaluating the respective similarity measures $\sigma ({\mathcal{F}}_i,{\mathcal{F}}_k,{\widehat{T}}_{ik})$, where ${\widehat{T}}_{ik}$ can be approximated by accumulating the rigid transformations along the path from ${\mathcal{F}}_i$ to ${\mathcal{F}}_k$ (refer to (a) and (b) of Figure 4).

If the largest one among the reevaluated measures is greater than ${\tau }_{repair}$, we delete the edge between ${\mathcal{F}}_i$ and ${\mathcal{F}}_j$ from the maximum spanning tree, and instead insert a new one between ${\mathcal{F}}_i$ and the corresponding ancestor ${\mathcal{F}}_{k^{*}}$ (e.g., ${\mathcal{F}}_{312}$ in Figure 4b). When the rigid transformation $T_{i{k}^{*}}$ is to be estimated for the new edge, we use the rigid transformation accumulated from ${\mathcal{F}}_i$ to ${\mathcal{F}}_{k^{*}}$ as an initial value for the iterative optimization process, which usually results in a more accurate pose estimate than using the identity transformation. Once the repair process is done, we use only those edges with similarity measures greater than ${\tau }_{good}$ for 3D reconstruction, as explained in Section 4.4. See Figure 4 for an example of the tree repair.

5.2. Component-Wise Camera Tracking

When the parameter ${\tau }_{good}$ is set to a high value, say, $0.85$, and the edge set of the similarity graph is restricted to those with weights above ${\tau }_{good}$, the graph tends to get disconnected into multiple connected components. In such case, the quality of the point cloud generated for each component is quite high, and thus it is desirable to align the major components to each other to form a larger one.

Let ${\mathcal{C}}_i$ and ${\mathcal{C}}_j$ be two connected components of a similarity graph, for each of which the 3D geometry has successfully been reproduced using the similarity graph method (see Figure 5). Assume further that ${\mathcal{F}}_i$ and ${\mathcal{F}}_j$ are the root frames of the minimum spanning trees that were built in the respective components. Being separated as two components means that there was not a single highly reliable edge between them that allows for deriving the rigid transformation $T_{ij}$ between the two frames. However, there often exist edges that connect the two components with a fair, although not sufficient, amount of similarity. Consider those edges whose similarity measures are greater than a given threshold ${\tau }_{fair}$, which is set to a value smaller than ${\tau }_{good}$ (e.g., ${\tau }_{fair}=0.7$). Then, although each of them may not lead to a sufficiently accurate frame-to-frame camera tracking, they often, although not always, can be collectively used to estimate the relative rigid-body motion $T_{ij}$.

In this component-to-component tracking approach, for each such edge $({\mathcal{F}}_{i_l},{\mathcal{F}}_{j_l})$ ($l=1,2,\dots ,n_l$), every valid pixel ${\mathbf{u}}_m$ of the $i_l$th frame contributes to the error vector $\mathbf{g}\left(\xi \right)$ in basically the same way as the frame-to-frame tracking discussed in Section 3. Again, let ${\mathbf{p}}_m$ be the point in the $i_l$th frame’s camera space that corresponds to ${\mathbf{u}}_m$. Then, the only difference in the formulation of the error function for finding $T_{ij}$ is the way of transforming ${\mathbf{p}}_m$ to the $j_l$th frame’s camera space, where the warping function now becomes ${\mathbf{v}}_m=(\mu \circ h_m^l)\left(\xi \right)$ where $h_m^l\left(\xi \right)=T_{j,j_l}T\left(\xi \right)T_{ij}^{\left(k\right)}{T}_{i_l,i}{\left({\mathbf{p}}_m^{\top }1\right)}^{\top }$. It should be noted that, during the minimization of the new cost function, the Jacobian matrix $J_{\mathbf{g}}\left(\xi \right)\in {\mathbb{R}}^{2n_{tp}\times 6}$, where $n_{tp}$ is the total number of pixels collected from the $n_l$ frames in ${\mathcal{C}}_i$, must be constructed slightly differently using the modified $3\times 6$ Jacobian of $h_m^l\left(\xi \right)$, which can be expressed as

$$\begin{array}{ccc}\hfill J_{h_m^l}\left(\xi \right)& =& R\left(T_{j,j_l}\right)\left[\begin{array}{cc}{I}_{3\times 3}& -{\left[{\overline{\mathbf{p}}}_m\right]}_{\times }\end{array}\right].\hfill \end{array}$$

(5)

Here, $R\left(T_{j,j_l}\right)$ is the $3\times 3$ rotation matrix of $T_{j,j_l}$, $I_{3\times 3}$ is the $3\times 3$ identity matrix, and ${\overline{\mathbf{p}}}_m\in {\mathbb{R}}^3$ is the point mapped from ${\mathbf{p}}_m$ via $T_{ij}^{\left(k\right)}{T}_{i_l,i}$. See Figure 6 for an example of the component-wise camera tracking.

6. Experiments

To demonstrate its effectiveness and applicability, the presented similarity graph scheme was tested with several RGB-D sequences, where all the test datasets, including those shown in the previous sections, were produced by storing into files the live RGB-D streams of $320\times 180$ pixels, captured using a Lenovo Phab 2 Pro smartphone. Compared to the Microsoft Kinect v2 sensor, the used mobile RGB-D sensor tended to suffer more from the limited depth precision and low pixel resolution. In addition, we often observed visually annoying temporal/spatial mismatches between the pixels of the intensity and depth images presumably due to the difference in their image acquisition rates, making the 3D reconstruction process more challenging. As mentioned in the Introduction section, for high-quality 3D reconstruction, our method is oriented to choose best possible frames and their camera tracking order from input sequences. Therefore, it is better suited for the input streams in which areas of interest are scanned rather densely so as to provide sufficient RGB-D information. Therefore, we found that such standard benchmark datasets as the ICL-NUIM or TUM datasets [34,35] are not best suited to evaluate our method.

6.1. Computational Costs

The effectiveness of the presented method only comes with the computational cost of finding the sequence of frame pairs that enables the robust and accurate camera tracking.

Table 1. Computational costs for the similarity graph-based pose estimation (unit: sec). For each stream made of $n_{fr}$ RGB-D images of $320\times 180$ pixels, the four columns titled by “sim-gr”, “mst”, “paths”, and “pose-est” respectively represent the total times consumed in the four major steps of our method: the construction of similarity graph, the construction of maximum spanning tree, the extraction of frame pair sequences via the tree traversal, and the actual frame-to-frame camera tracking. In the parentheses in the fifth column, the average per-frame pose estimation times are given. Finally, the last column “amortized” indicates the amortized per-frame tracking costs reflecting all computations.

${\mathit{n}}_{\mathit{fr}}$	Sim-Gr	Mst	Paths	Pose-Est	Amortized
200	0.77	0.0068	0.0022	5.15 (0.0258)	0.0297
400	2.86	0.0332	0.0044	10.62 (0.0266)	0.0338
800	10.83	0.1879	0.0128	22.59 (0.0282)	0.0420
1600	40.52	1.0450	0.0520	46.50 (0.0291)	0.0551

reveals the computational complexity of our method, where the timings were collected on a PC with an Intel Core i7-8700K CPU with 64 GB of main memory and an Nvidia GeForce GTX 1080 Ti GPU with 11 GB of graphics memory. To build a similarity graph for an input RGB-D stream of $n_{fr}$ frames, the similarity measure should be evaluated for each of the $\frac{n_{fr}(n_{fr}-1)}{2}$ pairs of frames, whose number increases quadratically with respect to the input size. In fact, this evaluation process has a high degree of parallelism, both frame-pair-wise and pixel-wise that allows efficient parallel processing. Despite our GPU implementation using the OpenCL, however, most computation time was spent in constructing the similarity graph as clearly shown in the table. Note that, for the stream of $n_{fr}$ frames, the frame-to-frame pose estimation needs to be carried out $n_{fr}-1$ times whether the similarity graph scheme is applied or not. Therefore, the naïve, drift-prone, frame-to-frame tracking can be regarded as consuming roughly the same per-frame pose estimation time as that given in the parentheses of the fifth column. Considering the substantially improved stability and accuracy achieved by our method, we believe that the increase in the amortized per-frame camera tracking cost was quite acceptable. In the final section, we discuss how the burdensome computation of the similarity graph construction can be done progressively during image capture in the future work.

6.2. Comparison to a Frame-to-Frame Tracking Method

In order to see how effectively the careful selection of source-target frame pairs from an input stream improves the quality of pose estimation, we first compared our method to an extended frame-to-frame tracking method of An et al. [31], which uses a multi-level pose-error correction scheme. The experiments indicated that both methods worked well when the camera was moved smoothly and rather slowly during image capture. When the camera movement exceeded an acceptable level, however, moving irregularly in jerky motions, the An et al.’s method could no more handle such adverse situations despite its effort to correct the pose estimation errors. On the other hand, the similarity graph enabled us to find an effective set of source-target frame pairs, from which quite acceptable frame-to-frame camera tracking results were obtained without needing to use such sophisticated tools as the frame-to-model tracking and/or global pose optimization (see Figure 7).

6.3. Comparison to the ElasticFusion Method

We next compared our method to a state-of-the-art method, called the ElasticFusion. As proposed by Whelan et al. [27], the camera tracking system employs a surfel map-based frame-to-model tracking approach, augmented with several mechanisms like geometry deformation and local/global loop closure. Unlike ours, it was developed as a real-time online system that performs all calculations on the fly without any preprocessing. However, we carried out this analysis to understand how the effort of selecting appropriate pairs of frames from the entire RGB-D stream compares to that of applying various mechanisms to ensure the accuracy of the estimated poses while processing the input frames in acquisition order. In doing the experiment, we used the codes courteously provided by [27].

As expected, under normal circumstances in which the camera movement allowed each input frame to share sufficient geometry with the accumulated model, the ElasticFusion system always completed its mission successfully. However, when we sporadically moved the camera sharply or suddenly during image capture so that the camera for some input frames looked at the space that hardly contained the accumulated model, it sometimes produced inaccurate pose estimates (see Figure 8). On the other hand, our method could still find sufficient numbers of frame pairs for stable 3D reconstruction, producing point clouds with fewer geometric errors. Note that since our method used only those frame pairs whose corresponding edges had sufficient similarity, a sparser point cloud was usually produced for the input stream containing more radical camera movements.

When the real world is scanned for 3D reconstruction, it might be necessary to generate additional RGB-D streams to supplement the poorly scanned regions. The property of being independent of the order of incoming frames implies that our method can also be used effectively to combine multiple independently scanned streams if they contain frames to each other that have enough similarity. Note that the pairs of source and target frames resulting from the similarity graph are aimed to reduce the possibility of improper pose estimations. Therefore, if a robust 3D scanning is the primary concern, the presented similarity graph scheme may also be used as a preprocessor that feeds refined RGB-D streams into the other state-of-the-art techniques like the ElasticFusion.

6.4. Comparison to the BundleFusion Method

We also performed a comparison with another state-of-the-art method by Dai et al. [22], called the BundleFusion that, based on the bundle adjustment framework, estimates globally optimized camera poses and produces 3D polygonal models in real-time. Our experiment showed that this global optimization framework generally performs very well in terms of both speed and scan quality. Nevertheless, for some input streams, it was desirable to filter out frames more aggressively that may introduce pose estimation errors.

Figure 9 displays a test result carried out for an RGB-D sequence with some very jerky camera movements. (The BundleFusion codes were courteously provided by [22]. As shown in Figure 9a, the BundleFusion method successfully generated globally consistent pose estimates despite the intractable camera motions. However, we observed that frames corresponding to the fast and abrupt camera movements sometimes influence negatively on the quality of 3D reconstruction via the global optimization process as revealed in Figure 9b. On the other hand, the similarity graph-based technique focused more on finding a set of interrelated frames that would collectively create a better quality of 3D reconstruction (compare Figure 9b and Figure 9c).

6.5. Towards 3D World Modeling in a Mixed-Reality Environment

Despite the effort to select good pairs of frames offering stable camera tracking, the pose estimation errors from our method are inevitably accumulated over time. In general, this is true for most sophisticated state-of-the-art camera tracking models when the scan area grows beyond their capability, sometimes with jerky camera movements. Note that, given a sequence of RGB-D images, the similarity graph method uses only those frames that would result in robust camera tracking and 3D reconstruction. Thus, depending on how a scene is scanned, it often produces a collection of separate components that correspond to faithfully reconstructed, local surface regions. While additional scanning would provide extra information for connecting them automatically, another approach worth investigating is to let the user assemble the well-built components manually using the real world for guidance.

Figure 10 shows the 3D modeling methodology that is currently being used to test our method in an experimental mixed-reality environment. Here, we set up the Microsoft HoloLens system to share physical space with the HTC Vive system so that the user wearing a HoloLens headset can freely use Vive’s controllers. Then, the user interactively selects each component of reconstructed 3D geometry and places it on the real object finely while seeing both virtual and real geometries through the holographic glasses. In this way, the user can utilize the mixed-reality technology to convert the real world into geometric models, which can in turn be used effectively to develop various mixed-reality applications (see Figure 11 for an example 3D scene modeling).

6.6. Towards Progressive 3D Reconstruction from a Live RGB-D Stream

The presented method is basically an offline approach, which requires all RGB-D frames to be available for 3D reconstruction. As shown in Table 1 in Section 6.1, the method spends most of its computation time on building the similarity graph. However, the graph construction can be done incrementally in parallel with the image acquisition process so that, for every incoming frame, the similarity measure is evaluated for each of the preceding frames, adding the respective weighed edge to the graph. Figure 12 proposes a computational pipeline, in which the user would be able to check the progressively growing 3D geometry on the fly and accordingly decide which region to scan further for more effective 3D reconstruction. In this scenario, an independent CPU thread iteratively takes each incoming frame and updates the similarity graph ( Incremental similarity graph update), which can be easily accelerated on the GPU. At every given time interval (e.g., at every 30 frames), as soon as the graph update is over, another CPU thread starts finding a maximum spanning tree (Maximum spanning tree construction), and performs the frame-to-frame pose estimation indicated by the tree (Pose estimation).

The new tree usually shares many edges with the earlier tree built in the previous round. Thus, it is sufficient to carry out the registration task only for the newly found tree edges, whose computation can also be parallelized on multithreaded CPU hardware. When the camera pose for a frame has been modified or newly generated, a per-frame point set is created for 3D reconstruction from its RGB-D image using yet another CPU thread ( Point cloud generation). Once all the needed point sets are collected for display, the screen is refreshed (Display refresh).

7. Conclusions

In this paper, we proposed a novel approach that reconstructs 3D geometry from a stream of RGB-D images taken by a consumer-grade mobile RGB-D camera. Although our method is not confined to low-cost mobile sensors, such as the one on the tested smartphone, its low pixel resolution and limited depth precision often hindered high-quality 3D reconstruction. Therefore, the presented similarity graph technique was designed to carefully select only the frames and their registration order from the input sequence that would produce accurate pose estimation and robust 3D reconstruction.

The proposed method sometimes produced multiple separate components of 3D reconstruction in spite of the effort to automatically merge them via the component-wise camera tracking technique (Section 5.2). This was often due to the fact that there were not RGB-D frames which would help connect the disconnected components with sufficient precision using our, basically local, pose estimation technique. One solution to this would be to apply a state-of-the-art global optimization method for the pose graph constructed over the set of well-built point clouds. On the other hand, another effective solution was, as proposed in Section 6.5, to allow the user to interactively put the point cloud of each component in place with real objects for guidance in the mixed-reality environment. As the mobile RGB-D sensors and the mixed-reality devices are evolving rapidly, we believe that the 3D modeling method based on the latter approach is prospective, which will enable a smartphone user to scan his/her environments easily with the help of mixed-reality technology.