Indoor Scene Understanding with RGB-D Images: Bottom-up Segmentation, Object Detection and Semantic Segmentation

Abstract

In this paper, we address the problems of contour detection, bottom-up grouping, object detection and semantic segmentation on RGB-D data. We focus on the challenging setting of cluttered indoor scenes, and evaluate our approach on the recently introduced NYU-Depth V2 (NYUD2) dataset (Silberman et al., ECCV, 2012). We propose algorithms for object boundary detection and hierarchical segmentation that generalize the $gPb-ucm$ approach of Arbelaez et al. (TPAMI, 2011) by making effective use of depth information. We show that our system can label each contour with its type (depth, normal or albedo). We also propose a generic method for long-range amodal completion of surfaces and show its effectiveness in grouping. We train RGB-D object detectors by analyzing and computing histogram of oriented gradients on the depth image and using them with deformable part models (Felzenszwalb et al., TPAMI, 2010). We observe that this simple strategy for training object detectors significantly outperforms more complicated models in the literature. We then turn to the problem of semantic segmentation for which we propose an approach that classifies superpixels into the dominant object categories in the NYUD2 dataset. We design generic and class-specific features to encode the appearance and geometry of objects. We also show that additional features computed from RGB-D object detectors and scene classifiers further improves semantic segmentation accuracy. In all of these tasks, we report significant improvements over the state-of-the-art.

Appendices

Appendix 1: Extracting a Geocentric Coordinate Frame

We note that the direction of gravity imposes a lot of structure on how the real world looks (the floor and other supporting surfaces are always horizontal, the walls are always vertical). Hence, to leverage this structure, we develop a simple algorithm to determine the direction of gravity.

Note that this differs from the Manhattan World assumption made by, e.g. Gupta et al. (2011) in the past. The assumption that there are 3 principal mutually orthogonal directions is not universally valid. On the other hand the role of the gravity vector in architectural design is equally important for a hut in Zimbabwe or an apartment in Manhattan.

Since we have depth data available, we propose a simple yet robust algorithm to estimate the direction of gravity. Intuitively, the algorithm tries to find the direction which is the most aligned to or most orthogonal to locally estimated surface normal directions at as many points as possible. The algorithm starts with an estimate of the gravity vector and iteratively refines the estimate via the following 2 steps.

• 1. Using the current estimate of the gravity direction $\varvec{g}_{i-1}$, make hard-assignments of local surface normals to aligned set $\mathcal {N}_{\Vert }$ and orthogonal set $\mathcal {N}_{\bot }$, (based on a threshold $d$ on the angle made by the local surface normal with $\varvec{g}_{i-1}$). Stack the vectors in $\mathcal {N}_{\Vert }$ to form a matrix $N_{\Vert }$, and similarly in $\mathcal {N}_{\bot }$ to form $N_{\bot }$.

  $$\begin{aligned}&\mathcal {N}_{\Vert } = \{\varvec{n}: \theta (\varvec{n},\varvec{g}_{i-1}) < d \text{ or } \theta (\varvec{n},\varvec{g}_{i-1}) > 180^\circ -d\} \\&\mathcal {N}_{\bot } = \{\varvec{n}: 90^\circ -d < \theta (\varvec{n},\varvec{g}_{i-1}) < 90^\circ + d\} \\&\text{ where, } \theta (\varvec{a},\varvec{b}) = \text{ Angle } \text{ between } \varvec{a} \text{ and } \varvec{b}\text{. } \end{aligned}$$

  Typically, $\mathcal {N}_{\Vert }$ would contain normals from points on the floor and table-tops and $\mathcal {N}_{\bot }$ would contain normals from points on the walls.

• 2. Solve for a new estimate of the gravity vector $\varvec{g}_i$ which is as aligned to normals in the aligned set and as orthogonal to the normals in the orthogonal set as possible. This corresponds to solving the following optimization problem, which simplifies into finding the eigen-vector with the smallest eigen value of the $3\times 3$ matrix, $N_{\bot }N^t_{\bot } - N_{\Vert }N^t_{\Vert }$.

  $$\begin{aligned} \min _{\varvec{g}:\Vert \varvec{g}\Vert _2 = 1} {\sum _{\varvec{n}\in \mathcal {N}_{\bot }} {\cos ^2(\theta (\varvec{n},\varvec{g}))} + \sum _{\varvec{n}\in \mathcal {N}_{\Vert }} {\sin ^2(\theta (\varvec{n},\varvec{g}))}} \end{aligned}$$

Our initial estimate for the gravity vector $\varvec{g}_0$ is the Y-axis, and we run 5 iterations with $d = 45^\circ$ followed by 5 iterations with $d = 15^\circ$.

To benchmark the accuracy of our gravity direction, we use the metric of Silberman et al. (2012). We rotate the point cloud to align the Y-axis with the estimated gravity direction and look at the angle the floor makes with the Y-axis. We show the cumulative distribution of the angle of the floor with the Y-axis in Fig. 5. Note that our gravity estimate is within $5^\circ$ of the actual direction for 90 % of the images, and works as well as the method of Silberman et al. (2012), while being significantly simpler.

Fig. 5

Cumulative distribution of angle of the floor with the estimated gravity direction (Color figure online)

Appendix 2: Histogram of Depth Gradients

Suppose we are looking at a plane $N_XX+ N_YY+N_ZZ + d = 0$ in space. A point $\left( X,Y,Z\right)$ in the world gets imaged at the point $\left( x=\frac{fX}{Z},y=\frac{fY}{Z}\right)$, where $f$ is the focal length of the camera. Using this in the first equation, we get the relation, $N_X\frac{Zx}{f}+ N_Y\frac{Zy}{f}+N_ZZ + d = 0$, which simplifies to give $Z = \frac{-fd}{fN_Z+N_Yy+N_Xx}$. Differentiating this with respect to the image gradient gives us,

$$\begin{aligned} \frac{\partial Z}{\partial x}&= \frac{N_XZ^2}{df} \end{aligned}$$

(1)

$$\begin{aligned} \frac{\partial Z}{\partial y}&= \frac{N_YZ^2}{df} \end{aligned}$$

(2)

Using this with the relation that relates disparity $\delta$ with depth value $Z, \delta = \frac{bf}{Z}$, where $b$ is the baseline for the Kinect, gives us the derivatives for the disparity $\delta$ to be

$$\begin{aligned} \frac{\partial \delta }{\partial x}&= \frac{-bN_X}{d} \end{aligned}$$

(3)

$$\begin{aligned} \frac{\partial \delta }{\partial y}&= \frac{-bN_Y}{d} \end{aligned}$$

(4)

Thus, the gradient orientation for both the disparity and the depth image comes out to be $tan^{-1}(\frac{N_Y}{N_X})$ (although the contrast is swapped). The gradient magnitude for the depth image is $\frac{Z^2\sqrt{1-N_Z^2}}{df} = \frac{Z^2\sin (\theta )}{df}$, and for the disparity image is $\frac{b\sqrt{1-N_Z^2}}{d} = \frac{b\sin (\theta )}{d}$, where $\theta$ is the angle that the normal at this point makes with the image plane.

Note that, the gradient magnitude for the disparity and the depth image differ in that the depth gradient has a factor of $Z^2$, which makes points further away have a larger gradient magnitude. This agrees well with the noise model for the Kinect (quantization of the disparity value, which leads to an error in $Z$ which value proportional to $Z^2$). In this sense, the disparity gradient is much better behaved than the gradient of the depth image.

Note that, the subsequent contrast normalization step in standard HOG computation, essentially gets rid of this difference between these 2 quantities (assuming that the close by cells have more or less comparable $Z$ values).

Appendix 3: Visualization for RGB-D Detector Parts

One interesting thing to visualize is what the DPM is learning with these features. The question that we want to ask here is that whether the parts that we get semantically meaningful? The hope is that with access to depth data, the parts that get discovered should be more meaningful than ones you get with purely appearance data.

In Fig. 6, we visualize the various DPM parts for the bed, chair and toilet detector. We run the detector on a set of images that the detector did not see at train time, pick the top few detections based on the detector score. We then crop out the part of the image that a particular part of the DPM got placed at, and visualize these image patches for the different DPM parts.

Fig. 6

Visualization for the DPM parts for bed, chairs and toilets

We observe that the parts are tight semantically—that is, a particular part likes semantically similar regions of the object class. For comparison, we also provide visualizations for the parts that get learnt for an appearance only DPM. As expected, the parts from our DPM are semantically tighter than the parts from an appearance only DPM. Recently, in the context of intensity images there has been a lot of work in trying to get mid-level parts in an unsupervised manner from weak annotations like that of bounding boxes in intensity images (Endres et al. 2013), and in a supervised manner from strong annotations like that of keypoint annotation (Bourdev et al. 2010). These visualizations suggest that it may be possible to get very reasonable mid-level parts from weak annotations in RGB-D images, which can be used to train appearance only part detectors.

We also visualize what the HHG features learn. In Fig. 7, we see that the model as expected picks on the shape cues. There is a flat horizontal surface along the sides and on the middle portion which corresponds to the floor and the top of the bed and there are vertical surfaces going from the horizontal floor to the top of the bed.

Fig. 7

Root and part filters for the bed. We can see that the model captures the shape for a bed. Horizontal lines correspond to horizontal surfaces and the vertical lines correspond to vertical surface. We can see that the model learnt a box which we are looking at towards one of its corners

Appendix 4: Scene Classification

We address the task of indoor scene classification based on the idea that a scene can be recognized by identifying the objects in it. Thus, we use our predicted semantic segmentation maps as features for this task. We use the spatial pyramid (SPM) formulation of Lazebnik et al. (2006), but instead of using histograms of vector quantized SIFT descriptors as features, we use the average presence of each semantic class (as predicted by our algorithm) in each pyramid cell as our feature.

To evaluate our performance, we use the scene labels provided by the NYUD2. The dataset has 27 scene categories but only a few are well represented. Hence, we reduce the 27 categories into 10 categories (9 most common categories and the rest). As before, we train on the 795 images from the train set and test on the remaining 654 images. We report the diagonal of the confusion matrix for each of the scene class and use the mean of the diagonal, and overall accuracy as aggregate measures of performance.

We use a $1, 2\times 2, 4\times 4$ spatial pyramid and use a SVM with an additive kernel as our classifier (Maji et al. 2013). We use the 40 category output of our algorithm. We compare against an appearance-only baseline based on SPM on vector quantized color SIFT descriptors (van de Sande et al. 2010), a geometry-only baseline based on SPM on geocentric textons (introduced in Sect. 5.1.2), and a third baseline which uses both SIFT and Geocentric Textons in the SPM.

We report the performance we achieve in Table 9.

Table 9 Performance on the scene classification task: we report the diagonal entry of the confusion matrix for each category; and the mean diagonal of the confusion matrix and the overall accuracy as aggregate metrics