Color-indoor: Incorporating Depth into Room Decoration Visualization
Authors: 
Zhaoxiang Wei, Shikui Wei, Ting Liu, Sen Xu, Yao ZhaoAuthors Info & Claims
Zhaoxiang Wei, Shikui Wei, Ting Liu, Sen Xu, Yao ZhaoAuthors Info & ClaimsArticle No.: 236, Pages 1 - 8
Abstract
Combined with computer vision technology, we propose an system to automatically visualize the decoration effect of 3D complex indoor scenes, named Color-indoor. Given a preferred color and RGB-D images, the Color-indoor system can be used for color replacement, editing texture, and synthesizing 3D result for specified semantic regions of the input image. The key idea of the proposed Color-indoor is leveraging depth information to guide the entire segmentation process and 3D data synthesis. We propose an depth-fusion criss-cross attention semantic segmentation framework (DFCCN) for parsing the indoor semantic scene, and introduce a depth branch to better extracted geometry information from different semantic areas. We utilize DFCCN to extract and fuse features from RGB branch and depth branch, so that the segmentation network can obtain more geometry information and enrich the structural details of features. Located the specified semantic regions, a simple yet effective editing algorithm is proposed for color and texture replacement. Combined the camera parameters, the 3D data synthesis algorithm are used to generate 3D results from edited images and depth images. For training and testing, we set up a new RGB-D dataset upon NYUv2 including 6 semantic labels. The experimental and visual results are demonstrated that our proposed Color-indoor can generate harmonious 3D results.
References
[1]
Silberman, Nathan, “Indoor segmentation and support inference from rgbd images,” European conference on computer vision. Springer, Berlin, Heidelberg, 2012.
[2]
Ren, Xiaofeng, Liefeng Bo, and Dieter Fox, “Rgb-(d) scene labeling: Features and algorithms,” 2012 IEEE Conference on Computer Vision and Pattern Recognition. IEEE, 2012.
[3]
Qi, Charles R., "Pointnet: Deep learning on point sets for 3d classification and segmentation," Proceedings of the IEEE conference on computer vision and pattern recognition, 2017.
[4]
Hazirbas, Caner, "Fusenet: Incorporating depth into semantic segmentation via fusion-based cnn architecture," Asian conference on computer vision. Springer, Cham, 2016.
[5]
Huang, Zilong, "Ccnet: Criss-cross attention for semantic segmentation," Proceedings of the IEEE International Conference on Computer Vision. 2019.
[6]
Jonathan Long, Evan Shelhamer, and Trevor Darrell, “Fully convolutional networks for semantic segmentation,” In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3431-3440, 2015.
[7]
Changqian Yu, Jingbo Wang, Chao Peng, Changxin Gao, Gang Yu, and Nong Sang, “Learning a discriminative feature network for semantic segmentation,” arXiv preprint arXiv:1804.09337, 2018.
[8]
Guosheng Lin, Anton Milan, Chunhua Shen, and Ian D Reidm “Refinenet: Multi-path refinement networks for highresolution semantic segmentation,” In Cvpr, vol. 1, p. 5, 2017.
[9]
Olaf Ronneberger, Philipp Fischer, and Thomas Brox, “Unet: Convolutional networks for biomedical image segmentation,” In International Conference on Medical image computing and computer-assisted intervention, pp. 234-241. Springer, 2015.
[10]
Eigen, D., and Fergus, R., “Predicting depth, surface normals and semantic labels with a common multi-scale convolutional architecture,” In: ICCV, 2015.
[11]
Ma, L., Stueckler, J., Kerl, C., and Cremers, D., “Multi-view deep learning for consistent semantic mapping with rgb-d cameras,” In: IROS, 2017.
[12]
Li, Z., Gan, Y., Liang, X., Yu, Y., Cheng, H., and Lin, L., “Lstm-cf: Unifying context modeling and fusion with lstms for rgb-d scene labeling,” In: ECCV, 2016.
[13]
Park, S.J., Hong, K.S., and Lee, S., “Rdfnet: Rgb-d multi-level residual feature fusion for indoor semantic segmentation,” In: ICCV, 2017.
[14]
Cheng, Y., Cai, R., Li, Z., Zhao, X., and Huang, K., “Locality-sensitive deconvolution networks with gated fusion for rgb-d indoor semantic segmentation,” In: CVPR, 2017.
[15]
Song, S., Yu, F., Zeng, A., Chang, A.X., Savva, M., and Funkhouser, T., “Semantic scene completion from a single depth image,” In: CVPR, 2017.
[16]
Song, S., and Xiao, J., “Deep Sliding Shapes for amodal 3D object detection in RGB-D images,” In: CVPR, 2016.
[17]
Wang, W., Yu, R., Huang, Q., and Neumann, U., “Sgpn: Similarity group proposal network for 3d point cloud instance segmentation,” CVPR, 2018.
[18]
Huang, Q., Wang, W., and Neumann, U., “Recurrent slice networks for 3d segmentation on point clouds,” CVPR, 2018.
[19]
Qi, C.R., Yi, L., Su, H., and Guibas, L.J., “Pointnet++: Deep hierarchical feature learning on point sets in a metric space,” In: NIPS, 2017.
[20]
Deng, Jia, "Imagenet: A large-scale hierarchical image database," 2009 IEEE conference on computer vision and pattern recognition, Ieee, 2009.
[21]
Ioffe, Sergey, and Christian Szegedy, “Batch normalization: Accelerating deep network training by reducing internal covariate shift,” arXiv preprint arXiv: 1502.03167, 2015.
[22]
Jiao, Jianbo, “Geometry-aware distillation for indoor semantic segmentation,” Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2019.
[23]
Chen, Liang-Chieh, “Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs,” IEEE transactions on pattern analysis and machine intelligence vol. 40, no. 4, pp. 834-848, 2017.
[24]
Qi, Xiaojuan, “3d graph neural networks for rgbd semantic segmentation,” Proceedings of the IEEE International Conference on Computer Vision, 2017.
Recommendations
Indoor semantic segmentation based on Swin-Transformer
AbstractIn recent years, with the rapid development of Transformer in the field of natural language processing, many researchers have realized its potential and gradually applied it to the field of computer vision, with a proliferation of theoretical ...
Indoor Scene Understanding with RGB-D Images: Bottom-up Segmentation, Object Detection and Semantic Segmentation
In this paper, we address the problems of contour detection, bottom-up grouping, object detection and semantic segmentation on RGB-D data. We focus on the challenging setting of cluttered indoor scenes, and evaluate our approach on the recently ...
RGB-D Gate-guided edge distillation for indoor semantic segmentation
AbstractFusing the RGB and depth information can significantly improve the performance of semantic segmentation since the depth data represents the geometric information. In this paper, we propose a novel Gate-guided Edge Distillation (GED) based approach ...
Comments
Information & Contributors
Information
Published In
May 2021
2053 pages
ISBN:9781450390200
DOI:10.1145/3469213
Copyright © 2021 ACM.
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 18 August 2021
Check for updates
Author Tags
Qualifiers
- Research-article
- Research
- Refereed limited
Conference
ICAIIS 2021
ICAIIS 2021: 2021 2nd International Conference on Artificial Intelligence and Information Systems
May 28 - 30, 2021
Chongqing, China
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 35Total Downloads
- Downloads (Last 12 months)11
- Downloads (Last 6 weeks)1
Reflects downloads up to 04 Oct 2024
Other Metrics
Citations
View Options
Get Access
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign inFull Access
View options
View or Download as a PDF file.
PDFeReader
View online with eReader.
eReaderHTML Format
View this article in HTML Format.
HTML Format