TwinTex: Geometry-Aware Texture Generation for Abstracted 3D Architectural Models
Authors: 
Weidan Xiong, Hongqian Zhang, Botao Peng, Ziyu Hu, Yongli Wu, Jianwei Guo, 
Hui HuangAuthors Info & Claims
Weidan Xiong, Hongqian Zhang, Botao Peng, Ziyu Hu, Yongli Wu, Jianwei Guo, Hui HuangAuthors Info & ClaimsArticle No.: 227, Pages 1 - 14
Abstract
Coarse architectural models are often generated at scales ranging from individual buildings to scenes for downstream applications such as Digital Twin City, Metaverse, LODs, etc. Such piece-wise planar models can be abstracted as twins from 3D dense reconstructions. However, these models typically lack realistic texture relative to the real building or scene, making them unsuitable for vivid display or direct reference. In this paper, we present TwinTex, the first automatic texture mapping framework to generate a photorealistic texture for a piece-wise planar proxy. Our method addresses most challenges occurring in such twin texture generation. Specifically, for each primitive plane, we first select a small set of photos with greedy heuristics considering photometric quality, perspective quality and facade texture completeness. Then, different levels of line features (LoLs) are extracted from the set of selected photos to generate guidance for later steps. With LoLs, we employ optimization algorithms to align texture with geometry from local to global. Finally, we fine-tune a diffusion model with a multi-mask initialization component and a new dataset to inpaint the missing region. Experimental results on many buildings, indoor scenes and man-made objects of varying complexity demonstrate the generalization ability of our algorithm. Our approach surpasses state-of-the-art texture mapping methods in terms of high-fidelity quality and reaches a human-expert production level with much less effort.
Supplementary Material
References
[1]
Hendrik Baatz, Jonathan Granskog, Marios Papas, Fabrice Rousselle, and Jan Novák. 2022. NeRF-Tex: Neural Reflectance Field Textures. Comp. Graph. Forum 41, 6 (2022), 287--301.
[2]
Connelly Barnes, Eli Shechtman, Adam Finkelstein, and Dan B Goldman. 2009. Patch-Match: A randomized correspondence algorithm for structural image editing. ACM Trans. Graph. 28, 3 (2009), 24:1--24:11.
[3]
Jean-Philippe Bauchet and Florent Lafarge. 2020. Kinetic shape reconstruction. ACM Trans. Graph. 39, 5 (2020), 156:1--156:14.
[4]
Fausto Bernardini, Ioana M. Martin, and Holly Rushmeier. 2001. High-quality texture reconstruction from multiple scans. IEEE Trans. on Vis. and Comp. Graph. 7, 4 (2001), 318--332.
[5]
Paul J Besl and Neil D McKay. 1992. Method for registration of 3-D shapes. In Sensor fusion IV: control paradigms and data structures, Vol. 1611. 586--606.
[6]
Sai Bi, Nima Khademi Kalantari, and Ravi Ramamoorthi. 2017. Patch-based optimization for image-based texture mapping. ACM Trans. Graph. (SIGGRAPH) 36, 4 (2017), 106:1--106:11.
[7]
Vasileios Bouzas, Hugo Ledoux, and Liangliang Nan. 2020. Structure-aware building mesh polygonization. ISPRS Journal of Photogrammetry and Remote Sensing 167 (2020), 432--442.
[8]
Yuri Boykov, Olga Veksler, and Ramin Zabih. 2001. Fast approximate energy minimization via graph cuts. IEEE Trans. Pattern Anal. Mach. Intell. 23, 11 (2001), 1222--1239.
[9]
Matthew Brown and David G Lowe. 2007. Automatic panoramic image stitching using invariant features. Int. Journal of Computer Vision 74, 1 (2007), 59--73.
[10]
Marco Callieri, Paolo Cignoni, Massimiliano Corsini, and Roberto Scopigno. 2008. Masked photo blending: Mapping dense photographic data set on high-resolution sampled 3D models. Computers & Graphics 32, 4 (2008), 464--473.
[11]
Duygu Ceylan, Niloy J Mitra, Hao Li, Thibaut Weise, and Mark Pauly. 2012. Factored facade acquisition using symmetric line arrangements. Comp. Graph. Forum (Proc. EUROGRAPHICS) 31, 2pt3 (2012), 671--680.
[12]
Duygu Ceylan, Niloy J Mitra, Youyi Zheng, and Mark Pauly. 2014. Coupled structure-from-motion and 3D symmetry detection for urban facades. ACM Trans. Graph. 33, 1 (2014), 1--15.
[13]
Che-Han Chang, Yoichi Sato, and Yung-Yu Chuang. 2014. Shape-preserving half-projective warps for image stitching. In IEEE Computer Vision and Pattern Recognition (CVPR). 3254--3261.
[14]
David Cohen-Steiner, Pierre Alliez, and Mathieu Desbrun. 2004. Variational shape approximation. In Proc. ACM SIGGRAPH. 905--914.
[15]
Florinel-Alin Croitoru, Vlad Hondru, Radu Tudor Ionescu, and Mubarak Shah. 2023. Diffusion models in vision: A survey. IEEE Trans. Pattern Anal. Mach. Intell. (2023).
[16]
Paul Debevec, Yizhou Yu, and George Borshukov. 1998. Efficient view-dependent image-based rendering with projective texture-mapping. In Eurographics Workshop on Rendering Techniques. 105--116.
[17]
Paul E Debevec, Camillo J Taylor, and Jitendra Malik. 1996. Modeling and rendering architecture from photographs: A hybrid geometry-and image-based approach. In Proc. ACM SIGGRAPH. 11--20.
[18]
Prafulla Dhariwal and Alexander Nichol. 2021. Diffusion models beat gans on image synthesis. Proc. International Conference on Neural Information Processing Systems 34 (2021), 8780--8794.
[19]
Omar Elharrouss, Noor Almaadeed, Somaya Al-Maadeed, and Younes Akbari. 2020. Image Inpainting: A Review. Neural Process. Letters 51, 2 (2020), 2007--2028.
[20]
Hao Fang and Florent Lafarge. 2020. Connect-and-Slice: an hybrid approach for reconstructing 3D objects. In IEEE Computer Vision and Pattern Recognition (CVPR). 13490--13498.
[21]
Yanping Fu, Qingan Yan, Jie Liao, and Chunxia Xiao. 2020. Joint texture and geometry optimization for RGB-D reconstruction. In IEEE Computer Vision and Pattern Recognition (CVPR). 5950--5959.
[22]
Yanping Fu, Qingan Yan, Long Yang, Jie Liao, and Chunxia Xiao. 2018. Texture mapping for 3d reconstruction with RGB-D sensor. In IEEE Computer Vision and Pattern Recognition (CVPR). 4645--4653.
[23]
Ran Gal, Yonatan Wexler, Eyal Ofek, Hugues Hoppe, and Daniel Cohen-Or. 2010. Seamless montage for texturing models. Comp. Graph. Forum 29, 2 (2010), 479--486.
[24]
Junhong Gao, Seon Joo Kim, and Michael S Brown. 2011. Constructing image panoramas using dual-homography warping. In IEEE Computer Vision and Pattern Recognition (CVPR). 49--56.
[25]
Ignacio Garcia-Dorado, Ilke Demir, and Daniel G Aliaga. 2013. Automatic urban modeling using volumetric reconstruction with surface graph cuts. Computers & Graphics 37, 7 (2013), 896--910.
[26]
Michael Garland and Paul S Heckbert. 1997. Surface simplification using quadric error metrics. In Proc. ACM SIGGRAPH. 209--216.
[27]
Yiangos Georgiou, Melinos Averkiou, Tom Kelly, and Evangelos Kalogerakis. 2021. Projective Urban Texturing. In International Conference on 3D Vision (3DV). 1034--1043.
[28]
Christine Guillemot and Olivier Le Meur. 2014. Image Inpainting: Overview and Recent Advances. IEEE Signal Processing Magazine 31, 1 (2014), 127--144.
[29]
Jianwei Guo, Yanchao Liu, Xin Song, Haoyu Liu, Xiaopeng Zhang, and Zhanglin Cheng. 2022. Line-Based 3D Building Abstraction and Polygonal Surface Reconstruction From Images. IEEE Trans. on Vis. and Comp. Graph. (2022), 1--15.
[30]
Jingwei Huang, Angela Dai, Leonidas J Guibas, and Matthias Nießner. 2017. 3Dlite: towards commodity 3D scanning for content creation. ACM Trans. Graph. (SIGGRAPH Asia) 36, 6 (2017), 203:1--203:14.
[31]
Jia-Bin Huang, Sing Bing Kang, Narendra Ahuja, and Johannes Kopf. 2014. Image completion using planar structure guidance. ACM Trans. Graph. (SIGGRAPH) 33, 4 (2014), 129:1--129:10.
[32]
Jireh Jam, Connah Kendrick, Kevin Walker, Vincent Drouard, Jison Gee-Sern Hsu, and Moi Hoon Yap. 2021. A comprehensive review of past and present image inpainting methods. Computer vision and image understanding 203 (2021), 103147.
[33]
Qi Jia, ZhengJun Li, Xin Fan, Haotian Zhao, Shiyu Teng, Xinchen Ye, and Longin Jan Latecki. 2021. Leveraging line-point consistence to preserve structures for wide parallax image stitching. In IEEE Computer Vision and Pattern Recognition (CVPR). 12186--12195.
[34]
Sergey Kastryulin, Jamil Zakirov, Denis Prokopenko, and Dmitry V Dylov. 2022. PyTorch Image Quality: Metrics for Image Quality Assessment. arXiv preprint arXiv:2208.14818 (2022).
[35]
Tom Kelly, Paul Guerrero, Anthony Steed, Peter Wonka, and Niloy J. Mitra. 2018. FrankenGAN: Guided Detail Synthesis for Building Mass Models Using Style-Synchonized GANs. ACM Trans. Graph. (SIGGRAPH Asia) 37, 6 (2018), 216:1--216:14.
[36]
Vivek Kwatra, Arno Schödl, Irfan Essa, Greg Turk, and Aaron Bobick. 2003. Graphcut textures: Image and video synthesis using graph cuts. ACM Trans. Graph. 22, 3 (2003), 277--286.
[37]
Kyu-Yul Lee and Jae-Young Sim. 2020. Warping residual based image stitching for large parallax. In IEEE Computer Vision and Pattern Recognition (CVPR). 8198--8206.
[38]
Victor Lempitsky and Denis Ivanov. 2007. Seamless mosaicing of image-based texture maps. In IEEE Computer Vision and Pattern Recognition (CVPR). 1--6.
[39]
Nan Li, Yifang Xu, and Chao Wang. 2017. Quasi-homography warps in image stitching. IEEE Trans. Multimedia 20, 6 (2017), 1365--1375.
[40]
Shiwei Li, Lu Yuan, Jian Sun, and Long Quan. 2015. Dual-feature warping-based motion model estimation. In IEEE International Conference on Computer Vision (ICCV). 4283--4291.
[41]
Tianli Liao and Nan Li. 2019. Single-perspective warps in natural image stitching. IEEE Trans. Image Process. 29 (2019), 724--735.
[42]
Chung-Ching Lin, Sharathchandra U Pankanti, Karthikeyan Natesan Ramamurthy, and Aleksandr Y Aravkin. 2015. Adaptive as-natural-as-possible image stitching. In IEEE Computer Vision and Pattern Recognition (CVPR). 1155--1163.
[43]
Liqiang Lin, Yilin Liu, Yue Hu, Xingguang Yan, Ke Xie, and Hui Huang. 2022. Capturing, Reconstructing, and Simulating: the UrbanScene3D Dataset. In European Conference on Computer Vision (ECCV). 93--109.
[44]
Wen-Yan Lin, Siying Liu, Yasuyuki Matsushita, Tian-Tsong Ng, and Loong-Fah Cheong. 2011. Smoothly varying affine stitching. In IEEE Computer Vision and Pattern Recognition (CVPR). 345--352.
[45]
Andreas Lugmayr, Martin Danelljan, Andres Romero, Fisher Yu, Radu Timofte, and Luc Van Gool. 2022. Repaint: Inpainting using denoising diffusion probabilistic models. In IEEE Computer Vision and Pattern Recognition (CVPR). 11461--11471.
[46]
Robert Maier, Kihwan Kim, Daniel Cremers, Jan Kautz, and Matthias Nießner. 2017. Intrinsic3D: High-quality 3D reconstruction by joint appearance and geometry optimization with spatially-varying lighting. In IEEE International Conference on Computer Vision (ICCV). 3114--3122.
[47]
Jalpa D Mehta and SG Bhirud. 2011. Image stitching techniques. In Thinkquest˜ 2010: Proceedings of the First International Conference on Contours of Computing Technology. 74--80.
[48]
Gal Metzer, Elad Richardson, Or Patashnik, Raja Giryes, and Daniel Cohen-Or. 2022. Latent-NeRF for Shape-Guided Generation of 3D Shapes and Textures. arXiv preprint arXiv:2211.07600 (2022).
[49]
Aron Monszpart, Nicolas Mellado, Gabriel J Brostow, and Niloy J Mitra. 2015. RAPter: rebuilding man-made scenes with regular arrangements of planes. ACM Trans. Graph. (SIGGRAPH) 34, 4 (2015), 103:1--103:12.
[50]
Thomas Müller, Alex Evans, Christoph Schied, and Alexander Keller. 2022. Instant Neural Graphics Primitives with a Multiresolution Hash Encoding. ACM Trans. Graph. (SIGGRAPH) 41, 4 (2022), 102:1--102:15.
[51]
Przemyslaw Musialski, Christian Luksch, Michael Schwärzler, Matthias Buchetics, Stefan Maierhofer, and Werner Purgathofer. 2010. Interactive Multi-View Facade Image Editing. In Vision, Modeling, and Visualization. 131--138.
[52]
Przemyslaw Musialski, Peter Wonka, Daniel G Aliaga, Michael Wimmer, Luc Van Gool, and Werner Purgathofer. 2013. A survey of urban reconstruction. Comp. Graph. Forum 32, 6 (2013), 146--177.
[53]
Liangliang Nan and Peter Wonka. 2017. PolyFit: Polygonal surface reconstruction from point clouds. In IEEE International Conference on Computer Vision (ICCV). 2353--2361.
[54]
Shanshan Pan, Jiahui Lv, Hao Fang, and Hui Huang. 2022. Efficient and robust 3D structure-aware reconstruction. Journal of Image and Graphics 27, 2 (2022), 421--434.
[55]
Patrick Pérez, Michel Gangnet, and Andrew Blake. 2003. Poisson image editing. In Proc. ACM SIGGRAPH. 313--318.
[56]
David Josué Barrientos Rojas, Bruno José Torres Fernandes, and Sergio Murilo Maciel Fernandes. 2020. A Review on Image Inpainting Techniques and Datasets. In SIBGRAPI Conference on Graphics, Patterns and Images. 240--247.
[57]
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. 2015. Imagenet large scale visual recognition challenge. Int. Journal of Computer Vision 115, 3 (2015), 211--252.
[58]
David Salinas, Florent Lafarge, and Pierre Alliez. 2015. Structure-aware mesh decimation. Comp. Graph. Forum 34, 6 (2015), 211--227.
[59]
Sudipta N Sinha, Drew Steedly, Richard Szeliski, Maneesh Agrawala, and Marc Pollefeys. 2008. Interactive 3D architectural modeling from unordered photo collections. ACM Trans. Graph. 27, 5 (2008), 159:1--159:10.
[60]
Neil Smith, Nils Moehrle, Michael Goesele, and Wolfgang Heidrich. 2018. Aerial Path Planning for Urban Scene Reconstruction: A Continuous Optimization Method and Benchmark. ACM Trans. Graph. (SIGGRAPH Asia) 37, 6 (2018), 183:1--183:15.
[61]
Iago Suárez, José M Buenaposada, and Luis Baumela. 2022. ELSED: Enhanced line SEgment drawing. Pattern Recognition 127 (2022), 108619.
[62]
Richard Szeliski et al. 2007. Image alignment and stitching: A tutorial. Foundations and Trends® in Computer Graphics and Vision 2, 1 (2007), 1--104.
[63]
Yannick Verdie, Florent Lafarge, and Pierre Alliez. 2015. LOD generation for urban scenes. ACM Trans. Graph. 34, 3 (2015), 30:1--30:14.
[64]
Michael Waechter, Mate Beljan, Simon Fuhrmann, Nils Moehrle, Johannes Kopf, and Michael Goesele. 2017. Virtual Rephotography: Novel View Prediction Error for 3D Reconstruction. ACM Trans. Graph. 36, 1, Article 8 (2017), 11 pages.
[65]
Michael Waechter, Nils Moehrle, and Michael Goesele. 2014. Let there be color! Large-scale texturing of 3D reconstructions. In European Conference on Computer Vision (ECCV). 836--850.
[66]
Bin Wang, Pan Pan, Qinjie Xiao, Likang Luo, Xiaofeng Ren, Rong Jin, and Xiaogang Jin. 2018. Seamless Color Mapping for 3D Reconstruction with Consumer-Grade Scanning Devices. In European Conference on Computer Vision (ECCV) Workshops. 633--648.
[67]
Chao Wang and Xiaohu Guo. 2018. Plane-based optimization of geometry and texture for RGB-D reconstruction of indoor scenes. In International Conference on 3D Vision (3DV). 533--541.
[68]
Tian-Zhu Xiang, Gui-Song Xia, Xiang Bai, and Liangpei Zhang. 2018. Image stitching by line-guided local warping with global similarity constraint. Pattern recognition 83 (2018), 481--497.
[69]
Cem Yuksel, Sylvain Lefebvre, and Marco Tarini. 2019. Rethinking texture mapping. Comp. Graph. Forum 38, 2 (2019), 535--551.
[70]
Julio Zaragoza, Tat-Jun Chin, Michael S Brown, and David Suter. 2013. As-projective-as-possible image stitching with moving DLT. In IEEE Computer Vision and Pattern Recognition (CVPR). 2339--2346.
[71]
Guofeng Zhang, Yi He, Weifeng Chen, Jiaya Jia, and Hujun Bao. 2016. Multi-viewpoint panorama construction with wide-baseline images. IEEE Trans. Image Process. 25, 7 (2016), 3099--3111.
[72]
Han Zhang, Yucong Yao, Ke Xie, Chi-Wing Fu, Hao Zhang, and Hui Huang. 2021. Continuous Aerial Path Planning for 3D Urban Scene Reconstruction. ACM Trans. Graph. (SIGGRAPH Asia) 40, 6 (2021), 225:1--225:15.
[73]
Lilian Zhang and Reinhard Koch. 2013. An efficient and robust line segment matching approach based on LBD descriptor and pairwise geometric consistency. Journal of Visual Communication and Image Representation 24, 7 (2013), 794--805.
[74]
Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. 2018. The unreasonable effectiveness of deep features as a perceptual metric. In IEEE Computer Vision and Pattern Recognition (CVPR). 586--595.
[75]
Hengshuang Zhao, Jianping Shi, Xiaojuan Qi, Xiaogang Wang, and Jiaya Jia. 2017. Pyramid scene parsing network. In IEEE Computer Vision and Pattern Recognition (CVPR). 2881--2890.
[76]
Qian-Yi Zhou and Vladlen Koltun. 2014. Color map optimization for 3D reconstruction with consumer depth cameras. ACM Trans. Graph. (SIGGRAPH) 33, 4 (2014), 155:1--155:10.
[77]
Xiaohui Zhou, Ke Xie, Kai Huang, Yilin Liu, Yang Zhou, Minglun Gong, and Hui Huang. 2020. Offsite Aerial Path Planning for Efficient Urban Scene Reconstruction. ACM Trans. Graph. (SIGGRAPH Asia) 39, 6 (2020), 192:1--192:16.
Index Terms
- TwinTex: Geometry-Aware Texture Generation for Abstracted 3D Architectural Models
Recommendations
Texture map generation for 3D reconstructed scenes
We present a novel method for generating texture maps for 3D geometric models reconstructed using consumer RGB-D sensors. Our method generates a texture map for a simplified 3D mesh of the reconstructed scene using spatially and temporally sub-sampled ...
Space-Optimized Texture Atlases for 3D Scenes with Per-polygon Textures
PACIFIC_GRAPHICS '10: Proceedings of the 2010 18th Pacific Conference on Computer Graphics and ApplicationsReal-time rendering of massively textured urban scenes is still a challenging problem in computer graphics. Texture atlases and the recently-introduced texture arrays have proven to be a powerful tool to accelerate the rendering of such scenes by ...
Texture analysis and repacking for improved storage efficiency
VRST '16: Proceedings of the 22nd ACM Conference on Virtual Reality Software and TechnologyTextures are widely used in modern computer graphics. Their size, however, is often a limiting factor. Considering the widespread adaptation of mobile virtual and augmented reality applications, efficient storage of textures has become an important ...
Comments
Information & Contributors
Information
Published In
Copyright © 2023 ACM.
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 05 December 2023
Published in TOG Volume 42, Issue 6
Check for updates
Author Tags
Qualifiers
- Research-article
Funding Sources
- DEGP Innovation Team
- NSFC
- Shenzhen Science and Technology Program
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 144Total Downloads
- Downloads (Last 12 months)144
- Downloads (Last 6 weeks)19
Reflects downloads up to 26 Jul 2024
Other Metrics
Citations
View Options
Get Access
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in