research-article

BigSUR: large-scale structured urban reconstruction

Authors:

Niloy J. MitraAuthors Info & Claims

ACM Transactions on Graphics (TOG), Volume 36, Issue 6

Article No.: 204, Pages 1 - 16

https://doi.org/10.1145/3130800.3130823

Published: 20 November 2017 Publication History

Abstract

The creation of high-quality semantically parsed 3D models for dense metropolitan areas is a fundamental urban modeling problem. Although recent advances in acquisition techniques and processing algorithms have resulted in large-scale imagery or 3D polygonal reconstructions, such data-sources are typically noisy, and incomplete, with no semantic structure. In this paper, we present an automatic data fusion technique that produces high-quality structured models of city blocks. From coarse polygonal meshes, street-level imagery, and GIS footprints, we formulate a binary integer program that globally balances sources of error to produce semantically parsed mass models with associated facade elements. We demonstrate our system on four city regions of varying complexity; our examples typically contain densely built urban blocks spanning hundreds of buildings. In our largest example, we produce a structured model of 37 city blocks spanning a total of 1, 011 buildings at a scale and quality previously impossible to achieve automatically.

Supplementary Material

ZIP File (a204-kelly.zip)

Supplemental material.

Download
16.74 MB

MP4 File (a204-kelly.mp4)

Download
268.90 MB

References

[1]

Lama Affara, Liangliang Nan, Bernard Ghanem, and Peter Wonka. 2016. Large Scale Asset Extraction for Urban Images. ECCV (2016), 437--452.

[2]

Oswin Aichholzer, Franz Aurenhammer, David Alberts, and Bernd Gärtner. 1996. A novel type of skeleton for polygons. In The Journal of Universal Computer Science. Springer, 752--761.

[3]

Dragomir Anguelov, Carole Dulong, Daniel Filip, Christian Frueh, Stéphane Lafon, Richard Lyon, Abhijit Ogale, Luc Vincent, and Josh Weaver. 2010. Google street view: Capturing the world at street level. Computer 43, 6 (2010), 32--38.

Digital Library

[4]

Vijay Badrinarayanan, Alex Kendall, and Roberto Cipolla. 2017. SegNet: A Deep Convolutional Encoder-Decoder Architecture for Image Segmentation. IEEE TPAMI (2017).

[5]

Therese Biedl, Stefan Huber, and Peter Palfrader. 2016. Planar matchings for weighted straight skeletons. International Journal of Computational Geometry & Applications 26, 03n04 (2016), 211--229.

Digital Library

[6]

Claus Brenner. 2005. Building reconstruction from images and laser scanning. International Journal of Applied Earth Observation and Geoinformation 6, 3 (2005), 187--198.

[7]

Gabriel J. Brostow, Jamie Shotton, Julien Fauqueur, and Roberto Cipolla. 2008. Segmentation and Recognition Using Structure from Motion Point Clouds. ECCV (2008), 44--57.

Digital Library

[8]

Duygu Ceylan, Minh Dang, Niloy J. Mitra, Boris Neubert, and Mark Pauly. 2016. Discovering Structured Variations Via Template Matching. CGF (01 2016).

Digital Library

[9]

Duygu Ceylan, Niloy J. Mitra, Youyi Zheng, and Mark Pauly. 2013. Coupled Structure-from-Motion and 3D Symmetry Detection for Urban Facades. ACM TOG (2013), 15.

Digital Library

[10]

John W. Chinneck. 2008. Feasibility and Infeasibility in Optimization: Algorithms and Computational Methods. Springer.

Digital Library

[11]

Andrea Cohen, Alexander G Schwing, and Marc Pollefeys. 2014. Efficient structured parsing of facades using dynamic programming. IEEE CVPR (2014), 3206--3213.

Digital Library

[12]

Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. 2016. The Cityscapes Dataset for Semantic Urban Scene Understanding. In IEEE CVPR.

[13]

Dengxin Dai, Mukta Prasad, Gerhard Schmitt, and Luc Van Gool. 2012. Learning domain knowledge for facade labelling. ECCV (2012), 710--723.

Digital Library

[14]

Paul E Debevec, Camillo J Taylor, and Jitendra Malik. 1996. Modeling and rendering architecture from photographs: A hybrid geometry-and image-based approach. ACM SIGGRAPH (1996), 11--20.

Digital Library

[15]

Anthony R Dick, Philip HS Torr, and Roberto Cipolla. 2004. Modelling and interpretation of architecture from several images. IJCV 60, 2 (2004), 111--134.

Digital Library

[16]

David Eigen and Rob Fergus. 2015. Predicting depth, surface normals and semantic labels with a common multi-scale convolutional architecture. IEEE ICCV (2015), 2650--2658.

Digital Library

[17]

David Eppstein and Jeff Erickson. 1999. Raising roofs, crashing cycles, and playing pool: Applications of a data structure for finding pairwise interactions. Discrete & Computational Geometry 22, 4 (1999), 569--592.

[18]

Tian Fang, Zhexi Wang, Honghui Zhang, and Long Quan. 2013. Image-based modeling of unwrappable facades. IEEE TVCG 19, 10 (2013), 1720--1731.

[19]

K. Fukunaga and L. Hostetler. 1975. The estimation of the gradient of a density function, with applications in pattern recognition. IEEE TIT 21, 1 (January 1975), 32--40.

Digital Library

[20]

Yasutaka Furukawa and Jean Ponce. 2010. Accurate, dense, and robust multiview stereopsis. IEEE PAMI 32, 8 (2010), 1362--1376.

Digital Library

[21]

Ignacio Garcia-Dorado, Ilke Demir, and Daniel G Aliaga. 2013. Automatic urban modeling using volumetric reconstruction with surface graph cuts. Computers & Graphics 37, 7 (2013), 896--910.

Digital Library

[22]

Aleksey Golovinskiy, Vladimir G Kim, and Thomas Funkhouser. 2009. Shape-based recognition of 3D point clouds in urban environments. IEEE ICCV (2009), 2154--2161.

[23]

Gurobi. 2016. Gurobi Optimizer Reference Manual. (2016). http://www.gurobi.com

[24]

Yangqing Jia, Evan Shelhamer, Jeff Donahue, Sergey Karayev, Jonathan Long, Ross Girshick, Sergio Guadarrama, and Trevor Darrell. 2014. Caffe: Convolutional Architecture for Fast Feature Embedding. arXiv preprint arXiv:1408.5093 (2014).

[25]

Haiyong Jiang, Liangliang Nan, Dong-Ming Yan, Weiming Dong, Xiaopeng Zhang, and Peter Wonka. 2016. Automatic constraint detection for 2D layout regularization. IEEE TVCG 22, 8 (2016), 1933--1944.

Digital Library

[26]

Michael Kazhdan, Matthew Bolitho, and Hugues Hoppe. 2006. Poisson Surface Reconstruction. SGP (2006), 61--70.

Digital Library

[27]

Michael Kazhdan and Hugues Hoppe. 2013. Screened poisson surface reconstruction. ACM TOG 32, 3 (2013), 29.

Digital Library

[28]

Tom Kelly and Peter Wonka. 2011. Interactive architectural modeling with procedural extrusions. ACM TOG 30, 2 (2011), 14.

Digital Library

[29]

Alex Kendall, Vijay Badrinarayanan, and Roberto Cipolla. 2015. Bayesian SegNet: Model Uncertainty in Deep Convolutional Encoder-Decoder Architectures for Scene Understanding. arXiv preprint arXiv.1511.02680 (2015).

[30]

Mateusz Kozinski, Raghudeep Gadde, Sergey Zagoruyko, Guillaume Obozinski, and Renaud Marlet. 2015. A MRF shape prior for facade parsing with occlusions. IEEE CVPR (2015), 2820--2828.

[31]

Florent Lafarge, Xavier Descombes, Josiane Zerubia, and Marc Pierrot-Deseilligny. 2010. Structural approach for building reconstruction from a single DSM. IEEE TPAMI 32, 1 (2010), 135--147.

Digital Library

[32]

Robert G Laycock and AM Day. 2003. Automatically generating large urban environments based on the footprint data of buildings. ACM SMA (2003), 346--351.

Digital Library

[33]

Minglei Li, Peter Wonka, and Liangliang Nan. 2016. Manhattan-world urban reconstruction from point clouds. ECCV (2016), 54--69.

[34]

Yangyan Li, Xiaokun Wu, Yiorgos Chrysathou, Andrei Sharf, Daniel Cohen-Or, and Niloy J Mitra. 2011a. Globfit: Consistently fitting primitives by discovering global relations. ACM SIGGRAPH 30, 4 (2011), 52.

Digital Library

[35]

Yangyan Li, Qian Zheng, Andrei Sharf, Daniel Cohen-Or, Baoquan Chen, and Niloy J Mitra. 2011b. 2D-3D fusion for layer decomposition of urban facades. IEEE ICCV (2011), 882--889.

Digital Library

[36]

Hui Lin, Jizhou Gao, Yu Zhou, Guiliang Lu, Mao Ye, Chenxi Zhang, Ligang Liu, and Ruigang Yang. 2013. Semantic decomposition and reconstruction of residential scenes from LiDAR data. ACM SIGGRAPH 32, 4 (2013), 66.

Digital Library

[37]

Andelo Martinović, Markus Mathias, Julien Weissenberg, and Luc Van Gool. 2012. A three-layered approach to facade parsing. ECCV (2012), 416--429.

Digital Library

[38]

Bogdan C Matei, Harpreet S Sawhney, Supun Samarasekera, Janet Kim, and Rakesh Kumar. 2008. Building segmentation for densely built urban regions using aerial lidar data. IEEE CVPR (2008), 1--8.

[39]

Peter Miller et al. 2017. Buildings - OpenStreetMap Wiki. (2017). Retrieved August 8, 2017 from http://wiki.openstreetmap.org/wiki/Buildings

[40]

Aron Monszpart, Nicolas Mellado, Gabriel J Brostow, and Niloy J Mitra. 2015. RAPter: rebuilding man-made scenes with regular arrangements of planes. ACM SIGGRAPH 34, 4 (2015), 103--1.

Digital Library

[41]

Pascal Müller, Peter Wonka, Simon Haegler, Andreas Ulmer, and Luc Van Gool. 2006. Procedural modeling of buildings. ACM SIGGRAPH 25, 3 (2006), 614--623.

Digital Library

[42]

Pascal Müller, Gang Zeng, Peter Wonka, and Luc Van Gool. 2007. Image-based procedural modeling of facades. ACM SIGGRAPH 26, 3 (2007), 85.

Digital Library

[43]

Przemyslaw Musialski, Peter Wonka, Daniel G Aliaga, Michael Wimmer, L v Gool, and Werner Purgathofer. 2013. A survey of urban reconstruction. CGF 32, 6 (2013), 146--177.

Digital Library

[44]

Liangliang Nan, Caigui Jiang, Bernard Ghanem, and Peter Wonka. 2015. Template assembly for detailed urban reconstruction. CGF Eurographics 34, 2 (2015), 217--228.

Digital Library

[45]

Liangliang Nan, Andrei Sharf, Hao Zhang, Daniel Cohen-Or, and Baoquan Chen. 2010. Smartboxes for interactive urban reconstruction. ACM SIGGRAPH 29, 4 (2010), 93.

Digital Library

[46]

Gen Nishida, Ignacio Garcia-Dorado, Daniel G Aliaga, Bedrich Benes, and Adrien Bousseau. 2016. Interactive sketching of urban procedural models. ACM SIGGRAPH 35, 4 (2016), 130.

Digital Library

[47]

Charalambos Poullis and Suya You. 2009. Automatic reconstruction of cities from remote sensor data. IEEE CVPR (2009), 2775--2782.

[48]

Hayko Riemenschneider, Ulrich Krispel, Wolfgang Thaller, Michael Donoser, Sven Havemann, Dieter Fellner, and Horst Bischof. 2012. Irregular lattices for complex shape grammar facade parsing. IEEE CVPR (2012), 1640--1647.

Digital Library

[49]

David Salinas, Florent Lafarge, and Pierre Alliez. 2015. Structure-Aware Mesh Decimation. CGF 34, 6 (2015), 211--227.

Digital Library

[50]

Chao-Hui Shen, Shi-Sheng Huang, Hongbo Fu, and Shi-Min Hu. 2011. Adaptive partitioning of urban facades. ACM SIGGRAPH Asia 30, 6 (2011), 184.

Digital Library

[51]

Noah Snavely, Steven M Seitz, and Richard Szeliski. 2006. Photo tourism: exploring photo collections in 3D. ACM SIGGRAPH 25, 3 (2006), 835--846.

Digital Library

[52]

Olivier Teboul, Iasonas Kokkinos, Loic Simon, Panagiotis Koutsourakis, and Nikos Paragios. 2013. Parsing facades with shape grammars and reinforcement learning. IEEE TPAMI 35, 7 (2013), 1744--1756.

Digital Library

[53]

Radim Tylecek. 2012. The CMP facade database. Technical Report. Tech. rep., CTU-CMP-2012-24, Czech Technical University.

[54]

Carlos A Vanegas, Daniel G Aliaga, and Bedřich Beneš. 2010. Building reconstruction using manhattan-world grammars. IEEE CVPR (2010), 358--365.

[55]

Yannick Verdie, Florent Lafarge, and Pierre Alliez. 2015. LOD generation for urban scenes. Technical Report 3. 30:1--30:14 pages.

[56]

Jianxiong Xiao, Tian Fang, Ping Tan, Peng Zhao, Eyal Ofek, and Long Quan. 2008. Image-based façade modeling. ACM SIGGRAPH Asia 27, 5 (2008), 161.

[57]

Chao Yang, Tian Han, Long Quan, and Chiew-Lan Tai. 2012. Parsing façade with rank-one approximation. IEEE CVPR (2012), 1720--1727.

Digital Library

[58]

Qian Zheng, Andrei Sharf, Guowei Wan, Yangyan Li, Niloy J Mitra, Daniel Cohen-Or, and Baoquan Chen. 2010. Non-local scan consolidation for 3D urban scenes. ACM SIGGRAPH 29, 4 (2010), 94.

Digital Library

[59]

Qian-Yi Zhou and Ulrich Neumann. 2010. 2.5D dual contouring: a robust approach to creating building models from aerial lidar point clouds. ECCV (2010), 115--128.

Digital Library

Cited By

Ogawa YNakamura RSato GMaeda HSekimoto Y(2024)End-to-End Framework for the Automatic Matching of Omnidirectional Street Images and Building Data and the Creation of 3D Building ModelsRemote Sensing10.3390/rs1611185816:11(1858)Online publication date: 23-May-2024
https://doi.org/10.3390/rs16111858
Gong HSun QFang CSun LSu R(2024)TreeDetector: Using Deep Learning for the Localization and Reconstruction of Urban Trees from High-Resolution Remote Sensing ImagesRemote Sensing10.3390/rs1603052416:3(524)Online publication date: 30-Jan-2024
https://doi.org/10.3390/rs16030524
Ogawa YSato GSekimoto Y(2024)Geometric-based approach for linking various building measurement data to a 3D city modelPLOS ONE10.1371/journal.pone.029644519:1(e0296445)Online publication date: 5-Jan-2024
https://doi.org/10.1371/journal.pone.0296445
Show More Cited By

Index Terms

BigSUR: large-scale structured urban reconstruction
1. Applied computing
  1. Arts and humanities
    1. Architecture (buildings)
2. Computing methodologies
  1. Artificial intelligence
    1. Computer vision
      1. Computer vision tasks
        Scene understanding
  2. Computer graphics
    1. Shape modeling
      1. Mesh models
      2. Shape analysis

Recommendations

Automatic Deep Inference of Procedural Cities from Global-scale Spatial Data

Recent advances in big spatial data acquisition and deep learning allow novel algorithms that were not possible several years ago. We introduce a novel inverse procedural modeling algorithm for urban areas that addresses the problem of spatial data ...
Proceduralization of Buildings at City Scale
3DV '14: Proceedings of the 2014 2nd International Conference on 3D Vision - Volume 01

We present a framework for the conversion of existing 3D unstructured urban models into a compact procedural representation that enables model synthesis, querying, and simplification of large urban areas. During the de-instancing phase, a dissimilarity-...
Procedural Urban Modeling in Practice

Film and game studios can no longer meet audience demand for visual content by increasing production budgets. Instead they are turning to procedural modeling, particularly for modeling cities. The authors review procedural modeling, examine the ...

Comments

Information & Contributors

Information

Published In

cover image ACM Transactions on Graphics

ACM Transactions on Graphics Volume 36, Issue 6

December 2017

973 pages

ISSN:0730-0301

EISSN:1557-7368

DOI:10.1145/3130800

Editor:
Kavita Bala

Issue’s Table of Contents

Copyright © 2017 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 20 November 2017

Published in TOG Volume 36, Issue 6

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

Salt River Project Agricultural Improvement and Power District Cooperative
KAUST Office of Sponsored Research
ERC

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

54
Total Citations
View Citations
1,035
Total Downloads

Downloads (Last 12 months)59
Downloads (Last 6 weeks)4

Reflects downloads up to 23 Sep 2024

Other Metrics

View Author Metrics

Citations

Cited By

Ogawa YNakamura RSato GMaeda HSekimoto Y(2024)End-to-End Framework for the Automatic Matching of Omnidirectional Street Images and Building Data and the Creation of 3D Building ModelsRemote Sensing10.3390/rs1611185816:11(1858)Online publication date: 23-May-2024
https://doi.org/10.3390/rs16111858
Gong HSun QFang CSun LSu R(2024)TreeDetector: Using Deep Learning for the Localization and Reconstruction of Urban Trees from High-Resolution Remote Sensing ImagesRemote Sensing10.3390/rs1603052416:3(524)Online publication date: 30-Jan-2024
https://doi.org/10.3390/rs16030524
Ogawa YSato GSekimoto Y(2024)Geometric-based approach for linking various building measurement data to a 3D city modelPLOS ONE10.1371/journal.pone.029644519:1(e0296445)Online publication date: 5-Jan-2024
https://doi.org/10.1371/journal.pone.0296445
DONG MXIE KHUANG H(2024)Coupled study on instance segmentation and structural reconstruction of buildings in 3D urban scenesSCIENTIA SINICA Informationis10.1360/SSI-2023-0221Online publication date: 7-Feb-2024
https://doi.org/10.1360/SSI-2023-0221
Wang LHu HShang QZeng HZhu Q(2024)StructuredMesh: 3-D Structured Optimization of Façade Components on Photogrammetric Mesh Models Using Binary Integer ProgrammingIEEE Transactions on Geoscience and Remote Sensing10.1109/TGRS.2023.334849262(1-12)Online publication date: 2024
https://doi.org/10.1109/TGRS.2023.3348492
Shang QHu HYu HXu BWang LZhu Q(2024)Semantic Image Translation for Repairing the Texture Defects of Building ModelsIEEE Transactions on Geoscience and Remote Sensing10.1109/TGRS.2023.333896262(1-20)Online publication date: 2024
https://doi.org/10.1109/TGRS.2023.3338962
Li JChen DHu FWang YLi PPeethambaran J(2024)Shape-preserving mesh decimation for 3D building modelingInternational Journal of Applied Earth Observation and Geoinformation10.1016/j.jag.2023.103623126(103623)Online publication date: Feb-2024
https://doi.org/10.1016/j.jag.2023.103623
Wang BZhang JZhang RLi YLi LNakashima Y(2024)Improving facade parsing with vision transformers and line integrationAdvanced Engineering Informatics10.1016/j.aei.2024.10246360(102463)Online publication date: Apr-2024
https://doi.org/10.1016/j.aei.2024.102463
Yu HCao JLiu XChen Z(2024)Regularity-constrained point cloud reconstruction of building models via global alignmentThe Visual Computer10.1007/s00371-023-03241-xOnline publication date: 20-Jan-2024
https://doi.org/10.1007/s00371-023-03241-x
Wang FZhou GHu HWang YFu BLi SXie J(2023)Reconstruction of LoD-2 Building Models Guided by Façade Structures from Oblique Photogrammetric Point CloudRemote Sensing10.3390/rs1502040015:2(400)Online publication date: 9-Jan-2023
https://doi.org/10.3390/rs15020400
Show More Cited By

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Issue’s Table of Contents