research-article

Open access

Nostalgin: Extracting 3D City Models from Historical Image Data

Authors:

Raimondas KiverisAuthors Info & Claims

KDD '19: Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining

Pages 2565 - 2575

https://doi.org/10.1145/3292500.3330743

Published: 25 July 2019 Publication History

Abstract

What did it feel like to walk through a city from the past? In this work, we describe Nostalgin (Nostalgia Engine), a method that can faithfully reconstruct cities from historical images. Unlike existing work in city reconstruction, we focus on the task of reconstructing 3D cities from historical images. Working with historical image data is substantially more difficult, as there are significantly fewer buildings available and the details of the camera parameters which captured the images are unknown. Nostalgin can generate a city model even if there is only a single image per facade, regardless of viewpoint or occlusions. To achieve this, our novel architecture combines image segmentation, rectification, and inpainting. We motivate our design decisions with experimental analysis of individual components of our pipeline, and show that we can improve on baselines in both speed and visual realism. We demonstrate the efficacy of our pipeline by recreating two 1940s Manhattan city blocks. We aim to deploy Nostalgin as an open source platform where users can generate immersive historical experiences from their own photos.

Supplementary Material

MP4 File (p2565-kapoor.mp4)

Download
1076.67 MB

References

[1]

Sameer Agarwal, Noah Snavely, Steven M Seitz, and Richard Szeliski. 2010. Bundle adjustment in the large. In European conference on computer vision. Springer, 29--42.

Digital Library

[2]

Sameer Agarwal, Noah Snavely, Ian Simon, Steven M Seitz, and Richard Szeliski. 2009. Building rome in a day. In Computer Vision, 2009 IEEE 12th International Conference on. IEEE, 72--79.

[3]

Connelly Barnes, Eli Shechtman, Adam Finkelstein, and Dan B Goldman. 2009. PatchMatch: A randomized correspondence algorithm for structural image editing. ACM Transactions on Graphics (ToG), Vol. 28, 3 (2009), 24.

Digital Library

[4]

Marcelo Bertalmio, Andrea L Bertozzi, and Guillermo Sapiro. 2001. Navier-stokes, fluid dynamics, and image and video inpainting. In Computer Vision and Pattern Recognition, 2001. CVPR 2001. Proceedings of the 2001 IEEE Computer Society Conference on, Vol. 1. IEEE, I--I.

[5]

Angel X Chang, Thomas Funkhouser, Leonidas Guibas, Pat Hanrahan, Qixing Huang, Zimo Li, Silvio Savarese, Manolis Savva, Shuran Song, Hao Su, et almbox. 2015. Shapenet: An information-rich 3d model repository. arXiv preprint arXiv:1512.03012 (2015).

[6]

Martin A. Fischler and Robert C. Bolles. 1981. Random Sample Consensus: A Paradigm for Model Fitting with Applications to Image Analysis and Automated Cartography. Commun. ACM, Vol. 24, 6 (June 1981), 381--395.

Digital Library

[7]

Kota Hara, Raviteja Vemulapalli, and Rama Chellappa. 2017. Designing deep convolutional neural networks for continuous object orientation estimation. arXiv preprint arXiv:1702.01499 (2017).

[8]

Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. 2017. Mask R-CNN. arXiv:1703.06870 (2017).

[9]

Youichi Horry, Ken-Ichi Anjyo, and Kiyoshi Arai. 1997. Tour into the picture: using a spidery mesh interface to make animation from a single image. In Proceedings of the 24th annual conference on Computer graphics and interactive techniques. ACM Press/Addison-Wesley Publishing Co., 225--232.

Digital Library

[10]

Arnold Irschara, Christopher Zach, and Horst Bischof. 2007. Towards wiki-based dense city modeling. In Computer Vision, 2007. ICCV 2007. IEEE 11th International Conference on. IEEE, 1--8.

[11]

Nianjuan Jiang, Ping Tan, and Loong-Fah Cheong. 2009. Symmetric architecture modeling with a single image. ACM Transactions on Graphics (TOG), Vol. 28, 5 (2009), 113.

Digital Library

[12]

Tom Kelly, John Femiani, Peter Wonka, and Niloy J Mitra. 2017. BigSUR: large-scale structured urban reconstruction. ACM Transactions on Graphics (TOG), Vol. 36, 6 (2017), 204.

Digital Library

[13]

Thommen Korah, Swarup Medasani, and Yuri Owechko. 2011. Strip histogram grid for efficient lidar segmentation from urban environments. In Computer Vision and Pattern Recognition Workshops (CVPRW), 2011 IEEE Computer Society Conference on. IEEE, 74--81.

[14]

Jana Kovs ecká and Wei Zhang. 2002. Video compass. In European conference on computer vision. Springer, 476--490.

Digital Library

[15]

Jana Kovs ecká and Wei Zhang. 2005. Extraction, matching, and pose recovery based on dominant rectangular structures. Computer Vision and Image Understanding, Vol. 100, 3 (2005), 274--293.

Digital Library

[16]

Anat Levin, Dani Lischinski, and Yair Weiss. 2008. A Closed Form Solution to Natural Image Matting. IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 30, 2 (2008), 228--242.

Digital Library

[17]

Hantang Liu, Jialiang Zhang, Jianke Zhu, and Steven CH Hoi. 2017. Deepfacade: A deep learning approach to facade parsing. (2017).

[18]

Przemyslaw Musialski, Peter Wonka, Daniel G Aliaga, Michael Wimmer, Luc Van Gool, and Werner Purgathofer. 2013. A survey of urban reconstruction. In Computer graphics forum, Vol. 32. Wiley Online Library, 146--177.

Digital Library

[19]

Gen Nishida, Adrien Bousseau, and Daniel G Aliaga. 2018. Procedural Modeling of a Building from a Single Image. In Computer Graphics Forum, Vol. 37. Wiley Online Library, 415--429.

[20]

Byong Mok Oh, Max Chen, Julie Dorsey, and Frédo Durand. 2001. Image-based modeling and photo editing. In Proceedings of the 28th annual conference on Computer graphics and interactive techniques. ACM, 433--442.

Digital Library

[21]

Carsten Rother. 2002. A new approach to vanishing point detection in architectural environments. Image and Vision Computing, Vol. 20, 9--10 (2002), 647--655.

[22]

Carlos A Vanegas, Daniel G Aliaga, Peter Wonka, Pascal Müller, Paul Waddell, and Benjamin Watson. 2010. Modelling the appearance and behaviour of urban spaces. In Computer Graphics Forum, Vol. 29. Wiley Online Library, 25--42.

[23]

Rafael Grompone von Gioi, Jeremie Jakubowicz, Jean-Michel Morel, and Gregory Randall. 2010. LSD: A Fast Line Segment Detector with a False Detection Control. IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 32, 4 (2010), 722--732.

Digital Library

[24]

Raymond A Yeh, Chen Chen, Teck Yian Lim, Alexander G Schwing, Mark Hasegawa-Johnson, and Minh N Do. 2017. Semantic image inpainting with deep generative models. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 5485--5493.

[25]

Xuetao Yin, Peter Wonka, and Anshuman Razdan. 2009. Generating 3d building models from architectural drawings: A survey. IEEE computer graphics and applications, Vol. 29, 1 (2009).

[26]

Jiahui Yu, Zhe Lin, Jimei Yang, Xiaohui Shen, Xin Lu, and Thomas S Huang. 2018a. Free-Form Image Inpainting with Gated Convolution. arXiv preprint arXiv:1806.03589 (2018).

[27]

Jiahui Yu, Zhe Lin, Jimei Yang, Xiaohui Shen, Xin Lu, and Thomas S Huang. 2018b. Generative image inpainting with contextual attention. arXiv preprint (2018).

[28]

Zhengyou Zhang. {n. d.}. Single-View Geometry of A Rectangle With Application to Whiteboard Image Rectification.

Cited By

Rajan VKoeva MKuffer MDa Silva Mano AMishra S(2023)Three-Dimensional Modelling of Past and Present Shahjahanabad through Multi-Temporal Remotely Sensed DataRemote Sensing10.3390/rs1511292415:11(2924)Online publication date: 3-Jun-2023
https://doi.org/10.3390/rs15112924
Farella EÖzdemir ERemondino F(2021)4D Building Reconstruction with Machine Learning and Historical MapsApplied Sciences10.3390/app1104144511:4(1445)Online publication date: 5-Feb-2021
https://doi.org/10.3390/app11041445
Ding DYu XWang Z(2021)The Evolution of the Living Environment in Suzhou in the Ming and Qing Dynasties Based on Historical PaintingsJournal on Computing and Cultural Heritage 10.1145/343070014:2(1-14)Online publication date: 19-Apr-2021
https://dl.acm.org/doi/10.1145/3430700
Show More Cited By

Index Terms

Nostalgin: Extracting 3D City Models from Historical Image Data

Recommendations

Oblique Aerial Image Acquisition, 3D City Modeling, 3D City Guide Project for Konya Metropolitan Municipality

Usage of aerial oblique cameras and oblique images in generation of 3D city models has become popular all over the world in recent years and various solutions has been developed involving specialized methods and softwares. The first comprehensive step ...
Determining Realism of Procedurally Generated City Road Networks
Image and Vision Computing
Abstract
Cities are both expansive and relatively homogeneous, making them well suited to procedural generation. Road networks are one of the most important components of generated cities, and rely on both exogenous and endogenous factors such as the ...
Technical Section: Image-based three-dimensional model reconstruction for Chinese treasure-Jadeite Cabbage with Insects

This paper presents a novel 3D reconstruction system for the famous Chinese treasure, Jadeite Cabbage with Insects, from uncalibrated image sequences. There are two major challenges for this 3D model reconstruction problem. The first is the difficult ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

KDD '19: Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining

July 2019

3305 pages

ISBN:9781450362016

DOI:10.1145/3292500

General Chairs:
Ankur Teredesai
KenSci
,
Vipin Kumar
University of Minnesota
,
Program Chairs:
Ying Li
EV Analysis Corporation
,
Rómer Rosales
LinkedIn
,
Evimaria Terzi
Boston University
,
George Karypis
University of Minnesota

Copyright © 2019 Owner/Author.

Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for third-party components of this work must be honored. For all other uses, contact the Owner/Author.

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 25 July 2019

Check for updates

Author Tags

Qualifiers

Research-article

Conference

KDD '19

Sponsor:

KDD '19: The 25th ACM SIGKDD Conference on Knowledge Discovery and Data Mining

August 4 - 8, 2019

AK, Anchorage, USA

Acceptance Rates

KDD '19 Paper Acceptance Rate 110 of 1,200 submissions, 9%;

Overall Acceptance Rate 1,133 of 8,635 submissions, 13%

Upcoming Conference

KDD '25

Sponsor:
sigkdd
sigkdd

The 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining

August 3 - 7, 2025

Toronto , ON , Canada

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

5
Total Citations
View Citations
640
Total Downloads

Downloads (Last 12 months)77
Downloads (Last 6 weeks)14

Reflects downloads up to 20 Feb 2025

Other Metrics

View Author Metrics

Citations

Cited By

Rajan VKoeva MKuffer MDa Silva Mano AMishra S(2023)Three-Dimensional Modelling of Past and Present Shahjahanabad through Multi-Temporal Remotely Sensed DataRemote Sensing10.3390/rs1511292415:11(2924)Online publication date: 3-Jun-2023
https://doi.org/10.3390/rs15112924
Farella EÖzdemir ERemondino F(2021)4D Building Reconstruction with Machine Learning and Historical MapsApplied Sciences10.3390/app1104144511:4(1445)Online publication date: 5-Feb-2021
https://doi.org/10.3390/app11041445
Ding DYu XWang Z(2021)The Evolution of the Living Environment in Suzhou in the Ming and Qing Dynasties Based on Historical PaintingsJournal on Computing and Cultural Heritage 10.1145/343070014:2(1-14)Online publication date: 19-Apr-2021
https://dl.acm.org/doi/10.1145/3430700
Amigo EGonzalo JMizzaro S(2021)What is my Problem Identifying Formal Tasks and Metrics in Data Mining on the Basis of Measurement TheoryIEEE Transactions on Knowledge and Data Engineering10.1109/TKDE.2021.3109823(1-1)Online publication date: 2021
https://doi.org/10.1109/TKDE.2021.3109823
Tavakkol SShahabi CHan FKiveris R(2020)Piaget: A Probabilistic Inference Approach for Geolocating Historical Buildings2020 IEEE International Conference on Big Data (Big Data)10.1109/BigData50022.2020.9378093(971-978)Online publication date: 10-Dec-2020
https://doi.org/10.1109/BigData50022.2020.9378093

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Figures

Tables

Media

View Table of Conten