research-article

Public Access

What is it like down there?: generating dense ground-level views and image features from overhead imagery using conditional generative adversarial networks

Authors:

Shawn NewsamAuthors Info & Claims

SIGSPATIAL '18: Proceedings of the 26th ACM SIGSPATIAL International Conference on Advances in Geographic Information Systems

Pages 43 - 52

https://doi.org/10.1145/3274895.3274969

Published: 06 November 2018 Publication History

Abstract

This paper investigates conditional generative adversarial networks (cGANs) to overcome a fundamental limitation of using geotagged media for geographic discovery, namely its sparse and uneven spatial distribution. We train a cGAN to generate ground-level views of a location given overhead imagery. We show the "fake" ground-level images are natural looking and are structurally similar to the real images. More significantly, we show the generated images are representative of the locations and that the representations learned by the cGANs are informative. In particular, we show that dense feature maps generated using our framework are more effective for land-cover classification than approaches which spatially interpolate features extracted from sparse ground-level images. To our knowledge, ours is the first work to use cGANs to generate ground-level views given overhead imagery in order to explore the benefits of the learned representations.

References

[1]

M. Arjovsky, S. Chintala, and L. Bottou. 2017. Wasserstein Generative Adversarial Networks. In Proceedings of the 34th International Conference on Machine Learning, Doina Precup and Yee Whye Teh (Eds.), Vol. 70. PMLR, International Convention Centre, Sydney, Australia, 214--223.

[2]

D. Berthelot, T. Schumm, and L. Metz. 2017. BEGAN: Boundary Equilibrium Generative Adversarial Networks. ArXiv e-prints (March 2017). arXiv:cs.LG/1703.10717

[3]

K. Bousmalis, N. Silberman, D. Dohan, D. Erhan, and D. Krishnan Krishnan. 2017. Unsupervised Pixel-Level Domain Adaptation with Generative Adversarial Networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 95--104.

[4]

Y. Chen, C. Shen, X.-S. Wei, L. Liu, and J. Yang. 2017. Adversarial PoseNet: A Structure-aware Convolutional Network for Human Pose Estimation. ArXiv e-prints (April 2017). arXiv:cs.CV/1705.00389

[5]

X. Deng, Y. Zhu, and S. Newsam. 2018. Spatial Morphing Kernel Regression for Feature Interpolation. In 25th IEEE International Conference on Image Processing (ICIP). 2182--2186.

[6]

E. Denton, S. Chintala, A. Szlam, and R. Fergus. 2015. Deep Generative Image Models Using a Laplacian Pyramid of Adversarial Networks. In Proceedings of 28th International Conference on Neural Information Processing Systems. 1486--1494.

Digital Library

[7]

H. Dong, S. Yu, C. Wu, and Y. Guo. 2017. Semantic Image Synthesis via Adversarial Learning. In Proceedings of the IEEE International Conference on Computer Vision. 5707--5715.

[8]

A. Dosovitskiy, P. Fischer, J. Springenberg, M. Riedmiller, and T. Brox. 2016. Discriminative Unsupervised Feature Learning with Exemplar Convolutional Neural Networks. IEEE Transactions on Pattern Analysis and Machine Intelligence 38, 9 (2016), 1734--1747.

[9]

I. Goodfellow. 2017. NIPS 2016 Tutorial: Generative Adversarial Networks. ArXiv e-prints (Dec. 2017). arXiv:cs.LG/1701.00160

[10]

I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio. 2014. Generative Adversarial Nets. In Advances in Neural Information Processing Systems. 2672--2680.

Digital Library

[11]

K. He, X. Zhang, S. Ren, and J. Sun. 2016. Deep Residual Learning for Image Recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 770--778.

[12]

T. Hu, J. Yang, X. Li, and P. Gong. 2016. Mapping Urban Land Use by Using Landsat Images and Open Social Data. Remote Sensing 8 (2016), 151.

[13]

R. Huang, S. Zhang, T. Li, and R. He. 2017. Beyond Face Rotation: Global and Local Perception GAN for Photorealistic and Identity Preserving Frontal View Synthesis. In Proceedings of the IEEE International Conference on Computer Vision. 2458--2467.

[14]

S. Ioffe and C. Szegedy. 2015. Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift. In 32nd International Conference on Machine Learning (Proceedings of Machine Learning Research), Francis Bach and David Blei (Eds.), Vol. 37. PMLR, Lille, France, 448--456.

Digital Library

[15]

P. Isola, J. Zhu, T. Zhou, and A. A Efros. 2017. Image-to-Image Translation with Conditional Adversarial Networks. In Proceedings of the IEEE International Conference on Computer Vision.

[16]

D. P. Kingma and J. Ba. 2014. Adam: A Method for Stochastic Optimization. CoRR abs/1412.6980 (2014).

[17]

C. Ledig, L. Theis, F. HuszÃąr, J. Caballero, A. Cunningham, A. Acosta, A. Aitken, A. Tejani, J. Totz, Z. Wang, and W. Shi. 2017. Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.

[18]

D. Leung and S. Newsam. 2010. Proximate Sensing: Inferring What-Is-Where From Georeferenced Photo Collections. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2955--2962.

[19]

Y. Li, J. Huang, and J. Luo. 2015. Using User Generated Online Photos to Estimate and Monitor Air Pollution in Major Cities. In Proceedings of the 7th International Conference on Internet Multimedia Computing and Service. 79.

Digital Library

[20]

Y. Li, S. Liu, J. Yang, and M. Yang. 2017. Generative Face Completion. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.

[21]

X. Liu, J. He, Y. Yao, J. Zhang, H. Liang, H. Wang, and Y. Hong. 2017. Classifying Urban Land Use by Integrating Remote Sensing and Social Media Data. International Journal of Geographical Information Science 31, 8 (2017), 1675--1696.

Digital Library

[22]

L. Ma, X. Jia, Q. Sun, B. Schiele, T. Tuytelaars, and L. Van Gool. 2017. Pose Guided Person Image Generation. In Advances in Neural Information Processing Systems (NIPS). 405--415.

Digital Library

[23]

X. Mao, Q. Li, H. Xie, R. Y. K. Lau, Z. Wang, and S. P. Smolley. 2016. Least Squares Generative Adversarial Networks. ArXiv e-prints (Nov. 2016). arXiv:cs.CV/1611.04076

[24]

M. Mirza and S. Osindero. 2014. Conditional Generative Adversarial Nets. arXiv preprint arXiv:1411.1784 (2014).

[25]

S. Mohamed and B. Lakshminarayanan. 2016. Learning in Implicit Generative Models. ArXiv e-prints (Oct. 2016). arXiv:stat.ML/1610.03483

[26]

A. Radford, L. Metz, and S. Chintala. 2016. Unsupervised representation learning with deep convolutional generative adversarial networks. In International Conference on Representation Learning (ICRL).

[27]

S. Reed, Z. Akata, X. Yan, L. Logeswaran, B. Schiele, and H. Lee. 2016. Generative Adversarial Text to Image Synthesis. In Proceedings of the 33rd International Conference on Machine Learning (ICML) (Proceedings of Machine Learning Research), Maria Florina Balcan and Kilian Q. Weinberger (Eds.), Vol. 48. PMLR, New York, New York, USA, 1060--1069.

Digital Library

[28]

K. Regmi and A. Borji. 2018. Cross-View Image Synthesis Using Conditional GANs. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.

[29]

K. Regmi and A. Borji. 2018. Cross-view image synthesis using geometry-guided conditional GANs. ArXiv e-prints (Aug. 2018). arXiv:cs.CV/1808.05469

[30]

T. Salimans, I. Goodfellow, W. Zaremba, V.i Cheung, A. Radford, and X. Chen. 2016. Improved Techniques for Training GANs. In Advances in Neural Information Processing Systems (NIPS). 2234--2242.

Digital Library

[31]

L. Tran, X. Yin, and X. Liu. 2017. Representation Learning by Rotating Your Faces. ArXiv e-prints (May 2017). arXiv:cs.CV/1705.11136

[32]

J. Wang, M. Korayem, S. Blanco, and D. Crandall. 2016. Tracking Natural Events Through Social Media and Computer Vision. In 2016 ACM on Multimedia Conference. 1097--1101.

Digital Library

[33]

S. Workman, M. Zhai, D. J. Crandall, and N. Jacobs. 2017. A Unified Model for Near and Remote Sensing. In Proceedings of the IEEE International Conference on Computer Vision. 2707--2716.

[34]

R. A. Yeh*, C. Chen*, T. Lim, A. G. Schwing, M. Hasegawa-Johnson, and M. N. Do. 2017. Semantic Image Inpainting with Deep Generative Models. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.* equal contribution.

[35]

M. Zhai, Z. Bessinger, S. Workman, and N. Jacobs. 2017. Predicting Ground-Level Scene Layout from Aerial Imagery. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.

[36]

H. Zhang, T. Xu, H. Li, S. Zhang, X. Huang, X. Wang, and D. Metaxas. 2017. Stackgan: Text to Photo-Realistic Image Synthesis with Stacked Generative Adversarial Networks. In Proceedings of the IEEE International Conference on Computer Vision. 5907--5915.

[37]

J. Zhu, T. Park, P. Isola, and A. A Efros. 2017. Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks. In Proceedings of the IEEE International Conference on Computer Vision.

[38]

Y. Zhu, X. Deng, and S. Newsam. 2018. Fine-Grained Land Use Classification at the City Scale Using Ground-Level Images. ArXiv e-prints (Feb. 2018). arXiv:cs.CV/1802.02668

[39]

Y. Zhu, S. Liu, and S. Newsam. 2017. Large-Scale Mapping of Human Activity Using Geo-Tagged Videos. In Proceedings of the 25th ACM SIGSPATIAL International Conference on Advances in Geographic Information Systems. 68:1--68:4.

Digital Library

[40]

Y. Zhu and S. Newsam. 2015. Land Use Classification Using Convolutional Neural Networks Applied to Ground-Level Images. In Proceedings of the 23rd SIGSPATIAL International Conference on Advances in Geographic Information Systems. 61.

Digital Library

[41]

Y. Zhu and S. Newsam. 2016. Spatio-Temporal Sentiment Hotspot Detection Using Geotagged Photos. In Proceedings of the 24th ACM SIGSPATIAL International Conference on Advances in Geographic Information Systems. 76.

Digital Library

Cited By

Wyawahare MDasare SBhadange ABhattad R(2023)An Ensemble for Satellite Image to Map Layout TranslationInventive Communication and Computational Technologies10.1007/978-981-99-5166-6_69(1023-1035)Online publication date: 4-Oct-2023
https://doi.org/10.1007/978-981-99-5166-6_69
Grujicic DBlaschko M(2022)2-D latent space models: Layer-wise perceptual training and spatial grounding2022 26th International Conference on Pattern Recognition (ICPR)10.1109/ICPR56361.2022.9956349(2437-2443)Online publication date: 21-Aug-2022
https://doi.org/10.1109/ICPR56361.2022.9956349
Himeur YRimal BTiwary AAmira A(2022)Using artificial intelligence and data fusion for environmental monitoring: A review and future perspectivesInformation Fusion10.1016/j.inffus.2022.06.00386-87(44-75)Online publication date: Oct-2022
https://doi.org/10.1016/j.inffus.2022.06.003
Show More Cited By

Index Terms

What is it like down there?: generating dense ground-level views and image features from overhead imagery using conditional generative adversarial networks
1. Computing methodologies
  1. Artificial intelligence
    1. Computer vision
      1. Computer vision representations
        Image representations
  2. Machine learning
    1. Machine learning approaches
      1. Neural networks
2. Information systems
  1. Information systems applications
    1. Spatial-temporal systems
      1. Geographic information systems

Recommendations

Rice Seed Image-to-Image Translation Using Generative Adversarial Networks to Improve Weedy Rice Image Classification
Machine Learning and Knowledge Extraction
Abstract
Rice is a staple food for more than half of the world’s population. Furthermore, rice is the main export crop of Thailand which produces 21% world’s market share. However, weedy rice is a major counterproductive plant that reduces rice ...
Monitoring crop phenology with street-level imagery using computer vision
Highlights
- Method to automatically collect side looking imagery for agriculture parcels.
- Crop types and phenology obtained from computer vision.
- Demonstration over monthly surveyed totalling 400,000 pictures for 17 crop types.
- Deep ...
Abstract
Street-level imagery holds a significant potential to scale-up in-situ data collection. This is enabled by combining the use of cheap high-quality cameras with recent advances in deep learning compute solutions to derive relevant thematic ...
Weed detection in soybean crops using ConvNets

The use of Deep Learning for the detection of weeds in soybean crops is proposed.The approach uses ConvNets in images segmented by the SLIC Superpixels algorithm.An image database was created using photographs captured by UAVs.The performance of ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

SIGSPATIAL '18: Proceedings of the 26th ACM SIGSPATIAL International Conference on Advances in Geographic Information Systems

November 2018

655 pages

ISBN:9781450358897

DOI:10.1145/3274895

General Chairs:
Farnoush Banaei-Kashani
University of Colorado, Denver
,
Erik Hoel
Esri
,
Program Chairs:
Ralf Hartmut Güting
FernUniversität in Hagen, Germany
,
Roberto Tamassia
Brown University
,
Li Xiong
Emory University

Copyright © 2018 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGSPATIAL: ACM Special Interest Group on Spatial Information

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 06 November 2018

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

National Science Foundation

Conference

SIGSPATIAL '18

Sponsor:

SIGSPATIAL

SIGSPATIAL '18: 26th ACM SIGSPATIAL International Conference on Advances in Geographic Information Systems

November 6 - 9, 2018

Washington, Seattle

Acceptance Rates

SIGSPATIAL '18 Paper Acceptance Rate 30 of 150 submissions, 20%;

Overall Acceptance Rate 220 of 1,116 submissions, 20%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

12
Total Citations
View Citations
567
Total Downloads

Downloads (Last 12 months)106
Downloads (Last 6 weeks)16

Reflects downloads up to 09 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Wyawahare MDasare SBhadange ABhattad R(2023)An Ensemble for Satellite Image to Map Layout TranslationInventive Communication and Computational Technologies10.1007/978-981-99-5166-6_69(1023-1035)Online publication date: 4-Oct-2023
https://doi.org/10.1007/978-981-99-5166-6_69
Grujicic DBlaschko M(2022)2-D latent space models: Layer-wise perceptual training and spatial grounding2022 26th International Conference on Pattern Recognition (ICPR)10.1109/ICPR56361.2022.9956349(2437-2443)Online publication date: 21-Aug-2022
https://doi.org/10.1109/ICPR56361.2022.9956349
Himeur YRimal BTiwary AAmira A(2022)Using artificial intelligence and data fusion for environmental monitoring: A review and future perspectivesInformation Fusion10.1016/j.inffus.2022.06.00386-87(44-75)Online publication date: Oct-2022
https://doi.org/10.1016/j.inffus.2022.06.003
Wu AStouffs RBiljecki F(2022)Generative Adversarial Networks in the built environment: A comprehensive review of the application of GANs across data types and scalesBuilding and Environment10.1016/j.buildenv.2022.109477223(109477)Online publication date: Sep-2022
https://doi.org/10.1016/j.buildenv.2022.109477
Liu YWang WFang FZhou LSun CZheng YChen Z(2021)CscGAN: Conditional Scale-Consistent Generation Network for Multi-Level Remote Sensing Image to Map TranslationRemote Sensing10.3390/rs1310193613:10(1936)Online publication date: 15-May-2021
https://doi.org/10.3390/rs13101936
Tuia DRoscher RWegner JJacobs NZhu XCamps-Valls G(2021)Toward a Collective Agenda on AI for Earth Science Data AnalysisIEEE Geoscience and Remote Sensing Magazine10.1109/MGRS.2020.30435049:2(88-104)Online publication date: Jun-2021
https://doi.org/10.1109/MGRS.2020.3043504
De Giacomo Gdos Santos MDrews PBotelho S(2021)Guided Sonar-to-Satellite TranslationJournal of Intelligent & Robotic Systems10.1007/s10846-021-01324-2101:3Online publication date: 13-Feb-2021
https://doi.org/10.1007/s10846-021-01324-2
Mahabir RSchuchard RCrooks ACroitoru AStefanidis A(2020)Crowdsourcing Street View Imagery: A Comparison of Mapillary and OpenStreetCamISPRS International Journal of Geo-Information10.3390/ijgi90603419:6(341)Online publication date: 26-May-2020
https://doi.org/10.3390/ijgi9060341
Tsagkatakis GAidini AFotiadou KGiannopoulos MPentari ATsakalides P(2019)Survey of Deep-Learning Approaches for Remote Sensing Observation EnhancementSensors10.3390/s1918392919:18(3929)Online publication date: 12-Sep-2019
https://doi.org/10.3390/s19183929
Miao XYuan XPu YAthitsos V(2019)lambda-Net: Reconstruct Hyperspectral Images From a Snapshot Measurement2019 IEEE/CVF International Conference on Computer Vision (ICCV)10.1109/ICCV.2019.00416(4058-4068)Online publication date: Oct-2019
https://doi.org/10.1109/ICCV.2019.00416
Show More Cited By

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Media

Figures

Other

Tables

View Table of Contents