Article

A three-layered approach to facade parsing

Authors:

Anđelo Martinović,

Markus Mathias,

Julien Weissenberg,

Luc Van GoolAuthors Info & Claims

ECCV'12: Proceedings of the 12th European conference on Computer Vision - Volume Part VII

Pages 416 - 429

https://doi.org/10.1007/978-3-642-33786-4_31

Published: 07 October 2012 Publication History

Abstract

We propose a novel three-layered approach for semantic segmentation of building facades. In the first layer, starting from an oversegmentation of a facade, we employ the recently introduced machine learning technique Recursive Neural Networks (RNN) to obtain a probabilistic interpretation of each segment. In the second layer, initial labeling is augmented with the information coming from specialized facade component detectors. The information is merged using a Markov Random Field. In the third layer, we introduce weak architectural knowledge, which enforces the final reconstruction to be architecturally plausible and consistent. Rigorous tests performed on two existing datasets of building facades demonstrate that we significantly outperform the current-state of the art, even when using outputs from earlier layers of the pipeline. Also, we show how the final output of the third layer can be used to create a procedural reconstruction.

References

[1]

Teboul, O., Simon, L., Koutsourakis, P., Paragios, N.: Segmentation of building facades using procedural shape priors. In: CVPR (2010).

[2]

Socher, R., Lin, C.C., Ng, A.Y., Manning, C.D.: Parsing Natural Scenes and Natural Language with Recursive Neural Networks. In: ICML (2011).

[3]

Teboul, O.: Ecole centrale paris facades database (2010), http://www.mas.ecp.fr/vision/Personnel/teboul/data.php.

[4]

Zhao, P., Fang, T., Xiao, J., Zhang, H., Zhao, Q., Quan, L.: Rectilinear parsing of architecture in urban environment. In: CVPR (2010).

[5]

Wendel, A., Donoser, M., Bischof, H.: Unsupervised Facade Segmentation Using Repetitive Patterns. In: Goesele, M., Roth, S., Kuijper, A., Schiele, B., Schindler, K. (eds.) DAGM 2010. LNCS, vol. 6376, pp. 51-60. Springer, Heidelberg (2010).

Digital Library

[6]

Recky, M., Wendel, A., Leberl, F.: Façade segmentation in a multi-view scenario. In: 3DIMPVT (2011).

Digital Library

[7]

Mathias, M., Martinovic, A., Weissenberg, J., Haegler, S., Gool, L.V.: Automatic architectural style recognition. In: 3D-ARCH (2011).

[8]

Korč, F., Förstner, W.: eTRIMS Image Database for interpreting images of man-made scenes. Technical Report TR-IGG-P-2009-01 (April 2009).

[9]

Xiao, J., Fang, T., Tan, P., Zhao, P., Ofek, E., Quan, L.: Image-based façade modeling. In: SIGGRAPH Asia (2008).

Digital Library

[10]

Xiao, J., Fang, T., Zhao, P., Lhuillier, M., Quan, L.: Image-based street-side city modeling. SIGGRAPH 28(5) (2009).

Digital Library

[11]

Korah, T., Rasmussen, C.: Analysis of Building Textures for Reconstructing Partially Occluded Facades. In: Forsyth, D., Torr, P., Zisserman, A. (eds.) ECCV 2008, Part I. LNCS, vol. 5302, pp. 359-372. Springer, Heidelberg (2008).

Digital Library

[12]

Mayer, H., Reznik, S.: Mcmc linked with implicit shape models and plane sweeping for 3d building facade interpretation in image sequences. ISPRS (2006).

[13]

Dick, A.R., Torr, P.H.S., Cipolla, R.: Modelling and interpretation of architecture from several images. IJCV 60 (2004).

Digital Library

[14]

Muller, P., Zeng, G., Wonka, P., Van Gool, L.: Image-based procedural modeling of facades. SIGGRAPH 26(3) (2007).

Digital Library

[15]

Gool, L.J.V., Zeng, G., den Borre, F.V., Müller, P.: Towards mass-produced building models. In: PIA (2007).

[16]

Alegre, O., Dellaert, F.: A probabilistic approach to the semantic interpretation of building facades. In: Workshop on Vision Techniques Applied to the Rehabilitation of City Centres (2004).

[17]

Ripperda, N., Brenner, C.: Reconstruction of Façade Structures Using a Formal Grammar and RjMCMC. In: Franke, K., Müller, K.-R., Nickolay, B., Schäfer, R. (eds.) DAGM 2006. LNCS, vol. 4174, pp. 750-759. Springer, Heidelberg (2006).

Digital Library

[18]

Han, F., Zhu, S.C.: Bottom-up/top-down image parsing with attribute grammar. IEEE TPAMI 31(1) (2009).

Digital Library

[19]

Teboul, O., Kokkinos, I., Simon, L., Koutsourakis, P., Paragios, N.: Shape grammar parsing via reinforcement learning. In: CVPR (2011).

Digital Library

[20]

Aliaga, D.G., Rosen, P.A., Bekins, D.R.: Style grammars for interactive visualization of architecture. TVCG 13(4) (2007).

Digital Library

[21]

Bokeloh, M., Wand, M., Seidel, H.P.: A connection between partial symmetry and inverse procedural modeling. SIGGRAPH 29(4) (2010).

Digital Library

[22]

Yang, M.Y., Förstner, W.: Regionwise Classification of Building Facade Images. In: Stilla, U., Rottensteiner, F., Mayer, H., Jutzi, B., Butenuth, M. (eds.) PIA 2011. LNCS, vol. 6952, pp. 209-220. Springer, Heidelberg (2011).

Digital Library

[23]

Liebowitz, D., Zisserman, A.: Metric rectification for perspective images of planes. In: CVPR (1998).

Digital Library

[24]

Comaniciu, D., Meer, P.: Mean shift: A robust approach toward feature space analysis. IEEE TPAMI 24(5) (2002).

Digital Library

[25]

Gould, S., Fulton, R., Koller, D.: Decomposing a scene into geometric and semantically consistent regions. In: ICCV (2009).

[26]

Gould, S., Russakovsky, O., Goodfellow, I., Baumstarck, P., Ng, A.Y., Koller, D.: The stair vision library, v2.2 (2009), http://ai.stanford.edu/˜sgould/svl.

[27]

Dollar, P., Tu, Z., Perona, P., Belongie, S.: Integral channel features. In: BMVC (2009).

[28]

Benenson, R., Mathias, M., Timofte, R., Van Gool, L.: Pedestrian detection at 100 frames per second. In: CVPR (2012).

Digital Library

[29]

Boykov, Y., Veksler, O., Zabih, R.: Fast approximate energy minimization via graph cuts. IEEE TPAMI 23(11) (2001).

Digital Library

[30]

Mathias, M., Martinovic, A., Weissenberg, J., Gool, L.V.: Procedural 3d building reconstruction using shape grammars and detectors. In: 3DIMPVT (2011).

Digital Library

[31]

Shechtman, E., Irani, M.: Matching local self-similarities across images and videos. In: CVPR (2007).

[32]

Teboul, O.: Shape Grammar Parsing: Application to Image-based Modeling. PhD thesis, Ecole Centrale Paris (2011).

[33]

Procedural: CityEngine (2010), http://www.procedural.com/

Cited By

Nauata NFurukawa Y(2020)Vectorizing World Buildings: Planar Graph Reconstruction by Primitive Detection and Relationship InferenceComputer Vision – ECCV 202010.1007/978-3-030-58598-3_42(711-726)Online publication date: 23-Aug-2020
https://dl.acm.org/doi/10.1007/978-3-030-58598-3_42
Dogan TKnutins MRakha TTurrin M(2018)CitySeekProceedings of the Symposium on Simulation for Architecture and Urban Design10.5555/3289750.3289786(1-8)Online publication date: 4-Jun-2018
https://dl.acm.org/doi/10.5555/3289750.3289786
Lian YShen XHu Y(2018)Detecting and inferring repetitive elements with accurate locations and shapes from façadesThe Visual Computer: International Journal of Computer Graphics10.1007/s00371-017-1355-z34:4(491-506)Online publication date: 1-Apr-2018
https://dl.acm.org/doi/10.1007/s00371-017-1355-z
Show More Cited By

Recommendations

Puzzle facade: a site-specific urban technological intervention
DIS Companion '14: Proceedings of the 2014 companion publication on Designing interactive systems

Media façades are becoming part of our urban landscapes, challenging media artists and designers to create content for them. The diversity of resolution and spatial properties of these façades hasn't stopped content creators to adapt their projects to ...
The media façade toolkit: prototyping and simulating interaction with media façades
UbiComp '13: Proceedings of the 2013 ACM international joint conference on Pervasive and ubiquitous computing

Digital technologies are rapidly finding their way into urban spaces. One prominent example is media façades. Due to their size, visibility and their technical capabilities, they offer great potential for interaction and for becoming the future displays ...
Façade map: continuous interaction with media façades using cartographic map projections
UbiComp '12: Proceedings of the 2012 ACM Conference on Ubiquitous Computing

The increasing number of media façades is a prominent example of the digital augmentation of urban spaces. Many media façades cover most of the outer shell of a building and come with a 3D form factor. They offer great potential for remote interaction ...

Comments

Information & Contributors

Information

Published In

cover image Guide Proceedings

ECCV'12: Proceedings of the 12th European conference on Computer Vision - Volume Part VII

October 2012

489 pages

ISBN:9783642337857

Editors:
Andrew Fitzgibbon
Microsoft Research Ltd., Cambridge, UK
,
Svetlana Lazebnik
Dept. of Computer Science, University of North Carolina, Chapel Hill, NC, UK
,
Pietro Perona
Dept. of Computer Science, California Institute of Technology, Pasadena, CA, UK
,
Yoichi Sato
Institute of Industrial Science, The University of Tokyo, Tokyo, CA, Japan
,
Cordelia Schmid
Institute of Industrial Science, INRIA, Montbonnot, CA, France

Sponsors

Adobe
TOYOTA: TOYOTA
Google Inc.
IBMR: IBM Research
Microsoft Reasearch: Microsoft Reasearch

Publisher

Springer-Verlag

Berlin, Heidelberg

Publication History

Published: 07 October 2012

Qualifiers

Article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

11
Total Citations
View Citations
0
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 25 Oct 2024

Other Metrics

View Author Metrics

Citations

Cited By

Nauata NFurukawa Y(2020)Vectorizing World Buildings: Planar Graph Reconstruction by Primitive Detection and Relationship InferenceComputer Vision – ECCV 202010.1007/978-3-030-58598-3_42(711-726)Online publication date: 23-Aug-2020
https://dl.acm.org/doi/10.1007/978-3-030-58598-3_42
Dogan TKnutins MRakha TTurrin M(2018)CitySeekProceedings of the Symposium on Simulation for Architecture and Urban Design10.5555/3289750.3289786(1-8)Online publication date: 4-Jun-2018
https://dl.acm.org/doi/10.5555/3289750.3289786
Lian YShen XHu Y(2018)Detecting and inferring repetitive elements with accurate locations and shapes from façadesThe Visual Computer: International Journal of Computer Graphics10.1007/s00371-017-1355-z34:4(491-506)Online publication date: 1-Apr-2018
https://dl.acm.org/doi/10.1007/s00371-017-1355-z
Kelly TFemiani JWonka PMitra N(2017)BigSURACM Transactions on Graphics10.1145/3130800.313082336:6(1-16)Online publication date: 20-Nov-2017
https://dl.acm.org/doi/10.1145/3130800.3130823
Li CWand M(2015)Approximate Translational Building Blocks for Image Decomposition and SynthesisACM Transactions on Graphics10.1145/275728734:5(1-16)Online publication date: 3-Nov-2015
https://dl.acm.org/doi/10.1145/2757287
Verdie YLafarge FAlliez P(2015)LOD Generation for Urban ScenesACM Transactions on Graphics10.1145/273252734:3(1-14)Online publication date: 8-May-2015
https://dl.acm.org/doi/10.1145/2732527
Weissenberg JGygli MRiemenschneider HVan Gool LSchmid FKray CFritze H(2014)Navigation using special buildings as signpostsProceedings of the 2nd ACM SIGSPATIAL International Workshop on Interacting with Maps10.1145/2677068.2677070(8-14)Online publication date: 4-Nov-2014
https://dl.acm.org/doi/10.1145/2677068.2677070
Fan LMusialski PLiu LWonka P(2014)Structure completion for facade layoutsACM Transactions on Graphics10.1145/2661229.266126533:6(1-11)Online publication date: 19-Nov-2014
https://dl.acm.org/doi/10.1145/2661229.2661265
Boulch AHoullier SMarlet RTournaire OFalcidieno BSpagnuolo MLipman YZhang H(2013)Semantizing complex 3D scenes using constrained attribute grammarsProceedings of the Eleventh Eurographics/ACMSIGGRAPH Symposium on Geometry Processing10.5555/2600289.2600294(33-42)Online publication date: 3-Jul-2013
https://dl.acm.org/doi/10.5555/2600289.2600294
Musialski PWonka PAliaga DWimmer MGool LPurgathofer W(2013)A Survey of Urban ReconstructionComputer Graphics Forum10.1111/cgf.1207732:6(146-177)Online publication date: 1-Sep-2013
https://dl.acm.org/doi/10.1111/cgf.12077
Show More Cited By

View Options

View options

Media

Figures

Other

Tables

View Table of Contents