research-article

3-Sweep: extracting editable objects from a single photo

Authors:

Daniel Cohen-OrAuthors Info & Claims

ACM Transactions on Graphics (TOG), Volume 32, Issue 6

Article No.: 195, Pages 1 - 10

https://doi.org/10.1145/2508363.2508378

Published: 01 November 2013 Publication History

Abstract

We introduce an interactive technique for manipulating simple 3D shapes based on extracting them from a single photograph. Such extraction requires understanding of the components of the shape, their projections, and relations. These simple cognitive tasks for humans are particularly difficult for automatic algorithms. Thus, our approach combines the cognitive abilities of humans with the computational accuracy of the machine to solve this problem. Our technique provides the user the means to quickly create editable 3D parts---human assistance implicitly segments a complex object into its components, and positions them in space. In our interface, three strokes are used to generate a 3D component that snaps to the shape's outline in the photograph, where each stroke defines one dimension of the component. The computer reshapes the component to fit the image of the object in the photograph as well as to satisfy various inferred geometric constraints imposed by its global 3D structure. We show that with this intelligent interactive modeling tool, the daunting task of object extraction is made simple. Once the 3D object has been extracted, it can be quickly edited and placed back into photos or 3D scenes, permitting object-driven photo editing tasks which are impossible to perform in image-space. We show several examples and present a user study illustrating the usefulness of our technique.

Supplementary Material

ZIP File (a195-chen.zip)

Supplemental material.

Download
48.72 MB

References

[1]

Andre, A., and Saito, S. 2011. Single-view sketch based modeling. In Proceedings of the Eighth Eurographics Symposium on Sketch-Based Interfaces and Modeling, 133--140.

Digital Library

[2]

Angelidis, A., Canif, M., Wyvill, G., and King, S. 2004. Swirling-sweepers: Constant-volume modeling. In 12th Pacific Conference on Computer Graphics and Applications, 2004. PG 2004. Proceedings, 10--15.

Digital Library

[3]

Arbelaez, P., Maire, M., Fowlkes, C., and Malik, J. 2011. Contour detection and hierarchical image segmentation. IEEE Transactions on Pattern Analysis and Machine Intelligence 33, 5, 898--916.

Digital Library

[4]

Arikan, M., Schwärzler, M., Flöry, S., Wimmer, M., and Maierhofer, S. 2013. O-snap: Optimization-based snapping for modeling architecture. ACM Transactions on Graphics (TOG) 32, 1, 6.

Digital Library

[5]

Barrett, W., and Cheney, A. 2002. Object-based image editing. In ACM Transactions on Graphics (TOG), vol. 21, 777--784.

Digital Library

[6]

Benko, P., Kós, G., Várady, T., Andor, L., and Martin, R. 2002. Constrained fitting in reverse engineering. Computer Aided Geometric Design 19, 3, 173--205.

Digital Library

[7]

Binford, T. O. 1971. Visual perception by computer. In IEEE conference on Systems and Control, vol. 261, 262.

[8]

Bolle, R. M., and Vemuri, B. C. 1991. On three-dimensional surface reconstruction methods. IEEE Transactions on Pattern Analysis and Machine Intelligence 13, 1, 1--13.

Digital Library

[9]

Cheng, M., Zhang, F., Mitra, N., Huang, X., and Hu, S. 2010. Repfinder: finding approximately repeated scene elements for image editing. ACM Transactions on Graphics (TOG) 29, 4, 83.

Digital Library

[10]

Cheng, M. 2009. Curve structure extraction for cartoon images. In Proceedings of The 5th Joint Conference on Harmonious Human Machine Environment, 13--25.

[11]

Choi, B., and Lee, C. 1990. Sweep surfaces modelling via coordinate transformation and blending. Computer-Aided Design 22, 2, 87--96.

Digital Library

[12]

Debevec, P. E., Taylor, C. J., and Malik, J. 1996. Modeling and rendering architecture from photographs: A hybrid geometry-and image-based approach. In Proceedings of the 23rd annual conference on Computer graphics and interactive techniques, 11--20.

Digital Library

[13]

Eitz, M., Sorkine, O., and Alexa, M. 2007. Sketch based image deformation. In Proceedings of Vision, Modeling and Visualization (VMV), 135--142.

[14]

Gal, R., Sorkine, O., Mitra, N., and Cohen-Or, D. 2009. iwires: an analyze-and-edit approach to shape manipulation. In ACM Transactions on Graphics (TOG), vol. 28, ACM, 33.

Digital Library

[15]

Gingold, Y., Igarashi, T., and Zorin, D. 2009. Structured annotations for 2d-to-3d modeling. In ACM Transactions on Graphics (TOG), vol. 28, ACM, 148.

Digital Library

[16]

Goldberg, C., Chen, T., Zhang, F., Shamir, A., and Hu, S. 2012. Data-driven object manipulation in images. In Computer Graphics Forum, vol. 31, 265--274.

Digital Library

[17]

Jiang, N., Tan, P., and Cheong, L. 2009. Symmetric architecture modeling with a single image. ACM Transactions on Graphics (TOG) 28, 5, 113.

Digital Library

[18]

Lalonde, J., Hoiem, D., Efros, A., Rother, C., Winn, J., and Criminisi, A. 2007. Photo clip art. In ACM Transactions on Graphics (TOG), vol. 26, 3.

Digital Library

[19]

Langbein, F., Marshall, A., and Martin, R. 2004. Choosing consistent constraints for beautification of reverse engineered geometric models. Computer-Aided Design 36, 3, 261--278.

[20]

Li, Y., Wu, X., Chrysathou, Y., Sharf, A., Cohen-Or, D., and Mitra, N. 2011. Globfit: Consistently fitting primitives by discovering global relations. In ACM Transactions on Graphics (TOG), vol. 30, 52.

Digital Library

[21]

Merrell, P., and Manocha, D. 2011. Model synthesis: A general procedural modeling algorithm. IEEE Transactions on Visualization and Computer Graphics 17, 6, 715--728.

Digital Library

[22]

Metaxas, D. N. 1996. Physics-based deformable models: applications to computer vision, graphics, and medical imaging. Kluwer Academic Publishers.

Digital Library

[23]

Mille, J., Boné, R., and Cohen, L. D. 2008. Region-based 2d deformable generalized cylinder for narrow structures segmentation. In ECCV 2008. 392--404.

Digital Library

[24]

Oh, B., Chen, M., Dorsey, J., and Durand, F. 2001. Image-based modeling and photo editing. In Proceedings of the 28th annual conference on Computer graphics and interactive techniques, 433--442.

Digital Library

[25]

Olsen, L., and Samavati, F. F. 2010. Image-assisted modeling from sketches. In Proceedings of Graphics Interface 2010, Canadian Information Processing Society, 225--232.

Digital Library

[26]

Oswald, M. R., Toppe, E., and Cremers, D. 2012. Fast and globally optimal single view reconstruction of curved objects. In IEEE CVPR, 534--541.

Digital Library

[27]

Russell, B., and Torralba, A. 2009. Building a database of 3d scenes from user annotations. In IEEE CVPR, 2711--2718.

[28]

Schmidt, R., and Wyvill, B. 2005. Generalized sweep templates for implicit modeling. In Proceedings of the 3rd international conference on Computer graphics and interactive techniques in Australasia and South East Asia, 187--196.

Digital Library

[29]

Schmidt, R., Wyvill, B., and Sousa, M. C. 2005. Sketch-based modeling with the blob tree. In ACM SIGGRAPH 2005 Sketches, 90.

Digital Library

[30]

Schmidt, R., Khan, A., Singh, K., and Kurtenbach, G. 2009. Analytic drawing of 3d scaffolds. In ACM Transactions on Graphics (TOG), vol. 28, 149.

Digital Library

[31]

Seitz, S., Curless, B., Diebel, J., Scharstein, D., and Szeliski, R. 2006. A comparison and evaluation of multi-view stereo reconstruction algorithms. In IEEE CVPR, vol. 1, 519--528.

Digital Library

[32]

Shtof, A., Agathos, A., Gingold, Y., Shamir, A., and Cohen-Or, D. 2013. Geosemantic snapping for sketch-based modeling. In Eurographics.

[33]

Snavely, N. 2011. Scene reconstruction and visualization from internet photo collections: A survey. IPSJ Transactions on Computer Vision and Applications 3, 0, 44--66.

[34]

Tan, P., Zeng, G., Wang, J., Kang, S. B., and Quan, L. 2007. Image-based tree modeling. ACM Transactions on Graphics (TOG) 26, 3, 87.

Digital Library

[35]

Terzopoulos, D., Witkin, A., and Kass, M. 1988. Constraints on deformable models: Recovering 3d shape and nonrigid motion. Artificial intelligence 36, 1, 91--123.

Digital Library

[36]

Tsang, S., Balakrishnan, R., Singh, K., and Ranjan, A. 2004. A suggestive interface for image guided 3d sketching. In Proceedings of the SIGCHI conference on Human factors in computing systems, 591--598.

Digital Library

[37]

Xiao, C., Liu, M., Yongwei, N., and Dong, Z. 2011. Fast exact nearest patch matching for patch-based image editing and processing. IEEE Transactions on Visualization and Computer Graphics 17, 8, 1122--1134.

Digital Library

[38]

Xu, K., Zheng, H., Zhang, H., Cohen-Or, D., Liu, L., and Xiong, Y. 2011. Photo-inspired model-driven 3d object modeling. In ACM Transactions on Graphics (TOG), vol. 30, 80.

Digital Library

[39]

Xu, K., Zhang, H., Cohen-Or, D., and Chen, B. 2012. Fit and diverse: Set evolution for inspiring 3d shape galleries. ACM Transactions on Graphics (TOG) 31, 4, 57.

Digital Library

[40]

Xue, T., Liu, J., and Tang, X. 2011. Symmetric piecewise planar object reconstruction from a single image. In IEEE CVPR, 2577--2584.

Digital Library

[41]

Zeleznik, R. C., Herndon, K. P., and Hughes, J. F. 1996. Sketch: an interface for sketching 3d scenes. In ACM SIGGRAPH, 163--170.

Digital Library

[42]

Zhang, X., Gao, Y., and Caelli, T. 2010. Primitive-based 3d structure inference from a single 2d image for insect modeling: Towards an electronic field guide for insect identification. In International Conference on Control Automation Robotics & Vision (ICARCV), 866--871.

[43]

Zheng, Y., Fu, H., Cohen-Or, D., Au, O., and Tai, C. 2011. Component-wise controllers for structure-preserving shape manipulation. In Computer Graphics Forum, vol. 30, 563--572.

[44]

Zheng, Y., Chen, X., Cheng, M., Zhou, K., Hu, S., and Mitra, N. 2012. Interactive images: cuboid proxies for smart image manipulation. ACM Transactions on Graphics (TOG) 31, 4, 99.

Digital Library

[45]

Zhou, S., Fu, H., Liu, L., Cohen-Or, D., and Han, X. 2010. Parametric reshaping of human bodies in images. ACM Transactions on Graphics (TOG) 29, 4, 126.

Digital Library

Cited By

Zhu XChen ZHu RHan X(2024)Controllable Shape Modeling with Neural Generalized CylinderSIGGRAPH Asia 2024 Conference Papers10.1145/3680528.3687617(1-11)Online publication date: 3-Dec-2024
https://dl.acm.org/doi/10.1145/3680528.3687617
Yenphraphai JPan XLiu SPanozzo DXie S(2024)Image Sculpting: Precise Object Editing with 3D Geometry Control2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)10.1109/CVPR52733.2024.00406(4241-4251)Online publication date: 16-Jun-2024
https://doi.org/10.1109/CVPR52733.2024.00406
Puhachov IMartens CKry PBessmeltsev M(2023)Reconstruction of Machine-Made Shapes from Bitmap SketchesACM Transactions on Graphics10.1145/361836142:6(1-16)Online publication date: 5-Dec-2023
https://dl.acm.org/doi/10.1145/3618361
Show More Cited By

Index Terms

3-Sweep: extracting editable objects from a single photo
1. Computing methodologies
  1. Computer graphics
    1. Shape modeling
2. Theory of computation
  1. Randomness, geometry and discrete structures

Recommendations

Exposing photo manipulation with inconsistent reflections

The advent of sophisticated photo editing software has made it increasingly easier to manipulate digital images. Often visual inspection cannot definitively distinguish the resulting forgeries from authentic photographs. In response, forensic techniques ...
Extracting 3D objects from photographs using 3-sweep

We introduce an interactive technique to extract and manipulate simple 3D shapes in a single photograph. Such extraction requires an understanding of the shape's components, their projections, and their relationships. These cognitive tasks are simple ...
Differential-Based Geometry and Texture Editing with Brushes

We present an interactive modeling framework for 3D shapes and for texture maps. The technique combines a differential-based deformation method with the idea of geometry brushes that allow to interactively apply modifications by painting on the ...

Comments

Information & Contributors

Information

Published In

cover image ACM Transactions on Graphics

ACM Transactions on Graphics Volume 32, Issue 6

November 2013

671 pages

ISSN:0730-0301

EISSN:1557-7368

DOI:10.1145/2508363

Issue’s Table of Contents

Copyright © 2013 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 01 November 2013

Published in TOG Volume 32, Issue 6

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

168
Total Citations
View Citations
2,030
Total Downloads

Downloads (Last 12 months)59
Downloads (Last 6 weeks)4

Reflects downloads up to 22 Dec 2024

Other Metrics

View Author Metrics

Citations

Cited By

Zhu XChen ZHu RHan X(2024)Controllable Shape Modeling with Neural Generalized CylinderSIGGRAPH Asia 2024 Conference Papers10.1145/3680528.3687617(1-11)Online publication date: 3-Dec-2024
https://dl.acm.org/doi/10.1145/3680528.3687617
Yenphraphai JPan XLiu SPanozzo DXie S(2024)Image Sculpting: Precise Object Editing with 3D Geometry Control2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)10.1109/CVPR52733.2024.00406(4241-4251)Online publication date: 16-Jun-2024
https://doi.org/10.1109/CVPR52733.2024.00406
Puhachov IMartens CKry PBessmeltsev M(2023)Reconstruction of Machine-Made Shapes from Bitmap SketchesACM Transactions on Graphics10.1145/361836142:6(1-16)Online publication date: 5-Dec-2023
https://dl.acm.org/doi/10.1145/3618361
Kim HLee GChoi YKim JZhu J(2023)3D-aware Blending with Generative NeRFs2023 IEEE/CVF International Conference on Computer Vision (ICCV)10.1109/ICCV51070.2023.02094(22849-22861)Online publication date: 1-Oct-2023
https://doi.org/10.1109/ICCV51070.2023.02094
Deng KYang GRamanan DZhu J(2023)3D-aware Conditional Image Synthesis2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)10.1109/CVPR52729.2023.00431(4434-4445)Online publication date: Jun-2023
https://doi.org/10.1109/CVPR52729.2023.00431
Sawada TPizlo Z(2023)Geometrical properties of a generalized cone and its 2D imageJournal of Mathematical Psychology10.1016/j.jmp.2023.102765114(102765)Online publication date: Jun-2023
https://doi.org/10.1016/j.jmp.2023.102765
Li J(2022)Semantic Segmentation-Based Adaptive Mining Algorithm for Ceramic Cultural Resource DataMobile Information Systems10.1155/2022/28150772022Online publication date: 1-Jan-2022
https://dl.acm.org/doi/10.1155/2022/2815077
Antoine AMalacria SVogel DCasiez G(2022)Étude de l’influence de la taille des sphères virtuelles de contrôle sur les rotations 3DProceedings of the 33rd Conference on l'Interaction Humain-Machine10.1145/3500866.3516370(1-10)Online publication date: 5-Apr-2022
https://dl.acm.org/doi/10.1145/3500866.3516370
Du XHe YYang XChang CXie H(2022)Sketch-based 3D shape modeling from sparse point cloudsInternational Workshop on Advanced Imaging Technology (IWAIT) 202210.1117/12.2626116(119)Online publication date: 1-May-2022
https://doi.org/10.1117/12.2626116
Wang JHafi LTaniguchi AHagiwara YTaniguchi T(2022)Extending HoloGAN by Embedding Image Content into Latent Vectors for Novel View Synthesis2022 IEEE/SICE International Symposium on System Integration (SII)10.1109/SII52469.2022.9708823(383-389)Online publication date: 9-Jan-2022
https://doi.org/10.1109/SII52469.2022.9708823
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Issue’s Table of Contents