research-article

3D object manipulation in a single photograph using stock 3D models

Authors:

Natasha Kholgade,

Yaser SheikhAuthors Info & Claims

ACM Transactions on Graphics (TOG), Volume 33, Issue 4

Article No.: 127, Pages 1 - 12

https://doi.org/10.1145/2601097.2601209

Published: 27 July 2014 Publication History

Abstract

Photo-editing software restricts the control of objects in a photograph to the 2D image plane. We present a method that enables users to perform the full range of 3D manipulations, including scaling, rotation, translation, and nonrigid deformations, to an object in a photograph. As 3D manipulations often reveal parts of the object that are hidden in the original photograph, our approach uses publicly available 3D models to guide the completion of the geometry and appearance of the revealed areas of the object. The completion process leverages the structure and symmetry in the stock 3D model to factor out the effects of illumination, and to complete the appearance of the object. We demonstrate our system by producing object manipulations that would be impossible in traditional 2D photo-editing programs, such as turning a car over, making a paper-crane flap its wings, or manipulating airplanes in a historical photograph to change its story.

Supplementary Material

ZIP File (a127-kholgade.zip)

Supplemental material.

Download
341.61 MB

MP4 File (a127-sidebyside.mp4)

Download
20.69 MB

References

[1]

Aubry, M., Maturana, D., Efros, A., Russell, B., and Sivic, J. 2014. Seeing 3D chairs: exemplar part-based 2D-3D alignment using a large dataset of CAD models. In Proc. CVPR.

[2]

Avidan, S., and Shamir, A. 2007. Seam carving for content-aware image resizing. In Proc. ACM SIGGRAPH.

Digital Library

[3]

Barnes, C., Shechtman, E., Finkelstein, A., and Goldman, D. B. 2009. Patchmatch: a randomized correspondence algorithm for structural image editing. In Proc. ACM SIGGRAPH, 24:1--24:11.

Digital Library

[4]

Barrett, W. A., and Cheney, A. S. 2002. Object-based image editing. In Proc. ACM SIGGRAPH, 777--784.

Digital Library

[5]

Barron, J. T. 2012. Shape, albedo, and illumination from a single image of an unknown object. In CVPR, 334--341.

Digital Library

[6]

Blanz, V., and Vetter, T. 1999. A morphable model for the synthesis of 3d faces. In Proc. ACM SIGGRAPH, 187--194.

Digital Library

[7]

Bokeloh, M., Wand, M., Koltun, V., and Seidel, H.-P. 2011. Pattern-aware shape deformation using sliding dockers. ACM Trans. Graph. 30, 6 (Dec.), 123:1--123:10.

Digital Library

[8]

Chen, J., Paris, S., Wang, J., Matusik, W., Cohen, M., and Durand, F. 2011. The video mesh: A data structure for image-based three-dimensional video editing. In ICCP, 1--8.

[9]

Chen, T., Zhu, Z., Shamir, A., Hu, S.-M., and Cohen-Or, D. 2013. 3-sweep: Extracting editable objects from a single photo. ACM Trans. Graph. 32, 6, to appear.

Digital Library

[10]

Debevec, P. E., Taylor, C. J., and Malik, J. 1996. Modeling and rendering architecture from photographs: a hybrid geometry- and image-based approach. In Proc. ACM SIGGRAPH, 11--20.

Digital Library

[11]

Debevec, P. 1998. Rendering synthetic objects into real scenes: bridging traditional and image-based graphics with global illumination and high dynamic range photography. In Proc. ACM SIGGRAPH, 189--198.

Digital Library

[12]

Durand, F. 2002. An invitation to discuss computer depiction. In Proc. ACM NPAR, 111--124.

Digital Library

[13]

Fang, H., and Hart, J. C. 2004. Textureshop: texture synthesis as a photograph editing tool. Proc. ACM SIGGRAPH, 354--359.

Digital Library

[14]

Fisher, R. 1953. Dispersion on a sphere. In Proceedings of the Royal Society of London. Series A, Mathematical and Physical Sciences, vol. 217, 295--305.

[15]

Gal, R., and Cohen-Or, D. 2006. Salient geometric features for partial shape matching and similarity. ACM Trans. Graph. 25, 1 (Jan.), 130--150.

Digital Library

[16]

Gal, R., Wexler, Y., Ofek, E., Hoppe, H., and Cohen-Or, D. 2010. Seamless montage for texturing models. Comput. Graph. Forum 29, 2, 479--486.

[17]

Goldberg, C., Chen, T., Zhang, F.-L., Shamir, A., and Hu, S.-M. 2012. Data-driven object manipulation in images. Computer Graphics Forum 31, 2pt1, 265--274.

Digital Library

[18]

Haber, T., Fuchs, C., Bekaert, P., Seidel, H.-P., Goesele, M., and Lensch, H. P. A. 2009. Relighting objects from image collections. In CVPR, IEEE, 627--634.

[19]

Hara, K., Nishino, K., and Ikeuchi, K. 2008. Mixture of spherical distributions for single-view relighting. IEEE Trans. Pattern Anal. Mach. Intell. 30, 1, 25--35.

Digital Library

[20]

Hong, W., Yang, A. Y., Huang, K., and Ma, Y. 2004. On symmetry and multiple-view geometry: Structure, pose, and calibration from a single image. IJCV 60, 3 (Dec.), 241--265.

Digital Library

[21]

Karsch, K., Hedau, V., Forsyth, D., and Hoiem, D. 2011. Rendering synthetic objects into legacy photographs. In Proc. ACM SIGGRAPH Asia, 157:1--157:12.

Digital Library

[22]

Khan, E. A., Reinhard, E., Fleming, R. W., and Bülthoff, H. H. 2006. Image-based material editing. In Proc. ACM SIGGRAPH, 654--663.

Digital Library

[23]

Kim, V. G., Lipman, Y., and Funkhouser, T. 2012. Symmetry-guided texture synthesis and manipulation. ACM Trans. Graph. 31, 3 (June), 22:1--22:14.

Digital Library

[24]

Kolmogorov, V. 2006. Convergent tree-reweighted message passing for energy minimization. IEEE TPAMI 28, 10 (Oct.), 1568--1583.

Digital Library

[25]

Kopf, J., Neubert, B., Chen, B., Cohen, M., Cohen-Or, D., Deussen, O., Uyttendaele, M., and Lischinski, D. 2008. Deep photo: model-based photograph enhancement and viewing. In Proc. ACM SIGGRAPH Asia, 116:1--116:10.

Digital Library

[26]

Kraevoy, V., Sheffer, A., and Gotsman, C. 2003. Match-maker: constructing constrained texture maps. ACM Trans. Graph. 22, 3 (July), 326--333.

Digital Library

[27]

Kraevoy, V., Sheffer, A., and van de Panne, M. 2009. Modeling from contour drawings. In Proceedings of the 6th Eurographics Symposium on Sketch-Based Interfaces and Modeling, ACM, New York, NY, USA, SBIM '09, 37--44.

Digital Library

[28]

Kwatra, V., Schodl, A., Essa, I., Turk, G., and Bobick, A. 2003. Graphcut textures: Image and video synthesis using graph cuts. ACM Trans. Graph. 22, 3 (July), 277--286.

Digital Library

[29]

Lalonde, J.-F., Hoiem, D., Efros, A. A., Rother, C., Winn, J., and Criminisi, A. 2007. Photo clip art. Proc. ACM SIGGRAPH 26, 3 (August), 3.

Digital Library

[30]

Land, E. H., John, and Mccann, J. 1971. Lightness and retinex theory. Journal of the Optical Society of America, 1--11.

[31]

Lepetit, V., Moreno-Noguer, F., and Fua, P. 2009. Epnp: An accurate o(n) solution to the pnp problem. Int. J. Comput. Vision 81, 155--166.

Digital Library

[32]

Lim, J., Piriavash, H., and Torralba, A. 2013. Parsing ikea objects: Fine pose estimation. In ICCV.

Digital Library

[33]

Mei, X., Ling, H., and Jacobs, D. 2009. Sparse representation of cast shadows via l1-regularized least squares. In ICCV.

[34]

Mitra, N. J., and Pauly, M. 2008. Symmetry for architectural design. In Advances in Architectural Geometry, 13--16.

[35]

Mitra, N. J., Guibas, L. J., and Pauly, M. 2006. Partial and approximate symmetry detection for 3d geometry. In Proc. ACM SIGGRAPH, 560--568.

Digital Library

[36]

Nealen, A., Sorkine, O., Alexa, M., and Cohen-Or, D. 2005. A sketch-based interface for detail-preserving mesh editing. ACM Trans. Graph. 24, 3 (July), 1142--1147.

Digital Library

[37]

Ng, R., Ramamoorthi, R., and Hanrahan, P. 2003. All-frequency shadows using non-linear wavelet lighting approximation. In Proc. ACM SIGGRAPH, 376--381.

Digital Library

[38]

Oh, B. M., Chen, M., Dorsey, J., and Durand, F. 2001. Image-based modeling and photo editing. In Proc. ACM SIGGRAPH, 433--442.

Digital Library

[39]

Okabe, T., Sato, I., and Sato, Y. 2004. Spherical harmonics vs. haar wavelets: Basis for recovering illumination from cast shadows. In CVPR, 50--57.

[40]

Panagopoulos, A., Samaras, D., and Paragios, N. 2009. Robust shadow and illumination estimation using a mixture model. In CVPR, 651--658.

[41]

Pauly, M., Mitra, N. J., Giesen, J., Gross, M., and Guibas, L. J. 2005. Example-based 3d scan completion. In Proc. SGP.

Digital Library

[42]

Prasad, M., Zisserman, A., and Fitzgibbon, A. W. 2006. Single view reconstruction of curved surfaces. In CVPR.

Digital Library

[43]

Ramamoorthi, R., and Hanrahan, P. 2001. On the relationship between radiance and irradiance: determining the illumination from images of a convex lambertian object. J. Opt. Soc. Am. A 18, 10, 2448--2459.

[44]

Romeiro, F., and Zickler, T. 2010. Blind reflectometry. In ECCV, 45--58.

Digital Library

[45]

Simpson, J., 2003. Oxford English Dictionary Online, 2nd edition. http://www.oed.com/, July.

[46]

Sorkine, O., and Alexa, M. 2007. As-rigid-as-possible surface modeling. In Proc. SGP, 109--116.

Digital Library

[47]

Terzopoulos, D., Witkin, A., and Kass, M. 1987. Symmetry-seeking models and 3d object reconstruction. International Journal of Computer Vision 1, 211--221.

[48]

Tzur, Y., and Tal, A. 2009. Flexistickers: photogrammetric texture mapping using casual images. ACM Trans. Graph. 28, 3 (July), 45:1--45:10.

Digital Library

[49]

Xu, K., Zheng, H., Zhang, H., Cohen-Or, D., Liu, L., and Xiong, Y. 2011. Photo-inspired model-driven 3d object modeling. ACM Transactions on Graphics 30, 4.

Digital Library

[50]

Zheng, Y., Chen, X., Cheng, M.-M., Zhou, K., Hu, S.-M., and Mitra, N. J. 2012. Interactive images: cuboid proxies for smart image manipulation. ACM Trans. Graph. 31, 4 (July), 99:1--99:11.

Digital Library

[51]

Zou, H., and Hastie, T. 2005. Regularization and variable selection via the elastic net. Journal of the Royal Statistical Society, Series B 67, 301--320.

Cited By

Chang YKim YSeo SYi JKwak N(2024)Fast Sun-aligned Outdoor Scene Relighting based on TensoRF2024 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV)10.1109/WACV57701.2024.00359(3614-3624)Online publication date: 3-Jan-2024
https://doi.org/10.1109/WACV57701.2024.00359
Zhang PYang LXie XLai J(2024)Pose Guided Person Image Generation Via Dual-Task Correlation and Affinity LearningIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2023.328639430:8(5111-5128)Online publication date: 1-Aug-2024
https://dl.acm.org/doi/10.1109/TVCG.2023.3286394
Zhou HMa RZhang LGao LMahdavi-Amiri AZhang H(2024)SAC-GAN: Structure-Aware Image CompositionIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2022.322668930:7(3151-3165)Online publication date: Jul-2024
https://doi.org/10.1109/TVCG.2022.3226689
Show More Cited By

Index Terms

3D object manipulation in a single photograph using stock 3D models
1. Computing methodologies
  1. Computer graphics
    1. Graphics systems and interfaces
      1. Virtual reality

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Transactions on Graphics

ACM Transactions on Graphics Volume 33, Issue 4

July 2014

1366 pages

ISSN:0730-0301

EISSN:1557-7368

DOI:10.1145/2601097

Issue’s Table of Contents

Copyright © 2014 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 27 July 2014

Published in TOG Volume 33, Issue 4

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

Google

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

120
Total Citations
View Citations
2,379
Total Downloads

Downloads (Last 12 months)51
Downloads (Last 6 weeks)1

Reflects downloads up to 27 Dec 2024

Other Metrics

View Author Metrics

Citations

Cited By

Chang YKim YSeo SYi JKwak N(2024)Fast Sun-aligned Outdoor Scene Relighting based on TensoRF2024 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV)10.1109/WACV57701.2024.00359(3614-3624)Online publication date: 3-Jan-2024
https://doi.org/10.1109/WACV57701.2024.00359
Zhang PYang LXie XLai J(2024)Pose Guided Person Image Generation Via Dual-Task Correlation and Affinity LearningIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2023.328639430:8(5111-5128)Online publication date: 1-Aug-2024
https://dl.acm.org/doi/10.1109/TVCG.2023.3286394
Zhou HMa RZhang LGao LMahdavi-Amiri AZhang H(2024)SAC-GAN: Structure-Aware Image CompositionIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2022.322668930:7(3151-3165)Online publication date: Jul-2024
https://doi.org/10.1109/TVCG.2022.3226689
Pandey KGuerrero PGadelha MHold-Geoffroy YSingh KMitra N(2024)Diffusion Handles Enabling 3D Edits for Diffusion Models by Lifting Activations to 3D2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)10.1109/CVPR52733.2024.00735(7695-7704)Online publication date: 16-Jun-2024
https://doi.org/10.1109/CVPR52733.2024.00735
Yenphraphai JPan XLiu SPanozzo DXie S(2024)Image Sculpting: Precise Object Editing with 3D Geometry Control2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)10.1109/CVPR52733.2024.00406(4241-4251)Online publication date: 16-Jun-2024
https://doi.org/10.1109/CVPR52733.2024.00406
Shen ISu LWu YChen B(2024)StylePart: image-based shape part manipulationThe Visual Computer10.1007/s00371-024-03310-9Online publication date: 2-Apr-2024
https://doi.org/10.1007/s00371-024-03310-9
Canet Tarrés GLin ZZhang ZZhang JSong YRuta DGilbert ACollomosse JKim S(2024)Thinking Outside the BBox: Unconstrained Generative Object CompositingComputer Vision – ECCV 202410.1007/978-3-031-73033-7_27(476-495)Online publication date: 29-Sep-2024
https://dl.acm.org/doi/10.1007/978-3-031-73033-7_27
Gouvatsos AXiao Z(2024)Sketch-Based Posing for 3D AnimationEncyclopedia of Computer Graphics and Games10.1007/978-3-031-23161-2_47(1663-1672)Online publication date: 5-Jan-2024
https://doi.org/10.1007/978-3-031-23161-2_47
Kim HLee GChoi YKim JZhu J(2023)3D-aware Blending with Generative NeRFs2023 IEEE/CVF International Conference on Computer Vision (ICCV)10.1109/ICCV51070.2023.02094(22849-22861)Online publication date: 1-Oct-2023
https://doi.org/10.1109/ICCV51070.2023.02094
Wang XZhu JYe QHuo YRan YZhong ZChen J(2023)Seal-3D: Interactive Pixel-Level Editing for Neural Radiance Fields2023 IEEE/CVF International Conference on Computer Vision (ICCV)10.1109/ICCV51070.2023.01621(17637-17647)Online publication date: 1-Oct-2023
https://doi.org/10.1109/ICCV51070.2023.01621
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Issue’s Table of Contents