research-article

Sketch2Photo: internet image montage

Authors:

Ming-Ming Cheng,

Shi-Min HuAuthors Info & Claims

ACM Transactions on Graphics (TOG), Volume 28, Issue 5

Pages 1 - 10

https://doi.org/10.1145/1618452.1618470

Published: 01 December 2009 Publication History

Abstract

We present a system that composes a realistic picture from a simple freehand sketch annotated with text labels. The composed picture is generated by seamlessly stitching several photographs in agreement with the sketch and text labels; these are found by searching the Internet. Although online image search generates many inappropriate results, our system is able to automatically select suitable photographs to generate a high quality composition, using a filtering scheme to exclude undesirable images. We also provide a novel image blending algorithm to allow seamless image composition. Each blending result is given a numeric score, allowing us to find an optimal combination of discovered images. Experimental results show the method is very successful; we also evaluate our system using the results from two user studies.

Supplementary Material

Supplemental material. (124-chen.zip)

Download
96.59 MB

References

[1]

Belongie, S., Malik, J., and Puzicha, J. 2002. Shape matching and object recognition using shape contexts. IEEE Trans. Pattern Anal. Mach. Intell. 24, 4, 509--522.

Digital Library

[2]

Ben-Haim, N., Babenko, B., and Belongie, S. 2006. Improvingweb-based image search via content based clustering. In Proc. of CVPR Workshop.

Digital Library

[3]

Diakopoulos, N., Essa, I., and Jain, R. 2004. Content based image synthesis. In Proc. of International Conference on Image and Video Retrieval (CIVR).

[4]

Eitz, M., Hildebrand, K., Boubekeur, T., and Alexa, M. 2009. Photosketch: A sketch based image query and compositing system. In SIGGRAPH 2009 Talk Program.

Digital Library

[5]

Farbman, Z., Hoffer, G., Lipman, Y., Cohen-Or, D., and Lischinski, D. 2009. Coordinates for instant image cloning. ACM Transactions on Graphics 28, 3 (Aug.), 67.

Digital Library

[6]

Felzenszwalb, P. F., and Huttenlocher, D. P. 2004. Efficient graph-based image segmentation. Int. J. of Comput. Vision 59, 2, 167--181.

Digital Library

[7]

Fergus, R., Fei-Fei, L., Perona, P., and Zisserman, A. 2005. Learning object categories from google's image search. In Proc. of ICCV.

Digital Library

[8]

Georgescu, B., Shimshoni, I., and Meer, P. 2003. Mean shift based clustering in high dimensions: A texture classification example. In Proc. of ICCV.

Digital Library

[9]

Hays, J. H., and Efros, A. A. 2007. Scene completion using millions of photographs. ACM Transactions on Graphics 26, 3 (July), 4.

Digital Library

[10]

Hou, X., and Zhang, L. 2007. Saliency detection: A spectral residual approach. In Proc. of CVPR.

[11]

Jacobs, C., Finkelstein, A., and Salesin, D. 1995. Fast multiresolution image querying. In SIGGRAPH 1995.

Digital Library

[12]

Jia, J., Sun, J., Tang, C.-K., and Shum, H.-Y. 2006. Drag-and-drop pasting. SIGGRAPH 2006.

Digital Library

[13]

Johnson, M., Brostow, G. J., Shotton, J., Arandjelović, O., Kwatra, V., and Cipolla, R. 2006. Semantic photo synthesis. Proc. of Eurographics.

[14]

Lalonde, J.-F., Hoiem, D., Efros, A. A., Rother, C., Winn, J., and Criminisi, A. 2007. Photo clip art. ACM Transactions on Graphics 26, 3 (Aug.), 3.

Digital Library

[15]

Levin, A., Lischinski, D., and Weiss, Y. 2008. A closed-form solution to natural image matting. IEEE Trans. Pattern Anal. Mach. Intell. 30, 2, 228--242.

Digital Library

[16]

Li, Y., Sun, J., Tang, C.-K., and Shum, H.-Y. 2004. Lazy snapping. SIGGRAPH 2004.

Digital Library

[17]

Liu, T., Sun, J., Zheng, N.-N., Tang, X., and Shum, H.-Y. 2007. Learning to detect a salient object. In Proc. of CVPR.

[18]

Manjunath, B. S., and Ma, W. Y. 1996. Texture features for browsing and retrieval of image data. IEEE Trans. Pattern Anal. Mach. Intell. 18, 8, 837--842.

Digital Library

[19]

Pérez, P., Gangnet, M., and Blake, A. 2003. Poisson image editing. SIGGRAPH 2003.

Digital Library

[20]

Rajendran, R., and Chang, S. 2000. Image retrieval with sketches and compositions. In Proc. of International Conference on Multimedia&Expo (ICME).

[21]

Rother, C., Kolmogorov, V., and Blake, A. 2004. "grabcut": interactive foreground extraction using iterated graph cuts. SIGGRAPH2004.

Digital Library

[22]

Saxena, A., Chung, S. H., and Ng, A. Y. 2008. 3-d depth reconstruction from a single still image. Int. J. of Comput. Vision 76, 1, 53--69.

Digital Library

[23]

Smeulders, A., Worring, M., Santini, S., Gupta, A., and Jain, R. 2000. Content-based image retrieval at the end of the early years. IEEE Trans. Pattern Anal. Mach. Intell. 22, 12, 1349--1380.

Digital Library

[24]

Wang, J., and Cohen, M. 2007. Simultaneous matting and compositing. In Proc. of CVPR, 1--8.

Cited By

Liu FFu HLai YGao L(2024)SketchDream: Sketch-based Text-To-3D Generation and EditingACM Transactions on Graphics10.1145/365812043:4(1-13)Online publication date: 19-Jul-2024
https://dl.acm.org/doi/10.1145/3658120
Cheng ZWu LLi XMeng X(2024)SMFS‐GAN: Style‐Guided Multi‐class Freehand Sketch‐to‐Image SynthesisComputer Graphics Forum10.1111/cgf.15190Online publication date: 7-Aug-2024
https://doi.org/10.1111/cgf.15190
Chen JCong RIp HKwong S(2024)KepSalinst: Using Peripheral Points to Delineate Salient InstancesIEEE Transactions on Cybernetics10.1109/TCYB.2023.332616554:6(3392-3405)Online publication date: Jun-2024
https://doi.org/10.1109/TCYB.2023.3326165
Show More Cited By

Index Terms

Sketch2Photo: internet image montage
1. Computing methodologies
  1. Artificial intelligence
    1. Computer vision
      1. Image and video acquisition
        3D imaging
    2. Knowledge representation and reasoning
  2. Computer graphics
2. Theory of computation
  1. Logic

Recommendations

Sketch2Photo: internet image montage
SIGGRAPH Asia '09: ACM SIGGRAPH Asia 2009 papers

We present a system that composes a realistic picture from a simple freehand sketch annotated with text labels. The composed picture is generated by seamlessly stitching several photographs in agreement with the sketch and text labels; these are found ...
Sketch2Photo: Synthesizing photo-realistic images from sketches via global contexts
Abstract
Sketch-to-image synthesis aims to generate realistic images that match the input sketches or edge maps exactly. Most known sketch-to-image synthesis methods use various generative adversarial networks (GANs) that are trained with ...
Highlights
- We propose a novel generator for sketch-to-image synthesis, which can capture global context information in the early layers of the network and maintain the ...
An art-directed wrinkle system for CG character clothing and skin

We present a kinematic system for creating art-directed clothing and skin wrinkles on CG characters used in the production of computer-animated feature films. This system employs a curve-based method for generating wrinkles on reference poses, which are ...

Comments

Information & Contributors

Information

Published In

cover image ACM Transactions on Graphics

ACM Transactions on Graphics Volume 28, Issue 5

December 2009

646 pages

ISSN:0730-0301

EISSN:1557-7368

DOI:10.1145/1618452

Issue’s Table of Contents

Copyright © 2009 ACM.

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 01 December 2009

Published in TOG Volume 28, Issue 5

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Qualifiers

Research-article

Funding Sources

Singapore FRC
Ministry of Science and Technology of the People's Republic of China
FDCT, Macau

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

391
Total Citations
View Citations
2,652
Total Downloads

Downloads (Last 12 months)113
Downloads (Last 6 weeks)9

Reflects downloads up to 30 Aug 2024

Other Metrics

View Author Metrics

Citations

Cited By

Liu FFu HLai YGao L(2024)SketchDream: Sketch-based Text-To-3D Generation and EditingACM Transactions on Graphics10.1145/365812043:4(1-13)Online publication date: 19-Jul-2024
https://dl.acm.org/doi/10.1145/3658120
Cheng ZWu LLi XMeng X(2024)SMFS‐GAN: Style‐Guided Multi‐class Freehand Sketch‐to‐Image SynthesisComputer Graphics Forum10.1111/cgf.15190Online publication date: 7-Aug-2024
https://doi.org/10.1111/cgf.15190
Chen JCong RIp HKwong S(2024)KepSalinst: Using Peripheral Points to Delineate Salient InstancesIEEE Transactions on Cybernetics10.1109/TCYB.2023.332616554:6(3392-3405)Online publication date: Jun-2024
https://doi.org/10.1109/TCYB.2023.3326165
Nazarieh FFeng ZAwais MWang WKittler J(2024)A Survey of Cross-Modal Visual Content GenerationIEEE Transactions on Circuits and Systems for Video Technology10.1109/TCSVT.2024.335160134:8(6814-6832)Online publication date: Aug-2024
https://doi.org/10.1109/TCSVT.2024.3351601
Hasan MZillanee ARabiul Alam M(2024)Generating Photorealistic Images from Human-Generated Sketches: A GAN-based Synthesis Approach for Enhanced Visual Realism2024 6th International Conference on Electrical Engineering and Information & Communication Technology (ICEEICT)10.1109/ICEEICT62016.2024.10534434(160-165)Online publication date: 2-May-2024
https://doi.org/10.1109/ICEEICT62016.2024.10534434
Zhao LSong DChen WKang Q(2023)Coloring and fusing architectural sketches by combining a Y‐shaped generative adversarial network and a denoising diffusion implicit modelComputer-Aided Civil and Infrastructure Engineering10.1111/mice.1311639:7(1003-1018)Online publication date: 26-Oct-2023
https://dl.acm.org/doi/10.1111/mice.13116
An ZYu JLiu RWang CYu Q(2023)SketchInverter: Multi-Class Sketch-Based Image Generation via GAN Inversion2023 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV)10.1109/WACV56688.2023.00430(4308-4318)Online publication date: Jan-2023
https://doi.org/10.1109/WACV56688.2023.00430
Woo Kim MIk Cho N(2023)WHFL: Wavelet-Domain High Frequency Loss for Sketch-to-Image Translation2023 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV)10.1109/WACV56688.2023.00081(744-754)Online publication date: Jan-2023
https://doi.org/10.1109/WACV56688.2023.00081
Li BHsiao H(2023)A Hybrid Convolutional and Transformer Network for Salient Object Detection2023 IEEE International Conference on Visual Communications and Image Processing (VCIP)10.1109/VCIP59821.2023.10402625(1-5)Online publication date: 4-Dec-2023
https://doi.org/10.1109/VCIP59821.2023.10402625
Tan ZChu QChai MChen DLiao JLiu QLiu BHua GYu N(2023)Semantic Probability Distribution Modeling for Diverse Semantic Image SynthesisIEEE Transactions on Pattern Analysis and Machine Intelligence10.1109/TPAMI.2022.321008545:5(6247-6264)Online publication date: 1-May-2023
https://dl.acm.org/doi/10.1109/TPAMI.2022.3210085
Show More Cited By

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Issue’s Table of Contents