Understanding mystery behind example-based image synthesis
Authors: 
J. Lu, M. Lukáč, D. SýkoraAuthors Info & Claims
J. Lu, M. Lukáč, D. SýkoraAuthors Info & ClaimsPages 5 - 8
Abstract
This tutorial presents a concise overview of development in the field of example-based image synthesis that over the last two decades rapidly evolved into a powerful tool enabling the production of synthetic imagery often indistinguishable from the source exemplar We discuss not only the basic algorithmic concepts but also their further improvements which lead to significant reduction of computational overhead as well as better visual quality. We also demonstrate numerous applications including texture synthesis, hole-filling, video completion, retargeting, reshuffling, morphing, melding, painting by feature, appearance transfer to fluid animations or artistic style transfer to 3D models and facial animations.
References
[1]
{BCK*13} Bénard P., Cole F., Kass M., Mordatch I., Hegarty J., Senn M. S., Fleischer K., Pesare D., Breeden K.: Stylizing animation by example. ACM Transactions on Graphics 32, 4 (2013), 119. 2
[2]
{BSFG09} Barnes C., Shechtman E., Finkelstein A., Goldman D. B.: PatchMatch: A randomized correspondence algorithm for structural image editing. ACM Transactions on Graphics 28, 3 (2009), 24. 2
[3]
{BZ17} Barnes C., Zhang F.-L.: A survey of the state-of-the-art in patch-based synthesis. Computational Visual Media 3, 1 (2017), 3--20. 1
[4]
{CBAF08} Cho T. S., Butman M., Avidan S., Freeman W. T.: The patch transform and its applications to image editing. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (2008). 2
[5]
{CGMP11} Cao F., Gousseau Y., Masnou S., Pérez P.: Geometrically guided exemplar-based inpainting. SIAM Journal on Imaging Sciences 4, 4 (2011), 1143--1179. 2
[6]
{DLTD08} Dong Y., Lefebvre S., Tong X., Drettakis G.: Lazy solid texture synthesis. Computer Graphics Forum 27, 4 (2008), 1165--1174. 2
[7]
{DSB*12} Darabi S., Shechtman E., Barnes C., Goldman D. B., Sen P.: Image Melding: Combining inconsistent images using patch-based synthesis. ACM Transactions on Graphics 31, 4 (2012), 82. 2
[8]
{EL99} Efros A. A., Leung T. K.: Texture synthesis by non-parametric sampling. In Proceedings of IEEE International Conference on Computer Vision (1999), pp. 1033--1038. 2
[9]
{FJL*16} Fišer J., Jamriška O., Lukáč M., Shechtman E., Asente P., Lu J., Sýkora D.: StyLit: Illumination-guided example-based stylization of 3D renderings. ACM Transactions on Graphics 35, 4 (2016), 92. 1, 2, 3
[10]
{FJS*17} Fišer J., Jamriška O., Simons D., Shechtman E., Lu J., Asente P., Lukáč M., Sýkora D.: Example-based synthesis of stylized facial animations. ACM Transactions on Graphics 36, 4 (2017), 155. 1, 2, 3
[11]
{FLJ*14} Fišer J., Lukáč M., Jamriška O., Čadík M., Gingold Y., Asente P., Sýkora D.: Color Me Noisy: Example-based rendering of hand-colored animations with temporal noise control. Computer Graphics Forum 33, 4 (2014), 1--10. 2
[12]
{GKT*12} Granados M., Kim K. I., Tompkin J., Kautz J., Theobalt C.: Background inpainting for videos with dynamic objects and a free-moving camera. In Proceedings of European Conference on Computer Vision (2012), pp. 682--695. 2
[13]
{GM14} Guillemot C., Meur O. L.: Image inpainting: Overview and recent advances. IEEE Signal Processing Magazine 31, 1 (2014), 127--144. 2
[14]
{HE07} Hays J., Efros A. A.: Scene completion using millions of photographs. ACM Transactions on Graphics 26, 3 (2007). 3
[15]
{HJO*01} Hertzmann A., Jacobs C. E., Oliver N., Curless B., Salesin D. H.: Image analogies. In SIGGRAPH Conference Proceedings (2001), pp. 327--340. 2
[16]
{HKAK14} Huang J.-B., Kang S. B., Ahuja N., Kopf J.: Image completion using planar structure guidance. ACM Transactions on Graphics 33, 4 (2014), 129. 2
[17]
{ISSI17} Iizuka S., Simo-Serra E., Ishikawa H.: Globally and locally consistent image completion. ACM Transactions on Graphics 36, 4 (jul 2017), 107:1--107:14. 3
[18]
{JFA*15} Jamriška O., Fišer J., Asente P., Lu J., Shechtman E., Sýkora D.: LazyFluids: Appearance transfer for fluid animations. ACM Transactions on Graphics 34, 4 (2015), 92. 1, 3
[19]
{KEBK05} Kwatra V., Essa I. A., Bobick A. F., Kwatra N.: Texture optimization for example-based synthesis. ACM Transactions on Graphics 24, 3 (2005), 795--802. 2, 3
[20]
{KFCO*07} Kopf J., Fu C.-W., Cohen-Or D., Deussen O., Lischinski D., Wong T.-T.: Solid texture synthesis from 2D exemplars. ACM Transactions on Graphics 26, 3 (2007), 2. 2, 3
[21]
{KNL*15} Kaspar A., Neubert B., Lischinski D., Pauly M., Kopf J.: Self tuning texture optimization. Computer Graphics Forum 34, 2 (2015), 349--360. 3
[22]
{LFA*15} Lukáč M., Fišer J., Asente P., Lu J., Shechtman E., Sýkora D.: Brushables: Example-based edge-aware directional texture painting. Computer Graphics Forum 34, 7 (2015), 257--268. 1, 2
[23]
{LFB*13} Lukáč M., Fišer J., Bazin J.-C., Jamriška O., Sorkine-Hornung A., Sýkora D.: Painting by feature: Texture boundaries for example-based image creation. ACM Transactions on Graphics 32, 4 (2013), 116. 2
[24]
{LHGM05} Lai Y.-K., Hu S.-M., Gu D. X., Martin R. R.: Geometric texture synthesis and transfer via geometry images. In Proceedings of ACM Symposium on Solid and Physical Modeling (2005), pp. 15--26. 2
[25]
{LPKK12} Lee S.-H., Park T., Kim J.-H., Kim C.-H.: Adaptive synthesis of distance fields. IEEE Transactions on Visualization and Computer Graphics 18, 7 (2012), 1135--1145. 2
[26]
{LYY*17} Liao J., Yao Y., Yuan L., Hua G., Kang S. B.: Visual attribute transfer through deep image analogy. ACM Transactions on Graphics 36, 4 (2017), 120. 3
[27]
{NAF*14} Newson A., Almansa A., Fradet M., Gousseau Y., Pérez P.: Video inpainting of complex scenes. SIAM Journal of Imaging Science 7, 4 (2014), 1993--2019. 2
[28]
{PKD*16} Pathak D., Krähenbühl P., Donahue J., Darrell T., Efros A.: Context encoders: Feature learning by inpainting. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (2016). 3
[29]
{PS00} Portilla J., Simoncelli E. P.: A parametric texture model based on joint statistics of complex wavelet coefficients. International Journal of Computer Vision 40, 1 (2000), 49--70. 2
[30]
{RCOL09} Rosenberger A., Cohen-Or D., Lischinski D.: Layered Shape Synthesis: Automatic generation of control maps for non-stationary textures. ACM Transactions on Graphics 28, 5 (2009), 107. 3
[31]
{RLC*06} Ritter L., Li W., Curless B., Agrawala M., Salesin D.: Painting with texture. In Proceedings of Eurographics Symposium on Rendering (2006), pp. 371--376. 2
[32]
{SCSI08} Simakov D., Caspi Y., Shechtman E., Irani M.: Summarizing visual data using bidirectional similarity. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (2008). 2, 3
[33]
{WHZ*08} Wei L.-Y., Han J., Zhou K., Bao H., Guo B., Shum H.-Y.: Inverse texture synthesis. ACM Transactions on Graphics 27, 3 (2008). 3
[34]
{WLKT09} Wei L.-Y., Lefebvre S., Kwatra V., Turk G.: State of the art in example-based texture synthesis. In Eurographics (STARs) (2009), pp. 93--117. 1
[35]
{WSI07} Wexler Y., Shechtman E., Irani M.: Space-time completion of video. IEEE Transactions on Pattern Analysis and Machine Intelligence 29, 3 (2007), 463--476. 2
Recommendations
Example-based wrinkle synthesis for clothing animation
SIGGRAPH '10: ACM SIGGRAPH 2010 papersThis paper describes a method for animating the appearance of clothing, such as pants or a shirt, that fits closely to a figure's body. Compared to flowing cloth, such as loose dresses or capes, these types of garments involve nearly continuous ...
Example-based wrinkle synthesis for clothing animation
This paper describes a method for animating the appearance of clothing, such as pants or a shirt, that fits closely to a figure's body. Compared to flowing cloth, such as loose dresses or capes, these types of garments involve nearly continuous ...
Interactive example-based urban layout synthesis
SIGGRAPH Asia '08: ACM SIGGRAPH Asia 2008 papersWe present an interactive system for synthesizing urban layouts by example. Our method simultaneously performs both a structure-based synthesis and an image-based synthesis to generate a complete urban layout with a plausible street network and with ...
Comments
Information & Contributors
Information
Published In
Publisher
Eurographics Association
Goslar, Germany
Publication History
Published: 16 April 2018
Qualifiers
- Research-article
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 0Total Downloads
- Downloads (Last 12 months)0
- Downloads (Last 6 weeks)0
Reflects downloads up to 10 Nov 2024
Other Metrics
Citations
View Options
View options
Get Access
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in