Abstract
Everyone has the dream of being in the center of famous art paintings, admired by numerous future generations. However, the dream came true at a huge cost of the painter’s commission in old days. In our paper, another practical choice is provided for everyone to achieve that dream – an automatic portrait oil painter transferring some artistic styles from one single reference painting. To address this issue, we propose a joint-domain image stylization approach, particularly for portrait oil paintings. From the view of artistic appreciation, we analyze an amount of oil painting art works and summarize three critical factors to depict the figure, i.e. color, structure and texture. Based on this point, we separate and represent an artistic work into these three domains. Then, considering their intrinsic properties and following an art creation route, we propose the corresponding approaches to jointly model and transfer the features in these domains. First, a swatch-based color adjustment is proposed to recolor the tone of the input image based on semantic regions corresponding to the references. Second, the main structures of the input image is maintained by sparse reconstruction. Third, a coarse-to-fine texture synthesis is used to enhance the detail oil painting patterns. Extensive experimental results demonstrate that the proposed method achieves desirable results compared with state-of-the-art methods in not only transferring the styles from references but also keeping consistent contents with the given portrait.
Similar content being viewed by others
Explore related subjects
Discover the latest articles, news and stories from top researchers in related subjects.References
Ashikhmin M (2003) Fast texture transfer. IEEE Comput Graph Appl 23(4):38–43
Berger I, Shamir A, Mahler M, Carter E, Hodgins J (2013) Style and abstraction in portrait sketching. ACM Trans Graph 32(4):96–96
Bhujle H, Chaudhuri S (2014) Novel speed-up strategies for non-local means denoising with patch and edge patch based dictionaries. IEEE Trans Image Process 23(1):356–365
Bousseau A, Kaplan M, Thollot J, Sillion F (1983) Interactive watercolor rendering with temporal coherence and abstraction. In: International Symposium on Non-photorealistic Animation and Rendering, pp 141–149
Curtis CJ, Anderson SE, Seims JE, Fleischer KW, Salesin D (1997) Computer-generated watercolor. In: Conference on Computer Graphics and Interactive Techniques, pp 421–430
Dezeustre G (2014) Glaze app https://itunes.apple.com/us/app/glaze/id521573656
Donoho DL (2006) Compressed sensing. IEEE Trans Inf Theory 52(4):1289–1306
Efros A, Leung TK et al (1999) Texture synthesis by non-parametric sampling. In: Proceedings of the seventh IEEE International Conference on Computer Vision, vol 2. IEEE, pp 1033–1038
Efros AA, Freeman WT (2001) Image quilting for texture synthesis and transfer. In: Proceedings of the 28th Annual Conference on Computer Graphics and Interactive Techniques. ACM, pp 341–346
Flanagan P, Cavanagh P, Favreau OE (1990) Independent orientation-selective mechanisms for the cardinal directions of colour space. Vis Res 30(5):769–778
Frigo O, Sabater N, Delon J, Hellier P (2016) Split and match: example-based adaptive patch sampling for unsupervised style transfer. In: IEEE Conference on Computer Vision and Pattern Recognition, pp 553–561
Gatys LA, Ecker AS, Bethge M (2016) Image style transfer using convolutional neural networks. In: The IEEE Conference on Computer Vision and Pattern Recognition (CVPR)
Greenfield GR, House DH (2003) Image recoloring induced by palette color associations. Journal of Wscg 11:189–196
He K, Sun J (2014) Image completion approaches using the statistics of similar patches. IEEE Trans Pattern Anal Mach Intell 36(12):2423–2435
Hertzmann A (1998) Painterly rendering with curved brush strokes of multiple sizes. In: Proceedings of the 25th Annual Conference on Computer Graphics and Interactive Techniques. ACM, pp 453–460
Hertzmann A, Jacobs CE, Oliver N, Curless B, Salesin D (2001) Image analogies. In: Proceedings of the 28th Annual Conference on Computer Graphics and Interactive Techniques. ACM, pp 327–340
Huang TW, Chen HT (2009) Landmark-based sparse color representations for color transfer. In: IEEE 12Th International Conference on Computer Vision. IEEE, pp 199–204
Jia K, Wang X, Tang X (2013) Image transformation based on learning dictionaries across image spaces. IEEE Trans Pattern Anal Mach Intell 35(2):367–380
Kyprianidis JE, Collomosse J, Wang T, Isenberg T (2013) State of the art: a taxonomy of artistic stylization techniques for images and video. IEEE Trans Vis Comput Graph 19(5):866–885
Lee H, Seo S, Ryoo S, Yoon K (2010) Directional texture transfer. In: Proceedings of the 8th International Symposium on Non-Photorealistic Animation and Rendering. ACM, pp 43–48
Levin A, Lischinski D, Weiss Y (2004) Colorization using optimization. ACM Trans Graph 23(3):689–694
Levin A, Lischinski D, Weiss Y (2008) A closed-form solution to natural image matting. IEEE Trans Pattern Anal Mach Intell 30(2):228–242
Li C, Wand M (2016) Combining markov random fields and convolutional neural networks for image synthesis. In: IEEE Conference on Computer Vision and Pattern Recognition, pp 2479–2486
Liu Y, Zheng Y, Liang Y, Liu S, Rosenblum DS (2016) Urban water quality prediction based on multi-task multi-view learning. In: Proceedings of the International Joint Conference on Artificial Intelligence
Liu X, Ma L, Liu Y (2017) Global tone: using tone to draw in pen-and-ink illustration. Multimed Tool Appl 76(10):12853–12869
Lo KH, Wang YCF, Hua KL (2016) Example-based image textural style transfer. IEEE Multimed 23(4):60–66
Lu Y, Wei Y, Liu L, Zhong J, Sun L, Liu Y (2017) Towards unsupervised physical activity recognition using smartphone accelerometers. Multimed Tool Appl 76(8):10,701–10,719
Marr D (1982) Vision: a computational investigation into the human representation and processing of visual information. Henry Holt and Co., Inc., New York
Meitu app. https://itunes.apple.com/cn/app/id463422433
Paget R, Longstaff D (1995) Texture synthesis via a non-parametric markov random field. In: Proceedings of DICTA-95, Digital Image Computing: Techniques and Applications, vol 1, pp 547–552
Pérez P, Gangnet M, Blake A (2003) Poisson image editing. ACM Trans Graph 22(3):313–318
Porter T, Duff T (1984) Compositing digital images. ACM Siggraph Comput Graph 18(3):253–259
Protter M, Elad M (2009) Image sequence denoising via sparse and redundant representations. IEEE Trans Image Process 18(1):27–35
Reinhard E, Ashikhmin M, Gooch B, Shirley P (2001) Color transfer between images. IEEE Comput Graph Appl 21(5):34–41
Rosales R, Achan K, Frey B (2003) Unsupervised image translation. In: Proceedings of IEEE International Conference on Computer Vision, pp 472–472
Rother C, Kolmogorov V, Blake A (2004) Grabcut: interactive foreground extraction using iterated graph cuts. ACM Trans Graph 23(3):309–314
Ruderman DL, Cronin TW, Chiao CC (1998) Statistics of cone responses to natural images: implications for visual coding. J Opt Soc Am 15(8):2036–2045
Salisbury M, Anderson CR, Lischinski D, Salesin D (1996) Scale-dependent reproduction of pen-and-ink illustrations. Br J Sports Med 48(7):622
Saragih JM, Lucey S, Cohn JF (2009) Face alignment through subspace constrained mean-shifts. In: IEEE 12th International Conference on Computer Vision. IEEE, pp 1034–1041
Shih YC, Paris S, Barnes C, Freeman WT, Durand F (2014) Style transfer for headshot portraits. ACM Trans Graph 33(4):1–14
Sunkavalli K, Johnson MK, Matusik W, Pfister H (2010) Multi-scale image harmonization. ACM Trans Graph 29(4):125
Wang X, Tang X (2009) Face photo-sketch synthesis and recognition. IEEE Trans Pattern Anal Mach Intell 31(11):1955–1967
Wang S, Zhang L, Liang Y, Pan Q (2012) Semi-coupled dictionary learning with applications to image super-resolution and photo-sketch synthesis. In: IEEE Conference on Computer Vision and Pattern Recognition. IEEE, pp 2216–2223
Wang T, Collomosse J, Hunter A, Greig D (2013) Learnable stroke models for example-based portrait painting. In: British Machine Vision Conference, pp 36.1–36.11
Wang Q, Chen D, Li S, Wu Q, Zhang Q (2017) An adaptive cartoon-like stylization for color video in real time. Multimed Tool Appl 76(15):16767–16782
Welsh T, Ashikhmin M, Mueller K (2002) Transferring color to greyscale images. ACM Trans Graph 21(3):277–280
Winkenbach GA, Salesin D (1994) Computer-generated pen-and-ink illustration. In: Conference on Computer Graphics and Interactive Techniques, pp 91–100
Yang J, Wright J, Huang TS, Ma Y (2010) Image super-resolution via sparse representation. IEEE Trans Image Process 19(11):2861–2873
Yang L, Chen WB, Zhang C, Johnstone JK, Gao S, Warner G (2012) Profiling online auction sellers using image-editing styles. IEEE Multimed 19(1):29–29
Yang Y, Zhao H, You L, Tu R, Wu X, Jin X (2017) Semantic portrait color transfer with internet images. Multimed Tool Appl 76:523–541
Zhang W, Cao C, Chen S, Liu J (2013) Style transfer via image component analysis. IEEE Trans Multimed 15(7):1594–1601
Zhao M, Zhu SC (2010) Sisley the abstract painter. In: Proceedings of the 8th International Symposium on Non-Photorealistic Animation and Rendering. ACM, pp 99–107
Zhao M, Zhu SC (2011) Portrait painting using active templates. In: International Symposium on Non-Photorealistic Animation and Rendering 2009, Vancouver, Bc, Canada, August, pp 117–124
Zhao M, Zhu SC (2013) Abstract painting with interactive control of perceptual entropy. ACM Transactions on Applied Perception 10(1):5
Acknowledgments
This work was supported by the National Natural Science Foundation of China under Contract 61472011.
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Yang, S., Yang, S., Yang, W. et al. Automatic portrait oil painter: joint domain stylization for portrait images. Multimed Tools Appl 77, 16113–16130 (2018). https://doi.org/10.1007/s11042-017-5190-z
Received:
Revised:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11042-017-5190-z