Location via proxy:   [ UP ]  
[Report a bug]   [Manage cookies]                
skip to main content
research-article
Open access

Differentiable vector graphics rasterization for editing and learning

Published: 27 November 2020 Publication History

Abstract

We introduce a differentiable rasterizer that bridges the vector graphics and raster image domains, enabling powerful raster-based loss functions, optimization procedures, and machine learning techniques to edit and generate vector content. We observe that vector graphics rasterization is differentiable after pixel prefiltering. Our differentiable rasterizer offers two prefiltering options: an analytical prefiltering technique and a multisampling anti-aliasing technique. The analytical variant is faster but can suffer from artifacts such as conflation. The multisampling variant is still efficient, and can render high-quality images while computing unbiased gradients for each pixel with respect to curve parameters.
We demonstrate that our rasterizer enables new applications, including a vector graphics editor guided by image metrics, a painterly rendering algorithm that fits vector primitives to an image by minimizing a deep perceptual loss function, new vector graphics editing algorithms that exploit well-known image processing methods such as seam carving, and deep generative models that generate vector content from raster-only supervision under a VAE or GAN training objective.

Supplementary Material

ZIP File (a193-li.zip)
Supplemental material.
MP4 File (a193-li.mp4)
MP4 File (3414685.3417871.mp4)
Presentation video

References

[1]
Adobe Inc. 2006. PDF Reference, six edition. https://www.adobe.com/content/dam/acom/en/devnet/pdf/pdf_reference_archive/pdf_reference_1-7.pdf.
[2]
Alessandro Artusi, Francesco Banterle, Alejandro Moreo, and Fabio Carrara. 2019. Efficient Evaluation of Image Quality via Deep-Learning Approximation of Perceptual Metrics. IEEE Trans. Image Process. 29 (2019), 1843--1855.
[3]
Shai Avidan and Ariel Shamir. 2007. Seam Carving for Content-Aware Image Resizing. ACM Trans. Graph. (Proc. SIGGRAPH) 26, 3 (2007), 10.
[4]
Samaneh Azadi, Matthew Fisher, Vladimir G Kim, Zhaowen Wang, Eli Shechtman, and Trevor Darrell. 2018. Multi-content GAN for few-shot font style transfer. In Computer Vision and Pattern Recognition. 7564--7573.
[5]
Dejan Azinović, Tzu-Mao Li, Anton Kaplanyan, and Matthias Nießner. 2019. Inverse Path Tracing for Joint Material and Lighting Estimation. In Computer Vision and Pattern Recognition.
[6]
Francesco Banterle, Massimiliano Corsini, Paolo Cignoni, and Roberto Scopigno. 2012. A low-memory, straightforward and fast bilateral filter through subsampling in spatial domain. In Comput. Graph. Forum, Vol. 31. Wiley Online Library, 19--32.
[7]
Vineet Batra, Mark J. Kilgard, Harish Kumar, and Tristan Lorach. 2015. Accelerating Vector Graphics Rendering Using the Graphics Hardware Pipeline. ACM Trans. Graph. (Proc. SIGGRAPH) 34, 4 (2015), 146:1--146:15.
[8]
Bradley M Bell and James V Burke. 2008. Algorithmic differentiation of implicit functions and optimal values. In Advances in Automatic Differentiation. Springer.
[9]
Gilbert Louis Bernstein and Wilmot Li. 2015. Lillicon: Using Transient Widgets to Create Scale Variations of Icons. ACM Trans. Graph. (Proc. SIGGRAPH) 34, 4 (2015).
[10]
John C Bowers, Jonathan Leahey, and Rui Wang. 2011. A ray tracing approach to diffusion curves. 30, 4 (2011), 1345--1352.
[11]
Edwin Catmull. 1978. A Hidden-Surface Algorithm with Anti-Aliasing. Comput. Graph. (Proc. SIGGRAPH) 12, 3 (1978), 6--11.
[12]
Ravi Chugh, Brian Hempel, Mitchell Spradlin, and Jacob Albers. 2016. Programmatic and direct manipulation, together at last. 51, 6 (2016), 341--354.
[13]
Adrian Colomitchi. 2006. Approximating cubic Bezier curves by quadratic ones. http://caffeineowl.com/graphics/2d/vectorial/cubic2quad01.html#pseudoQuadDist. Accessed: 2020-05-21.
[14]
Robert L. Cook. 1986. Stochastic Sampling in Computer Graphics. ACM Trans. Graph. 5, 1 (1986), 51--72.
[15]
Joe Cridge. 2015. Approximating Arcs Using Cubic Bézier Curves. https://www.joecridge.me/content/pdf/bezier-arcs.pdf
[16]
Martin de La Gorce, David J Fleet, and Nikos Paragios. 2011. Model-based 3D hand pose estimation from monocular video. IEEE Trans. Pattern Anal. Mach. Intell. 33, 9 (2011), 1793--1805.
[17]
Mark A. Z. Dippé and Erling Henry Wold. 1985. Antialiasing through Stochastic Sampling. Comput. Graph. (Proc. SIGGRAPH) 19, 3 (1985), 69--78.
[18]
Tom Duff. 1989. Polygon scan conversion by exact convolution. In International Conference On Raster Imaging and Digital Typography. 154--168.
[19]
David Eberly. 2011. Distance from a point to an ellipse, an ellipsoid, or a hyperellipsoid. Geometric Tools, LLC (2011).
[20]
Kevin Ellis, Daniel Ritchie, Armando Solar-Lezama, and Joshua B. Tenenbaum. 2018. Learning to Infer Graphics Programs from Hand-drawn Images. In Advances in Neural Information Processing Systems. Curran Associates Inc., 6062--6071.
[21]
A. E. Fabris and A. R. Forrest. 1997. Antialiasing of Curves by Discrete Pre-Filtering. In SIGGRAPH. ACM, 317--326.
[22]
Rida T Farouki and C Andrew Neff. 1990. Algebraic properties of plane offset curves. Computer Aided Geometric Design 7, 1--4 (1990), 101--127.
[23]
Harley Flanders. 1973. Differentiation under the integral sign. The American Mathematical Monthly 80, 6 (1973), 615--627.
[24]
Francisco Ganacim, Rodolfo S. Lima, Luiz Henrique de Figueiredo, and Diego Nehab. 2014. Massively-parallel Vector Graphics. ACM Trans. Graph. (Proc. SIGGRAPH Asia) 33, 6 (2014), 229:1--229:14.
[25]
Yaroslav Ganin, Tejas Kulkarni, Igor Babuschkin, S. M. Ali Eslami, and Oriol Vinyals. 2018. Synthesizing Programs for Images using Reinforced Adversarial Learning. In International Conference on Machine Learning. 1666--1675.
[26]
Leon A Gatys, Alexander S Ecker, and Matthias Bethge. 2016. Image style transfer using convolutional neural networks. In Computer Vision and Pattern Recognition. IEEE, 2414--2423.
[27]
Michael Gleicher. 1992. Briar: A Constraint-Based Drawing Program. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM.
[28]
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. 2014. Generative adversarial nets. In Advances in Neural Information Processing Systems. 2672--2680.
[29]
Andreas Griewank and Andrea Walther. 2008. Evaluating Derivatives: Principles and Techniques of Algorithmic Differentiation (second ed.). Society for Industrial and Applied Mathematics, Philadelphia, PA, USA.
[30]
Ishaan Gulrajani, Faruk Ahmed, Martin Arjovsky, Vincent Dumoulin, and Aaron C Courville. 2017. Improved training of wasserstein gans. In Advances in Neural Information Processing Systems. 5767--5777.
[31]
Satish Gupta and Robert F. Sproull. 1981. Filtering Edges for Gray-Scale Displays. Comput. Graph. (Proc. SIGGRAPH) 15, 3 (1981), 1--5.
[32]
David Ha and Douglas Eck. 2018. A Neural Representation of Sketch Drawings. In International Conference on Learning Representations.
[33]
Aaron Hertzmann. 1998. Painterly rendering with curved brush strokes of multiple sizes. In SIGGRAPH. ACM, 453--460.
[34]
Aaron Hertzmann. 2003. A survey of stroke-based rendering. IEEE Comput. Graph. Appl. 4 (2003), 70--81.
[35]
Zhewei Huang, Wen Heng, and Shuchang Zhou. 2019. Learning to paint with modelbased deep reinforcement learning. In International Conference on Computer Vision. 8709--8718.
[36]
Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A Efros. 2017. Image-to-image translation with conditional adversarial networks. In Computer Vision and Pattern Recognition. 1125--1134.
[37]
Hiroharu Kato, Yoshitaka Ushiku, and Tatsuya Harada. 2018. Neural 3D Mesh Renderer. In Computer Vision and Pattern Recognition. IEEE, 3907--3916.
[38]
Mark J. Kilgard. 2020. Polar Stroking: New Theory and Methods for Stroking Paths. ACM Trans. Graph. (Proc. SIGGRAPH) 39, 4 (2020).
[39]
Mark J. Kilgard and Jeff Bolz. 2012. GPU-accelerated Path Rendering. ACM Trans. Graph. (Proc. SIGGRAPH) 31, 6 (2012), 172:1--172:10.
[40]
Diederick P Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In International Conference on Learning Representations.
[41]
Diederik P Kingma and Max Welling. 2014. Auto-encoding variational Bayes. In International Conference on Learning Representations.
[42]
Günter Klambauer, Thomas Unterthiner, Andreas Mayr, and Sepp Hochreiter. 2017. Self-normalizing neural networks. In Advances in Neural Information Processing Systems. 971--980.
[43]
Yoshiyuki Kokojima, Kaoru Sugita, Takahiro Saito, and Takashi Takemoto. 2006. Resolution Independent Rendering of Deformable Vector Objects Using Graphics Hardware. In ACM SIGGRAPH 2006 Sketches. ACM, 118.
[44]
Christian Lauterbach, Michael Garland, Shubhabrata Sengupta, David Luebke, and Dinesh Manocha. 2009. Fast BVH construction on GPUs. Comput. Graph. Forum (Proc. Eurographics) 28, 2 (2009), 375--384.
[45]
Gregory Lecot and Bruno Levy. 2006. Ardeco: Automatic Region DEtection and COnversion. (2006).
[46]
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. 1998. Gradient-based learning applied to document recognition. Proc. IEEE 86, 11 (1998), 2278--2324.
[47]
Raph Levien. 2020. Fast 2D rendering on GPU. https://raphlinus.github.io/rust/graphics/gpu/2020/06/13/fast-2d-rendering.html Accessed: 2020-08-24.
[48]
Rui Li, Qiming Hou, and Kun Zhou. 2016. Efficient GPU Path Rendering Using Scanline Rasterization. ACM Trans. Graph. (Proc. SIGGRAPH Asia) 35, 6 (2016), 228:1--228:12.
[49]
Tzu-Mao Li. 2019. Differentiable Visual Computing. Ph.D. Dissertation. Massachusetts Institute of Technology. Advisor(s) Durand, Frédo.
[50]
Tzu-Mao Li, Miika Aittala, Frédo Durand, and Jaakko Lehtinen. 2018. Differentiable Monte Carlo Ray Tracing through Edge Sampling. ACM Trans. Graph. (Proc. SIGGRAPH Asia) 37, 6 (2018), 222:1--222:11.
[51]
Hsueh-Ti Derek Liu, Michael Tao, and Alec Jacobson. 2018. Paparazzi: Surface Editing by Way of Multi-view Image Processing. ACM Trans. Graph. (Proc. SIGGRAPH Asia) 37, 6 (2018), 221:1--221:11.
[52]
Shichen Liu, Tianye Li, Weikai Chen, and Hao Li. 2019. Soft Rasterizer: A Differentiable Renderer for Image-based 3D Reasoning. International Conference on Computer Vision (2019).
[53]
Charles Loop and Jim Blinn. 2005. Resolution independent curve rendering using programmable graphics hardware. ACM Trans. Graph. (Proc. SIGGRAPH) 24, 3 (2005), 1000--1009.
[54]
Matthew M. Loper and Michael J. Black. 2014. OpenDR: An Approximate Differentiable Renderer. In European Conference on Computer Vision, Vol. 8695. ACM, 154--169.
[55]
Raphael Gontijo Lopes, David Ha, Douglas Eck, and Jonathon Shlens. 2019. A Learned Representation for Scalable Vector Graphics. In International Conference on Computer Vision.
[56]
Josiah Manson and Scott Schaefer. 2013. Analytic rasterization of curves with polynomial filters. Comput. Graph. Forum (Proc. Eurographics) 32, 24 (2013), 499--507.
[57]
Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida. 2018. Spectral Normalization for Generative Adversarial Networks. In International Conference on Learning Representations.
[58]
Reiichiro Nakano. 2019. Neural Painters: A learned differentiable constraint for generating brushstroke paintings. arXiv preprint arXiv:1904.08410 (2019).
[59]
Diego Nehab. 2020. Converting Stroked Primitives to Filled Primitives. ACM Trans. Graph. (Proc. SIGGRAPH) 39, 4 (2020).
[60]
Diego Nehab and Hugues Hoppe. 2008. Random-access Rendering of General Vector Graphics. ACM Trans. Graph. (Proc. SIGGRAPH Asia) 27, 5 (2008), 135:1--135:10.
[61]
Alexandrina Orzan, Adrien Bousseau, Holger Winnemöller, Pascal Barla, Joëlle Thollot, and David Salesin. 2008. Diffusion Curves: A Vector Representation for Smooth-shaded Images. ACM Trans. Graph. (Proc. SIGGRAPH) 27, 3 (2008), 92:1--92:8.
[62]
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. 2019. PyTorch: An Imperative Style, High-Performance Deep Learning Library. In Advances in Neural Information Processing Systems. 8024--8035.
[63]
Thomas Porter and Tom Duff. 1984. Compositing digital images. (1984), 253--259.
[64]
William H. Press, Saul A. Teukolsky, William T. Vetterling, and Brian P. Flannery. 2007. Numerical Recipes 3rd Edition: The Art of Scientific Computing (3 ed.). Cambridge University Press, USA.
[65]
Osborne Reynolds, Arthur William Brightmore, and William Henry Moorby. 1903. The sub-mechanics of the universe. Vol. 3. University Press.
[66]
Oswaldo Rio Branco de Oliveira. 2012. The Implicit and the Inverse Function theorems: easy proofs. arXiv preprint arXiv:1212.2066 (2012).
[67]
Thomas W Sederberg and Geng-Zhe Chang. 1994. Isolator polynomials. In Algebraic Geometry and Its Applications. Springer, 507--512.
[68]
Peter Selinger. 2003. Potrace: a polygon-based tracing algorithm. http://potrace.sourceforge.net/potrace.pdf
[69]
Jian Sun, Lin Liang, Fang Wen, and Heung-Yeung Shum. 2007. Image Vectorization Using Optimized Gradient Meshes. ACM Trans. Graph. (Proc. SIGGRAPH) 26, 3 (2007).
[70]
Ivan E. Sutherland. 1964. Sketch Pad: a Man-machine Graphical Communication System. In Proceedings of the SHARE Design Automation Workshop. 6.329--6.346.
[71]
K. Turkowski. 1982. Anti-Aliasing through the Use of Coordinate Transformations. ACM Trans. Graph. 1, 3 (1982), 215--234.
[72]
Zhou Wang, Alan C Bovik, Hamid R Sheikh, Eero P Simoncelli, et al. 2004. Image quality assessment: from error visibility to structural similarity. IEEE Trans. Image Process. 13, 4 (2004), 600--612.
[73]
Tian Xia, Binbin Liao, and Yizhou Yu. 2009. Patch-Based Image Vectorization with Automatic Curvilinear Feature Alignment. ACM Trans. Graph. (Proc. SIGGRAPH Asia) (2009).
[74]
Guofu Xie, Xin Sun, Xin Tong, and Derek Nowrouzezahrai. 2014. Hierarchical Diffusion Curves for Accurate Automatic Image Vectorization. ACM Trans. Graph. (Proc. SIGGRAPH Asia) 33, 6 (2014), 230:1--230:11.
[75]
Ming Yang, Hongyang Chao, Chi Zhang, Jun Guo, Lu Yuan, and Jian Sun. 2015. Effective clipart image vectorization through direct optimization of bezigons. IEEE Trans. Vis. Comput. Graph. 22, 2 (2015), 1063--1075.
[76]
Gao Yue, Guo Yuan, Lian Zhouhui, Tang Yingmin, and Xiao Jianguo. 2019. Artistic Glyph Image Synthesis via One-Stage Few-Shot Learning. ACM Trans. Graph. (Proc. SIGGRAPH Asia) 38, 6 (2019).
[77]
Cheng Zhang, Lifan Wu, Changxi Zheng, Ioannis Gkioulekas, Ravi Ramamoorthi, and Shuang Zhao. 2019. A differential theory of radiative transfer. ACM Trans. Graph. (Proc. SIGGRAPH Asia) 38, 6 (2019), 227.
[78]
Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. 2018. The Unreasonable Effectiveness of Deep Features as a Perceptual Metric. In Computer Vision and Pattern Recognition. 586--595.
[79]
Shuang Zhao, Frédo Durand, and Changxi Zheng. 2018. Inverse diffusion curves using shape optimization. IEEE Trans. Vis. Comput. Graph. 24, 7 (2018), 2153--2166.
[80]
Ningyuan Zheng, Yifan Jiang, and Dingjiang Huang. 2019. StrokeNet: A Neural Painting Environment. In International Conference on Learning Representations.
[81]
Chenyang Zhu, Kai Xu, Siddhartha Chaudhuri, Renjiao Yi, and Hao Zhang. 2018. SCORES: Shape Composition with Recursive Substructure Priors. ACM Trans. Graph. (Proc. SIGGRAPH Asia) 37, 6 (2018), 14.

Cited By

View all
  • (2024)FontCLIP: A Semantic Typography Visual‐Language Model for Multilingual Font ApplicationsComputer Graphics Forum10.1111/cgf.1504343:2Online publication date: 30-Apr-2024
  • (2024)A Collaborative, Interactive and Context-Aware Drawing Agent for Co-Creative DesignIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2023.329385330:8(5525-5537)Online publication date: Aug-2024
  • (2024)A Survey of Smooth Vector Graphics: Recent Advances in Repr esentation, Creation, Rasterization, and Image VectorizationIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2022.322057530:3(1652-1671)Online publication date: Mar-2024
  • Show More Cited By

Index Terms

  1. Differentiable vector graphics rasterization for editing and learning

        Recommendations

        Comments

        Information & Contributors

        Information

        Published In

        cover image ACM Transactions on Graphics
        ACM Transactions on Graphics  Volume 39, Issue 6
        December 2020
        1605 pages
        ISSN:0730-0301
        EISSN:1557-7368
        DOI:10.1145/3414685
        Issue’s Table of Contents
        Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for third-party components of this work must be honored. For all other uses, contact the Owner/Author.

        Publisher

        Association for Computing Machinery

        New York, NY, United States

        Publication History

        Published: 27 November 2020
        Published in TOG Volume 39, Issue 6

        Check for updates

        Author Tags

        1. differentiable rendering
        2. image vectorization
        3. vector graphics

        Qualifiers

        • Research-article

        Funding Sources

        Contributors

        Other Metrics

        Bibliometrics & Citations

        Bibliometrics

        Article Metrics

        • Downloads (Last 12 months)1,391
        • Downloads (Last 6 weeks)157
        Reflects downloads up to 10 Oct 2024

        Other Metrics

        Citations

        Cited By

        View all
        • (2024)FontCLIP: A Semantic Typography Visual‐Language Model for Multilingual Font ApplicationsComputer Graphics Forum10.1111/cgf.1504343:2Online publication date: 30-Apr-2024
        • (2024)A Collaborative, Interactive and Context-Aware Drawing Agent for Co-Creative DesignIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2023.329385330:8(5525-5537)Online publication date: Aug-2024
        • (2024)A Survey of Smooth Vector Graphics: Recent Advances in Repr esentation, Creation, Rasterization, and Image VectorizationIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2022.322057530:3(1652-1671)Online publication date: Mar-2024
        • (2024)MARVEL: Raster Gray-level Manga Vectorization via Primitive-wise Deep Reinforcement LearningIEEE Transactions on Circuits and Systems for Video Technology10.1109/TCSVT.2023.3309786(1-1)Online publication date: 2024
        • (2024)Segmentation-Based Parametric Painting2024 IEEE International Conference on Multimedia and Expo Workshops (ICMEW)10.1109/ICMEW63481.2024.10645421(1-6)Online publication date: 15-Jul-2024
        • (2024)Text-Based Vector Sketch Editing with Image Editing Diffusion Prior2024 IEEE International Conference on Multimedia and Expo (ICME)10.1109/ICME57554.2024.10688086(1-6)Online publication date: 15-Jul-2024
        • (2024)SAMVG: A Multi-Stage Image Vectorization Model with the Segment-Anything ModelICASSP 2024 - 2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)10.1109/ICASSP48485.2024.10447396(4350-4354)Online publication date: 14-Apr-2024
        • (2024)Wired Perspectives: Multi-View Wire Art Embraces Generative AI2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)10.1109/CVPR52733.2024.00588(6149-6158)Online publication date: 16-Jun-2024
        • (2024)NIVeL: Neural Implicit Vector Layers for Text-to-Vector Generation2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)10.1109/CVPR52733.2024.00439(4589-4597)Online publication date: 16-Jun-2024
        • (2024)SVGDreamer: Text Guided SVG Generation with Diffusion Model2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)10.1109/CVPR52733.2024.00435(4546-4555)Online publication date: 16-Jun-2024
        • Show More Cited By

        View Options

        View options

        PDF

        View or Download as a PDF file.

        PDF

        eReader

        View online with eReader.

        eReader

        Get Access

        Login options

        Full Access

        Media

        Figures

        Other

        Tables

        Share

        Share

        Share this Publication link

        Share on social media