Cited By
View all- Wang QMao ZGao JZhang Y(2024)Document-level Relation Extraction with Progressive Self-distillationACM Transactions on Information Systems10.1145/365616842:6(1-34)Online publication date: 25-Jun-2024
Patternshop: Editing Point Patterns by Image Manipulation
Authors: 
Xingchang Huang, Tobias Ritschel, Hans-Peter Seidel, Pooran Memari, Gurprit SinghAuthors Info & Claims
Xingchang Huang, Tobias Ritschel, Hans-Peter Seidel, Pooran Memari, Gurprit SinghAuthors Info & ClaimsArticle No.: 53, Pages 1 - 14
Abstract
Point patterns are characterized by their density and correlation. While spatial variation of density is well-understood, analysis and synthesis of spatially-varying correlation is an open challenge. No tools are available to intuitively edit such point patterns, primarily due to the lack of a compact representation for spatially varying correlation. We propose a low-dimensional perceptual embedding for point correlations. This embedding can map point patterns to common three-channel raster images, enabling manipulation with off-the-shelf image editing software. To synthesize back point patterns, we propose a novel edge-aware objective that carefully handles sharp variations in density and correlation. The resulting framework allows intuitive and backward-compatible manipulation of point patterns, such as recoloring, relighting to even texture synthesis that have not been available to 2D point pattern design before. Effectiveness of our approach is tested in several user experiments. Code is available at https://github.com/xchhuang/patternshop.
Supplementary Material
References
[1]
Rameen Abdal, Yipeng Qin, and Peter Wonka. 2019. Image2stylegan: How to embed images into the stylegan latent space?. In ICCV. 4432--4441.
[2]
Abdalla GM Ahmed, Jianwei Guo, Dong-Ming Yan, Jean-Yves Franceschia, Xiaopeng Zhang, and Oliver Deussen. 2017. A simple push-pull algorithm for blue-noise sampling. IEEE Trans. Vis and Comp. Graph. 23, 12 (2017).
[3]
Abdalla GM Ahmed, Hui Huang, and Oliver Deussen. 2015. AA patterns for point sets with controlled spectral properties. ACM Trans. Graph. 34, 6 (2015).
[4]
Abdalla GM Ahmed, Hélène Perrier, David Coeurjolly, Victor Ostromoukhov, Jianwei Guo, Dong-Ming Yan, Hui Huang, and Oliver Deussen. 2016. Low-discrepancy blue noise sampling. ACM Trans. Graph. 35, 6 (2016).
[5]
Xiaobo An, Xin Tong, Jonathan D. Denning, and Fabio Pellacini. 2011. AppWarp: Retargeting Measured Materials by Appearance-Space Warping. ACM Trans. Graph. 30, 6 (2011), 1--10.
[6]
Michael Balzer, Thomas Schlömer, and Oliver Deussen. 2009. Capacity-constrained point distributions: a variant of Lloyd's method. ACM Trans. Graph. 28, 3 (2009).
[7]
Marcelo Bertalmio, Guillermo Sapiro, Vincent Caselles, and Coloma Ballester. 2000. Image Inpainting. In Proc. SIGGRAPH. 417--424.
[8]
György Büttner and Barbara Kosztra. 2017. CLC2018 Technical Guidelines. Technical Report. European Environment Agency.
[9]
Jiating Chen, Xiaoyin Ge, Li-Yi Wei, Bin Wang, Yusu Wang, Huamin Wang, Yun Fei, Kang-Lai Qian, Jun-Hai Yong, and Wenping Wang. 2013. Bilateral blue noise sampling. ACM Trans. Graph. 32, 6 (2013), 1--11.
[10]
Jiawen Chen, Sylvain Paris, and Frédo Durand. 2007. Real-time edge-aware image processing with the bilateral grid. ACM Trans. Graph. 26, 3 (2007), 103--113.
[11]
Yunjey Choi, Youngjung Uh, Jaejun Yoo, and Jung-Woo Ha. 2020. Stargan v2: Diverse image synthesis for multiple domains. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 8188--8197.
[12]
ClipDrop. 2023. Relight API. https://clipdrop.co/relight
[13]
Robert L Cook. 1986. Stochastic sampling in computer graphics. ACM Trans. Graph. 5, 1 (1986).
[14]
Fernando De Goes, Katherine Breeden, Victor Ostromoukhov, and Mathieu Desbrun. 2012. Blue noise through optimal transport. ACM Trans. Graph. 31, 6 (2012).
[15]
Oliver Deussen, Stefan Hiller, Cornelius Van Overveld, and Thomas Strothotte. 2000. Floating points: A method for computing stipple drawings. In Comp. Graph. Forum, Vol. 19. 41--50.
[16]
Oliver Deussen and Tobias Isenberg. 2013. Halftoning and Stippling. In Image and Video-Based Artistic Stylisation, Paul Rosin and John Collomosse (Eds.). Springer London, 45--61.
[17]
Francesco Di Renzo, Claudio Calabrese, and Fabio Pellacini. 2014. AppIm: Linear Spaces for Image-Based Appearance Editing. ACM Trans. Graph. 33, 6 (2014).
[18]
Mark A. Z. Dippé and Erling Henry Wold. 1985. Antialiasing through Stochastic Sampling. In SIGGRAPH. 69--78.
[19]
John Duncan and Glyn W Humphreys. 1989. Visual search and stimulus similarity. Psychological review 96, 3 (1989), 433.
[20]
Frédo Durand. 2011. A frequency analysis of Monte-Carlo and other numerical integration schemes. Technical Report TR-2011-052. MIT CSAIL.
[21]
Alexei A. Efros and William T. Freeman. 2001. Image Quilting for Texture Synthesis and Transfer. In SIGGRAPH. 341--346.
[22]
Arnaud Emilien, Ulysse Vimont, Marie-Paule Cani, Pierre Poulin, and Bedrich Benes. 2015. Worldbrush: Interactive example-based synthesis of procedural virtual worlds. ACM Trans. Graph. 34, 4 (2015), 1--11.
[23]
Mark D Fairchild. 2013. Color appearance models. John Wiley & Sons.
[24]
Raanan Fattal. 2011. Blue-noise point sampling using kernel density model. ACM Trans. Graph. 30, 4 (2011).
[25]
Leon A. Gatys, Alexander S. Ecker, and Matthias Bethge. 2016. Image Style Transfer Using Convolutional Neural Networks. In CVPR. 2414--2423.
[26]
Paul Guerrero, Gilbert Bernstein, Wilmot Li, and Niloy J. Mitra. 2016. PATEX: Exploring Pattern Variations. ACM Trans. Graph. 35, 4 (2016).
[27]
Daniel Heck, Thomas Schlömer, and Oliver Deussen. 2013. Blue noise sampling with controlled aliasing. ACM Trans. Graph. (Proc. SIGGRAPH) 32, 3 (2013).
[28]
Philipp Henzler, Niloy J Mitra, and Tobias Ritschel. 2019. Learning a Neural 3D Texture Space from 2D Exemplars. In CVPR.
[29]
Pedro Hermosilla, Tobias Ritschel, Pere-Pau Vázquez, Àlvar Vinacua, and Timo Ropinski. 2018. Monte Carlo Convolution for Learning on Non-uniformly Sampled Point Clouds. ACM Trans. Graph (Proc. SIGGRAPH Asia) 37, 5 (2018).
[30]
Aaron Hertzmann, Charles E Jacobs, Nuria Oliver, Brian CNOurless, and David H Salesin. 2001. Image analogies. In SIGGRAPH. 327--340.
[31]
Chen-Yuan Hsu, Li-Yi Wei, Lihua You, and Jian Jun Zhang. 2020. Autocomplete element fields. In Proc. CHI. 1--13.
[32]
Xingchang Huang, Pooran Memari, Hans-Peter Seidel, and Gurprit Singh. 2022. Point-Pattern Synthesis using Gabor and Random Filters. In Comp. Graph. Forum, Vol. 41. 169--179.
[33]
Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A Efros. 2017. Image-to-Image Translation with Conditional Adversarial Networks. CVPR (2017).
[34]
Adrian Jarabo, Belen Masia, Adrien Bousseau, Fabio Pellacini, and Diego Gutierrez. 2014. How Do People Edit Light Fields? ACM Trans. Graph. 33, 4 (2014).
[35]
Henrik Wann Jensen. 2001. Realistic image synthesis using photon mapping. AK Peters/crc Press.
[36]
Bhavya Kailkhura, Jayaraman J Thiagarajan, Peer-Timo Bremer, and Pramod K Varshney. 2016. Stair blue noise sampling. ACM Trans. Graph. 35, 6 (2016).
[37]
Konrad Kapp, James Gain, Eric Guérin, Eric Galin, and Adrien Peytavie. 2020. Data-driven authoring of large-scale ecosystems. ACM Trans. Graph. 39, 6 (2020), 1--14.
[38]
Sung Ye Kim, Ross Maciejewski, Tobias Isenberg, William M. Andrews, Wei Chen, Mario Costa Sousa, and David S. Ebert. 2009. Stippling by Example. In Proc. NPAR. 41--50.
[39]
Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014).
[40]
Hans Knutsson and C-F Westin. 1993. Normalized and differential convolution. In CVPR. 515--523.
[41]
Johannes Kopf, Daniel Cohen-Or, Oliver Deussen, and Dani Lischinski. 2006. Recursive Wang tiles for real-time blue noise. ACM Trans. Graph. (Proc. SIGGRAPH) 25, 3 (2006).
[42]
Ares Lagae and Philip Dutre. 2008. A Comparison of Methods for Generating Poisson Disk Distributions. Comp. Graph. Forum 27, 1 (2008).
[43]
Daniel L. Lau, Gonzalo R. Arce, and Neal C. Gallagher. 1999. Digital halftoning by means of green-noise masks. J OSA 16, 7 (1999), 1575--1586.
[44]
Cheng-Han Lee, Ziwei Liu, Lingyun Wu, and Ping Luo. 2020. MaskGAN: Towards Diverse and Interactive Facial Image Manipulation. In CVPR.
[45]
Thomas Leimkühler, Gurprit Singh, Karol Myszkowski, Hans-Peter Seidel, and Tobias Ritschel. 2019. Deep Point Correlation Design. ACM Trans. Graph. 38, 6 (2019).
[46]
Norbert Lindow, Daniel Baum, and Hans-Christian Hege. 2012. Perceptually linear parameter variations. In Comp. Graph. Forum, Vol. 31. 535--544.
[47]
Yang Liu, Wenping Wang, Bruno Lévy, Feng Sun, Dong-Ming Yan, Lin Lu, and Chenglei Yang. 2009. On centroidal Voronoi tessellation - energy smoothness and fast computation. ACM Trans. Graph. 28, 4 (2009).
[48]
Stuart Lloyd. 1982. Least squares quantization in PCM. IEEE Trans Inform. Theory 28, 2 (1982).
[49]
Chongyang Ma, Li-Yi Wei, and Xin Tong. 2011. Discrete Element Textures. ACM Trans. Graph. 30, 4 (2011).
[50]
Domingo Martín, Germán Arroyo, Alejandro Rodríguez, and Tobias Isenberg. 2017. A survey of digital stippling. Computers & Graphics 67 (2017), 24--44.
[51]
Chuong H Nguyen, Tobias Ritschel, and Hans-Peter Seidel. 2015. Data-driven color manifolds. ACM Trans. Graph. 34, 2 (2015), 1--9.
[52]
Victor Ostromoukhov, Charles Donohue, and Pierre-Marc Jodoin. 2004. Fast hierarchical importance sampling with blue noise properties. ACM Trans. Graph. 23, 3 (2004).
[53]
A Cengiz Öztireli and Markus Gross. 2012. Analysis and synthesis of point distributions based on pair correlation. ACM Trans. Graph. 31, 6 (2012).
[54]
Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. 2017. Automatic differentiation in pytorch. (2017).
[55]
Fabio Pellacini. 2010. envyLight: An Interface for Editing Natural Illumination. ACM Trans. Graph. (SIGGRAPH) (2010).
[56]
Fabio Pellacini, James A Ferwerda, and Donald P Greenberg. 2000. Toward a psychophysically-based light reflection model for image synthesis. In SIGGRAPH. 55--64.
[57]
Fabio Pellacini and Jason Lawrence. 2007. AppWand: Editing Measured Materials Using Appearance-Driven Optimization. ACM Trans. Graph. 26, 3 (2007), 54--64.
[58]
Georg Petschnigg, Richard Szeliski, Maneesh Agrawala, Michael Cohen, Hugues Hoppe, and Kentaro Toyama. 2004. Digital photography with flash and no-flash image pairs. ACM Trans. Graph. 23, 3 (2004), 664--672.
[59]
Michal Piovarči, David I.W. Levin, Danny Kaufman, and Piotr Didyk. 2018. Perception-Aware Modeling and Fabrication of Digital Drawing Tools. ACM Trans. Graph. (SIGGRAPH) 37, 4 (2018).
[60]
Louis CW Pols, LJ Th Van der Kamp, and Reinier Plomp. 1969. Perceptual and physical space of vowel sounds. J ASA 46, 2B (1969), 458--467.
[61]
Javier Portilla and Eero P Simoncelli. 2000. A parametric texture model based on joint statistics of complex wavelet coefficients. Int. J Computer Vision 40 (2000), 49--70.
[62]
Charles R Qi, Hao Su, Kaichun Mo, and Leonidas J Guibas. 2017. Pointnet: Deep learning on point sets for 3D classification and segmentation. CVPR (2017).
[63]
Hongxing Qin, Yi Chen, Jinlong He, and Baoquan Chen. 2017. Wasserstein Blue Noise Sampling. ACM Trans. Graph. 36, 5 (2017).
[64]
Pradyumna Reddy, Paul Guerrero, Matt Fisher, Wilmot Li, and Niloy J. Mitra. 2020. Discovering Pattern Structure Using Differentiable Compositing. ACM Trans. Graph. 39, 6 (2020).
[65]
Bernhard Reinert, Tobias Ritschel, and Hans-Peter Seidel. 2013. Interactive By-Example Design of Artistic Packing Layouts. ACM Trans. Graph. 32, 6 (2013).
[66]
Paul Rosin and John Collomosse. 2012. Image and Video-Based Artistic Stylisation. Springer Publishing Company, Incorporated.
[67]
Tamar Rott Shaham, Tali Dekel, and Tomer Michaeli. 2019. SinGAN: Learning a Generative Model from a Single Natural Image. In ICCV.
[68]
Riccardo Roveri, A. Cengiz Öztireli, and Markus Gross. 2017. General Point Sampling with Adaptive Density and Correlations. Comp. Graph. Forum 36, 2 (2017), 107--117.
[69]
Corentin Salaün, Iliyan Georgiev, Hans-Peter Seidel, and Gurprit Singh. 2022. Scalable Multi-Class Sampling via Filtered Sliced Optimal Transport. ACM Trans. Graph. (SIGGRAPH) 41, 6 (2022).
[70]
Christian Schmaltz, Pascal Gwosdek, Andres Bruhn, and Joachim Weickert. 2010. Electrostatic Halftoning. Comp. Graph. Forum (2010).
[71]
Christoph Schulz, Kin Chung Kwan, Michael Becher, Daniel Baumgartner, Guido Reina, Oliver Deussen, and Daniel Weiskopf. 2021. Multi-Class Inverted Stippling. ACM Trans. Graph. 40, 6 (2021).
[72]
Adrian Secord. 2002. Weighted Voronoi stippling. In Proc. NPAR.
[73]
Omry Sendik and Daniel Cohen-Or. 2017. Deep Correlations for Texture Synthesis. ACM Trans. Graph. 36, 5 (2017).
[74]
Eero P Simoncelli and Bruno A Olshausen. 2001. Natural image statistics and neural representation. Ann. Review Neuroscience 24, 1 (2001).
[75]
Karen Simonyan and Andrew Zisserman. 2015. Very Deep Convolutional Networks for Large-Scale Image Recognition. In Proc. ICLR, Yoshua Bengio and Yann LeCun (Eds.).
[76]
Gurprit Singh, Cengiz Oztireli, Abdalla G.M. Ahmed, David Coeurjolly, Kartic Subr, Oliver Deussen, Victor Ostromoukhov, Ravi Ramamoorthi, and Wojciech Jarosz. 2019. Analysis of Sample Correlations for Monte Carlo Rendering. Comp. Graph Form. (Proc. EGSR) 38, 2 (2019).
[77]
Šárka Sochorová and Ondřej Jamriška. 2021. Practical pigment mixing for digital painting. ACM Trans. Graph. 40, 6 (2021), 1--11.
[78]
Marc Spicker, Franz Hahn, Thomas Lindemeier, Dietmar Saupe, and Oliver Deussen. 2017. Quantifying Visual Abstraction Quality for Stipple Drawings. In Proc. NPAR.
[79]
Steve Strassmann. 1986. Hairy brushes. SIGGRAPH 20, 4 (1986), 225--232.
[80]
Tiancheng Sun, Jonathan T. Barron, Yun-Ta Tsai, Zexiang Xu, Xueming Yu, Graham Fyffe, Christoph Rhemann, Jay Busch, Paul Debevec, and Ravi Ramamoorthi. 2019. Single Image Portrait Relighting. ACM Trans. Graph. 38, 4 (2019).
[81]
Peihan Tu, Dani Lischinski, and Hui Huang. 2019. Point Pattern Synthesis via Irregular Convolution. Comp. Graph. Forum 38 (2019).
[82]
Robert Ulichney. 1987. Digital Halftoning. MIT Press.
[83]
Florent Wachtel, Adrien Pilleboue, David Coeurjolly, Katherine Breeden, Gurprit Singh, Gaël Cathelin, Fernando De Goes, Mathieu Desbrun, and Victor Ostromoukhov. 2014. Fast tile-based adaptive sampling with user-specified Fourier spectra. ACM Trans. Graph. 33, 4 (2014).
[84]
Li-Yi Wei. 2010. Multi-class blue noise sampling. ACM Trans. Graph. 29, 4 (2010).
[85]
Li-Yi Wei and Rui Wang. 2011. Differential domain analysis for non-uniform sampling. ACM Trans. Graph. 30, 4 (2011).
[86]
Josh Wills, Sameer Agarwal, David Kriegman, and Serge Belongie. 2009. Toward a perceptual space for gloss. ACM Trans. Graph. 28, 4 (2009), 1--15.
[87]
Yifan Xu, Tianqi Fan, Yi Yuan, and Gurprit Singh. 2020. Ladybird: Quasi-Monte Carlo Sampling for Deep Implicit Field Based 3D Reconstruction with Symmetry. ECCV (2020), 248--263.
[88]
Dong-Ming Yan, Jian-Wei Guo, Bin Wang, Xiao-Peng Zhang, and Peter Wonka. 2015. A survey of blue-noise sampling and its applications. Journal of Comp. Sci. and Tech. 30, 3 (2015).
[89]
John I Yellott. 1983. Spectral consequences of photoreceptor sampling in the rhesus retina. Science 221, 4608 (1983).
[90]
Fisher Yu, Ari Seff, Yinda Zhang, Shuran Song, Thomas Funkhouser, and Jianxiong Xiao. 2015. Lsun: Construction of a large-scale image dataset using deep learning with humans in the loop. arXiv preprint arXiv:1506.03365 (2015).
[91]
Cheng Zhang, Cengiz Öztireli, Stephan Mandt, and Giampiero Salvi. 2019. Active Mini-Batch Sampling Using Repulsive Point Processes. AAAI.
[92]
Yahan Zhou, Haibin Huang, Li-Yi Wei, and Rui Wang. 2012. Point sampling with general noise spectrum. ACM Trans. Graph. 31, 4 (2012).
[93]
Yang Zhou, Zhen Zhu, Xiang Bai, Dani Lischinski, Daniel Cohen-Or, and Hui Huang. 2018. Non-Stationary Texture Synthesis by Adversarial Expansion. ACM Trans. Graph. 37, 4 (2018).
Index Terms
- Patternshop: Editing Point Patterns by Image Manipulation
Recommendations
Plenoptic Image Editing
This paper presents a new class of interactive image editing operations designed to maintain consistency between multiple images of a physical 3D scene. The distinguishing feature of these operations is that edits to any one image propagate ...
Inverse image editing: recovering a semantic editing history from a before-and-after image pair
We study the problem of inverse image editing, which recovers a semantically-meaningful editing history from a source image and an edited copy. Our approach supports a wide range of commonly-used editing operations such as cropping, object insertion and ...
Comments
Information & Contributors
Information
Published In
Copyright © 2023 Owner/Author(s).
This work is licensed under a Creative Commons Attribution-NonCommercial International 4.0 License.
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 26 July 2023
Published in TOG Volume 42, Issue 4
Check for updates
Author Tags
Qualifiers
- Research-article
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- View Citations1Total Citations
- 646Total Downloads
- Downloads (Last 12 months)382
- Downloads (Last 6 weeks)42
Reflects downloads up to 12 Nov 2024
Other Metrics
Citations
Cited By
View all- Wang QMao ZGao JZhang Y(2024)Document-level Relation Extraction with Progressive Self-distillationACM Transactions on Information Systems10.1145/365616842:6(1-34)Online publication date: 25-Jun-2024
View Options
Get Access
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in