research-article

Creating Word Paintings Jointly Considering Semantics, Attention, and Aesthetics

Authors:

Linchengyu Jin,

Zhitang Lu, and

Jinhui YuAuthors Info & Claims

ACM Transactions on Applied Perceptions (TAP), Volume 19, Issue 3

Article No.: 13, Pages 1 - 21

https://doi.org/10.1145/3539610

Published: 02 September 2022 Publication History

Abstract

In this article, we present a content-aware method for generating a word painting. Word painting is a composite artwork made from the assemblage of words extracted from a given text, which carries similar semantics and visual features to a given source image. However, word painting, usually created by skilled artists, involves tedious manual processes, especially when generating streamlines and laying out text. Hence, we provide an easy method to create word paintings for users. How to design textural layout that simultaneously conveys the input image and enables easy access to the semantic theme is the key challenge to generating a visually pleasing word painting. To address this issue, given an image and its content-related text, we first decompose the input image into several regions and approximate each region with a smooth vector field. At the same time, by analyzing the input text, we extract some weighted keywords as the graphic elements. Then, to measure the likelihood of positions in the input image that attract the observers’ attention, we generate a saliency map with our trained visual attention model. Finally, jointly considering visual attention and aesthetic rules, we propose an energy-based optimization framework to arrange extracted keywords into the decomposed regions and synthesize a word painting. Experimental results and user studies show that this method is able to generate a fashionable and appealing word painting.

References

[1]

Guoning Chen, Vivek Kwatra, Li Yi Wei, Charles D. Hansen, and Eugene Zhang. 2012. Design of 2D time-varying vector fields. IEEE Trans. Visual. Comput. Graph. 18, 10 (Oct. 2012), 1717–1730. DOI:

Digital Library

[2]

Mingte Chi, Shihsyun Lin, Shiang Yi Chen, Chao Hung Lin, and Tongyee Lee. 2015. Morphable word clouds for time-varying text data visualization. IEEE Trans. Visual. Comput. Graph. 21, 12 (Dec. 2015), 1415–1426. DOI:

Digital Library

[3]

Piotr Dollar and C. Lawrence Zitnick. 2013. Structured forests for fast edge detection. In Proceedings of the IEEE International Conference on Computer Vision (ICCV’13). IEEE Computer Society, Washington, DC, 1841–1848. DOI:

Digital Library

[4]

Simone Frintrop, Erich Rome, and Henrik I. Christensen. 2010. Computational visual attention systems and their cognitive foundations: A survey. ACM Trans. Appl. Percept. 7, 1, Article 6 (Jan. 2010), 39 pages. DOI:

Digital Library

[5]

Zhenzhen Hu, Si Liu, Jianguo Jiang, Richang Hong, Meng Wang, and Shuicheng Yan. 2014. PicWords: Render a picture by packing keywords. IEEE Trans. Multimedia 16, 4 (June 2014), 1156–1164. DOI:

Digital Library

[6]

IBM. 2019. Watson Natural Language Understanding. Retrieved from https://www.ibm.com/cloud/watson-natural-language-understanding.

[7]

Ali Jahanian, Jerry Liu, Qian Lin, Daniel R. Tretter, Eamonn Obrienstrain, Seungyon Claire Lee, Nicholas P. Lyons, and Jan P. Allebach. 2013. Recommendation system for automatic design of magazine covers. In Proceedings of the International Conference on Intelligent User Interfaces (IUI’13). ACM, New York, NY, 95–106. DOI:

Digital Library

[8]

Tilke Judd, Krista A. Ehinger, Fredo Durand, and Antonio Torralba. 2009. Learning to predict where humans look. In Proceedings of the IEEE International Conference on Computer Vision (ICCV’09). 2106–2113. DOI:

[9]

Jim Krause. 2004. Design Basics Index. David & Charles Plc.

[10]

Bongshin Lee, Nathalie Henry Riche, Amy K. Karlson, and M. Sheelagh T. Carpendale. 2010. SparkClouds: Visualizing trends in tag clouds. IEEE Trans. Visual. Comput. Graph. 16, 6 (Nov. 2010), 1182–1189. DOI:

Digital Library

[11]

Chenlu Li, Xiaoju Dong, and Xiaoru Yuan. 2018. Metro-wordle: An interactive visualization for urban text distributions based on wordle. Visual Inform. 2, 1 (2018), 50–59. DOI:

[12]

Ron Maharik, Mikhail Bessmeltsev, Alla Sheffer, Ariel Shamir, and Nathan A. Carr. 2011. Digital micrography. ACM Trans. Graph. 30, 4 (July 2011), 100:1–100:12. DOI:

Digital Library

[13]

Fernando Vieira Paulovich, Franklina Maria Bragion Toledo, Guilherme P. Telles, Rosane Minghim, and Luis Gustavo Nonato. 2012. Semantic wordification of document collections. Comput. Graph. Forum 31, 3 (June 2012), 1145–1153. DOI:

Digital Library

[14]

Reza Adhitya Saputra, Craig S. Kaplan, Paul Asente, and Radomir Mech. 2017. FLOWPAK: Flow-based ornamental element packing. In Proceedings of the Graphics Interface Conference (GI’17). Canadian Human-Computer Communications Society, School of Computer Science, University of Waterloo, Waterloo, Ontario, Canada, 8–15. DOI:

[15]

Jeremiah Still and Mary Still. 2019. Influence of visual salience on webpage product searches. ACM Trans. Appl. Percept. 16, 1, Article 3 (Feb. 2019), 11 pages. DOI:

Digital Library

[16]

Tatiana Surazhsky and Gershon Elber. 2002. Artistic surface rendering using layout of text. Comput. Graph. Forum 21, 2 (Aug. 2002), 99–110. DOI:

[17]

Fernanda B. Viegas, Martin Wattenberg, and Jonathan Feinberg. 2009. Participatory visualization with wordle. IEEE Trans. Visual. Comput. Graph. 15, 6 (Nov. 2009), 1137–1144. DOI:

Digital Library

[18]

Yunhai Wang, Bongshin Lee, Xiaowei Chu, Kaiyi Zhang, Chen Bao, Xiaotong Li, Jian Zhang, Chiwing Fu, Christophe Hurter, and Oliver Deussen. 2020. ShapeWordle: Tailoring wordles using shape-aware archimedean spirals. IEEE Trans. Visual. Comput. Graph. 26, 1 (Jan. 2020), 991–1000. DOI:

[19]

Jie Xu and Craig S. Kaplan. 2007. Calligraphic packing. In Proceedings of the Graphics Interface Conference (GI’07). ACM, New York, NY, 43–50. DOI:

Digital Library

[20]

Xuemiao Xu, Linling Zhang, and Tientsin Wong. 2010. Structure-based ASCII art. ACM Trans. Graph. 29, 4 (July 2010), 52:1–52:10. DOI:

Digital Library

[21]

Xuemiao Xu, Linyuan Zhong, Minshan Xie, Jing Qin, Yilan Chen, Qiang Jin, Tientsin Wong, and Guoqiang Han. 2015. Texture-aware ASCII art synthesis with proportional fonts. In Proceedings of the Workshop on Non-Photorealistic Animation and Rendering (NPAR’15). Eurographics Association, Aire-la-Ville, Switzerland, 183–193. DOI:

[22]

Xuyong Yang, Tao Mei, Yingqing Xu, Yong Rui, and Shipeng Li. 2016. Automatic generation of visual-textual presentation layout. ACM Trans. Multimedia Comput. Commun. Appl. 12, 2 (Feb. 2016), 33:1–33:22. DOI:

Digital Library

[23]

Yiting Yeh, Lingfeng Yang, Matthew Watson, Noah D. Goodman, and Pat Hanrahan. 2012. Synthesizing open worlds with constraints using locally annealed reversible jump MCMC. ACM Trans. Graph. 31, 4 (July 2012), 56:1–56:11. DOI:

Digital Library

[24]

Wenyuan Yin, Tao Mei, and Chang Wen Chen. 2013. Automatic generation of social media snippets for mobile browsing. In Proceedings of the ACM International Conference on Multimedia (MM’13). ACM, New York, NY, 927–936. DOI:

Digital Library

[25]

Xinru Zheng, Xiaotian Qiao, Ying Cao, and Rynson W. H. Lau. 2019. Content-aware generative modeling of graphic design layouts. ACM Trans. Graph. 38, 4 (July 2019), 133:1–133:15. DOI:

Digital Library

[26]

Changqing Zou, Junjie Cao, Warunika Ranaweera, Ibraheem Alhashim, Ping Tan, Alla Sheffer, and Hao Zhang. 2016. Legible compact calligrams. ACM Trans. Graph. 35, 4 (July 2016), 122:1–122:12. DOI:

Digital Library

Cited By

Lin WZhang YDai WLiu HSee JXiong H(2024)Scene Graph Lossless Compression with Adaptive Prediction for Objects and RelationsACM Transactions on Multimedia Computing, Communications, and Applications10.1145/364950320:7(1-23)Online publication date: 27-Mar-2024
https://dl.acm.org/doi/10.1145/3649503
Xing WYao JLiu ZLiu WZhang SWang L(2023)Contrastive JS: A Novel Scheme for Enhancing the Accuracy and Robustness of Deep ModelsIEEE Transactions on Multimedia10.1109/TMM.2022.323203025(7881-7893)Online publication date: 1-Jan-2023
https://dl.acm.org/doi/10.1109/TMM.2022.3232030
Tanveer MWang YMahdavi-Amiri AZhang H(2023)DS-Fusion: Artistic Typography via Discriminated and Stylized Diffusion2023 IEEE/CVF International Conference on Computer Vision (ICCV)10.1109/ICCV51070.2023.00041(374-384)Online publication date: 1-Oct-2023
https://doi.org/10.1109/ICCV51070.2023.00041

Index Terms

Creating Word Paintings Jointly Considering Semantics, Attention, and Aesthetics
1. Computing methodologies
  1. Computer graphics
    1. Graphics systems and interfaces
      1. Perception
2. Human-centered computing
  1. Interaction design
    1. Systems and tools for interaction design

Recommendations

Interpreting the Effect of Embellishment on Chart Visualizations
CHI '21: Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems

Infographics range from minimalism that aims to convey the raw data to elaborately decorated, or embellished, graphics that aim to engage readers by telling a story. Several studies have shown evidence to negative, but also positive, effects on ...
Read More
Creating Abstract Digital Paintings with Painting Camera Technique
IV '00: Proceedings of the International Conference on Information Visualisation

We present a simple and intuitive rendering technique to create abstract paintings. This technique allows the users to render a given animated scene with multiple cameras, using a method inspired by concepts used by cubist and futurist abstract ...
Read More
Creating for 3D Digital Chinese Ink-Wash Landscape Paintings Based on Maya
DMDCM '11: Proceedings of the 2011 Workshop on Digital Media and Digital Content Management

Putting forward the idea of simulating the "three-perspective method" and "cavalier perspective" layout in the Chinese landscape paintings theory based on "Oblique Projection", with a successful realization. A process of brush rendering and multi-layer ...
Read More

Comments

Information & Contributors

Information

Published In

cover image ACM Transactions on Applied Perception

ACM Transactions on Applied Perception Volume 19, Issue 3

July 2022

83 pages

ISSN:1544-3558

EISSN:1544-3965

DOI:10.1145/3543998

Editor:
Bobby Bodenheimer
Vanderbilt University USA

Issue’s Table of Contents

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 02 September 2022

Online AM: 04 July 2022

Accepted: 01 May 2022

Revised: 01 January 2022

Received: 01 November 2020

Published in TAP Volume 19, Issue 3

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Refereed

Funding Sources

National Nature Science Foundation of China

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

3
Total Citations
View Citations
415
Total Downloads

Downloads (Last 12 months)202
Downloads (Last 6 weeks)12

Other Metrics

View Author Metrics

Citations

Cited By

Lin WZhang YDai WLiu HSee JXiong H(2024)Scene Graph Lossless Compression with Adaptive Prediction for Objects and RelationsACM Transactions on Multimedia Computing, Communications, and Applications10.1145/364950320:7(1-23)Online publication date: 27-Mar-2024
https://dl.acm.org/doi/10.1145/3649503
Xing WYao JLiu ZLiu WZhang SWang L(2023)Contrastive JS: A Novel Scheme for Enhancing the Accuracy and Robustness of Deep ModelsIEEE Transactions on Multimedia10.1109/TMM.2022.323203025(7881-7893)Online publication date: 1-Jan-2023
https://dl.acm.org/doi/10.1109/TMM.2022.3232030
Tanveer MWang YMahdavi-Amiri AZhang H(2023)DS-Fusion: Artistic Typography via Discriminated and Stylized Diffusion2023 IEEE/CVF International Conference on Computer Vision (ICCV)10.1109/ICCV51070.2023.00041(374-384)Online publication date: 1-Oct-2023
https://doi.org/10.1109/ICCV51070.2023.00041

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Full Text

View this article in Full Text.

HTML Format

View this article in HTML Format.

Media

Figures

Other

Tables

View full text|Download PDF

View Issue’s Table of Contents