research-article

Interactive image filtering with multiple levels-of-control on mobile devices

Authors:

Tobias Dürschmid,

Matthias Trapp,

Mandy Klingbeil,

Jürgen Döllner,

Sebastian PasewaldtAuthors Info & Claims

SA '16: SIGGRAPH ASIA 2016 Mobile Graphics and Interactive Applications

Article No.: 2, Pages 1 - 8

https://doi.org/10.1145/2999508.2999521

Published: 28 November 2016 Publication History

Abstract

With the continuous development of mobile graphics hardware, interactive high-quality image stylization based on nonlinear filtering is becoming feasible and increasingly used in casual creativity apps. However, these apps often only serve high-level controls to parameterize image filters and generally lack support for low-level (artistic) control, thus automating art creation rather than assisting it. This work presents a GPU-based framework that enables to parameterize image filters at three levels of control: (1) presets followed by (2) global parameter adjustments can be interactively refined by (3) complementary on-screen painting that operates within the filters' parameter spaces for local adjustments. The framework provides a modular XML-based effect scheme to effectively build complex image processing chains-using these interactive filters as building blocks---that can be efficiently processed on mobile devices. Thereby, global and local parameterizations are directed with higher-level algorithmic support to ease the interactive editing process, which is demonstrated by state-of-the-art stylization effects, such as oil-paint filtering and watercolor rendering.

Supplementary Material

ZIP File (a2-semmo.zip)

Supplemental files.

Download
80.21 MB

References

[1]

Anjyo, K.-i., Wemler, S., and Baxter, W. 2006. Tweakable Light and Shade for Cartoon Animation. In Proc. NPAR, 133--139.

Digital Library

[2]

Benedetti, L., Winnemöller, H., Corsini, M., and Scopigno, R. 2014. Painting with Bob: Assisted Creativity for Novices. In Proc. ACM UIST, 419--428.

Digital Library

[3]

Bousseau, A., Kaplan, M., Thollot, J., and Sillion, F. X. 2006. Interactive Watercolor Rendering with Temporal Coherence and Abstraction. In Proc. NPAR, 141--149.

Digital Library

[4]

Brox, T., Weickert, J., Burgeth, B., and Mrázek, P. 2006. Nonlinear structure tensors. Image and Vision Computing 24, 1, 41--55.

Digital Library

[5]

Capin, T., Pulli, K., and Akenine-Mller, T. 2008. The State of the Art in Mobile Graphics Research. IEEE Computer Graphics and Applications 28, 4, 74--84.

Digital Library

[6]

Dev, K. 2013. Mobile Expressive Renderings: The State of the Art. IEEE Computer Graphics and Applications 33, 3, 22--31.

Digital Library

[7]

DiVerdi, S., Krishnaswamy, A., Mech, R., and Ito, D. 2013. Painting with Polygons: A Procedural Watercolor Engine. IEEE Trans. Vis. Comput. Graphics 19, 5, 723--735.

Digital Library

[8]

DiVerdi, S. 2013. A Brush Stroke Synthesis Toolbox. In Image and Video-Based Artistic Stylisation. Springer, 23--44.

[9]

Fischer, J., Haller, M., and Thomas, B. 2008. Stylized Depiction in Mixed Reality. International Journal of Virtual Reality 7, 4 (December), 71--79.

[10]

Gerl, M., and Isenberg, T. 2013. Interactive Example-based Hatching. Computers & Graphics 37, 1--2, 65--80.

Digital Library

[11]

Gooch, A. A., Long, J., Ji, L., Estey, A., and Gooch, B. S. 2010. Viewing Progress in Non-photorealistic Rendering through Heinlein's Lens. In Proc. NPAR, 165--171.

Digital Library

[12]

Hanrahan, P., and Haeberli, P. 1990. Direct WYSIWYG Painting and Texturing on 3D Shapes. Computer Graphics 24, 4, 215--223.

Digital Library

[13]

Hays, J., and Essa, I. 2004. Image and Video Based Painterly Animation. In Proc. NPAR, 113--120.

Digital Library

[14]

Isenberg, T. 2016. Interactive NPAR: What Type of Tools Should We Create? In Proc. NPAR, 89--96.

Digital Library

[15]

Kang, D., and Yoon, K. 2015. Interactive Painterly Rendering for Mobile Devices. In Entertainment Computing - ICEC 2015. Springer International Publishing, 445--450.

[16]

Kim, T. H., and Lin, I., 2012. Real-Time Non-photorealistic Viewfinder on the Tegra 3 Platform. Stanford University, unpublished.

[17]

Kopf, J., Cohen, M. F., Lischinski, D., and Uyttendaele, M. 2007. Joint Bilateral Upsampling. ACM Trans. Graph. 26, 3.

Digital Library

[18]

Kyprianidis, J. E., and Döllner, J. 2008. Image Abstraction by Structure Adaptive Filtering. In Proc. EG UK TPCG, 51--58.

[19]

Kyprianidis, J. E., Collomosse, J., Wang, T., and Isenberg, T. 2013. State of the 'Art': A Taxonomy of Artistic Stylization Techniques for Images and Video. IEEE Trans. Vis. Comput. Graphics 19, 5, 866--885.

Digital Library

[20]

Lum, E. B., and Ma, K.-L. 2002. Interactivity is the Key to Expressive Visualization. SIGGRAPH Comput. Graph. 36, 3, 5--9.

Digital Library

[21]

Mould, D., and Rosin, P. L. 2016. A Benchmark Image Set for Evaluating Stylization. In Proc. NPAR, 11--20.

Digital Library

[22]

Nielsen, J. 1993. Usability Engineering. Morgan Kaufmann Publishers Inc., San Francisco, CA, USA, 23--48.

Digital Library

[23]

Oh, J., Maeng, S., and Park, J. 2012. Efficient Watercolor Painting on Mobile Devices. International Journal of Contents 8, 4, 36--41.

[24]

Olsen, S. C., Maxwell, B. A., and Gooch, B. 2005. Interactive Vector Fields for Painterly Rendering. In Proc. Graphics Interface, 241--247.

Digital Library

[25]

Praun, E., Hoppe, H., Webb, M., and Finkelstein, A. 2001. Real-Time Hatching. In Proc. ACM SIGGRAPH, 581--586.

Digital Library

[26]

Rosin, P. L., and Lai, Y.-K. 2013. Non-photorealistic Rendering with Spot Colour. In Proc. CAe, 67--75.

Digital Library

[27]

Salesin, D. H., 2002. Non-Photorealistic Animation & Rendering: 7 Grand Challenges. Keynote talk at NPAR.

[28]

Salisbury, M. P., Wong, M. T., Hughes, J. F., and Salesin, D. H. 1997. Orientable Textures for Image-based Pen-and-ink Illustration. In Proc. ACM SIGGRAPH, 401--406.

Digital Library

[29]

Semmo, A., Döllner, J., and Schlegel, F. 2016. BeCasso: Image Stylization by Interactive Oil Paint Filtering on Mobile Devices. In Proc. ACM SIGGRAPH Appy Hour, 6:1--6:1.

Digital Library

[30]

Semmo, A., Limberger, D., Kyprianidis, J. E., and Dö llner, J. 2016. Image Stylization by Interactive Oil Paint Filtering. Computers & Graphics 55, 157--171.

Digital Library

[31]

Semmo, A., Trapp, M., Dürschmid, T., Döllner, J., and Pasewaldt, S. 2016. Interactive Multi-scale Oil Paint Filtering on Mobile Devices. In Proc. ACM SIGGRAPH Posters, 42:1--42:2.

Digital Library

[32]

Singhal, N., Yoo, J. W., Choi, H. Y., and Park, I. K. 2011. Design and Optimization of Image Processing Algorithms on Mobile GPU. In ACM SIGGRAPH Posters, 21:1--21:1.

Digital Library

[33]

Thabet, R., Mahmoudi, R., and Bedoui, M. H. 2014. Image processing on mobile devices: An overview. In Proc. IPAS, 1--8.

[34]

Todo, H., Anjyo, K.-i., Baxter, W., and Igarashi, T. 2007. Locally Controllable Stylized Shading. ACM Trans. Graph. 26, 3, 17:1--17:7.

Digital Library

[35]

Wang, M., Wang, B., Fei, Y., Qian, K., Wang, W., Chen, J., and Yong, J.-H. 2014. Towards Photo Watercolorization with Artistic Verisimilitude. IEEE Trans. Vis. Comput. Graphics 20, 10, 1451--1460.

[36]

Webb, M., Praun, E., Finkelstein, A., and Hoppe, H. 2002. Fine tone control in hardware hatching. In Proc. NPAR, 53--58.

Digital Library

[37]

Wexler, D., and Dezeustre, G. 2012. Intelligent Brush Strokes. In Proc. ACM SIGGRAPH Talks, 50:1--50:1.

Digital Library

[38]

Winnemöller, H., Olsen, S. C., and Gooch, B. 2006. RealTime Video Abstraction. ACM Trans. Graph. 25, 3, 1221--1226.

Digital Library

[39]

Winnemöller, H., Kyprianidis, J. E., and Olsen, S. C. 2012. XDoG: an extended difference-of-Gaussians compendium including advanced image stylization. Computers & Graphics 36, 6, 740--753.

Digital Library

[40]

Winnemöller, H. 2013. NPR in the Wild. In Image and Video-Based Artistic Stylisation. Springer, 353--374.

Cited By

Sramek ZSato AZhou ZHosio SYatani K(2023)SoundTraveller: Exploring Abstraction and Entanglement in Timbre Creation Interfaces for SynthesizersProceedings of the 2023 ACM Designing Interactive Systems Conference10.1145/3563657.3596089(95-114)Online publication date: 10-Jul-2023
https://dl.acm.org/doi/10.1145/3563657.3596089
Lötzsch WReimann MBüssemeyer MSemmo ADöllner JTrapp M(2022)WISE: Whitebox Image Stylization by Example-Based LearningComputer Vision – ECCV 202210.1007/978-3-031-19790-1_9(135-152)Online publication date: 24-Oct-2022
https://doi.org/10.1007/978-3-031-19790-1_9
Söchting MTrapp M(2022)Tracking Eye Movement for Controlling Real-Time Image-Abstraction TechniquesComputer Vision, Imaging and Computer Graphics Theory and Applications10.1007/978-3-030-94893-1_5(103-123)Online publication date: 22-Jan-2022
https://doi.org/10.1007/978-3-030-94893-1_5
Show More Cited By

Index Terms

Interactive image filtering with multiple levels-of-control on mobile devices
1. Computing methodologies
  1. Computer graphics
    1. Image manipulation

Recommendations

Challenges in user experience design of image filtering apps
SA '17: SIGGRAPH Asia 2017 Mobile Graphics & Interactive Applications

Photo filtering apps successfully deliver image-based stylization techniques to a broad audience, in particular in the ubiquitous domain (e.g., smartphones, tablet computers). Interacting with these inherently complex techniques has so far mostly been ...
Becasso: artistic image processing and editing on mobile devices
SA '16: SIGGRAPH ASIA 2016 Mobile Graphics and Interactive Applications

BeCasso is a mobile app that enables users to transform photos into high-quality, high-resolution non-photorealistic renditions, such as oil and watercolor paintings, cartoons, and colored pencil drawings, which are inspired by real-world paintings or ...
Interactive image filtering for level-of-abstraction texturing of virtual 3D scenes

Texture mapping is a key technology in computer graphics. For the visual design of 3D scenes, in particular, effective texturing depends significantly on how important contents are expressed, e.g., by preserving global salient structures, and how their ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

SA '16: SIGGRAPH ASIA 2016 Mobile Graphics and Interactive Applications

November 2016

76 pages

ISBN:9781450345514

DOI:10.1145/2999508

Conference Chairs:
Steven ZhiYing Zhou
National University of Singapore, Singapore
,
Börje Karlsson
Microsoft Research Asia, China

Copyright © 2016 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGGRAPH: ACM Special Interest Group on Computer Graphics and Interactive Techniques

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 28 November 2016

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

Bundesministerium für Bildung und Forschung

Conference

SA '16

Sponsor:

SIGGRAPH

SA '16: SIGGRAPH Asia 2016

December 5 - 8, 2016

Macau

Acceptance Rates

Overall Acceptance Rate 178 of 869 submissions, 20%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

14
Total Citations
View Citations
286
Total Downloads

Downloads (Last 12 months)11
Downloads (Last 6 weeks)0

Reflects downloads up to 16 Oct 2024

Other Metrics

View Author Metrics

Citations

Cited By

Sramek ZSato AZhou ZHosio SYatani K(2023)SoundTraveller: Exploring Abstraction and Entanglement in Timbre Creation Interfaces for SynthesizersProceedings of the 2023 ACM Designing Interactive Systems Conference10.1145/3563657.3596089(95-114)Online publication date: 10-Jul-2023
https://dl.acm.org/doi/10.1145/3563657.3596089
Lötzsch WReimann MBüssemeyer MSemmo ADöllner JTrapp M(2022)WISE: Whitebox Image Stylization by Example-Based LearningComputer Vision – ECCV 202210.1007/978-3-031-19790-1_9(135-152)Online publication date: 24-Oct-2022
https://doi.org/10.1007/978-3-031-19790-1_9
Söchting MTrapp M(2022)Tracking Eye Movement for Controlling Real-Time Image-Abstraction TechniquesComputer Vision, Imaging and Computer Graphics Theory and Applications10.1007/978-3-030-94893-1_5(103-123)Online publication date: 22-Jan-2022
https://doi.org/10.1007/978-3-030-94893-1_5
Reimann MBuchheim BSemmo ADollner JTrapp M(2021)Interactive Multi-level Stroke Control for Neural Style Transfer2021 International Conference on Cyberworlds (CW)10.1109/CW52790.2021.00009(1-8)Online publication date: Sep-2021
https://doi.org/10.1109/CW52790.2021.00009
Reimann MKlingbeil MPasewaldt SSemmo ATrapp MDöllner J(2019)Locally controllable neural style transfer on mobile devicesThe Visual Computer: International Journal of Computer Graphics10.1007/s00371-019-01654-135:11(1531-1547)Online publication date: 1-Nov-2019
https://dl.acm.org/doi/10.1007/s00371-019-01654-1
Trapp MPasewaldt SDürschmid TSemmo ADöllner JPost FZara J(2018)Teaching image-processing programming for mobile devicesProceedings of the 39th Annual European Association for Computer Graphics Conference: Education Papers10.5555/3297765.3297769(17-24)Online publication date: 16-Apr-2018
https://dl.acm.org/doi/10.5555/3297765.3297769
Montesdeoca SSeah HSemmo ABénard PVergne RThollot JBenvenuti DWyvill BFu H(2018)MNPRProceedings of the Joint Symposium on Computational Aesthetics and Sketch-Based Interfaces and Modeling and Non-Photorealistic Animation and Rendering10.1145/3229147.3229162(1-11)Online publication date: 17-Aug-2018
https://dl.acm.org/doi/10.1145/3229147.3229162
Gao CTang MLiang XSu ZZou C(2018)PencilArt: A Chromatic Penciling Style Generation FrameworkComputer Graphics Forum10.1111/cgf.1333437:6(395-409)Online publication date: 26-Mar-2018
https://doi.org/10.1111/cgf.13334
Reimann MKlingbeil MPasewaldt SSemmo ATrapp MDollner J(2018)MaeSTrO: A Mobile App for Style Transfer Orchestration Using Neural Networks2018 International Conference on Cyberworlds (CW)10.1109/CW.2018.00016(9-16)Online publication date: Oct-2018
https://doi.org/10.1109/CW.2018.00016
Dürschmid TSöchting MSemmo ATrapp MDöllner JBillinghurst MRungjiratananon W(2017)ProsumerFXSIGGRAPH Asia 2017 Mobile Graphics & Interactive Applications10.1145/3132787.3139208(1-8)Online publication date: 27-Nov-2017
https://dl.acm.org/doi/10.1145/3132787.3139208
Show More Cited By

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents