research-article

Open access

Band-Sifting Decomposition for Image-Based Material Editing

Authors:

Ivaylo Boyadzhiev,

Edward AdelsonAuthors Info & Claims

ACM Transactions on Graphics (TOG), Volume 34, Issue 5

Article No.: 163, Pages 1 - 16

https://doi.org/10.1145/2809796

Published: 03 November 2015 Publication History

Abstract

Photographers often “prep” their subjects to achieve various effects; for example, toning down overly shiny skin, covering blotches, etc. Making such adjustments digitally after a shoot is possible, but difficult without good tools and good skills. Making such adjustments to video footage is harder still. We describe and study a set of 2D image operations, based on multiscale image analysis, that are easy and straightforward and that can consistently modify perceived material properties. These operators first build a subband decomposition of the image and then selectively modify the coefficients within the subbands. We call this selection process band sifting.

We show that different siftings of the coefficients can be used to modify the appearance of properties such as gloss, smoothness, pigmentation, or weathering. The band-sifting operators have particularly striking effects when applied to faces; they can provide “knobs” to make a face look wetter or drier, younger or older, and with heavy or light variation in pigmentation. Through user studies, we identify a set of operators that yield consistent subjective effects for a variety of materials and scenes. We demonstrate that these operators are also useful for processing video sequences.

Supplementary Material

boyadzhiev (boyadzhiev.zip)

Supplemental movie, appendix, image and software files for, Band-Sifting Decomposition for Image-Based Material Editing

Download
541.74 MB

References

[1]

Arqball. 2014. Shoes. https://arqspin.com/gallery/shoes/.

[2]

S. Bae, S. Paris, and F. Durand. 2006. Two-scale tone management for photographic look. ACM Trans. Graph. 25, 3, 637--645.

Digital Library

[3]

S. Bell, P. Upchurch, N. Snavely, and K. Bala. 2013. OpenSurfaces: A richly annotated catalog of surface appearance. ACM Trans. Graph. 32, 4.

Digital Library

[4]

S. Bell, P. Upchurch, N. Snavely, and K. Bala. 2015. Material recognition in the wild with the materials in context database. In Proceedings of the Conference on Computer Vision and Pattern Recognition (CVPR'15).

[5]

P. J. Burt and E. H. Adelson. 1983. The Laplacian pyramid as a compact image code. IEEE Trans. Comm. 31, 4, 532--540.

[6]

V. Bychkovsky, S. Paris, E. Chan, and F. Durand. 2011. Learning photographic global tonal adjustment with a database of input/output image pairs. In Proceedings of the 24^th IEEE Conference on Computer Vision and Pattern Recognition (CVPR'11). 97--104.

Digital Library

[7]

J. Chen, S. Paris, and F. Durand. 2007. Real-time edge-aware image processing with the bilateral grid. ACM Trans. Graph. 26, 3.

Digital Library

[8]

P. Debevec, T. Hawkins, C. Tchou, H.-P. Duiker, W. Sarokin, and M. Sagar. 2000. Acquiring the reflectance field of a human face. In Proceedings of the 27^th Annual ACM Congerence on Computer Graphics and Interactive Techniques (SIGGRAPH'00). ACM Press/Addison-Wesley, New York, 145--156.

Digital Library

[9]

P. E. Debevec and J. Malik. 1997. Recovering high dynamic range radiance maps from photographs. In Proceedings of the 24^th Annual ACM Conference on Computer Graphics and Interactive Techniques (SIGGRAPH'97). ACM Press/Addison-Wesley, New York, 369--378.

Digital Library

[10]

D. Donoho. 1995. De-noising by soft-thresholding. IEEE Trans. Inf. Theory. 41, 3, 613--627.

Digital Library

[11]

F. Durand and J. Dorsey. 2002. Fast bilateral filtering for the display of high-dynamic rrange images. ACM Trans. Graph. 21, 3.

Digital Library

[12]

T. Ewan. 2009. Orange. https://www.flickr.com/photos/the_ewan/3958779607/.

[13]

Z. Farbman, R. Fattal, D. Lischinski, and R. Szeliski. 2008. Edge-preserving decomposition for mult-scale tone and detail manipulation. ACM Trans. Graph. 27, 3.

Digital Library

[14]

R. Fattal. 2009. Edge-avoiding wavelets and their applications. ACM Trans. Graph. 28, 3.

Digital Library

[15]

R. Fattal, M. Agrawala, and S. Rusinkiewicz. 2007. Multiscale shape and detail enhancement from multi-light image collections. ACM Trans. Graph. 26, 3.

Digital Library

[16]

R. W. Fleming and H. H. Bülthoff. 2005. Low-level image cues in the perception of translucent materials. ACM Trans. Appl. Percept. 2, 3.

Digital Library

[17]

R. W. Fleming, C. Wiebel, and K. Gegenfurtner. 2013. Perceptual qualities and material classes. J. Vis. 13, 8, 9.

[18]

E. S. L. Gastal and M. M. Oliveira. 2011. Domain transform for edge-aware image and video processing. ACM Trans. Graph. 30, 3.

Digital Library

[19]

E. S. L. Gastal and M. M. Oliveira. 2012. Adaptive manifolds for real-time high-dimensional filtering. ACM Trans. Graph. 31, 4.

Digital Library

[20]

L. Glondu, L. Muguercia, M. Marchal, C. Bosch, H. Rushmeier, G. Dumont, and G. Drettakis. 2012. Example-based fractured appearance. Comput. Graph. Forum 31, 4.

Digital Library

[21]

A. Golovinskiy, W. Matusik, H. Pfister, S. Rusinkiewicz, and T. Funkhouser. 2006. A statistical model for synthesis of detailed facial geometry. ACM Trans. Graph. 25, 3.

Digital Library

[22]

J. Hanika, H. Dammertz, and H. P. A. Lensch. 2011. Edge-optimized a-trous wavelets for local contrast enhancement with robust denoising. Comput. Graph. Forum 30, 7.

[23]

M. Harris, S. Sengupta, and J. D. Owens. 2007. Parallel prefix sum (scan) with CUDA. In GPU Gems 3, H. Nguyen, Ed., Addison-Wesley, 851--876.

[24]

W. Hartshorn. 2008. Sweet potatoes. https://www.flickr.com/photos/wallyhartshorn/2850269642/.

[25]

K. He, J. Sun, and X. Tang. 2010. Guided image filtering. In Proceedings of the European Conference on Computer Vision (ECCV'10).

Digital Library

[26]

D. J. Heeger and J. R. Bergen. 1995. Pyramid-based texture analysis/synthesis. In Proceedings of the Annual ACM Conference on Computer Graphics and Interactive Techniques (SIGGRAPH'95). ACM Press, New York.

Digital Library

[27]

R. Jill. 2006. Self portrait. https://www.flickr.com/photos/98972490N04/958207891.

[28]

E. Judy. 2009. Good grape crop. https://www.flickr.com/photos/659247400N00/5140439795/.

[29]

L. Karacan, E. Erdem, and A. Erdem. 2013. Structure preserving image smoothing via region covariances. ACM Trans. Graph. 32, 6.

Digital Library

[30]

E. A. Khan, E. Reinhard, R. W. Fleming, and H. H. Bülthoff. 2006. Image-based material editing. ACM Trans. Graph. 25, 3.

Digital Library

[31]

J. Kim, P. Marlow, and B. L. Anderson. 2011. The perception of gloss depends on highlight congruence with surface shading. J. Vis. 11, 9, 4.

[32]

E. H. Land and J. Mccann. 1971. Lightness and retinex theory. J. Optic. Soc. Amer. 1--11.

[33]

C. Liu, L. Sharan, E. H. Adelson, and R. Rosenholtz. 2010. Exploring features in a Bayesian framework for material recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR'10). 239--246.

[34]

S. P. Mallick, T. Zickler, P. N. Belhumeur, and D. J. Kriegman. 2006. Specularity removal in images and videos: A PDE approach. In Proceedings of the European Conference on Computer Vision (ECCV'06). 550--563.

Digital Library

[35]

R. Mantiuk, K. J. Kim, A. G. Rempel, and W. Heidrich. 2011. HDR-VDP-2: A calibrated visual metric for visibility and quality predictions in all luminance conditions. ACM Trans. Graph. 30, 4.

Digital Library

[36]

R. Mantiuk, K. Myszkowski, and H.-P. Seidel. 2006. A perceptual framework for contrast processing of high dynamic range images. ACM Trans. Appl. Percept. 3, 3, 286--308.

Digital Library

[37]

P. J. Marlow and B. L. Anderson. 2013. Generative constraints on image cues for perceived gloss. J. Vis. 13, 14, 2.

[38]

P. J. Marlow, J. Kim, and B. L. Anderson. 2012. The perception and misperception of specular surface reflectance. Curr. Biol. 22, 20, 1909--1913.

[39]

T. Mertens, J. Kautz, J. Chen, P. Bekaert, and F. Durand. 2006. Texture transfer using geometry correlation. In Proceedings of the Eurographics Conference on Rendering Techniques (EGSR'06). 273--284.

Digital Library

[40]

I. Motoyoshi, S. Nishida, L. Sharan, and E. H. Adelson. 2007. Image statistics and the perception of surface qualities. Nature 447, 206--209.

[41]

Ngan A, F. Durand, and W. Matusik. 2006. Image-driven navigation of analytical BRDF models. In Proceedings of the Eurographics Conference on Rendering Techniques (EGSR'06). 399--407.

Digital Library

[42]

F. Osu. 2012. How to answer “behavior based interview questions” - Interview tip. https:///www.youtube.com/watch?v=qKBubK0-798.

[43]

S. Paris, S. W. Hasinoff, and J. Kautz. 2011. Local Laplacian filters: Edge-aware image processing with a Laplacian pyramid. ACM Trans. Graph. 30, 4.

Digital Library

[44]

G. Patterson, C. Xu, H. Su, and J. Hays. 2014. The sun attribute database: Beyond categories for deeper scene understanding. Int. J. Comput. Vis. 108, 1--2, 59--81.

Digital Library

[45]

J. Portilla and E. P. Simoncelli. 2000. A parametric texture model based on joint statistics of complex wavelet coefficients. Int. J. Comput. Vis. 40, 1, 49--70.

Digital Library

[46]

L. Sharan, Y. Li, I. Motoyoshi, S. Nishida, and E. H. Adelson. 2008. Image statistics for surface reflectance perception. J. Optic. Soc. Amer. A25, 4.

[47]

L. Sharan, C. Liu, R. Rosenholtz, and E. H. Adelson. 2013. Recognizing materials using perceptually inspired features. Int. J. Comput. Vis. 103, 3, 348--371.

[48]

E. P. Simoncelli and E. H. Adelson. 1996. Noise removal via Bayesian wavelet coring. In Proceedings of the IEEE International Conference on Image Processing (ICIP'96). 379--382.

[49]

E. Stochter. 2013. Gargoyle. https://pixabay.com/en/gargoyle-fountain-lion-lion-head-205999/.

[50]

C. Tomasi and R. Manduchi. 1998. Bilateral filtering for gray and color images. In Proceedings of the IEEE International Conference on Computer Vision (ICCV'98).

Digital Library

[51]

M. Trentacoste, R. Mantuik, and W. Heidrich. 2011. Blur-aware image downsizing. Comput. Graph. Forum 30, 2, 573--582.

[52]

M. Trentacoste, R. Mantiuk, W. Heidrich, and F. Dufrot. 2012. Unsharp masking, countershading and halos: Enhancements or artifacts? Comput. Graph. Forum 30, 2, 573--582.

Digital Library

[53]

R. Vergne, P. Barla, R. Fleming, and X. Granier. 2012. Surface flows for image-based shading design. ACM Trans. Graph. 31, 3.

Digital Library

[54]

Videvo. 2014. Fresh grapes close up. http://www.videvo.net/video/fresh-grapes-close-up/2350.

[55]

J. Von Der Pahlen, J. Jimenez, E. Danvoye, P. Devevec, G. Fyffe, and O. Alexander. 2014. Digital IRA and beyond: Creating real-time photoreal digital actors. In ACM SIGGRAPH Courses. ACM Press, New York, 1:1--1:384.

Digital Library

[56]

G. J. Ward. 1992. Measuring and modeling anisotropic reflection. ACM Trans. Comput. Graph. 26, 2, 265--272.

Digital Library

[57]

L. West. 2006. Onion. https://www.flickr.com/photos/calliope/104314753/

[58]

L. Xu, C. Lu, Y. Xu, and J. Jia. 2011. Image smoothing via L0 gradient minimization. ACM Trans. Graph. 30, 6.

Digital Library

[59]

L. Xu, Q. Yan, Y. Xia, and J. Jia. 2012. Structure extraction from texture via relative total variation. ACM Trans. Graph. 31, 6.

Digital Library

Cited By

Stenvers VVangorp P(2024)Image-Based Material Editing Using Perceptual Attributes or Ground-Truth ParametersProceedings of 21st ACM SIGGRAPH Conference on Visual Media Production10.1145/3697294.3697301(1-9)Online publication date: 18-Nov-2024
https://dl.acm.org/doi/10.1145/3697294.3697301
Sharma PJampani VLi YJia XLagun DDurand FFreeman BMatthews M(2024)Alchemist: Parametric Control of Material Properties with Diffusion Models2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)10.1109/CVPR52733.2024.02278(24130-24141)Online publication date: 16-Jun-2024
https://doi.org/10.1109/CVPR52733.2024.02278
Cheng JWang LZhang LGao FZhang J(2024)Single-image SVBRDF estimation with auto-adaptive high-frequency feature extractionComputers & Graphics10.1016/j.cag.2024.104103(104103)Online publication date: Oct-2024
https://doi.org/10.1016/j.cag.2024.104103
Show More Cited By

Index Terms

Band-Sifting Decomposition for Image-Based Material Editing
1. Computing methodologies
  1. Artificial intelligence
    1. Computer vision
  2. Computer graphics
    1. Image manipulation

Recommendations

The Undecimated Wavelet Decomposition and its Reconstruction

This paper describes the undecimated wavelet transform and its reconstruction. In the first part, we show the relation between two well known undecimated wavelet transforms, the standard undecimated wavelet transform and the isotropic undecimated ...
Image-based material editing
SIGGRAPH '06: ACM SIGGRAPH 2006 Papers

Photo editing software allows digital images to be blurred, warped or re-colored at the touch of a button. However, it is not currently possible to change the material appearance of an object except by painstakingly painting over the appropriate pixels. ...
Low-light image enhancement based on sharpening-smoothing image filter
Abstract
Low-light images suffer from poor visibility, severe noise, low contrast, and low brightness. To overcome these issues, many image enhancement methods have been proposed. Few techniques solve these problems simultaneously. This paper presents a ...

Comments

Information & Contributors

Information

Published In

cover image ACM Transactions on Graphics

ACM Transactions on Graphics Volume 34, Issue 5

October 2015

188 pages

ISSN:0730-0301

EISSN:1557-7368

DOI:10.1145/2843519

Editor:
Kavita Bala
Cornell University

Issue’s Table of Contents

Copyright © 2015 Owner/Author.

Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for third-party components of this work must be honored. For all other uses, contact the Owner/Author.

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 03 November 2015

Accepted: 01 July 2015

Revised: 01 June 2015

Received: 01 December 2014

Published in TOG Volume 34, Issue 5

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

43
Total Citations
View Citations
1,425
Total Downloads

Downloads (Last 12 months)230
Downloads (Last 6 weeks)19

Reflects downloads up to 20 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Stenvers VVangorp P(2024)Image-Based Material Editing Using Perceptual Attributes or Ground-Truth ParametersProceedings of 21st ACM SIGGRAPH Conference on Visual Media Production10.1145/3697294.3697301(1-9)Online publication date: 18-Nov-2024
https://dl.acm.org/doi/10.1145/3697294.3697301
Sharma PJampani VLi YJia XLagun DDurand FFreeman BMatthews M(2024)Alchemist: Parametric Control of Material Properties with Diffusion Models2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)10.1109/CVPR52733.2024.02278(24130-24141)Online publication date: 16-Jun-2024
https://doi.org/10.1109/CVPR52733.2024.02278
Cheng JWang LZhang LGao FZhang J(2024)Single-image SVBRDF estimation with auto-adaptive high-frequency feature extractionComputers & Graphics10.1016/j.cag.2024.104103(104103)Online publication date: Oct-2024
https://doi.org/10.1016/j.cag.2024.104103
Tsuda HKawabata H(2023)materialmodifier: An R package of photo editing effects for material perception researchBehavior Research Methods10.3758/s13428-023-02116-256:3(2657-2674)Online publication date: 10-May-2023
https://doi.org/10.3758/s13428-023-02116-2
Morimoto TAkbarinia AStorrs KCheeseman JSmithson HGegenfurtner KFleming R(2023)Color and gloss constancy under diverse lighting environmentsJournal of Vision10.1167/jov.23.7.823:7(8)Online publication date: 11-Jul-2023
https://doi.org/10.1167/jov.23.7.8
Gokbudak FOztireli A(2023)One-shot Detail Retouching with Patch Space Neural Transformation BlendingProceedings of the 20th ACM SIGGRAPH European Conference on Visual Media Production10.1145/3626495.3626499(1-10)Online publication date: 30-Nov-2023
https://dl.acm.org/doi/10.1145/3626495.3626499
Chao CKlein JTan JEchevarria JGingold Y(2023)ColorfulCurves: Palette-Aware Lightness Control and Color Editing via Sparse OptimizationACM Transactions on Graphics10.1145/359240542:4(1-12)Online publication date: 26-Jul-2023
https://dl.acm.org/doi/10.1145/3592405
Deschaintre VGuerrero-Viu JGutierrez DBoubekeur TMasia B(2023)The Visual Language of FabricsACM Transactions on Graphics10.1145/359239142:4(1-15)Online publication date: 26-Jul-2023
https://dl.acm.org/doi/10.1145/3592391
Subias JLagunas M(2023)In‐the‐wild Material Appearance Editing using Perceptual AttributesComputer Graphics Forum10.1111/cgf.1476542:2(333-345)Online publication date: 23-May-2023
https://doi.org/10.1111/cgf.14765
Shi CJiang ZMa XLuo Q(2022)A Personalized Visual Aid for Selections of Appearance Building Products with Long-term EffectsProceedings of the 2022 CHI Conference on Human Factors in Computing Systems10.1145/3491102.3517659(1-18)Online publication date: 29-Apr-2022
https://dl.acm.org/doi/10.1145/3491102.3517659
Show More Cited By

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

Media

Figures

Other

Tables

View Issue’s Table of Contents