research-article

Multi-scale image harmonization

Authors:

Kalyan Sunkavalli,

Micah K. Johnson,

Wojciech Matusik,

Hanspeter PfisterAuthors Info & Claims

ACM Transactions on Graphics (TOG), Volume 29, Issue 4

Article No.: 125, Pages 1 - 10

https://doi.org/10.1145/1778765.1778862

Published: 26 July 2010 Publication History

Abstract

Traditional image compositing techniques, such as alpha matting and gradient domain compositing, are used to create composites that have plausible boundaries. But when applied to images taken from different sources or shot under different conditions, these techniques can produce unrealistic results. In this work, we present a framework that explicitly matches the visual appearance of images through a process we call image harmonization, before blending them. At the heart of this framework is a multi-scale technique that allows us to transfer the appearance of one image to another. We show that by carefully manipulating the scales of a pyramid decomposition of an image, we can match contrast, texture, noise, and blur, while avoiding image artifacts. The output composite can then be reconstructed from the modified pyramid coefficients while enforcing both alpha-based and seamless boundary constraints. We show how the proposed framework can be used to produce realistic composites with minimal user interaction in a number of different scenarios.

Supplementary Material

Supplemental material. (125.zip)

The auxillary material consists of a PDF document with larger versions of the results in the paper.

Download
36.05 MB

References

[1]

Agarwala, A., Dontcheva, M., Agrawala, M., Drucker, S., Colburn, A., Curless, B., Salesin, D., and Cohen, M. 2004. Interactive digital photomontage. ACM Transactions on Graphics 23, 3, 294--302.

Digital Library

[2]

Agarwala, A. 2007. Efficient gradient-domain compositing using quadtrees. ACM Transactions on Graphics 26, 3, 94.

Digital Library

[3]

Bae, S., Paris, S., and Durand, F. 2006. Two-scale tone management for photographic look. ACM Transactions on Graphics 25, 3, 637--645.

Digital Library

[4]

Burt, P. J., and Adelson, E. H. 1983. The Laplacian pyramid as a compact image code. IEEE Transactions on Communications COM-31, 4, 532--540.

[5]

Burt, P. J., and Adelson, E. H. 1983. A multiresolution spline with application to image mosaics. ACM Transactions on Graphics 2, 4, 217--236.

Digital Library

[6]

Chen, J., Tang, C.-K., and Wang, J. 2009. Noise brush: interactive high quality image-noise separation. ACM Transactions on Graphics 28, 5, 1--10.

Digital Library

[7]

Cohen-Or, D., Sorkine, O., Gal, R., Leyvand, T., and Xu, Y.-Q. 2006. Color harmonization. ACM Transactions on Graphics 25, 3, 624--630.

Digital Library

[8]

Davis, T. A. 2006. Direct Methods for Sparse Linear Systems (Fundamentals of Algorithms 2). Society for Industrial and Applied Mathematics, Philadelphia, PA, USA.

Digital Library

[9]

Farbman, Z., Fattal, R., Lischinski, D., and Szeliski, R. 2008. Edge-preserving decompositions for multi-scale tone and detail manipulation. ACM Transactions on Graphics 27, 3, 1--10.

Digital Library

[10]

Farbman, Z., Hoffer, G., Lipman, Y., Cohen-Or, D., and Lischinski, D. 2009. Coordinates for instant image cloning. ACM Transactions on Graphics 28, 3, 1--9.

Digital Library

[11]

Fattal, R., Agrawala, M., and Rusinkiewicz, S. 2007. Multiscale shape and detail enhancement from multi-light image collections. ACM Transactions on Graphics 26, 3, 51.

Digital Library

[12]

Georgiev, T. 2004. Photoshop healing brush: a tool for seamless cloning. In Workshop on Applications of Computer Vision (ECCV 2004), 1--8.

[13]

Gonzalez, R. C., and Woods, R. E. 2001. Digital Image Processing. Addison-Wesley Longman Publishing Co., Inc., Boston, MA, USA.

Digital Library

[14]

Heeger, D. J., and Bergen, J. R. 1995. Pyramid-based texture analysis/synthesis. SIGGRAPH '95: Proceedings of the 22nd annual conference on Computer graphics and interactive techniques, 229--238.

Digital Library

[15]

Jia, J., Sun, J., Tang, C.-K., and Shum, H.-Y. 2006. Drag-and-drop pasting. ACM Transactions on Graphics 25, 3, 631--637.

Digital Library

[16]

Lalonde, J.-F., and Efros, A. A. 2007. Using color compatibility for assessing image realism. In IEEE International Conference on Computer Vision.

[17]

Levin, A., Zomet, A., Peleg, S., and Weiss, Y. 2004. Seamless image stitching in the gradient domain. In European Conference on Computer Vision.

[18]

Li, Y., Sharan, L., and Adelson, E. H. 2005. Compressing and companding high dynamic range images with subband architectures. ACM Transactions on Graphics 24, 3, 836--844.

Digital Library

[19]

McCann, J., and Pollard, N. S. 2008. Real-time gradient-domain painting. ACM Transactions on Graphics 27, 3, 1--7.

Digital Library

[20]

Pérez, P., Gangnet, M., and Blake, A. 2003. Poisson image editing. ACM Transactions on Graphics 22, 3, 313--318.

Digital Library

[21]

Pitie, F., Kokaram, A. C., and Dahyot, R. 2005. N-dimensional probablility density function transfer and its application to colour transfer. In IEEE International Conference on Computer Vision.

Digital Library

[22]

Porter, T., and Duff, T. 1984. Compositing digital images. In SIGGRAPH '84: Proceedings of the 11th annual conference on Computer graphics and interactive techniques, 253--259.

Digital Library

[23]

Portilla, J., and Simoncelli, E. P. 2000. A parametric texture model based on joint statistics of complex wavelet coefficients. International Journal of Computer Vision 40, 1, 49--70.

Digital Library

[24]

Reinhard, E., Ashikhmin, M., Gooch, B., and Shirley, P. 2001. Color transfer between images. IEEE Computer Graphics and Applications 21, 5, 34--41.

Digital Library

[25]

Sun, J., Jia, J., Tang, C.-K., and Shum, H.-Y. 2004. Poisson matting. ACM Transactions on Graphics 23, 3, 315--321.

Digital Library

[26]

Szeliski, R. 2006. Locally adapted hierarchical basis preconditioning. ACM Transactions on Graphics 25, 3, 1135--1143.

Digital Library

[27]

Tomasi, C., and Manduchi, R. 1998. Bilateral filtering for gray and color images. In IEEE International Conference on Computer Vision.

Digital Library

[28]

Wang, J., Agrawala, M., and Cohen, M. F. 2007. Soft scissors: an interactive tool for realtime high quality matting. ACM Transactions on Graphics 26, 3, 9.

Digital Library

Cited By

Yuan JWu HXie LZhang LXing J(2025)Learning multi-color curve for image harmonizationEngineering Applications of Artificial Intelligence10.1016/j.engappai.2025.110277146(110277)Online publication date: Apr-2025
https://doi.org/10.1016/j.engappai.2025.110277
Zhou PFeng FLiu GLi RWang XCai JKankanhalli MPrabhakaran BBoll SSubramanian RZheng LSingh VCesar PXie LXu D(2024)DiffHarmony++: Enhancing Image Harmonization with Harmony-VAE and Inverse Harmonization ModelProceedings of the 32nd ACM International Conference on Multimedia10.1145/3664647.3681466(10592-10601)Online publication date: 28-Oct-2024
https://dl.acm.org/doi/10.1145/3664647.3681466
Zhou JYu ZBao ZFu GHe WLiang CXiao CCai JKankanhalli MPrabhakaran BBoll SSubramanian RZheng LSingh VCesar PXie LXu D(2024)Foreground Harmonization and Shadow Generation for Composite ImageProceedings of the 32nd ACM International Conference on Multimedia10.1145/3664647.3681355(8267-8276)Online publication date: 28-Oct-2024
https://dl.acm.org/doi/10.1145/3664647.3681355
Show More Cited By

Index Terms

Multi-scale image harmonization
1. Computing methodologies
  1. Artificial intelligence
    1. Computer vision
  2. Computer graphics
2. Human-centered computing
  1. Human computer interaction (HCI)
    1. Interaction devices
      1. Graphics input devices

Recommendations

Drag-and-drop pasting

In this paper, we present a user-friendly system for seamless image composition, which we call drag-and-drop pasting. We observe that for Poisson image editing [Perez et al. 2003] to work well, the user must carefully draw a boundary on the source image ...
Multi-scale image harmonization
SIGGRAPH '10: ACM SIGGRAPH 2010 papers

Traditional image compositing techniques, such as alpha matting and gradient domain compositing, are used to create composites that have plausible boundaries. But when applied to images taken from different sources or shot under different conditions, ...
Nonlinear multiresolution image blending

We study contrast enhancement for multiresolution image blending. In image compositing, image stitching, and image fusion, a blending operator combines coefficients of a pixel array, an image pyramid, a wavelet decomposition, or a gradient domain ...

Comments

Information & Contributors

Information

Published In

cover image ACM Transactions on Graphics

ACM Transactions on Graphics Volume 29, Issue 4

July 2010

942 pages

ISSN:0730-0301

EISSN:1557-7368

DOI:10.1145/1778765

Issue’s Table of Contents

Copyright © 2010 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 26 July 2010

Published in TOG Volume 29, Issue 4

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

National Science Foundation

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

155
Total Citations
View Citations
2,339
Total Downloads

Downloads (Last 12 months)54
Downloads (Last 6 weeks)3

Reflects downloads up to 19 Feb 2025

Other Metrics

View Author Metrics

Citations

Cited By

Yuan JWu HXie LZhang LXing J(2025)Learning multi-color curve for image harmonizationEngineering Applications of Artificial Intelligence10.1016/j.engappai.2025.110277146(110277)Online publication date: Apr-2025
https://doi.org/10.1016/j.engappai.2025.110277
Zhou PFeng FLiu GLi RWang XCai JKankanhalli MPrabhakaran BBoll SSubramanian RZheng LSingh VCesar PXie LXu D(2024)DiffHarmony++: Enhancing Image Harmonization with Harmony-VAE and Inverse Harmonization ModelProceedings of the 32nd ACM International Conference on Multimedia10.1145/3664647.3681466(10592-10601)Online publication date: 28-Oct-2024
https://dl.acm.org/doi/10.1145/3664647.3681466
Zhou JYu ZBao ZFu GHe WLiang CXiao CCai JKankanhalli MPrabhakaran BBoll SSubramanian RZheng LSingh VCesar PXie LXu D(2024)Foreground Harmonization and Shadow Generation for Composite ImageProceedings of the 32nd ACM International Conference on Multimedia10.1145/3664647.3681355(8267-8276)Online publication date: 28-Oct-2024
https://dl.acm.org/doi/10.1145/3664647.3681355
Yu ZZhou JBao ZFu GHe WLiang CXiao CCai JKankanhalli MPrabhakaran BBoll SSubramanian RZheng LSingh VCesar PXie LXu D(2024)CFDiffusion: Controllable Foreground Relighting in Image Compositing via Diffusion ModelProceedings of the 32nd ACM International Conference on Multimedia10.1145/3664647.3681283(3647-3656)Online publication date: 28-Oct-2024
https://dl.acm.org/doi/10.1145/3664647.3681283
Pham KChen JChen QCai JKankanhalli MPrabhakaran BBoll SSubramanian RZheng LSingh VCesar PXie LXu D(2024)TALE: Training-free Cross-domain Image Composition via Adaptive Latent Manipulation and Energy-guided OptimizationProceedings of the 32nd ACM International Conference on Multimedia10.1145/3664647.3681079(3160-3169)Online publication date: 28-Oct-2024
https://dl.acm.org/doi/10.1145/3664647.3681079
Li YJiang JYang XDing YZhang JCai JKankanhalli MPrabhakaran BBoll SSubramanian RZheng LSingh VCesar PXie LXu D(2024)Harmony Everything! Masked Autoencoders for Video HarmonizationProceedings of the 32nd ACM International Conference on Multimedia10.1145/3664647.3680800(8536-8545)Online publication date: 28-Oct-2024
https://dl.acm.org/doi/10.1145/3664647.3680800
Chang YLin T(2024)Subjective assessment for inverse rendered composite images in 360-deg imagesJournal of Electronic Imaging10.1117/1.JEI.33.1.01303733:01Online publication date: 1-Jan-2024
https://doi.org/10.1117/1.JEI.33.1.013037
Wang XNiu LCao JHong YZhang L(2024)Painterly Image Harmonization via Adversarial Residual Learning2024 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV)10.1109/WACV57701.2024.00506(5129-5138)Online publication date: 3-Jan-2024
https://doi.org/10.1109/WACV57701.2024.00506
Xiao ZZhu YFu XXiong Z(2024) TSA 2 : Temporal Segment Adaptation and Aggregation for Video Harmonization 2024 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV)10.1109/WACV57701.2024.00409(4124-4133)Online publication date: 3-Jan-2024
https://doi.org/10.1109/WACV57701.2024.00409
Weng SShi B(2024)Conditional Image RepaintingIEEE Transactions on Pattern Analysis and Machine Intelligence10.1109/TPAMI.2023.333127546:4(2285-2298)Online publication date: Apr-2024
https://doi.org/10.1109/TPAMI.2023.3331275
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Issue’s Table of Contents