Article

Image denoising and inpainting with deep neural networks

Authors:

Enhong ChenAuthors Info & Claims

NIPS'12: Proceedings of the 26th International Conference on Neural Information Processing Systems - Volume 1

Pages 341 - 349

Published: 03 December 2012 Publication History

Abstract

We present a novel approach to low-level vision problems that combines sparse coding and deep networks pre-trained with denoising auto-encoder (DA). We propose an alternative training scheme that successfully adapts DA, originally designed for unsupervised feature learning, to the tasks of image denoising and blind inpainting. Our method's performance in the image denoising task is comparable to that of KSVD which is a widely used sparse coding technique. More importantly, in blind image inpainting task, the proposed method provides solutions to some complex problems that have not been tackled before. Specifically, we can automatically remove complex patterns like superimposed text from an image, rather than simple patterns like pixels missing at random. Moreover, the proposed method does not need the information regarding the region that requires inpainting to be given a priori. Experimental results demonstrate the effectiveness of the proposed method in the tasks of image denoising and blind inpainting. We also show that our new training scheme for DA is more effective and can improve the performance of unsupervised feature learning.

References

[1]

J. Xu, K. Zhang, M. Xu, and Z. Zhou. An adaptive threshold method for image denoising based on wavelet domain. Proceedings of SPIE, the International Society for Optical Engineering, 7495:165, 2009.

[2]

J. Portilla, V. Strela, M.J. Wainwright, and E.P. Simoncelli. Image denoising using scale mixtures of Gaussians in the wavelet domain. Image Processing, IEEE Transactions on, 12(11):1338-1351, 2003.

Digital Library

[3]

F. Luisier, T. Blu, and M. Unser. A new SURE approach to image denoising: Interscale orthonormal wavelet thresholding. IEEE Transactions on Image Processing, 16(3):593-606, 2007.

Digital Library

[4]

B.A. Olshausen and D.J. Field. Sparse coding with an overcomplete basis set: A strategy employed by V1? Vision research, 37(23):3311-3325, 1997.

[5]

K. Kreutz-Delgado, J.F. Murray, B.D. Rao, K. Engan, T.W. Lee, and T.J. Sejnowski. Dictionary learning algorithms for sparse representation. Neural computation, 15(2):349-396, 2003.

Digital Library

[6]

M. Elad and M. Aharon. Image denoising via sparse and redundant representations over learned dictionaries. IEEE Transactions on Image Processing, 15(12):3736-3745, 2006.

Digital Library

[7]

J. Mairal, M. Elad, and G. Sapiro. Sparse representation for color image restoration. IEEE Transactions on Image Processing, 17(1):53-69, 2008.

Digital Library

[8]

X. Lu, H. Yuan, P. Yan, Y. Yuan, L. Li, and X. Li. Image denoising via improved sparse coding. Proceedings of the British Machine Vision Conference, pages 74-1, 2011.

[9]

J. Mairal, F. Bach, J. Ponce, and G. Sapiro. Online dictionary learning for sparse coding. Proceedings of the 26th Annual International Conference on Machine Learning, pages 689-696, 2009.

Digital Library

[10]

A. Criminisi, P. Pérez, and K. Toyama. Region filling and object removal by exemplar-based image inpainting. IEEE Transactions on Image Processing, 13(9):1200-1212, 2004.

Digital Library

[11]

M. Bertalmio, G. Sapiro, V. Caselles, and C. Ballester. Image inpainting. Proceedings of the 27th annual conference on Computer graphics and interactive techniques, pages 417-424, 2000.

Digital Library

[12]

A. Telea. An image inpainting technique based on the fast marching method. Journal of graphics tools., 9(1):23-34, 2004.

[13]

B. Dong, H. Ji, J. Li, Z. Shen, and Y. Xu. Wavelet frame based blind image inpainting. Applied and Computational Harmonic Analysis, 2011.

[14]

Y. Wang, A. Szlam, and G. Lerman. Robust locally linear analysis with applications to image denoising and blind inpainting. preprint, 2011.

[15]

M. Yan. Restoration of images corrupted by impulse noise using blind inpainting and l0 norm. preprint, 2011.

[16]

V. Jain and H.S. Seung. Natural image denoising with convolutional networks. Advances in Neural Information Processing Systems, 21:769-776, 2008.

Digital Library

[17]

H. Lee, C. Ekanadham, and A. Ng. Sparse deep belief net model for visual area V2. Advances in Neural Information Processing Systems 20, pages 873-880, 2008.

Digital Library

[18]

D. Erhan, Y. Bengio, A. Courville, P.A. Manzagol, P. Vincent, and S. Bengio. Why does unsupervised pre-training help deep learning? The Journal of Machine Learning Research, 11:625-660, 2010.

Digital Library

[19]

Y. Bengio. Learning deep architectures for AI. Foundations and Trends° in Machine Learning, 2(1):1-127, 2009.

Digital Library

[20]

R. Salakhutdinov and G.E. Hinton. Deep boltzmann machines. Proceedings of the international conference on artificial intelligence and statistics, 5(2):448-455, 2009.

[21]

P. Vincent, H. Larochelle, I. Lajoie, Y Bengio, and PA. Manzagol. Stacked denoising autoencoders: Learning useful representations in a deep network with a local denoising criterion. The Journal of Machine Learning Research, 11:3371-3408, 2010.

Digital Library

[22]

Q.V. Le, A. Coates, B. Prochnow, and A.Y. Ng. On optimization methods for deep learning. Learning, pages 265-272, 2011.

[23]

S. Roth and M.J. Adviser-Black. High-order markov random fields for low-level vision. Brown University Press, 2007.

Cited By

Peng G(2024)Learning the sparse priorWiley Interdisciplinary Reviews 10.1002/wics.164616:1Online publication date: 15-Feb-2024
https://dl.acm.org/doi/10.1002/wics.1646
Yin XHuang DFu ZWang YChen L(2023)Segmentation-Reconstruction-Guided Facial Image De-occlusion2023 IEEE 17th International Conference on Automatic Face and Gesture Recognition (FG)10.1109/FG57933.2023.10042570(1-8)Online publication date: 5-Jan-2023
https://dl.acm.org/doi/10.1109/FG57933.2023.10042570
Kuchida AGoto T(2022)Image Quality Improvement of Surveillance Camera Images by Learning-based Denoising Method Utilizing Noise2NoiseProceedings of the 2022 5th International Conference on Digital Medicine and Image Processing10.1145/3576938.3576943(24-30)Online publication date: 10-Nov-2022
https://dl.acm.org/doi/10.1145/3576938.3576943
Show More Cited By

Index Terms

Image denoising and inpainting with deep neural networks
1. Computing methodologies

Index terms have been assigned to the content through auto-classification.

Recommendations

A deep cascade of neural networks for image inpainting, deblurring and denoising

In recent years, we have witnessed the great success of deep learning on various problems both in low and high-level computer visions. The low-level vision problems, including inpainting, deblurring, denoising, super-resolution, and so on, are highly ...
Image inpainting based on deep learning: A review
Abstract
Image inpainting is an important research direction in the study of computer vision, and is widely used in image editing and photo inpainting etc. Traditional image inpainting algorithms are often difficult to deal with large-scale image deletion,...
Highlights
- Reviewing the image inpainting on deep learning of the past 15 years.
- Summarizing the prominent deep learning methods especially in feature fusion.
- Pointing out the problem and difficulty in image inpainting task.
- Providing ...
Deep Neural Network Compression for Image Inpainting
Computer Vision – ECCV 2022 Workshops
Abstract
Image inpainting techniques have recently been developed leveraging deep neural networks and have seen many real-world applications. However, image inpainting networks, which are typically based on generative adversarial network (GAN), suffer from ...

Comments

Information & Contributors

Information

Published In

cover image Guide Proceedings

NIPS'12: Proceedings of the 26th International Conference on Neural Information Processing Systems - Volume 1

December 2012

3328 pages

Publisher

Curran Associates Inc.

Red Hook, NY, United States

Publication History

Published: 03 December 2012

Qualifiers

Article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

60
Total Citations
View Citations
4
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 03 Feb 2025

Other Metrics

View Author Metrics

Citations

Cited By

Peng G(2024)Learning the sparse priorWiley Interdisciplinary Reviews 10.1002/wics.164616:1Online publication date: 15-Feb-2024
https://dl.acm.org/doi/10.1002/wics.1646
Yin XHuang DFu ZWang YChen L(2023)Segmentation-Reconstruction-Guided Facial Image De-occlusion2023 IEEE 17th International Conference on Automatic Face and Gesture Recognition (FG)10.1109/FG57933.2023.10042570(1-8)Online publication date: 5-Jan-2023
https://dl.acm.org/doi/10.1109/FG57933.2023.10042570
Kuchida AGoto T(2022)Image Quality Improvement of Surveillance Camera Images by Learning-based Denoising Method Utilizing Noise2NoiseProceedings of the 2022 5th International Conference on Digital Medicine and Image Processing10.1145/3576938.3576943(24-30)Online publication date: 10-Nov-2022
https://dl.acm.org/doi/10.1145/3576938.3576943
Saxena NRaman B(2022)Pansharpening Scheme Using Bi-dimensional Empirical Mode Decomposition and Neural NetworkACM Transactions on Multimedia Computing, Communications, and Applications10.1145/350670918:4(1-22)Online publication date: 4-Mar-2022
https://dl.acm.org/doi/10.1145/3506709
Shen YFu HDu ZChen XBurnaev EZorin DZhou KZheng Y(2022)GCN-Denoiser: Mesh Denoising with Graph Convolutional NetworksACM Transactions on Graphics10.1145/348016841:1(1-14)Online publication date: 9-Feb-2022
https://dl.acm.org/doi/10.1145/3480168
Waeldchen SMacdonald JHauch SKutyniok G(2021)The Computational Complexity of Understanding Binary Classifier DecisionsJournal of Artificial Intelligence Research10.1613/jair.1.1235970(351-387)Online publication date: 1-May-2021
https://dl.acm.org/doi/10.1613/jair.1.12359
Luo XZhang XYoo PMartin-Brualla RLawrence JSeitz S(2021)Time-travel rephotographyACM Transactions on Graphics10.1145/3478513.348048540:6(1-12)Online publication date: 10-Dec-2021
https://dl.acm.org/doi/10.1145/3478513.3480485
Li YYi YLiu DLi LLi ZLi H(2021)Neural-Network-Based Cross-Channel Intra PredictionACM Transactions on Multimedia Computing, Communications, and Applications10.1145/343425017:3(1-23)Online publication date: 22-Jul-2021
https://dl.acm.org/doi/10.1145/3434250
Maiti SKumar AJain SBhatnagar G(2021)A Novel Image Inpainting Framework Using RegressionACM Transactions on Internet Technology10.1145/340217721:3(1-16)Online publication date: 16-Jun-2021
https://dl.acm.org/doi/10.1145/3402177
Fang SNirjon SValois FJulien CMurphy AGnawali O(2020)SuperRF: Enhanced 3D RF Representation Using Stationary Low-Cost mmWave RadarProceedings of the 2020 International Conference on Embedded Wireless Systems and Networks10.5555/3400306.3400321(120-131)Online publication date: 17-Feb-2020
https://dl.acm.org/doi/10.5555/3400306.3400321
Show More Cited By

View Options

View options

Figures

Tables

Media

View Table of Conten