research-article

Deep joint demosaicking and denoising

Authors:

Michaël Gharbi,

Gaurav Chaurasia,

Frédo DurandAuthors Info & Claims

ACM Transactions on Graphics (TOG), Volume 35, Issue 6

Article No.: 191, Pages 1 - 12

https://doi.org/10.1145/2980179.2982399

Published: 05 December 2016 Publication History

Abstract

Demosaicking and denoising are the key first stages of the digital imaging pipeline but they are also a severely ill-posed problem that infers three color values per pixel from a single noisy measurement. Earlier methods rely on hand-crafted filters or priors and still exhibit disturbing visual artifacts in hard cases such as moiré or thin edges. We introduce a new data-driven approach for these challenges: we train a deep neural network on a large corpus of images instead of using hand-tuned filters. While deep learning has shown great success, its naive application using existing training datasets does not give satisfactory results for our problem because these datasets lack hard cases. To create a better training set, we present metrics to identify difficult patches and techniques for mining community photographs for such patches. Our experiments show that this network and training procedure outperform state-of-the-art both on noisy and noise-free data. Furthermore, our algorithm is an order of magnitude faster than the previous best performing techniques.

References

[1]

Akiyama, H., Tanaka, M., and Okutomi, M. 2015. Pseudo four-channel image denoising for noisy cfa raw data. In Image Processing (ICIP), 2015 IEEE International Conference on, 4778--4782.

[2]

Badrinarayanan, V., Handa, A., and Cipolla, R. 2015. Segnet: A deep convolutional encoder-decoder architecture for robust semantic pixel-wise labelling. arXiv preprint arXiv:1505.07293.

[3]

Bengio, Y., Louradour, J., Collobert, R., and Weston, J. 2009. Curriculum learning. In Proceedings of the 26th Annual International Conference on Machine Learning, ACM, New York, NY, USA, ICML '09, 41--48.

Digital Library

[4]

Buades, A., Coll, B., Morel, J.-M., and Sbert, C. 2009. Self-similarity driven color demosaicking. Image Processing, IEEE Transactions on 18, 6, 1192--1202.

Digital Library

[5]

Burger, H., Schuler, C., and Harmeling, S. 2012. Image denoising: Can plain neural networks compete with BM3D? In Computer Vision and Pattern Recognition (CVPR), 2012 IEEE Conference on, 2392--2399.

Digital Library

[6]

Bychkovsky, V., Paris, S., Chan, E., and Durand, F. 2011. Learning photographic global tonal adjustment with a database of input / output image pairs. In The Twenty-Fourth IEEE Conference on Computer Vision and Pattern Recognition.

Digital Library

[7]

Chang, L., and Tan, Y.-P. 2004. Effective use of spatial and spectral correlations for color filter array demosaicking. Consumer Electronics, IEEE Transactions on 50, 1, 355--365.

Digital Library

[8]

Cheng, Z., Yang, Q., and Sheng, B. 2015. Deep colorization. In Proceedings of the IEEE International Conference on Computer Vision, 415--423.

Digital Library

[9]

Cok, D. R., 1987. Signal processing method and apparatus for producing interpolated chrominance values in a sampled color image signal, Feb. 10. US Patent 4,642,678.

[10]

Condat, L., and Mosaddegh, S. 2012. Joint demosaicking and denoising by total variation minimization. In Image Processing (ICIP), 2012 19th IEEE International Conference on, IEEE, 2781--2784.

[11]

Condat, L. 2011. A new color filter array with optimal properties for noiseless and noisy color image acquisition. Image Processing, IEEE Transactions on 20, 8, 2200--2210.

Digital Library

[12]

Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., and FeiFei, L. 2009. Imagenet: A large-scale hierarchical image database. In Computer Vision and Pattern Recognition, 2009. CVPR 2009. IEEE Conference on, IEEE, 248--255.

[13]

Dosovitskiy, A., Fischer, P., Ilg, E., Hausser, P., Hazirbas, C., Golkov, V., v.d. Smagt, P., Cremers, D., and Brox, T. 2015. Flownet: Learning optical flow with convolutional networks. In IEEE International Conference on Computer Vision (ICCV).

Digital Library

[14]

Dosovitskiy, A., Tobias Springenberg, J., and Brox, T. 2015. Learning to generate chairs with convolutional neural networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 1538--1546.

[15]

Eigen, D., Krishnan, D., and Fergus, R. 2013. Restoring an image taken through a window covered with dirt or rain. In IEEE International Conference on Computer Vision, ICCV 2013, Sydney, Australia, December 1-8, 2013, 633--640.

Digital Library

[16]

Eigen, D., Puhrsch, C., and Fergus, R. 2014. Depth map prediction from a single image using a multi-scale deep network. In Advances in Neural Information Processing Systems 27, Z. Ghahramani, M. Welling, C. Cortes, N. D. Lawrence, and K. Q. Weinberger, Eds. Curran Associates, Inc., 2366--2374.

Digital Library

[17]

Flynn, J., Neulander, I., Philbin, J., and Snavely, N. 2016. Deepstereo: Learning to predict new views from the world's imagery. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[18]

Foi, A., Trimeche, M., Katkovnik, V., and Egiazarian, K. 2008. Practical poissonian-gaussian noise modeling and fitting for single-image raw-data. Image Processing, IEEE Transactions on 17, 10, 1737--1754.

Digital Library

[19]

Gatys, L. A., Ecker, A. S., and Bethge, M. 2016. Image style transfer using convolutional neural networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2414--2423.

[20]

Getreuer, P. 2011. Color demosaicing with contour stencils. In Digital Signal Processing (DSP), 2011 17th International Conference on, IEEE, 1--6.

[21]

Go, J., Sohn, K., and Lee, C. 2000. Interpolation using neural networks for digital still cameras. Consumer Electronics, IEEE Transactions on 46, 3, 610--616.

Digital Library

[22]

Gunturk, B. K., Altunbasak, Y., and Mersereau, R. M. 2002. Color plane interpolation using alternating projections. Image Processing, IEEE Transactions on 11, 9, 997--1013.

Digital Library

[23]

Hamilton Jr, J. F., and Adams Jr, J. E., 1997. Apparatus for utilizing a digitized image signal, May 13. US Patent 5,629,734.

[24]

Hasinoff, S. W., Durand, F., and Freeman, W. T. 2010. Noise-optimal capture for high dynamic range photography. In Computer Vision and Pattern Recognition (CVPR), 2010 IEEE Conference on, IEEE, 553--560.

[25]

He, F.-L., Wang, Y.-C. F., and Hua, K.-L. 2012. Self-learning approach to color demosaicking via support vector regression. In Image Processing (ICIP), 2012 19th IEEE International Conference on, IEEE, 2765--2768.

[26]

He, K., Zhang, X., Ren, S., and Sun, J. 2015. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In The IEEE International Conference on Computer Vision (ICCV).

Digital Library

[27]

He, K., Zhang, X., Ren, S., and Sun, J. 2016. Deep residual learning for image recognition. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[28]

Heide, F., Steinberger, M., Tsai, Y.-T., Rouf, M., Pajkak, D., Reddy, D., Gallo, O., Liu, J., Heidrich, W., Egiazarian, K., et al. 2014. Flexisp: a flexible camera image processing framework. ACM Transactions on Graphics (TOG) 33, 6, 231.

Digital Library

[29]

Hirakawa, K., and Parks, T. W. 2005. Adaptive homogeneity-directed demosaicing algorithm. Image Processing, IEEE Transactions on 14, 3, 360--369.

Digital Library

[30]

Hirakawa, K., and Parks, T. W. 2006. Joint demosaicing and denoising. Image Processing, IEEE Transactions on 15, 8, 2146--2157.

Digital Library

[31]

Huiskes, M. J., and Lew, M. S. 2008. The mir flickr retrieval evaluation. In Proceedings of the 1st ACM international conference on Multimedia information retrieval, ACM, 39--43.

Digital Library

[32]

Iizuka, S., Simo-Serra, E., and Ishikawa, H. 2016. Let there be Color!: Joint End-to-end Learning of Global and Local Image Priors for Automatic Image Colorization with Simultaneous Classification. ACM Transactions on Graphics (Proc. of SIGGRAPH 2016) 35, 4.

Digital Library

[33]

Jeon, G., and Dubois, E. 2013. Demosaicking of noisy bayer-sampled color images with least-squares luma-chroma demultiplexing and noise level estimation. Image Processing, IEEE Transactions on 22, 1, 146--156.

Digital Library

[34]

Jia, Y., Shelhamer, E., Donahue, J., Karayev, S., Long, J., Girshick, R., Guadarrama, S., and Darrell, T. 2014. Caffe: Convolutional architecture for fast feature embedding. In Proceedings of the 22Nd ACM International Conference on Multimedia, ACM, New York, NY, USA, MM '14, 675--678.

Digital Library

[35]

Kapah, O., and Hel-Or, H. Z. 2000. Demosaicking using artificial neural networks. In Electronic Imaging, International Society for Optics and Photonics, 112--120.

[36]

Khashabi, D., Nowozin, S., Jancsary, J., and Fitzgibbon, A. W. 2014. Joint demosaicing and denoising via learned non-parametric random fields. Image Processing, IEEE Transactions on 23, 12, 4968--4981.

[37]

Kingma, D., and Ba, J. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980.

[38]

Klatzer, T., Hammernik, K., Knobelreiter, P., and Pock, T. 2016. Learning joint demosaicing and denoising based on sequential energy minimization. In 2016 IEEE International Conference on Computational Photography (ICCP), IEEE, 1--11.

[39]

Krizhevsky, A., Sutskever, I., and Hinton, G. E. 2012. Imagenet classification with deep convolutional neural networks. In Advances in neural information processing systems, 1097--1105.

Digital Library

[40]

Kwan, C., and Xiaolin, W. 2004. A classification approach to color demosaicking. In International Conference on Image Processing.

[41]

Lansel, S., and Wandell, B. 2011. Local linear learned image processing pipeline. In Imaging Systems and Applications, Optical Society of America, IMC3.

[42]

Laroche, C. A., and Prescott, M. A., 1994. Apparatus and method for adaptively interpolating a full color image utilizing chrominance gradients, Dec. 13. US Patent 5,373,322.

[43]

Larsson, G., Maire, M., and Shakhnarovich, G. 2016. Learning representations for automatic colorization. In European Conference on Computer Vision (ECCV).

[44]

LeCun, Y., Bengio, Y., and Hinton, G. 2015. Deep learning. Nature 521, 7553, 436--444.

[45]

Levin, A., Nadler, B., Durand, F., and Freeman, W. T. 2012. Patch complexity, finite pixel correlations and optimal denoising. In Computer Vision-ECCV 2012. Springer, 73--86.

Digital Library

[46]

Li, X., Gunturk, B., and Zhang, L. 2008. Image demosaicing: A systematic survey. In Electronic Imaging 2008, International Society for Optics and Photonics, 68221J--68221J.

[47]

Li, X. 2005. Demosaicing by successive approximation. Image Processing, IEEE Transactions on 14, 3, 370--379.

Digital Library

[48]

Long, J., Shelhamer, E., and Darrell, T. 2015. Fully convolutional networks for semantic segmentation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 3431--3440.

[49]

Lu, Y. M., Karzand, M., and Vetterli, M. 2010. De-mosaicking by alternating projections: theory and fast one-step implementation. Image Processing, IEEE Transactions on 19, 8, 2085--2098.

Digital Library

[50]

Mantiuk, R., Kim, K. J., Rempel, A. G., and Heidrich, W. 2011. HDR-VDP-2: a calibrated visual metric for visibility and quality predictions in all luminance conditions. In ACM Transactions on Graphics (TOG), vol. 30, ACM, 40.

Digital Library

[51]

Noh, H., Hong, S., and Han, B. 2015. Learning deconvolution network for semantic segmentation. CoRR abs/1505.04366.

[52]

Park, S. H., Kim, H. S., Lansel, S., Parmar, M., and Wandell, B. A. 2009. A case for denoising before demosaicking color filter array data. In Signals, Systems and Computers, 2009 Conference Record of the Forty-Third Asilomar Conference on, IEEE, 860--864.

Digital Library

[53]

Pathak, D., Krähenbühl, P., Donahue, J., Darrell, T., and Efros, A. 2016. Context encoders: Feature learning by inpainting.

[54]

Ragan-Kelley, J., Barnes, C., Adams, A., Paris, S., Durand, F., and Amarasinghe, S. 2013. Halide: A language and compiler for optimizing parallelism, locality, and recomputation in image processing pipelines. SIGPLAN Not. 48, 6 (June), 519--530.

Digital Library

[55]

Sergej, T., and Mantiuk, R. 2014. Perceptual evaluation of demosaicing artefacts. In Image Analysis and Recognition. Springer, 38--45.

[56]

Simonyan, K., and Zisserman, A. 2014. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556.

[57]

Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., and Rabinovich, A. 2015. Going deeper with convolutions. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 1--9.

[58]

Tian, Q., Lansel, S., Farrell, J. E., and Wandell, B. A. 2014. Automating the design of image processing pipelines for novel color filter arrays: Local, linear, learned (13) method. In IS&T/SPIE Electronic Imaging, International Society for Optics and Photonics, 90230K--90230K.

[59]

Wang, X., Fouhey, D., and Gupta, A. 2015. Designing deep networks for surface normal estimation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 539--547.

[60]

Xu, L., Ren, J. S., Liu, C., and Jia, J. 2014. Deep convolutional neural network for image deconvolution. In Advances in Neural Information Processing Systems, 1790--1798.

Digital Library

[61]

Xu, L., Ren, J., Yan, Q., Liao, R., and Jia, J. 2015. Deep edge-aware filters. In Proceedings of the 32nd International Conference on Machine Learning (ICML-15), 1669--1678.

[62]

Zhang, L., and Wu, X. 2005. Color demosaicking via directional linear minimum mean square-error estimation. Image Processing, IEEE Transactions on 14, 12, 2167--2178.

Digital Library

[63]

Zhang, F., Wu, X., Yang, X., Zhang, W., and Zhang, L. 2009. Robust color demosaicking with adaptation to varying spectral correlations. Image Processing, IEEE Transactions on 18, 12, 2706--2717.

Digital Library

[64]

Zhang, L., Wu, X., Buades, A., and Li, X. 2011. Color demosaicking by local directional interpolation and nonlocal adaptive thresholding. Journal of Electronic Imaging 20, 2, 023016--023016.

[65]

Zhang, R., Isola, P., and Efros, A. A. 2016. Colorful image colorization. ECCV.

Cited By

Jeong KKim SKang M(2024)Multispectral Demosaicing Based on Iterative-Linear-Regression Model for Estimating Pseudo-Panchromatic ImageSensors10.3390/s2403076024:3(760)Online publication date: 24-Jan-2024
https://doi.org/10.3390/s24030760
Waseem IHabib MRehman EBibi RYousaf RAslam MJilani SYounis M(2024)Enhanced Learning Enriched Features Mechanism Using Deep Convolutional Neural Network for Image Denoising and Super-ResolutionApplied Sciences10.3390/app1414628114:14(6281)Online publication date: 18-Jul-2024
https://doi.org/10.3390/app14146281
Zhang TFu YZhang J(2024)Deep Guided Attention Network for Joint Denoising and Demosaicing in Real ImageChinese Journal of Electronics10.23919/cje.2022.00.41433:1(303-312)Online publication date: Jan-2024
https://doi.org/10.23919/cje.2022.00.414
Show More Cited By

Index Terms

Deep joint demosaicking and denoising
1. Computing methodologies
  1. Artificial intelligence
    1. Computer vision
      1. Image and video acquisition
        Computational photography
  2. Computer graphics
    1. Image manipulation
      1. Image processing

Recommendations

Joint demosaicking and denoising benefits from a two-stage training strategy
Abstract
Image demosaicking and denoising are the first two key steps of the color image production pipeline. The classical processing sequence has for a long time consisted of applying denoising first, and then demosaicking. Applying the operations in ...
RETRACTED ARTICLE: Lightweight deep dense Demosaicking and Denoising using convolutional neural networks
Abstract
A single sensor camera uses Color Filter Array (CFA) to capture single-color information at each pixel. Thus, to estimate the missing color samples and then to reconstruct an original image is known as CFA interpolation or demosaicking. Despite ...
Wavelet enabled convolutional autoencoder based deep neural network for hyperspectral image denoising
Abstract
Denoising of hyperspectral images (HSIs) is an important preprocessing step to enhance the performance of its analysis and interpretation. In reality, a remotely sensed HSI experiences disturbance from different sources and therefore gets affected ...

Comments

Information & Contributors

Information

Published In

cover image ACM Transactions on Graphics

ACM Transactions on Graphics Volume 35, Issue 6

November 2016

1045 pages

ISSN:0730-0301

EISSN:1557-7368

DOI:10.1145/2980179

Issue’s Table of Contents

Copyright © 2016 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 05 December 2016

Published in TOG Volume 35, Issue 6

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

Adobe

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

319
Total Citations
View Citations
3,636
Total Downloads

Downloads (Last 12 months)296
Downloads (Last 6 weeks)31

Reflects downloads up to 06 Aug 2024

Other Metrics

View Author Metrics

Citations

Cited By

Jeong KKim SKang M(2024)Multispectral Demosaicing Based on Iterative-Linear-Regression Model for Estimating Pseudo-Panchromatic ImageSensors10.3390/s2403076024:3(760)Online publication date: 24-Jan-2024
https://doi.org/10.3390/s24030760
Waseem IHabib MRehman EBibi RYousaf RAslam MJilani SYounis M(2024)Enhanced Learning Enriched Features Mechanism Using Deep Convolutional Neural Network for Image Denoising and Super-ResolutionApplied Sciences10.3390/app1414628114:14(6281)Online publication date: 18-Jul-2024
https://doi.org/10.3390/app14146281
Zhang TFu YZhang J(2024)Deep Guided Attention Network for Joint Denoising and Demosaicing in Real ImageChinese Journal of Electronics10.23919/cje.2022.00.41433:1(303-312)Online publication date: Jan-2024
https://doi.org/10.23919/cje.2022.00.414
Dong YXiong RZhao JZhang JFan XZhu SHuang T(2024)Learning a Deep Demosaicing Network for Spike Camera With Color Filter ArrayIEEE Transactions on Image Processing10.1109/TIP.2024.340305033(3634-3647)Online publication date: 2024
https://doi.org/10.1109/TIP.2024.3403050
Hou JGendy GChen GWang LHe G(2024)DTDeMo: A Deep Learning-Based Two-Stage Image Demosaicing Model With Interpolation and EnhancementIEEE Transactions on Computational Imaging10.1109/TCI.2024.342636010(1026-1039)Online publication date: 2024
https://doi.org/10.1109/TCI.2024.3426360
Ling MChang KHuang MLi HDang SLi B(2024)PRNet: Pyramid Restoration Network for RAW Image Super-ResolutionIEEE Transactions on Computational Imaging10.1109/TCI.2024.337408410(479-495)Online publication date: 2024
https://doi.org/10.1109/TCI.2024.3374084
Yuan NLi JSun B(2024)Consumer Camera Demosaicking and Denoising With a Collaborative Attention Fusion NetworkIEEE Transactions on Consumer Electronics10.1109/TCE.2023.334203570:1(509-521)Online publication date: Feb-2024
https://doi.org/10.1109/TCE.2023.3342035
Liu JZhu HChen ZLiu S(2024)Mutual Guidance Distillation for Joint Demosaicking and Denoising of Raw Images2024 Picture Coding Symposium (PCS)10.1109/PCS60826.2024.10566401(1-5)Online publication date: 12-Jun-2024
https://doi.org/10.1109/PCS60826.2024.10566401
Becker YNossek RPeleg T(2024)SDAT: Sub-Dataset Alternation Training for Improved Image DemosaicingIEEE Open Journal of Signal Processing10.1109/OJSP.2024.33951795(611-620)Online publication date: 2024
https://doi.org/10.1109/OJSP.2024.3395179
Liu CHe SXu JLi J(2024)Fine-Tuning for Bayer Demosaicking Through Periodic-Consistent Self-Supervised LearningIEEE Signal Processing Letters10.1109/LSP.2024.337048931(989-993)Online publication date: 2024
https://doi.org/10.1109/LSP.2024.3370489
Show More Cited By

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Issue’s Table of Contents