research-article

Joint demosaicking and denoising benefits from a two-stage training strategy

Authors:

Jean-Michel Morel,

Gabriele FaccioloAuthors Info & Claims

Volume 434, Issue C

https://doi.org/10.1016/j.cam.2023.115330

Published: 15 December 2023 Publication History

Abstract

Image demosaicking and denoising are the first two key steps of the color image production pipeline. The classical processing sequence has for a long time consisted of applying denoising first, and then demosaicking. Applying the operations in this order leads to oversmoothing and checkerboard effects. Yet, it was difficult to change this order, because once the image is demosaicked, the statistical properties of the noise are dramatically changed and hard to handle by traditional denoising models. In this paper, we address this problem by a hybrid machine learning method. We invert the traditional color filter array (CFA) processing pipeline by first demosaicking and then denoising. Our demosaicking algorithm, trained on noiseless images, combines a traditional method and a residual convolutional neural network (CNN). This first stage retains all known information, which is the key point to obtain faithful final results. The noisy demosaicked image is then passed through a second CNN restoring a noiseless full-color image. This pipeline order completely avoids checkerboard effects and restores fine image detail. Although CNNs can be trained to solve jointly demosaicking–denoising end-to-end, we find that this two-stage training performs better and is less prone to failure. It is shown experimentally to improve on the state of the art, both quantitatively and in terms of visual quality.

References

[1]

B.E. Bayer, Color Imaging Array, US Patent 3,971,065, 1976, Google Patents.

[2]

Jin Q., Facciolo G., Morel J., A review of an old dilemma: Demosaicking first, or denoising first?, in: Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recogn. Workshops, 2020, pp. 2169–2179,.

[3]

Kalevo O., Rantanen H., Noise reduction techniques for bayer-matrix images, in: Proc. Sensors and Camera Systems for Scientific, Industrial, and Digital Photography Applications III, Vol. 4669, International Society for Optics and Photonics, 2002, pp. 348–359.

[4]

Park S.H., Kim H.S., Lansel S., Parmar M., Wandell B.A., A case for denoising before demosaicking color filter array data, in: Proc. Conf. Rec. Asilomar Conf. Signals Syst. Comput., 2009, pp. 860–864,.

[5]

Lee M., Park S., Kang M., Denoising algorithm for CFA image sensors considering inter-channel correlation, Sensors 17 (6) (2017) 1236.

[6]

Condat L., A simple, fast and efficient approach to denoisaicking: Joint demosaicking and denoising, in: Proc. IEEE Int. Conf. Image Process., 2010, pp. 905–908,.

[7]

Condat L., Mosaddegh S., Joint demosaicking and denoising by total variation minimization, in: Proc. IEEE Int. Conf. Image Process., 2012, pp. 2781–2784,.

[8]

Condat L., A generic proximal algorithm for convex optimization—Application to total variation minimization, IEEE Signal Process. Lett. 21 (8) (2014) 985–989,.

[9]

Danielyan A., Vehvilainen M., Foi A., Katkovnik V., Egiazarian K., Cross-color BM3D filtering of noisy raw data, in: Proc. Int. Workshop Local Non-Local Approx. Image Process., 2009, pp. 125–129,.

[10]

K. He, X. Zhang, S. Ren, J. Sun, Deep Residual Learning for Image Recognition, in: Proc. IEEE Conf. Comput. Vis. Pattern Recognit., 2016, pp. 770–778.

[11]

C. Szegedy, S. Ioffe, V. Vanhoucke, A.A. Alemi, Inception-v4, Inception-ResNet and the Impact of Residual Connections on Learning, in: Thirty-First AAAI Conference on Artificial Intelligence, 2017, pp. 4278–4284.

[12]

Zhang K., Zuo W., Chen Y., Meng D., Zhang L., Beyond a Gaussian denoiser: Residual learning of deep CNN for image denoising, IEEE Trans. Image Process. 26 (7) (2017) 3142–3155.

Digital Library

[13]

Zhang K., Zuo W., Zhang L., Ffdnet: Toward a fast and flexible solution for CNN-based image denoising, IEEE Trans. Image Process. 27 (9) (2018) 4608–4622,.

[14]

Guo Y., Davy A., Facciolo G., Morel J.-M., Jin Q., Fast, nonlocal and neural: A lightweight high quality solution to image denoising, IEEE Signal Process. Lett. 28 (2021) 1515–1519,.

[15]

Fang F., Li J., Yuan Y., Zeng T., Zhang G., Multilevel edge features guided network for image denoising, IEEE Trans. Neural Netw. Learn. Syst. 32 (9) (2021) 3956–3970,.

[16]

Hou R., Li F., IDPCNN: Iterative denoising and projecting CNN for MRI reconstruction, J. Comput. Appl. Math. 406 (2022),.

Digital Library

[17]

Gharbi M., Chaurasia G., Paris S., Durand F., Deep joint demosaicking and denoising, ACM Trans. Graph. 35 (6) (2016) 191.

[18]

R. Tan, K. Zhang, W. Zuo, L. Zhang, Color image demosaicking via deep residual learning, in: Proc. IEEE Int. Conf. Multimedia Expo, ICME, 2017, pp. 793–798.

[19]

Tan D.S., Chen W., Hua K., DeepDemosaicking: Adaptive image demosaicking via multiple deep fully convolutional networks, IEEE Trans. Image Process. 27 (5) (2018) 2408–2419.

[20]

L. Liu, X. Jia, J. Liu, Q. Tian, Joint Demosaicing and Denoising With Self Guidance, in: Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit., 2020, pp. 2237–2246.

[21]

Guo S., Liang Z., Zhang L., Joint denoising and demosaicking with green channel prior for real-world burst images, IEEE Trans. Image Process. 30 (2021) 6930–6942,.

Digital Library

[22]

Fang F., Li J., Zeng T., Soft-edge assisted network for single image super-resolution, IEEE Trans. Image Process. 29 (2020) 4656–4668.

[23]

Wen Z., Guan J., Zeng T., Li Y., Residual network with detail perception loss for single image super-resolution, Comput. Vis. Image Underst. 199 (2020).

[24]

F. Kokkinos, S. Lefkimmiatis, Deep image demosaicking using a cascade of convolutional residual denoising networks, in: Proc. Eur. Conf. Comput. Vis., 2018, pp. 303–319.

[25]

W. Xing, K. Egiazarian, End-to-End Learning for Joint Image Demosaicing, Denoising and Super-Resolution, in: Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit., 2021, pp. 3507–3516.

[26]

C.A. Laroche, M.A. Prescott, Apparatus and Method for Adaptively Interpolating a Full Color Image Utilizing Chrominance Gradients, US Patent 5,373,322, 1994, Google Patents.

[27]

J.F. Hamilton, J.E. Adams, Adaptive Color Plan Interpolation in Single Sensor Color Electronic Camera, US Patent 5,629,734, 1997, Google Patents.

[28]

J.E. Adams, Design of practical color filter array interpolation algorithms for digital cameras .2, in: Proc. IEEE Int. Conf. Image Process., Vol. 1, 1998, pp. 488–492.

[29]

Jin Q., Guo Y., Morel J.-M., Facciolo G., A mathematical analysis and implementation of residual interpolation demosaicking algorithms, Image Process. Line 11 (2021) 234–283,.

[30]

Lei Zhang Q., Xiaolin Wu Y., Color demosaicking via directional linear minimum mean square-error estimation, IEEE Trans. Image Process. 14 (12) (2005) 2167–2178.

[31]

I. Pekkucuksen, Y. Altunbasak, Gradient based threshold free color filter array interpolation, in: Proc. IEEE Int. Conf. Image Process., 2010, pp. 137–140.

[32]

D. Kiku, Y. Monno, M. Tanaka, M. Okutomi, Residual interpolation for color image demosaicking, in: Proc. IEEE Int. Conf. Image Process., 2013, pp. 2304–2308.

[33]

He K., Sun J., Tang X., Guided image filtering, IEEE Trans. Pattern Anal. Mach. Intell. 35 (6) (2013) 1397–1409.

Digital Library

[34]

D. Kiku, Y. Monno, M. Tanaka, M. Okutomi, Minimized-Laplacian residual interpolation for color image demosaicking, in: SPIE, Vol. 9023, 2014, pp. 90230L–1–90230L–8.

[35]

Kiku D., Monno Y., Tanaka M., Okutomi M., Beyond color difference: Residual interpolation for color image demosaicking, IEEE Trans. Image Process. 25 (3) (2016) 1288–1300.

[36]

Monno Y., Kiku D., Tanaka M., Okutomi M., Adaptive residual interpolation for color and multispectral image demosaicking, Sensors 17 (12) (2017) 2787.

[37]

Buades A., Coll B., Morel J., Sbert C., Self-similarity driven color demosaicking, IEEE Trans. Image Process. 18 (6) (2009) 1192–1202.

[38]

Duran J., Buades A., Self-similarity and spectral correlation adaptive algorithm for color demosaicking, IEEE Trans. Image Process. 23 (9) (2014) 4031–4040.

[39]

J. Mairal, F. Bach, J. Ponce, G. Sapiro, A. Zisserman, Non-local sparse models for image restoration, in: Proc. IEEE Int. Conf. Comput. Vis., 2009, pp. 2272–2279.

[40]

Lu Y.M., Karzand M., Vetterli M., Demosaicking by alternating projections: Theory and fast one-step implementation, IEEE Trans. Image Process. 19 (8) (2010) 2085–2098.

[41]

Zhang J., Sheng A., Hirakawa K., A wavelet-GSM approach to demosaicking, IEEE Signal Process. Lett. 25 (6) (2018) 778–782.

[42]

Dubois E., Frequency-domain methods for demosaicking of Bayer-sampled color images, IEEE Signal Process. Lett. 12 (12) (2005) 847–850.

[43]

E. Dubois, Filter Design for Adaptive Frequency-Domain Bayer Demosaicking, in: Proc. Int. Conf. Image Process., 2006, pp. 2705–2708.

[44]

Hua K.-L., Hidayati S.C., He F.-L., Wei C.-P., Wang Y.-C.F., Context-aware joint dictionary learning for color image demosaicking, J. Vis. Commun. Image Represent. 38 (2016) 230–245.

[45]

Bai C., Li J., Lin Z., Demosaicking based on channel-correlation adaptive dictionary learning, J. Electron. Imaging 27 (4) (2018).

[46]

K. Cui, Z. Jin, E. Steinbach, Color Image Demosaicking Using a 3-Stage Convolutional Neural Network Structure, in: Proc. IEEE Int. Conf. Image Process., 2018, pp. 2177–2181.

[47]

Malvar H., wei He L., Cutler R., High-quality linear interpolation for demosaicing of Bayer-patterned color images, in: 2004 IEEE International Conference on Acoustics, Speech, and Signal Processing, Vol. 3, 2004, pp. iii–485,.

[48]

Syu N.-S., Chen Y.-S., Chuang Y.-Y., Learning deep convolutional networks for demosaicing, 2018, arXiv:1802.03769.

[49]

T. Yamaguchi, M. Ikehara, Image Demosaicking via Chrominance Images with Parallel Convolutional Neural Networks, in: ICASSP 2019 - 2019 IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP, 2019, pp. 1702–1706.

[50]

K. Mei, J. Li, J. Zhang, H. Wu, J. Li, R. Huang, HighEr-Resolution Network for Image Demosaicing and Enhancing, in: Proc. IEEE/CVF Int. Conf. Comput. Vis. Workshop, 2019, pp. 3441–3448.

[51]

L. Condat, S. Mosaddegh, Joint demosaicking and denoising by total variation minimization, in: Proc. IEEE Int. Conf. Image Process., 2012, pp. 2781–2784.

[52]

T. Klatzer, K. Hammernik, P. Knobelreiter, T. Pock, Learning joint demosaicing and denoising based on sequential energy minimization, in: Proc. IEEE Int. Conf. Comput. Photogr., 2016, pp. 1–11.

[53]

Khashabi D., Nowozin S., Jancsary J., Fitzgibbon A.W., Joint demosaicing and denoising via learned nonparametric random fields, IEEE Trans. Image Process. 23 (12) (2014) 4968–4981.

[54]

D. Menon, G. Calvagno, Joint demosaicking and denoisingwith space-varying filters, in: Proc. IEEE Int. Conf. Image Process., 2009, pp. 477–480.

[55]

Menon D., Calvagno G., Regularization approaches to demosaicking, IEEE Trans. Image Process. 18 (10) (2009) 2209–2220,.

Digital Library

[56]

H. Tan, X. Zeng, S. Lai, Y. Liu, M. Zhang, Joint demosaicing and denoising of noisy Bayer images with ADMM, in: Proc. IEEE Int. Conf. Image Process., 2017, pp. 2951–2955.

[57]

Lefkimmiatis S., Universal denoising networks : A novel CNN architecture for image denoising, in: 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2018, pp. 3204–3213,.

[58]

Kokkinos F., Lefkimmiatis S., Iterative joint image demosaicking and denoising using a residual denoising network, IEEE Trans. Image Process. 28 (8) (2019) 4177–4188.

[59]

T. Huang, F.F. Wu, W. Dong, G. Shi, X. Li, Lightweight Deep Residue Learning for Joint Color Image Demosaicking and Denoising, in: Proc. Int. Conf. Pattern Recognit., 2018, pp. 127–132.

[60]

T. Ehret, A. Davy, P. Arias, G. Facciolo, Joint Demosaicking and Denoising by Fine-Tuning of Bursts of Raw Images, in: Proc. IEEE/CVF Int. Conf. Comput. Vis., 2019, pp. 8867–8876.

[61]

Ma K., Duanmu Z., Wu Q., Wang Z., Yong H., Li H., Zhang L., Waterloo exploration database: New challenges for image quality assessment models, IEEE Trans. Image Process. 26 (2) (2017) 1004–1016.

Digital Library

[62]

K. He, X. Zhang, S. Ren, J. Sun, Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification, in: Proc. IEEE Int. Conf. Comput. Vis., 2015, pp. 1026–1034.

[63]

Kingma D.P., Ba J., Adam: A method for stochastic optimization, 2014, arXiv:1412.6980.

[64]

Zhang L., Wu X., Buades A., Li X., Color demosaicking by local directional interpolation and nonlocal adaptive thresholding, J. Electron. Imaging 20 (2) (2011).

[65]

J. Huang, A. Singh, N. Ahuja, Single image super-resolution from transformed self-exemplars, in: Proc. IEEE Conf. Comput. Vis. Pattern Recognit., 2015, pp. 5197–5206.

[66]

Alleysson D., Susstrunk S., Herault J., Linear demosaicing inspired by the human visual system, IEEE Trans. Image Process. 14 (4) (2005) 439–449.

[67]

Wang Z., Bovik A.C., Sheikh H.R., Simoncelli E.P., Image quality assessment: from error visibility to structural similarity, IEEE Trans. Image Process. 13 (4) (2004) 600–612.

Digital Library

[68]

K. Dabov, A. Foi, V. Katkovnik, K. Egiazarian, Color Image Denoising via Sparse 3D Collaborative Filtering with Grouping Constraint in Luminance-Chrominance Space, in: Proc. IEEE Int. Conf. Image Process., Vol. 1, 2007, pp. I–313–I–316.

[69]

T. Brooks, B. Mildenhall, T. Xue, J. Chen, D. Sharlet, J.T. Barron, Unprocessing Images for Learned Raw Denoising, in: Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit., 2019, pp. 11028–11037.

[70]

Plötz T., Roth S., Benchmarking denoising algorithms with real photographs, in: Proc. IEEE Conf. Comput. Vis. Pattern Recognit., 2017, pp. 2750–2759,.

[71]

Mittal A., Soundararajan R., Bovik A.C., Making a “completely blind” image quality analyzer, IEEE Signal Process. Lett. 20 (3) (2013) 209–212.

Index Terms

Joint demosaicking and denoising benefits from a two-stage training strategy
1. Computing methodologies

Index terms have been assigned to the content through auto-classification.

Recommendations

A case for denoising before demosaicking color filter array data
Asilomar'09: Proceedings of the 43rd Asilomar conference on Signals, systems and computers

Denoising algorithms are well developed for grayscale and color images, but not as well for color filter array (CFA) data. Consequently, the common color imaging pipeline demosaics CFA data before denoising. In this paper we explore the noise-related ...
Joint demosaicking and denoising with space-varying filters
ICIP'09: Proceedings of the 16th IEEE international conference on Image processing

In a digital camera usually images are captured using a Color Filter Array (CFA), requiring a demosaicking process to reconstruct the full resolution color image. Often also a denoising step is necessary to remove the noise introduced by the sensor. In ...
Deep joint demosaicking and denoising

Demosaicking and denoising are the key first stages of the digital imaging pipeline but they are also a severely ill-posed problem that infers three color values per pixel from a single noisy measurement. Earlier methods rely on hand-crafted filters or ...

Comments

Information & Contributors

Information

Published In

cover image Journal of Computational and Applied Mathematics

Journal of Computational and Applied Mathematics Volume 434, Issue C

Dec 2023

448 pages

ISSN:0377-0427

Issue’s Table of Contents

Elsevier B.V.

Publisher

Elsevier Science Publishers B. V.

Netherlands

Publication History

Published: 15 December 2023

Author Tags

Qualifiers

Research-article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

0
Total Citations
0
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 10 Oct 2024

Other Metrics

View Author Metrics

Citations

View Options

View options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Media

Figures

Other

Tables

View Issue’s Table of Contents