One-Pot Multi-frame Denoising
Authors: 
Lujia Jin, Qing Guo, Shi Zhao, Lei Zhu, Qian Chen, Qiushi Ren, Yanye LuAuthors Info & Claims
Lujia Jin, Qing Guo, Shi Zhao, Lei Zhu, Qian Chen, Qiushi Ren, Yanye LuAuthors Info & ClaimsPages 515 - 536
Abstract
The efficacy of learning-based denoising techniques is heavily reliant on the quality of clean supervision. Unfortunately, acquiring clean images in many scenarios is a challenging task. Conversely, capturing multiple noisy frames of the same field of view is feasible and natural in real-life scenarios. Thus, it is imperative to explore the potential of noisy data in model training and avoid the limitations imposed by clean labels. In this paper, we propose a novel unsupervised learning strategy called one-pot denoising (OPD), which is the first unsupervised multi-frame denoising method. OPD differs from traditional supervision schemes, such as supervised Noise2Clean, unsupervised Noise2Noise, and self-supervised Noise2Void, as it employs mutual supervision among all the multiple frames. This provides learning with more diverse supervision and allows models to better exploit the correlation among frames. Notably, we reveal that Noise2Noise is a special case of the proposed OPD. We provide two specific implementations, namely OPD-random coupling and OPD-alienation loss, to achieve OPD during model training based on data allocation and loss refine, respectively. Our experiments demonstrate that OPD outperforms other unsupervised denoising methods and is comparable to non-transformer-based supervised N2C methods for several classic noise patterns, including additive white Gaussian noise, signal-dependent Poisson noise, and multiplicative Bernoulli noise. Additionally, OPD shows remarkable performance in more challenging tasks such as mixed-blind denoising, denoising random-valued impulse noise, and text removal. The source code and pre-trained models are available at https://github.com/LujiaJin/One-Pot_Multi-Frame_Denoising.
References
[1]
Anwar, S., & Barnes, N. (2019). Real image denoising with feature attention. In Ieee/cvf International Conference on Computer Vision (iccv) (pp. 3155–3164).
[2]
Batson, J., & Royer, L. (2019). Noise2self: Blind denoising by self-supervision. In: International Conference on Machine Learning (icml) (pp. 524–533).
[3]
Bhat, G., Danelljan, M., Yu, F., Van Gool, L., & Timofte, R. (2021). Deep reparametrization of multi-frame super-resolution and denoising. In: Proceedings of the ieee/cvf International Conference on Computer Vision (pp. 2460–2470).
[4]
Buades, A., Coll, B., & Morel, J.-M. (2005a). A non-local algorithm for image denoising. In Ieee/cvf Conference on Computer Vision and Pattern Recognition (cvpr) (Vol. 2, pp. 60–65). IEEE.
[5]
Buades A, Coll B, and Morel J-M A review of image denoising algorithms, with a new one Multiscale Modeling & Simulation 2005 4 2 490-530
[6]
Buades, T., Lou, Y., Morel, J.-M., & Tang, Z. (2009). A note on multi-image denoising. In Ieee 2009 International Workshop on Local and Nonlocal Approximation in Image Processing (pp. 1–15). IEEE.
[7]
Calvarons, A. F. (2021). Improved noise2noise denoising with limited data. In: Ieee/cvf Conference on Computer Vision and Pattern Recognition (cvpr) (pp. 796–805).
[8]
Chen T, Ma K-K, and Chen L-H Tri-state median filter for image denoising IEEE Transactions on Image Processing 1999 8 12 1834-1838
[9]
Dabov K, Foi A, Katkovnik V, and Egiazarian K Image denoising by sparse 3-d transform-domain collaborative filtering IEEE Transactions on Image Processing (TIP) 2007 16 8 2080-2095
[10]
Dalsasso E, Denis L, and Tupin F Sar2sar: A semi-supervised despeckling algorithm for sar images IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing 2021 14 4321-4329
[11]
Dudhane, A., Zamir, S. W., Khan, S., Khan, F. S., & Yang, M.-H. (2022). Burst image restoration and enhancement. In Proceedings of the ieee/cvf Conference on Computer Vision and Pattern Recognition (pp. 5759–5768).
[12]
Godard, C., Matzen, K., & Uyttendaele, M. (2018). Deep burst denoising. In European Conference on Computer Vision (eccv) (pp. 538–554).
[13]
Guilloteau C, Oberlin T, Berné O, and Dobigeon N Hyperspectral and multispectral image fusion under spectrally varying spatial blurs-application to high dimensional infrared astronomical imaging IEEE Transactions on Computational Imaging (TCI) 2020 6 1362-1374
[14]
Guo, S., Yan, Z., Zhang, K., Zuo, W., & Zhang, L. (2019). Toward convolutional blind denoising of real photographs. In Ieee/cvf Conference on Computer Vision and Pattern Recognition (cvpr) (pp. 1712–1722).
[15]
Hasan AM, Mohebbian MR, Wahid KA, and Babyn P Hybrid-collaborative noise2noise denoiser for low-dose ct images IEEE Transactions on Radiation and Plasma Medical Sciences 2020 5 2 235-244
[16]
Hasinoff SW, Sharlet D, Geiss R, Adams A, Barron JT, Kainz F, and Levoy M Burst photography for high dynamic range and low-light imaging on mobile cameras ACM Transactions on Graphics (TOG) 2016 35 6 1-12
[17]
He, K., Zhang, X., Ren, S., & Sun, J. (2015). Delving deep into rectifiers: Surpassing humanlevel performance on imagenet classification. In Ieee/cvf International Conference on Computer Vision (iccv) (pp. 1026–1034).
[18]
Huang, G., Liu, Z., Van Der Maaten, L., & Weinberger, K. Q. (2017). Densely connected convolutional networks. In Proceedings of the ieee Conference on Computer Vision and Pattern Recognition (pp. 4700–4708).
[19]
Huang, T., Li, S., Jia, X., Lu, H., & Liu, J. (2021). Neighbor2neighbor: Self-supervised denoising from single noisy images. In Proceedings of the ieee/cvf Conference on Computer Vision and Pattern Recognition (pp. 14781–14790).
[20]
Huang T, Li S, Jia X, Lu H, and Liu J Neighbor2neighbor: A self-supervised framework for deep image denoising IEEE Transactions on Image Processing 2022 31 4023-4038
[21]
Isola, P., Zhu, J.-Y., Zhou, T., & Efros, A. A. (2017). Image-to-image translation with conditional adversarial networks. In Proceedings of the ieee Conference on Computer Vision and Pattern Recognition (pp. 1125–1134).
[22]
Jiang Z, Huang Z, Qiu B, Meng X, You Y, Liu X, et al. Comparative study of deep learning models for optical coherence tomography angiography Biomedical Optics Express 2020 11 3 1580-1597
[23]
Kingma, D. P., & Ba, J. (2014). Adam: A method for stochastic optimization. arXiv:1412.6980 .
[24]
Kong, Z., Yang, X., & He, L. (2020). A comprehensive comparison of multi-dimensional image denoising methods. arXiv:2011.03462 .
[25]
Kostadin, D., Alessandro, F., & Karen, E. (2007). Video denoising by sparse 3d transformdomain collaborative filtering. In Ieee 15th European Signal Processing Conference (espc) (Vol. 149, p. 2). IEEE.
[26]
Krull, A., Buchholz, T.-O., & Jug, F. (2019). Noise2void-learning denoising from single noisy images. In Ieee/cvf Conference on Computer Vision and Pattern Recognition (cvpr) (pp. 2129–2137).
[27]
Kuan DT, Sawchuk AA, Strand TC, and Chavel P Adaptive noise smoothing filter for images with signal-dependent noise IEEE Transactions on Pattern Analysis and Machine Intelligence 1985 2 165-177
[28]
Laine S, Karras T, Lehtinen J, and Aila T High-quality self-supervised deep image denoising Advances in Neural Information Processing Systems 2019 32 6970-6980
[29]
Lee J-S Refined filtering of image noise using local statistics Computer Graphics and Image Processing 1981 15 4 380-389
[30]
Lee J-S Digital image smoothing and the sigma filter Computer Vision, Graphics, and Image Processing 1983 24 2 255-269
[31]
Lefkimmiatis, S. (2018). Universal denoising networks: a novel cnn architecture for image denoising. In Ieee/cvf Conference on Computer Vision and Pattern Recognition (cvpr) (pp. 3204–3213).
[32]
Lehtinen, J., Munkberg, J., Hasselgren, J., Laine, S., Karras, T., Aittala, M., & Aila, T. (2018). Noise2noise: Learning image restoration without clean data. arXiv:1803.04189.
[33]
Liang, J., Cao, J., Sun, G., Zhang, K., Van Gool, L., & Timofte, R. (2021). Swinir: Image restoration using swin transformer. In Proceedings of the ieee/cvf International Conference on Computer Vision (pp. 1833–1844).
[34]
Liu Z, Yuan L, Tang X, Uyttendaele M, and Sun J Fast burst images denoising ACM Transactions on Graphics (TOG) 2014 33 6 1-9
[35]
Maggioni M, Boracchi G, Foi A, and Egiazarian K Video denoising using separable 4d nonlocal spatiotemporal transforms Image Processing: Algorithms and Systems ix 2011 7870
[36]
Maggioni M, Boracchi G, Foi A, and Egiazarian K Video denoising, deblocking, and enhancement through separable 4-d nonlocal spatiotemporal transforms IEEE Transactions on Image Processing (TIP) 2012 21 9 3952-3966
[37]
Marinč, T., Srinivasan, V., Gül, S., Hellge, C., & Samek, W. (2019). Multi-kernel prediction networks for denoising of burst images. In Ieee International Conference on Image Processing (icip) (pp. 2404–2408).
[38]
Martin, D., Fowlkes, C., Tal, D., & Malik, J. (2001). A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics. In Ieee/cvf International Conference on Computer Vision (iccv) (Vol. 2, pp. 416–423).
[39]
Meiniel W, Olivo-Marin J-C, and Angelini ED Denoising of microscopy images: A review of the state-of-the-art, and a new sparsity-based method IEEE Transactions on Image Processing 2018 27 8 3842-3856
[40]
Mildenhall, B., Barron, J. T., Chen, J., Sharlet, D., Ng, R., & Carroll, R. (2018). Burst denoising with kernel prediction networks. Ieee/cvf Conference on Computer Vision and Pattern Recognition (cvpr) (pp. 2502–2510).
[41]
Moran, N., Schmidt, D., Zhong, Y., & Coady, P. (2020). Noisier2noise: Learning to denoise from unpaired noisy data. In Ieee/cvf Conference on Computer Vision and Pattern Recognition (cvpr) (pp. 12064–12072).
[42]
Neshatavar, R., Yavartanoo, M., Son, S., & Lee, K. M. (2022). Cvf-sid: Cyclic multi-variate function for self-supervised image denoising by disentangling noise from image. In Proceedings of the ieee/cvf Conference on Computer Vision and Pattern Recognition (pp. 17583–17591).
[43]
Pang, T., Zheng, H., Quan, Y., & Ji, H. (2021). Recorrupted-to-recorrupted: unsupervised deep learning for image denoising. In Proceedings of the ieee/cvf Conference on Computer Vision and Pattern Recognition (pp. 2043–2052).
[44]
Paszke A, Gross S, Massa F, Lerer A, Bradbury J, and Chanan G Pytorch: An imperative style, highperformance deep learning library Advances in Neural Information Processing Systems 2019 32 8026-8037
[45]
Qiu B, Huang Z, Liu X, Meng X, You Y, Liu G, and Lu Y Noise reduction in optical coherence tomography images using a deep neural network with perceptuallysensitive loss function Biomedical Optics Express 2020 11 2 817-830
[46]
Quan, Y., Chen, M., Pang, T., & Ji, H. (2020). Self2self with dropout: Learning selfsupervised denoising from single image. In Proceedings of the ieee/cvf Conference on Computer Vision and Pattern Recognition (pp. 1890–1898).
[47]
Ronneberger, O., Fischer, P., & Brox, T. (2015). U-net: Convolutional networks for biomedical image segmentation. In International Conference on Medical Image Computing and Computer-Assisted Intervention (miccai) (pp. 234–241).
[48]
Russakovsky O, Deng J, Su H, Krause J, Satheesh S, Ma S, et al. Imagenet large scale visual recognition challenge International Journal of Computer Vision (IJCV) 2015 115 3 211-252
[49]
Su J, Xu B, and Yin H A survey of deep learning approaches to image restoration Neurocomputing 2022 487 46-65
[50]
Tian C, Fei L, Zheng W, Xu Y, Zuo W, and Lin C-W Deep learning on image denoising: An overview Neural Networks 2020 131 251-275
[51]
Tian C, Xu Y, Li Z, Zuo W, Fei L, and Liu H Attention-guided cnn for image denoising Neural Networks 2020 124 117-129
[52]
Tico, M. (2008). Multi-frame image denoising and stabilization. In Ieee 16th European Signal Processing Conference (ESPC) (pp. 1–4). IEEE.
[53]
Ulyanov, D., Vedaldi, A., & Lempitsky, V. (2018). Deep image prior. In Ieee/cvf Conference on Computer Vision and Pattern Recognition (cvpr) (pp. 9446–9454).
[54]
Ulyanov D, Vedaldi A, and Lempitsky V Deep image prior International Journal of Computer Vision 2020 128 1867-1888
[55]
Wang Z, Bovik AC, Sheikh HR, and Simoncelli EP Image quality assessment: From error visibility to structural similarity IEEE Transactions on Image Processing (TIP) 2004 13 4 600-612
[56]
Wang, Z., Cun, X., Bao, J., Zhou, W., Liu, J., & Li, H. (2022). Uformer: A general ushaped transformer for image restoration. In Proceedings of the ieee/cvf Conference on Computer Vision and Pattern Recognition (pp. 17683–17693).
[57]
Wu, D., Gong, K., Kim, K., Li, X., & Li, Q. (2019). Consensus neural network for medical imaging denoising with only noisy training samples. In International Conference on Medical Image Computing and Computer-Assisted Intervention (miccai) (pp. 741–749).
[58]
Xia, Z., Perazzi, F., Gharbi, M., Sunkavalli, K., & Chakrabarti, A. (2020). Basis prediction networks for effective burst denoising with large kernels. In Ieee/cvf Conference on Computer Vision and Pattern Recognition (cvpr) (pp. 11844–11853).
[59]
Xu J, Huang Y, Cheng M-M, Liu L, Zhu F, Xu Z, and Shao L Noisy-as-clean: Learning self-supervised denoising from corrupted image IEEE Transactions on Image Processing (TIP) 2020 29 9316-9329
[60]
Zamir, S. W., Arora, A., Khan, S., Hayat, M., Khan, F. S., & Yang, M. -H. (2022). Restormer: Efficient transformer for highresolution image restoration. In Proceedings of the ieee/cvf Conference on Computer Vision and Pattern Recognition (pp. 5728–5739).
[61]
Zbontar, J., Jing, L., Misra, I., LeCun, Y., & Deny, S. (2021). Barlow twins: Self-supervised learning via redundancy reduction. In International Conference on Machine Learning (pp. 12310–12320).
[62]
Zeyde, R., Elad, M., & Protter, M. (2010). On single image scale-up using sparse-representations. In International Conference on Curves and Surfaces (pp. 711–730).
[63]
Zhang K, Zuo W, Chen Y, Meng D, and Zhang L Beyond a gaussian denoiser: Residual learning of deep cnn for image denoising IEEE Transactions on Image Processing (TIP) 2017 26 7 3142-3155
[64]
Zhang M and Gunturk BK Multiresolution bilateral filtering for image denoising IEEE Transactions on Image Processing (TIP) 2008 17 12 2324-2333
Index Terms
- One-Pot Multi-frame Denoising
Index terms have been assigned to the content through auto-classification.
Recommendations
Dual-scale similarity-guided cycle generative adversarial network for unsupervised low-dose CT denoising
AbstractRemoving the noise in low-dose CT (LDCT) is crucial to improving the diagnostic quality. Previously, many supervised or unsupervised deep learning-based LDCT denoising algorithms have been proposed. Unsupervised LDCT denoising algorithms are more ...
Highlights- For high-quality unsupervised LDCT denoising, we propose dual-scale similarity-guided cycle generative adversarial network. We design the similarity-guided discrimination loss for the network, which can guarantee effective removal of noise ...
Multi-Instance Learning with One Side Label Noise
Multi-instance Learning (MIL) is a popular learning paradigm arising from many real applications. It assigns a label to a set of instances, which is called a bag, and the bag’s label is determined by the instances within it. A bag is positive if and only ...
A New Training Principle for Stacked Denoising Autoencoders
ICIG '13: Proceedings of the 2013 Seventh International Conference on Image and GraphicsIn this work, a new training principle is introduced for unsupervised learning that makes the learned representations more efficient and useful. Using partially corrupted inputs instead, the denoising Auto encoder can obtain more robust and ...
Comments
Information & Contributors
Information
Published In
© The Author(s), under exclusive licence to Springer Science+Business Media, LLC, part of Springer Nature 2023. Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
Publisher
Kluwer Academic Publishers
United States
Publication History
Published: 09 September 2023
Accepted: 10 August 2023
Received: 27 April 2023
Author Tags
Qualifiers
- Research-article
Funding Sources
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 0Total Downloads
- Downloads (Last 12 months)0
- Downloads (Last 6 weeks)0
Reflects downloads up to 03 Oct 2024
Other Metrics
Citations
View Options
View options
Get Access
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in