Restoration of Multiple Image Distortions using a Semi-dynamic Deep Neural Network
Pages 7871 - 7880
Abstract
Restoring multiple image distortions with a single model is difficult because different distortions require fundamentally different processing mechanisms, e.g., deblurring requires high-pass filtering, while denoising requires low-pass filtering operations. This paper presents a dynamic universal image restoration (DUIR) system capable of simultaneously processing multiple distortions. The new model features several innovative designs: (i) a distortion embedding module (DEM) to automatically encode the distortion information of an input, (ii) a distortion attention module (DAM) that uses a bi-directional long short-term memory (LSTM) to encode the distortion into a sequence of forward and backward interdependent modulating signals, and (iii) a dynamically adaptive image restoration deep convolutional neural network (DAIR-DCNN) featuring unique semi-dynamic layers (SDLs) in which part of their parameters are dynamically modulated by the distortion signals. DEM, DAM, and SDLs together make DAIR-DCNN adaptive to the distortions of the current input, which in turn equips the DUIR system with the capability of simultaneously processing multiple image distortions with a single trained model. We present extensive experimental results to show that the new technique achieves superior performance to state-of-the-art models on both synthetic and real data. We further demonstrate that a trained DUIR system can simultaneously handle different distortions, including those with conflicting demands, such as denoising, deblurring, and compression artifact removal.
Supplementary Material
In this video, we will talk about our work "Restoration of Multiple Image Distortions using a Semi-dynamic Deep Neural Network". The motivation, data observation and model analysis are introduced in the first part. Then, we will talk about our model construction and how to train this model. The final part showed a variety of results of experiments including model comparison in datasets and real-world datasets, our ablation study, as well as many visualisation results.
- Download
- 26.70 MB
References
[1]
Abdelrahman Abdelhamed, Stephen Lin, and Michael S. Brown. 2018. A High-Quality Denoising Dataset for Smartphone Cameras. In 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. 1692--1700. https://doi.org/10.1109/ CVPR.2018.00182
[2]
Eirikur Agustsson and Radu Timofte. 2017. NTIRE 2017 Challenge on Single Image Super-Resolution: Dataset and Study. In 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW). 1122--1131. https://doi.org/ 10.1109/CVPRW.2017.150
[3]
Yuval Bahat, Netalee Efrat, and Michal Irani. 2017. Non-uniform Blind Deblurring by Reblurring. In 2017 IEEE International Conference on Computer Vision (ICCV). 3306--3314. https://doi.org/10.1109/ICCV.2017.356
[4]
Hanting Chen, Yunhe Wang, Tianyu Guo, Chang Xu, Yiping Deng, Zhenhua Liu, Siwei Ma, Chunjing Xu, Chao Xu, and Wen Gao. 2021. Pre-Trained Image Processing Transformer. In 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 12294--12305. https://doi.org/10.1109/CVPR46437.2021.01212
[5]
Liangyu Chen, Xin Lu, Jie Zhang, Xiaojie Chu, and Chengpeng Chen. 2021. HINet: Half Instance Normalization Network for Image Restoration. In 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW). 182--192. https://doi.org/10.1109/CVPRW53098.2021.00027
[6]
Xiaojie Chu, Liangyu Chen, andWenqing Yu. 2022. NAFSSR: Stereo Image Super- Resolution Using NAFNet. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops. 1239--1248.
[7]
Bert De Brabandere, Xu Jia, Tinne Tuytelaars, and Luc Van Gool. 2016. Dynamic Filter Networks. In NIPS.
[8]
Chao Dong, Yubin Deng, Chen Change Loy, and Xiaoou Tang. 2015. Compression Artifacts Reduction by a Deep Convolutional Network. In 2015 IEEE International Conference on Computer Vision (ICCV). 576--584. https://doi.org/10.1109/ICCV. 2015.73
[9]
Qingnan Fan, Dongdong Chen, Lu Yuan, Gang Hua, Nenghai Yu, and Baoquan Chen. 2021. A General Decoupled Learning Framework for Parameterized Image Operators. IEEE Transactions on Pattern Analysis and Machine Intelligence 43, 1 (2021), 33--47. https://doi.org/10.1109/TPAMI.2019.2925793
[10]
Alessandro Foi, Vladimir Katkovnik, and Karen Egiazarian. 2007. Pointwise Shape-Adaptive DCT for High-Quality Denoising and Deblocking of Grayscale and Color Images. IEEE Transactions on Image Processing 16, 5 (2007), 1395--1411. https://doi.org/10.1109/TIP.2007.891788
[11]
Chunzhi Gu, Xuequan Lu, Ying He, and Chao Zhang. 2021. Blur Removal Via Blurred-Noisy Image Pair. IEEE Transactions on Image Processing 30 (2021), 345-- 359. https://doi.org/10.1109/TIP.2020.3036745
[12]
Zhenyu Guan, Qunliang Xing, Mai Xu, Ren Yang, Tie Liu, and Zulin Wang. 2021. MFQE 2.0: A New Approach for Multi-Frame Quality Enhancement on Compressed Video. IEEE Transactions on Pattern Analysis and Machine Intelligence 43, 3 (2021), 949--963. https://doi.org/10.1109/TPAMI.2019.2944806
[13]
David Ha, Andrew Dai, and Quoc Le. 2016. HyperNetworks. In ICLR.
[14]
Yizeng Han, Gao Huang, Shiji Song, Le Yang, Honghui Wang, and Yulin Wang. 2022. Dynamic Neural Networks: A Survey. IEEE Transactions on Pattern Analysis and Machine Intelligence 44, 11 (2022), 7436--7456. https://doi.org/10.1109/TPAMI. 2021.3117837
[15]
Jingwen He, Chao Dong, and Yu Qiao. 2019. Modulating Image Restoration With Continual Levels via Adaptive Feature Modification Layers. In 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 11048--11056. https://doi.org/10.1109/CVPR.2019.01131
[16]
Jingwen He, Chao Dong, and Yu Qiao. 2020. Interactive Multi-dimension Modulation with Dynamic Controllable Residual Learning for Image Restoration. In Computer Vision - ECCV 2020, Andrea Vedaldi, Horst Bischof, Thomas Brox, and Jan-Michael Frahm (Eds.). Springer International Publishing, Cham, 53--68.
[17]
Jia-Bin Huang, Abhishek Singh, and Narendra Ahuja. 2015. Single Image Super- Resolution From Transformed Self-Exemplars. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 5197--5206.
[18]
Jaesung Rim, Haeyun Lee, Jucheol Won, and Sunghyun Cho. 2020. Real-World Blur Dataset for Learning and Benchmarking Deblurring Algorithms. In Proceedings of the European Conference on Computer Vision (ECCV).
[19]
Orest Kupyn, Volodymyr Budzan, Mykola Mykhailych, Dmytro Mishkin, and Jiri Matas. 2018. DeblurGAN: Blind Motion Deblurring Using Conditional Adversarial Networks. In 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. 8183--8192. https://doi.org/10.1109/CVPR.2018.00854
[20]
Boyun Li, Xiao Liu, Peng Hu, Zhongqin Wu, Jiancheng Lv, and Xi Peng. 2022. All-In-One Image Restoration for Unknown Corruption. In IEEE Conference on Computer Vision and Pattern Recognition. New Orleans, LA.
[21]
Jingyun Liang, Jiezhang Cao, Guolei Sun, Kai Zhang, Luc Van Gool, and Radu Timofte. 2021. SwinIR: Image Restoration Using Swin Transformer. In 2021 IEEE/CVF International Conference on Computer VisionWorkshops (ICCVW). 1833--1844. https://doi.org/10.1109/ICCVW54120.2021.00210
[22]
Bee Lim, Sanghyun Son, Heewon Kim, Seungjun Nah, and Kyoung Mu Lee. 2017. Enhanced Deep Residual Networks for Single Image Super-Resolution. In 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW). 1132--1140. https://doi.org/10.1109/CVPRW.2017.151
[23]
Bee Lim, Sanghyun Son, Heewon Kim, Seungjun Nah, and Kyoung Mu Lee. 2017. Enhanced Deep Residual Networks for Single Image Super-Resolution. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Workshops.
[24]
Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. 2014. Microsoft COCO: Common Objects in Context. In Computer Vision - ECCV 2014, David Fleet, Tomas Pajdla, Bernt Schiele, and Tinne Tuytelaars (Eds.). Springer International Publishing, Cham, 740--755.
[25]
Wei Liu, Fei Zhou, Tao Lu, Jiang Duan, and Guoping Qiu. 2021. Image Defogging Quality Assessment: Real-World Database and Method. IEEE Transactions on Image Processing 30 (2021), 176--190. https://doi.org/10.1109/TIP.2020.3033402
[26]
Guo Lu, Xiaoyun Zhang, Wanli Ouyang, Dong Xu, Li Chen, and Zhiyong Gao. 2020. Deep Non-Local Kalman Network for Video Compression Artifact Reduction. IEEE Transactions on Image Processing 29 (2020), 1725--1737. https://doi.org/10.1109/TIP.2019.2943214
[27]
Kede Ma, Zhengfang Duanmu, Qingbo Wu, Zhou Wang, Hongwei Yong, Hongliang Li, and Lei Zhang. 2017. Waterloo Exploration Database: New Challenges for Image Quality Assessment Models. IEEE Transactions on Image Processing 26, 2 (Feb. 2017), 1004--1016.
[28]
Pavan C. Madhusudana, Neil Birkbeck, Yilin Wang, Balu Adsumilli, and Alan C. Bovik. 2022. Image Quality Assessment Using Contrastive Learning. IEEE Transactions on Image Processing 31 (2022), 4149--4161. https://doi.org/10.1109/ TIP.2022.3181496
[29]
D. Martin, C. Fowlkes, D. Tal, and J. Malik. 2001. A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics. In Proceedings Eighth IEEE International Conference on Computer Vision. ICCV 2001, Vol. 2. 416--423 vol.2. https://doi.org/10.1109/ICCV.2001.937655
[30]
Guoping Qiu. 2000. MLP for adaptive postprocessing block-coded images. IEEE Transactions on Circuits and Systems for Video Technology 10, 8 (2000), 1450--1454. https://doi.org/10.1109/76.889048
[31]
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. 2015. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision (IJCV) 115, 3 (2015), 211--252. https: //doi.org/10.1007/s11263-015-0816-y
[32]
H.R. Sheikh, M.F. Sabir, and A.C. Bovik. 2006. A Statistical Evaluation of Recent Full Reference Image Quality Assessment Algorithms. IEEE Transactions on Image Processing 15, 11 (2006), 3440--3451. https://doi.org/10.1109/TIP.2006.881959
[33]
Xingjian Shi, Zhourong Chen, Hao Wang, Dit-Yan Yeung, Wai-kin Wong, and Wang-chun Woo. 2015. Convolutional LSTM Network: A Machine Learning Approach for Precipitation Nowcasting. In Proceedings of the 28th International Conference on Neural Information Processing Systems - Volume 1 (Montreal, Canada) (NIPS'15). 802--810.
[34]
Patricia L. Suárez, Angel D. Sappa, Boris X. Vintimilla, and Riad I. Hammoud. 2018. Deep Learning Based Single Image Dehazing. In 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW). 1250--12507. https://doi.org/10.1109/CVPRW.2018.00162
[35]
Chunwei Tian, Yong Xu, Zuoyong Li, Wangmeng Zuo, Lunke Fei, and Hong Liu. 2020. Attention-Guided CNN for Image Denoising. Neural Netw. 124, C (apr 2020), 117--129. https://doi.org/10.1016/j.neunet.2019.12.024
[36]
Laurens van der Maaten and Geoffrey Hinton. 2008. Visualizing Data using t-SNE. Journal of Machine Learning Research 9, 86 (2008), 2579--2605. http: //jmlr.org/papers/v9/vandermaaten08a.html
[37]
Longguan gWang, Yingqian Wang, Xiaoyu Dong, Qingyu Xu, Jungang Yang, Wei An, and Yulan Guo. 2021. Unsupervised Degradation Representation Learning for Blind Super-Resolution. In CVPR.
[38]
Xintao Wang, Ke Yu, Chao Dong, and Chen Change Loy. 2018. Recovering Realistic Texture in Image Super-Resolution by Deep Spatial Feature Transform. In 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. 606--615. https://doi.org/10.1109/CVPR.2018.00070
[39]
Xintao Wang, Ke Yu, Chao Dong, Xiaoou Tang, and Chen Change Loy. 2019. Deep Network Interpolation for Continuous Imagery Effect Transition. In 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 1692--1701. https://doi.org/10.1109/CVPR.2019.00179
[40]
Haiyan Wu, Yanyun Qu, Shaohui Lin, Jian Zhou, Ruizhi Qiao, Zhizhong Zhang, Yuan Xie, and Lizhuang Ma. 2021. Contrastive Learning for Compact Single Image Dehazing. In 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 10546--10555. https://doi.org/10.1109/CVPR46437.2021.01041
[41]
SyedWaqas Zamir, Aditya Arora, Salman Khan, Munawar Hayat, Fahad Shahbaz Khan, and Ming-Hsuan Yang. 2022. Restormer: Efficient Transformer for High-Resolution Image Restoration. In CVPR.
[42]
Syed Waqas Zamir, Aditya Arora, Salman Khan, Munawar Hayat, Fahad Shahbaz Khan, Ming-Hsuan Yang, and Ling Shao. 2021. Multi-Stage Progressive Image Restoration. In 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 14816--14826. https://doi.org/10.1109/CVPR46437.2021.01458
[43]
Kai Zhang, Wangmeng Zuo, Yunjin Chen, Deyu Meng, and Lei Zhang. 2017. Beyond a Gaussian denoiser: Residual learning of deep CNN for image denoising. IEEE Transactions on Image Processing 26, 7 (2017), 3142--3155.
[44]
Kai Zhang, Wangmeng Zuo, and Lei Zhang. 2018. FFDNet: Toward a Fast and Flexible Solution for CNN based Image Denoising. IEEE Transactions on Image Processing (2018).
[45]
Lei Zhang, Xiaolin Wu, Antoni Buades, and Xin Li. 2011. Color Demosaicking by Local Directional Interpolation and Non-local Adaptive Thresholding. Journal of Electronic Imaging 20, 2 (2011), 1--15.
[46]
Hengyuan Zhao, Wenze Shao, Bingkun Bao, and Haibo Li. 2019. A Simple and Robust Deep Convolutional Approach to Blind Image Denoising. In 2019 IEEE/CVF International Conference on Computer Vision Workshop (ICCVW). 3943--3951. https://doi.org/10.1109/ICCVW.2019.00488
[47]
Xizhou Zhu, Han Hu, Stephen Lin, and Jifeng Dai. 2019. Deformable ConvNets V2: More Deformable, Better Results. In 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 9300--9308. https://doi.org/10.1109/CVPR.2019.00953
Index Terms
- Restoration of Multiple Image Distortions using a Semi-dynamic Deep Neural Network
Recommendations
Image Restoration Using a Multilayered Quantum Backpropagation Neural Network
CICN '11: Proceedings of the 2011 International Conference on Computational Intelligence and Communication NetworksThe problem of image restoration in presence of blur and noise has been a very important problem in the domain of digital image processing and computer vision. A quantum inspired back propagation neural network (QBPNN) architecture based on quantum ...
Image restoration
AbstractTrue images are usually degraded during image acquisition. Image restoration is for restoring true images from their observed but degraded versions; it is often used for preprocessing observed images so that subsequent image processing and ...
Comments
Information & Contributors
Information
Published In
October 2023
9913 pages
ISBN:9798400701085
DOI:10.1145/3581783
- General Chairs:
- Abdulmotaleb El Saddik,
- Tao Mei,
- Rita Cucchiara,
- Program Chairs:
- Marco Bertini,
- Diana Patricia Tobon Vallejo,
- Pradeep K. Atrey,
- M. Shamim Hossain
Copyright © 2023 ACM.
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].
Sponsors
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 27 October 2023
Check for updates
Author Tags
Qualifiers
- Research-article
Funding Sources
- Guangdong Basic and Applied Basic Research Foundation
- the National Natural Science Foundation of China
- the Shenzhen Research and Development Program
Conference
MM '23
Sponsor:
MM '23: The 31st ACM International Conference on Multimedia
October 29 - November 3, 2023
Ottawa ON, Canada
Acceptance Rates
Overall Acceptance Rate 995 of 4,171 submissions, 24%
Upcoming Conference
MM '24
- Sponsor:
- sigmm
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 184Total Downloads
- Downloads (Last 12 months)184
- Downloads (Last 6 weeks)6
Reflects downloads up to 11 Aug 2024
Other Metrics
Citations
View Options
Get Access
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in