D3U-Net: Dual-Domain Collaborative Optimization Deep Unfolding Network for Image Compressive Sensing
Authors: 
Kai Han, Jin Wang, Yunhui Shi, Nam Ling, Baocai YinAuthors Info & Claims
Kai Han, Jin Wang, Yunhui Shi, Nam Ling, Baocai YinAuthors Info & ClaimsPages 9952 - 9960
Abstract
Deep unfolding network (DUN) is a powerful technique for image compressive sensing that bridges the gap between optimization methods and deep networks. However, DUNs usually rely heavily on single-domain information, overlooking the inter-domain dependencies. Therefore, such DUNs often face the following challenges: 1) information loss due to the inefficient representation within a single domain, and 2) limited robustness due to the absence of inter-domain dependencies. To overcome these challenges, we propose a deep unfolding framework D^3U-Net that establishes a dual-domain collaborative optimization scheme. This framework introduces both visual representations from the image domain and multi-resolution analysis provided by the wavelet domain. Such dual-domain representations constrain the feasible region within the solution space more accurately. Specifically, we design a consistency-difference collaborative mechanism to capture inter-domain dependencies effectively. This mechanism not only enhances the fidelity of reconstruction but also enriches the depth and breadth of extracted features, improving the overall robustness and reconstruction quality. Moreover, we develop an inter-stage transmission pathway to minimize the information loss during transmission while broadcasting multi-scale features in a frequency-adaptive manner. Extensive experimental results on various benchmark datasets show the superior performance of our method.
References
[1]
Hemant K. Aggarwal, Merry P. Mani, and Mathews Jacob. 2019. MoDL: Model-Based Deep Learning Architecture for Inverse Problems. IEEE Transactions on Medical Imaging, Vol. 38, 2 (2019), 394--405.
[2]
Marco Bevilacqua, Aline Roumy, Christine M. Guillemot, and et al. 2012. Low-Complexity Single-Image Super-Resolution based on Nonnegative Neighbor Embedding. In British Machine Vision Conference, BMVC 2012. BMVA Press, 1--10.
[3]
Mark Borgerding, Philip Schniter, and Sundeep Rangan. 2017. AMP-Inspired Deep Networks for Sparse Linear Inverse Problems. IEEE Transactions on Signal Processing, Vol. 65, 16 (2017), 4293--4308.
[4]
Thuong Nguyen Canh and Byeungwoo Jeon. 2018. Multi-Scale Deep Compressive Sensing Network. In 2018 IEEE Visual Communications and Image Processing (VCIP). 1--4.
[5]
Chen Chen and Junzhou Huang. 2012. Compressive sensing MRI with wavelet tree sparsity. Advances in neural information processing systems, Vol. 25 (2012), 1124--1132.
[6]
Jiwei Chen, Yubao Sun, Qingshan Liu, and Rui Huang. 2020. Learning Memory Augmented Cascading Network for Compressed Sensing of Images. In European Conference on Computer Vision. 513--529.
[7]
Scott Saobing Chen, David L. Donoho, and Michael A. Saunders. 1998. Atomic Decomposition by Basis Pursuit. SIAM J. Sci. Comput., Vol. 20 (1998), 33--61.
[8]
Theodor Cheslerean-Boghiu, Felix C. Hofmann, Manuel Schultheiß, Franz Pfeiffer, Daniela Pfeiffer, and Tobias Lasser. 2023. WNet: A Data-Driven Dual-Domain Denoising Model for Sparse-View Computed Tomography With a Trainable Reconstruction Layer. IEEE Transactions on Computational Imaging, Vol. 9 (2023), 120--132.
[9]
Wenxue Cui, Shaohui Liu, Feng Jiang, and Debin Zhao. 2023. Image Compressed Sensing Using Non-Local Neural Network. IEEE Transactions on Multimedia, Vol. 25 (2023), 816--830.
[10]
Wenxue Cui, Xingtao Wang, Xiaopeng Fan, Shaohui Liu, Chen Ma, and Debin Zhao. 2023. G2-DUN: Gradient Guided Deep Unfolding Network for Image Compressive Sensing. In Proceedings of the 31st ACM International Conference on Multimedia. 7933--7942.
[11]
D.L. Donoho. 2006. Compressed sensing. IEEE Transactions on Information Theory, Vol. 52, 4 (2006), 1289--1306.
[12]
Jiang Du, Xuemei Xie, Chenye Wang, and et al. 2019. Fully convolutional measurement network for compressive sensing image reconstruction. Neurocomputing, Vol. 328 (2019), 105--112.
[13]
Hongping Gan, Minghe Shen, Yi Hua, Chunyan Ma, and Tao Zhang. 2023. From Patch to Pixel: A Transformer-Based Hierarchical Framework for Compressive Image Sensing. IEEE Transactions on Computational Imaging, Vol. 9 (2023), 133--146.
[14]
Davis Gilton, Gregory Ongie, and Rebecca Willett. 2021. Deep Equilibrium Architectures for Inverse Problems in Imaging. IEEE Transactions on Computational Imaging, Vol. 7 (2021), 1123--1133.
[15]
John R. Hershey, Jonathan Le Roux, and Felix Weninger. 2014. Deep Unfolding: Model-Based Inspiration of Novel Deep Architectures. ArXiv, Vol. abs/1409.2574 (2014).
[16]
Jia-Bin Huang, Abhishek Singh, and Narendra Ahuja. 2015. Single image super-resolution from transformed self-exemplars. In 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 5197--5206.
[17]
Tao Huang, Xin Yuan, Weisheng Dong, Jinjian Wu, and Guangming Shi. 2023. Deep Gaussian Scale Mixture Prior for Image Reconstruction. IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 45, 9 (2023), 10778--10794.
[18]
Kuldeep Kulkarni, Suhas Lohit, Pavan Turaga, Ronan Kerviche, and Amit Ashok. 2016. ReconNet: Non-Iterative Reconstruction of Images from Compressively Sensed Measurements. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 449--458.
[19]
Jingyun Liang, Jiezhang Cao, Guolei Sun, Kai Zhang, Luc Van Gool, and Radu Timofte. 2021. SwinIR: Image Restoration Using Swin Transformer. In 2021 IEEE/CVF International Conference on Computer Vision Workshops (ICCVW). 1833--1844.
[20]
Suhas Lohit, Kuldeep Kulkarni, Ronan Kerviche, Pavan Turaga, and Amit Ashok. 2018. Convolutional Neural Networks for Noniterative Reconstruction of Compressively Sensed Images. IEEE Transactions on Computational Imaging, Vol. 4, 3 (2018), 326--340.
[21]
Huynh Van Luong, Boris Joukovsky, and Nikos Deligiannis. 2021. Designing Interpretable Recurrent Neural Networks for Video Reconstruction via Deep Unfolding. IEEE Transactions on Image Processing, Vol. 30 (2021), 4099--4113.
[22]
Shiqian Ma, Wotao Yin, Yin Zhang, and Amit Chakraborty. 2008. An efficient algorithm for compressed MR imaging using total variation and wavelets. In 2008 IEEE Conference on Computer Vision and Pattern Recognition. 1--8.
[23]
Julien Mairal, Francis Bach, Jean Ponce, Guillermo Sapiro, and Andrew Zisserman. 2009. Non-local sparse models for image restoration. In 2009 IEEE 12th International Conference on Computer Vision. 2272--2279.
[24]
S. Mallat and Z. Zhang. 1992. Adaptive time-frequency decomposition with matching pursuits. In Proceedings of the IEEE-SP International Symposium on Time-Frequency and Time-Scale Analysis. 7--10.
[25]
Christopher A. Metzler, Ali Mousavi, and Richard G. Baraniuk. 2017. Learned D-AMP: Principled Neural Network Based Compressive Image Recovery. In Proceedings of the 31st International Conference on Neural Information Processing Systems. 1770--1781.
[26]
Ali Mousavi, Ankit B. Patel, and Richard G. Baraniuk. 2015. A deep learning approach to structured signal recovery. In 2015 53rd Annual Allerton Conference on Communication, Control, and Computing (Allerton). 1336--1343.
[27]
Sujit Kumar Sahoo and Anamitra Makur. 2015. Signal Recovery from Random Measurements via Extended Orthogonal Matching Pursuit. IEEE Transactions on Signal Processing, Vol. 63, 10 (2015), 2572--2581.
[28]
Minghe Shen, Hongping Gan, Chao Ning, Yi Hua, and Tao Zhang. 2022. TransCS: A Transformer-Based Hybrid Architecture for Image Compressed Sensing. IEEE Transactions on Image Processing, Vol. 31 (2022), 6991--7005.
[29]
Wuzhen Shi, Feng Jiang, Shaohui Liu, and Debin Zhao. 2019. Scalable Convolutional Neural Network for Image Compressed Sensing. In 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 12282--12291.
[30]
Wuzhen Shi, Feng Jiang, shaohui Liu, and Debin Zhao. 2020. Image compressed sensing using convolutional neural network. IEEE Transactions on Image Processing, Vol. 29 (2020), 375--388.
[31]
Jiechong Song, Bin Chen, and Jian Zhang. 2021. Memory-Augmented Deep Unfolding Network for Compressive Sensing. In Proceedings of the 29th ACM International Conference on Multimedia. 4249--4258.
[32]
Jiechong Song, Bin Chen, and Jian Zhang. 2023. Dynamic Path-Controllable Deep Unfolding Network for Compressive Sensing. IEEE Transactions on Image Processing, Vol. 32 (2023), 2202--2214.
[33]
Jiechong Song and Jian Zhang. 2023. SODAS-Net: Side-Information-Aided Deep Adaptive Shrinkage Network for Compressive Sensing. IEEE Transactions on Instrumentation and Measurement, Vol. 72 (2023), 1--12.
[34]
Yubao Sun, Jiwei Chen, Qingshan Liu, Bo Liu, and Guodong Guo. 2020. Dual-Path Attention Network for Compressed Sensing Image Reconstruction. IEEE Transactions on Image Processing, Vol. 29 (2020), 9482--9495.
[35]
Weiwen Wu, Dianlin Hu, Chuang Niu, Hengyong Yu, Varut Vardhanabhuti, and Ge Wang. 2021. DRONE: Dual-Domain Residual-based Optimization NEtwork for Sparse-View CT Reconstruction. IEEE Transactions on Medical Imaging, Vol. 40, 11 (2021), 3002--3014.
[36]
Xuemei Xie, Chenye Wang, Jiang Du, and Guangming Shi. 2018. Full Image Recover for Block-Based Compressive Sensing. In 2018 IEEE International Conference on Multimedia and Expo (ICME). 1--6.
[37]
Xuemei Xie, Yuxiang Wang, Guangming Shi, Chenye Wang, Jiang Du, and Xiao Han. 2017. Adaptive Measurement Network for CS Image Reconstruction. In Computer Vision. 407--417.
[38]
Jingyu Yang, Xiangjun Yin, Mengxi Zhang, Huihui Yue, Xingyu Cui, and Huanjing Yue. 2022. Learning Image Formation and Regularization in Unrolling AMP for Lensless Image Reconstruction. IEEE Transactions on Computational Imaging, Vol. 8 (2022), 479--489.
[39]
Yan Yang, Jian Sun, Huibin Li, and Zongben Xu. 2020. ADMM-CSNet: A Deep Learning Approach for Image Compressive Sensing. IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 42, 3 (2020), 521--538.
[40]
Yixiao Yang, Ran Tao, Kaixuan Wei, and Ying Fu. 2022. Dynamic proximal unrolling network for compressive imaging. Neurocomputing, Vol. 510 (2022), 203--217.
[41]
Hantao Yao, Feng Dai, Shiliang Zhang, Yongdong Zhang, Qi Tian, and Changsheng Xu. 2019. DR2-Net: Deep Residual Reconstruction Network for Image Compressive Sensing. Neurocomput., Vol. 359, C (2019), 483--493.
[42]
Di You, Jingfen Xie, and Jian Zhang. 2021. ISTA-NET: Flexible Deep Unfolding Network for Compressive Sensing. In 2021 IEEE International Conference on Multimedia and Expo (ICME). 1--6.
[43]
Di You, Jian Zhang, Jingfen Xie, Bin Chen, and Siwei Ma. 2021. COAST: COntrollable Arbitrary-Sampling NeTwork for Compressive Sensing. IEEE Transactions on Image Processing, Vol. 30 (2021), 6066--6080.
[44]
Roman Zeyde, Michael Elad, and Matan Protter. 2010. On Single Image Scale-Up Using Sparse-Representations. In Curves and Surfaces. 711--730.
[45]
Zhiyuan Zha, Bihan Wen, Xin Yuan, Jiantao Zhou, and Ce Zhu. 2020. Reconciliation Of Group Sparsity And Low-Rank Models For Image Restoration. In 2020 IEEE International Conference on Multimedia and Expo (ICME). 1--6.
[46]
Jian Zhang and Bernard Ghanem. 2018. ISTA-Net: Interpretable Optimization-Inspired Deep Network for Image Compressive Sensing. In 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. 1828--1837.
[47]
Jian Zhang, Chen Zhao, and Wen Gao. 2020. Optimization-Inspired Compact Deep Compressive Sensing. IEEE Journal of Selected Topics in Signal Processing, Vol. 14, 4 (2020), 765--774.
[48]
Yi Zhang, Hu Chen, Wenjun Xia, Yang Chen, Baodong Liu, Yan Liu, Huaiqiang Sun, and Jiliu Zhou. 2023. LEARN: Recurrent Dual-Domain Reconstruction Network for Compressed Sensing CT. IEEE Transactions on Radiation and Plasma Medical Sciences, Vol. 7, 2 (2023), 132--142.
[49]
Zhonghao Zhang, Yipeng Liu, Jiani Liu, Fei Wen, and Ce Zhu. 2021. AMP-Net: Denoising-Based Deep Unfolding for Compressive Image Sensing. IEEE Transactions on Image Processing, Vol. 30 (2021), 1487--1500.
[50]
Siwang Zhou, Yan He, Yonghe Liu, Chengqing Li, and Jianming Zhang. 2021. Multi-Channel Deep Networks for Block-Based Image Compressive Sensing. IEEE Transactions on Multimedia, Vol. 23 (2021), 2627--2640.
Index Terms
- D3U-Net: Dual-Domain Collaborative Optimization Deep Unfolding Network for Image Compressive Sensing
Recommendations
WTDUN: Wavelet Tree-Structured Sampling and Deep Unfolding Network for Image Compressed Sensing
Deep unfolding networks have gained increasing attention in the field of compressed sensing (CS) owing to their theoretical interpretability and superior reconstruction performance. However, most existing deep unfolding methods often face the following ...
SAUNet: Spatial-Attention Unfolding Network for Image Compressive Sensing
MM '23: Proceedings of the 31st ACM International Conference on MultimediaImage Compressive Sensing (CS) enables compressed capture of natural images via a spatial multiplexing camera and accurate reconstruction from few measurements via an advanced algorithm. Deep learning, especially deep unfolding, has recently achieved ...
WDU-Net: Wavelet-Guided Deep Unfolding Network for Image Compressed Sensing Reconstruction
Pattern Recognition and Computer VisionAbstractMore and more deep unfolding networks have been studied to obtain good interpretability for high-quality image Compressed Sensing (CS) reconstruction. However, most of these networks simply focus on transmitting information across adjacent stages ...
Comments
Information & Contributors
Information
Published In
October 2024
11719 pages
ISBN:9798400706868
DOI:10.1145/3664647
- General Chairs:
- Jianfei Cai,
- Mohan Kankanhalli,
- Balakrishnan Prabhakaran,
- Susanne Boll,
- Program Chairs:
- Ramanathan Subramanian,
- Liang Zheng,
- Vivek K. Singh,
- Pablo Cesar,
- Lexing Xie,
- Dong Xu
Copyright © 2024 ACM.
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].
Sponsors
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 28 October 2024
Check for updates
Author Tags
Qualifiers
- Research-article
Funding Sources
Conference
MM '24
Sponsor:
MM '24: The 32nd ACM International Conference on Multimedia
October 28 - November 1, 2024
Melbourne VIC, Australia
Acceptance Rates
MM '24 Paper Acceptance Rate 1,150 of 4,385 submissions, 26%;
Overall Acceptance Rate 2,145 of 8,556 submissions, 25%
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 66Total Downloads
- Downloads (Last 12 months)66
- Downloads (Last 6 weeks)40
Reflects downloads up to 23 Dec 2024
Other Metrics
Citations
View Options
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in