research-article

Dynamic Path-Controllable Deep Unfolding Network for Compressive Sensing

Authors:

Jian ZhangAuthors Info & Claims

IEEE Transactions on Image Processing, Volume 32

Pages 2202 - 2214

https://doi.org/10.1109/TIP.2023.3263100

Published: 01 January 2023 Publication History

Abstract

Deep unfolding network (DUN) that unfolds the optimization algorithm into a deep neural network has achieved great success in compressive sensing (CS) due to its good interpretability and high performance. Each stage in DUN corresponds to one iteration in optimization. At the test time, all the sampling images generally need to be processed by all stages, which comes at a price of computation burden and is also unnecessary for the images whose contents are easier to restore. In this paper, we focus on CS reconstruction and propose a novel Dynamic Path-Controllable Deep Unfolding Network (DPC-DUN). DPC-DUN with our designed path-controllable selector can dynamically select a rapid and appropriate route for each image and is slimmable by regulating different performance-complexity tradeoffs. Extensive experiments show that our DPC-DUN is highly flexible and can provide excellent performance and dynamic adjustment to get a suitable tradeoff, thus addressing the main requirements to become appealing in practice. Codes are available at <uri>https://github.com/songjiechong/DPC-DUN</uri>.

References

[1]

J. Zhang, B. Chen, R. Xiong, and Y. Zhang, “Physics-inspired compressive sensing: Beyond deep unrolling,” IEEE Signal Process. Mag., vol. 40, no. 1, pp. 58–72, Jan. 2023.

[2]

E. J. Candès, J. Romberg, and T. Tao, “Robust uncertainty principles: Exact signal reconstruction from highly incomplete frequency information,” IEEE Trans. Inf. Theory, vol. 52, no. 2, pp. 489–509, Feb. 2006.

Digital Library

[3]

A. C. Sankaranarayanan, C. Studer, and R. G. Baraniuk, “CS-MUVI: Video compressive sensing for spatial-multiplexing cameras,” in Proc. IEEE Int. Conf. Comput. Photography (ICCP), Apr. 2012, pp. 1–10.

[4]

A. Liutkuset al., “Imaging with nature: Compressive imaging using a multiply scattering medium,” Sci. Rep., vol. 4, p. 5552, Jul. 2014.

[5]

A. Zymnis, S. Boyd, and E. J. Candès, “Compressed sensing with quantized measurements,” IEEE Signal Process. Lett., vol. 17, no. 2, pp. 149–152, Feb. 2010.

[6]

T. P. Szczykutowicz and G.-H. Chen, “Dual energy CT using slow kVp switching acquisition and prior image constrained compressed sensing,” Phys. Med. Biol., vol. 55, no. 21, p. 6411, 2010.

[7]

J. Zhang, Z. Zhang, J. Xie, and Y. Zhang, “High-throughput deep unfolding network for compressive sensing MRI,” IEEE J. Sel. Topics Signal Process., vol. 16, no. 4, pp. 750–761, Jun. 2022.

[8]

Z. Chenet al., “Compressive sensing multi-layer residual coefficients for image coding,” IEEE Trans. Circuits Syst. Video Technol., vol. 30, no. 4, pp. 1109–1120, Apr. 2020.

[9]

M. F. Duarteet al., “Single-pixel imaging via compressive sampling,” IEEE Signal Process. Mag., vol. 25, no. 2, pp. 83–91, Mar. 2008.

[10]

F. Rousset, N. Ducros, A. Farina, G. Valentini, C. D’Andrea, and F. Peyrin, “Adaptive basis scan by wavelet prediction for single-pixel imaging,” IEEE Trans. Comput. Imag., vol. 3, no. 1, pp. 36–46, Mar. 2017.

[11]

H. Shen, X. Li, L. Zhang, D. Tao, and C. Zeng, “Compressed sensing-based inpainting of aqua moderate resolution imaging spectroradiometer band 6 using adaptive spectrum-weighted sparse Bayesian dictionary learning,” IEEE Trans. Geosci. Remote Sens., vol. 52, no. 2, pp. 894–906, Feb. 2014.

[12]

C. Mou and J. Zhang, “TransCL: Transformer makes strong and flexible compressive learning,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 45, no. 4, pp. 5236–5251, Apr. 2023.

Digital Library

[13]

Z. Wu, Z. Zhang, J. Song, and M. Zhang, “Spatial-temporal synergic prior driven unfolding network for snapshot compressive imaging,” in Proc. IEEE Int. Conf. Multimedia Expo (ICME), Jul. 2021, pp. 1–6.

[14]

Z. Wu, J. Zhang, and C. Mou, “Dense deep unfolding network with 3D-CNN prior for snapshot compressive sensing,” in Proc. IEEE Int. Conf. Comput. Vis. (ICCV), Oct. 2021, pp. 4892–4901.

[15]

Y. Kim, M. S. Nadar, and A. Bilgin, “Compressed sensing using a Gaussian scale mixtures model in wavelet domain,” in Proc. IEEE Int. Conf. Image Process. (ICIP), Sep. 2010, pp. 3365–3368.

[16]

C. Li, W. Win, H. Jing, and Y. Zhang, “An efficient augmented Lagrangian method with applications to total variation minimization,” Comput. Optim. Appl., vol. 56, no. 3, pp. 507–530, Dec. 2013.

[17]

J. Zhang, D. Zhao, and W. Gao, “Group-based sparse representation for image restoration,” IEEE Trans. Image Process., vol. 23, no. 8, pp. 3336–3351, Aug. 2014.

[18]

J. Zhang, C. Zhao, D. Zhao, and W. Gao, “Image compressive sensing recovery using adaptively learned sparsifying basis via L0 minimization,” Signal Process., vol. 103, pp. 114–126, Oct. 2014.

Digital Library

[19]

X. Gao, J. Zhang, W. Che, X. Fan, and D. Zhao, “Block-based compressive sensing coding of natural images by local structural measurement matrix,” in Proc. Data Compress. Conf., Apr. 2015, pp. 133–142.

[20]

C. A. Metzler, A. Maleki, and R. G. Baraniuk, “From denoising to compressed sensing,” IEEE Trans. Inf. Theory, vol. 62, no. 9, pp. 5117–5144, Sep. 2016.

Digital Library

[21]

C. Zhao, J. Zhang, R. Wang, and W. Gao, “CREAM: CNN-REgularized ADMM framework for compressive-sensed image reconstruction,” IEEE Access, vol. 6, pp. 76838–76853, 2018.

[22]

C. Zhao, S. Ma, and W. Gao, “Image compressive-sensing recovery using structured Laplacian sparsity in DCT domain and multi-hypothesis prediction,” in Proc. IEEE Int. Conf. Multimedia Expo (ICME), Jul. 2014, pp. 1–6.

[23]

C. Zhao, S. Ma, J. Zhang, R. Xiong, and W. Gao, “Video compressive sensing reconstruction via reweighted residual sparsity,” IEEE Trans. Circuits Syst. Video Technol., vol. 27, no. 6, pp. 1182–1195, Jun. 2017.

Digital Library

[24]

C. Zhao, J. Zhang, S. Ma, and W. Gao, “Nonconvex Lp nuclear norm based ADMM framework for compressed sensing,” in Proc. Data Compress. Conf. (DCC), Mar. 2016, pp. 161–170.

[25]

K. Kulkarni, S. Lohit, P. Turaga, R. Kerviche, and A. Ashok, “ReconNet: Non-iterative reconstruction of images from compressively sensed measurements,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR), Jun. 2016, pp. 449–458.

[26]

Y. Sun, J. Chen, Q. Liu, B. Liu, and G. Guo, “Dual-path attention network for compressed sensing image reconstruction,” IEEE Trans. Image Process., vol. 29, pp. 9482–9495, 2020.

[27]

C. Ren, X. He, C. Wang, and Z. Zhao, “Adaptive consistency prior based deep network for image denoising,” in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR), Jun. 2021, pp. 8596–8606.

[28]

J. Zhang and B. Ghanem, “ISTA-Net: Interpretable optimization-inspired deep network for image compressive sensing,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR), Jun. 2018, pp. 1828–1837.

[29]

J. Zhang, C. Zhao, and W. Gao, “Optimization-inspired compact deep compressive sensing,” IEEE J. Sel. Topics Signal Process., vol. 14, no. 4, pp. 765–774, May 2020.

[30]

D. You, J. Xie, and J. Zhang, “ISTA-NET++: Flexible deep unfolding network for compressive sensing,” in Proc. IEEE Int. Conf. Multimedia Expo (ICME), Jul. 2021, pp. 1–6.

[31]

D. You, J. Zhang, J. Xie, B. Chen, and S. Ma, “COAST: Controllable arbitrary-sampling NeTwork for compressive sensing,” IEEE Trans. Image Process., vol. 30, pp. 6066–6080, 2021.

[32]

Z. Zhang, Y. Liu, J. Liu, F. Wen, and C. Zhu, “AMP-Net: Denoising-based deep unfolding for compressive image sensing,” IEEE Trans. Image Process., vol. 30, pp. 1487–1500, 2021.

[33]

Y. Chen and T. Pock, “Trainable nonlinear reaction diffusion: A flexible framework for fast and effective image restoration,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 39, no. 6, pp. 1256–1272, Jun. 2017.

Digital Library

[34]

S. Lefkimmiatis, “Non-local color image denoising with convolutional neural networks,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR), Jul. 2017, pp. 3587–3596.

[35]

J. Kruse, C. Rother, and U. Schmidt, “Learning to push the limits of efficient FFT-based image deconvolution,” in Proc. IEEE Int. Conf. Comput. Vis. (ICCV), Oct. 2017, pp. 4596–4604.

[36]

H. Wanget al., “Stacking networks dynamically for image restoration based on the plug-and-play framework,” in Proc. Eur. Conf. Comput. Vis. (ECCV), Aug. 2020, pp. 446–462.

[37]

F. Kokkinos and S. Lefkimmiatis, “Deep image demosaicking using a cascade of convolutional residual denoising networks,” in Proc. Eur. Conf. Comput. Vis. (ECCV), Sep. 2018, pp. 317–333.

[38]

K. Zhang, W. Zuo, S. Gu, and L. Zhang, “Learning deep CNN denoiser prior for image restoration,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR), Jul. 2017, pp. 3929–3938.

[39]

W. Dong, P. Wang, W. Yin, and G. Shi, “Denoising prior driven deep neural network for image restoration,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 41, no. 10, pp. 2305–2318, Oct. 2019.

Digital Library

[40]

D. Gilton, G. Ongie, and R. Willett, “Neumann networks for linear inverse problems in imaging,” IEEE Trans. Comput. Imag., vol. 6, pp. 328–343, 2019.

[41]

J. Song, B. Chen, and J. Zhang, “Memory-augmented deep unfolding network for compressive sensing,” in Proc. 29th ACM Int. Conf. Multimedia, Oct. 2021, pp. 4249–4258.

[42]

J. Chen, Y. Sun, Q. Liu, and R. Huang, “Learning memory augmented cascading network for compressed sensing of images,” in Proc. Eur. Conf. Comput. Vis. (ECCV), Aug. 2020, pp. 513–529.

[43]

Y. Su and Q. Lian, “IPiano-Net: Nonconvex optimization inspired multi-scale reconstruction network for compressed sensing,” Signal Process., Image Commun., vol. 89, Nov. 2020, Art. no.

[44]

K. Yu, X. Wang, C. Dong, X. Tang, and C. C. Loy, “Path-restore: Learning network path selection for image restoration,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 44, no. 10, pp. 7078–7092, Oct. 2022.

Digital Library

[45]

Y. Han, G. Huang, S. Song, L. Yang, H. Wang, and Y. Wang, “Dynamic neural networks: A survey,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 44, no. 11, pp. 7436–7456, Nov. 2022.

[46]

M. Zhuet al., “Dynamic resolution network,” in Proc. Int. Conf. Neural Inf. Process. Syst. (NeurIPS), Dec. 2021, pp. 1–12.

[47]

C. Li, G. Wang, B. Wang, X. Liang, Z. Li, and X. Chang, “Dynamic slimmable network,” in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR), Jun. 2021, pp. 8607–8617.

[48]

Z. Wuet al., “BlockDrop: Dynamic inference paths in residual networks,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR), Jun. 2018, pp. 8817–8826.

[49]

X. Wang, F. Yu, Z.-Y. Dou, T. Darrell, and J. E. Gonzalez, “SkipNet: Learning dynamic routing in convolutional networks,” in Proc. Eur. Conf. Comput. Vis. (ECCV), Sep. 2018, pp. 420–436.

[50]

Y. Song, Y. Zhu, and X. Du, “Dynamic residual dense network for image denoising,” Sensors, vol. 19, no. 17, p. 3809, Sep. 2019.

[51]

K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR), Jun. 2016, pp. 770–778.

[52]

J. He, C. Dong, Y. Liu, and Y. Qiao, “Interactive multi-dimension modulation for image restoration,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 44, no. 12, pp. 9363–9379, Dec. 2022.

[53]

H. Cai, J. He, Y. Qiao, and C. Dong, “Toward interactive modulation for photo-realistic image restoration,” in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. Workshops (CVPRW), Jun. 2021.

[54]

J. Jiang, K. Zhang, and R. Timofte, “Towards flexible blind JPEG artifacts removal,” in Proc. IEEE/CVF Int. Conf. Comput. Vis. (ICCV), Oct. 2021, pp. 4977–4986.

[55]

Y. Choi, M. El-Khamy, and J. Lee, “Variable rate deep image compression with a conditional autoencoder,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR), Jun. 2019, pp. 3146–3154.

[56]

J. Linet al., “Variable-rate multi-frequency image compression using modulated generalized octave convolution,” in Proc. IEEE 22nd Int. Workshop Multimedia Signal Process. (MMSP), Sep. 2020, pp. 1–6.

[57]

E. Jang, S. Gu, and B. Poole, “Categorical reparameterization with Gumbel-Softmax,” in Proc. Int. Conf. Learn. Represent. (ICLR), Apr. 2017, pp. 1–13.

[58]

T. Dai, Y. Lv, B. Chen, Z. Wang, Z. Zhu, and S.-T. Xia, “Mix-order attention networks for image restoration,” in Proc. 29th ACM Int. Conf. Multimedia, Oct. 2021, pp. 2880–2888.

[59]

J. Hu, L. Shen, and G. Sun, “Squeeze-and-excitation networks,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR), Jun. 2018, pp. 7132–7141.

[60]

K. Zhang, Y. Li, W. Zuo, L. Zhang, L. V. Gool, and R. Timofte, “Plug-and-play image restoration with deep denoiser prior,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 44, no. 10, pp. 6360–6376, Oct. 2022.

Digital Library

[61]

E. Agustsson and R. Timofte, “NTIRE 2017 challenge on single image super-resolution: Dataset and study,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit. Workshops (CVPRW), Jul. 2017, pp. 126–135.

[62]

Y. Zhong, C. Zhang, F. Ren, H. Kuang, and P. Tang, “Scalable image compressed sensing with generator networks,” IEEE Trans. Comput. Imag., vol. 8, pp. 1025–1037, 2022.

[63]

C. Ma, J. T. Zhou, X. Zhang, and Y. Zhou, “Deep unfolding for compressed sensing with denoiser,” in Proc. IEEE Int. Conf. Multimedia Expo (ICME), Jul. 2022, pp. 1–6.

[64]

M. Shen, H. Gan, C. Ning, Y. Hua, and T. Zhang, “TransCS: A transformer-based hybrid architecture for image compressed sensing,” IEEE Trans. Image Process., vol. 31, pp. 6991–7005, 2022.

[65]

W. Shi, F. Jiang, S. Liu, and D. Zhao, “Image compressed sensing using convolutional neural network,” IEEE Trans. Image Process., vol. 29, pp. 375–388, 2020.

Digital Library

Cited By

Song HGong JMeng HLai YWooldridge MDy JNatarajan S(2024)Multi-cross sampling and frequency-division reconstruction for image compressed sensingProceedings of the Thirty-Eighth AAAI Conference on Artificial Intelligence and Thirty-Sixth Conference on Innovative Applications of Artificial Intelligence and Fourteenth Symposium on Educational Advances in Artificial Intelligence10.1609/aaai.v38i5.28294(4909-4917)Online publication date: 20-Feb-2024
https://dl.acm.org/doi/10.1609/aaai.v38i5.28294
Han KWang JShi YCai HLing NYin B(2024)WTDUN: Wavelet Tree-Structured Sampling and Deep Unfolding Network for Image Compressed SensingACM Transactions on Multimedia Computing, Communications, and Applications10.1145/370173121:1(1-22)Online publication date: 26-Oct-2024
https://dl.acm.org/doi/10.1145/3701731
Han KWang JShi YLing NYin BCai JKankanhalli MPrabhakaran BBoll SSubramanian RZheng LSingh VCesar PXie LXu D(2024)D3U-Net: Dual-Domain Collaborative Optimization Deep Unfolding Network for Image Compressive SensingProceedings of the 32nd ACM International Conference on Multimedia10.1145/3664647.3681532(9952-9960)Online publication date: 28-Oct-2024
https://dl.acm.org/doi/10.1145/3664647.3681532
Show More Cited By

Recommendations

Image compressive sensing via Truncated Schatten-p Norm regularization

Low-rank property as a useful image prior has attracted much attention in image processing communities. Recently, a nonlocal low-rank regularization (NLR) approach toward exploiting low-rank property has shown the state-of-the-art performance in ...
Compressive sensing via nonlocal low-rank tensor regularization

The aim of Compressing sensing (CS) is to acquire an original signal, when it is sampled at a lower rate than Nyquist rate previously. In the framework of CS, the original signal is often assumed to be sparse and correlated in some domain. Recently, ...
Block compressive sensing

Iterative re-weighted l1 norm minimization reconstruction algorithm.Joint reconstruction algorithm for multiview images and video frames.The correlation between multiple images is exploited through joint sparsity model. Compressive sensing provides ...

Comments

Information & Contributors

Information

Published In

cover image IEEE Transactions on Image Processing

IEEE Transactions on Image Processing Volume 32, Issue

2023

5324 pages

ISSN:1057-7149

Issue’s Table of Contents

1941-0042 © 2023 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See https://www.ieee.org/publications/rights/index.html for more information.

Publisher

IEEE Press

Publication History

Published: 01 January 2023

Qualifiers

Research-article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

11
Total Citations
View Citations
0
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 10 Feb 2025

Other Metrics

View Author Metrics

Citations

Cited By

Song HGong JMeng HLai YWooldridge MDy JNatarajan S(2024)Multi-cross sampling and frequency-division reconstruction for image compressed sensingProceedings of the Thirty-Eighth AAAI Conference on Artificial Intelligence and Thirty-Sixth Conference on Innovative Applications of Artificial Intelligence and Fourteenth Symposium on Educational Advances in Artificial Intelligence10.1609/aaai.v38i5.28294(4909-4917)Online publication date: 20-Feb-2024
https://dl.acm.org/doi/10.1609/aaai.v38i5.28294
Han KWang JShi YCai HLing NYin B(2024)WTDUN: Wavelet Tree-Structured Sampling and Deep Unfolding Network for Image Compressed SensingACM Transactions on Multimedia Computing, Communications, and Applications10.1145/370173121:1(1-22)Online publication date: 26-Oct-2024
https://dl.acm.org/doi/10.1145/3701731
Han KWang JShi YLing NYin BCai JKankanhalli MPrabhakaran BBoll SSubramanian RZheng LSingh VCesar PXie LXu D(2024)D3U-Net: Dual-Domain Collaborative Optimization Deep Unfolding Network for Image Compressive SensingProceedings of the 32nd ACM International Conference on Multimedia10.1145/3664647.3681532(9952-9960)Online publication date: 28-Oct-2024
https://dl.acm.org/doi/10.1145/3664647.3681532
Hui CZhu HYan SLiu SJiang FZhao DCai JKankanhalli MPrabhakaran BBoll SSubramanian RZheng LSingh VCesar PXie LXu D(2024)S2-CSNet: Scale-Aware Scalable Sampling Network for Image Compressive SensingProceedings of the 32nd ACM International Conference on Multimedia10.1145/3664647.3681323(2515-2524)Online publication date: 28-Oct-2024
https://dl.acm.org/doi/10.1145/3664647.3681323
Cui WFan XZhang JZhao D(2024)Deep Unfolding Network for Image Compressed Sensing by Content-Adaptive Gradient Updating and Deformation-Invariant Non-Local ModelingIEEE Transactions on Multimedia10.1109/TMM.2023.332142426(4012-4027)Online publication date: 1-Jan-2024
https://dl.acm.org/doi/10.1109/TMM.2023.3321424
Kong XChen YHe Z(2024)When Channel Correlation Meets Sparse Prior: Keeping Interpretability in Image Compressive SensingIEEE Transactions on Multimedia10.1109/TMM.2023.330582826(2953-2965)Online publication date: 1-Jan-2024
https://dl.acm.org/doi/10.1109/TMM.2023.3305828
Zheng BSun GDong LWang S(2024)LD-CSNetNeural Networks10.1016/j.neunet.2024.106541179:COnline publication date: 1-Nov-2024
https://dl.acm.org/doi/10.1016/j.neunet.2024.106541
Cui WWang XFan XLiu SMa CZhao DEl Saddik AMei TCucchiara RBertini MTobon Vallejo DAtrey PHossain M(2023)G2-DUN: Gradient Guided Deep Unfolding Network for Image Compressive SensingProceedings of the 31st ACM International Conference on Multimedia10.1145/3581783.3612339(7933-7942)Online publication date: 26-Oct-2023
https://dl.acm.org/doi/10.1145/3581783.3612339
Gan HWang XHe LLiu J(2023)Learned Two-Step Iterative Shrinkage Thresholding Algorithm for Deep Compressive SensingIEEE Transactions on Circuits and Systems for Video Technology10.1109/TCSVT.2023.332534034:5(3943-3956)Online publication date: 17-Oct-2023
https://dl.acm.org/doi/10.1109/TCSVT.2023.3325340
Yang JWang HTaniguchi IFan YZhou J(2023)aVCSR: Adaptive Video Compressive Sensing Using Region-of-Interest Detection in the Compressed DomainIEEE MultiMedia10.1109/MMUL.2023.334206231:1(19-32)Online publication date: 14-Dec-2023
https://dl.acm.org/doi/10.1109/MMUL.2023.3342062
Show More Cited By

View Options

View options

Figures

Tables

Media

View Issue’s Table of Contents