Cited By
View all- Wang YLu TYao YZhang YXiong Z(2024)Learning to Hallucinate Face in the DarkIEEE Transactions on Multimedia10.1109/TMM.2023.329480826(2314-2326)Online publication date: 1-Jan-2024
SMNet: Synchronous Multi-Scale Low Light Enhancement Network With Local and Global Concern
Abstract
Limited by objectively poor lighting conditions and hardware devices, low-light images with low visual quality and low visibility are inevitable in the real world. Accurate local details and reasonable global information play their essential and distinct roles in low-light image enhancement: local details contribute to fine textures, while global information is critical for a proper understanding of the global brightness level. In this article, we focus on integrating local and global aspects to achieve high-quality low-light image enhancement by proposing the synchronous multi-scale low-light enhancement network (SMNet). A synchronous multi-scale representation learning structure and a global feature recalibration module are adopted in SMNet. Different from the traditional multi-scale feature learning architecture, SMNet carries out the multi-scale representation learning in a synchronous way: we first calculate the rough contextual representations in a top-down manner and then learn multi-scale representations in a bottom-up way to generate representations with rich local details. To acquire global brightness information, a global feature recalibration module (GFRM) is applied after the synchronous multi-scale representations to perceive and exploit proper global information by global pooling and projection to recalibrate channel weights globally. The synchronous multi-scale representation and GFRM compose the basic local-and-global block. Experimental results on mainstream real-world dataset LOL and synthetic dataset MIT-Adobe FiveK show that the proposed SMNet not only leads the way on objective metrics (0.41/2.31 improvement of PSNR on two datasets) but is also superior in subjective comparisons compared with typical SoTA methods.
References
[1]
S.-C. Huang, F.-C. Cheng, and Y.-S. Chiu, “Efficient contrast enhancement using adaptive gamma correction with weighting distribution,” IEEE Trans. Image Process., vol. 22, no. 3, pp. 1032–1041, Mar. 2013.
[2]
S. M. Pizer et al., “Adaptive histogram equalization and its variations,” Comput. Vis., Graph., Image Process., vol. 39, no. 3, pp. 355–368, 1987.
[3]
E. D. Pisano et al., “Contrast limited adaptive histogram equalization image processing to improve the detection of simulated spiculations in dense mammograms,” J. Digit. Imag., vol. 11, no. 4, 1998, Art. no.
[4]
S. Wang, J. Zheng, H.-M. Hu, and B. Li, “Naturalness preserved enhancement algorithm for non-uniform illumination images,” IEEE Trans. Image Process., vol. 22, no. 9, pp. 3538–3548, Sep. 2013.
[5]
Z. Ying, G. Li, Y. Ren, R. Wang, and W. Wang, “A new low-light image enhancement algorithm using camera response model,” in Proc. IEEE Int. Conf. Comput. Vis. Workshops, 2017, pp. 3015–3022.
[6]
D. J. Jobson, Z.-u. Rahman, and G. A. Woodell, “Properties and performance of a center/surround retinex,” IEEE Trans. Image Process., vol. 6, no. 3, pp. 451–462, Mar. 1997.
[7]
D. J. Jobson, Z.-u. Rahman, and G. A. Woodell, “A multiscale retinex for bridging the gap between color images and the human observation of scenes,” IEEE Trans. Image Process., vol. 6, no. 7, pp. 965–976, Jul. 1997.
[8]
C.-H. Lee, J.-L. Shih, C.-C. Lien, and C.-C. Han, “Adaptive multiscale retinex for image contrast enhancement,” in Proc. Int. Conf. Signal-Image Technol. Internet-Based Syst., 2013, pp. 43–50.
[9]
C. Wei, W. Wang, W. Yang, and J. Liu, “Deep retinex decomposition for low-light enhancement,” in Proc. 29th Brit. Mach. Vis. Conf., 2018, pp. 1–12.
[10]
S. W. Zamir et al., “Learning enriched features for real image restoration and enhancement,” in Proc. Eur. Conf. Comput. Vis., 2020, pp. 492–511.
[11]
W. Yang, S. Wang, Y. Fang, Y. Wang, and J. Liu, “From fidelity to perceptual quality: A semi-supervised approach for low-light image enhancement,” in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit., 2020, pp. 3063–3072.
[12]
K. G. Lore, A. Akintayo, and S. Sarkar, “LLNet: A deep autoencoder approach to natural low-light image enhancement,” Pattern Recognit., vol. 61, pp. 650–662, 2017.
[13]
J. Liang et al., “Deep bilateral retinex for low-light image enhancement,” 2020, arXiv:2007.02018.
[14]
R. Wang et al., “Underexposed photo enhancement using deep illumination estimation,” in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit., 2019, pp. 6849–6857.
[15]
Y. Jiang et al., “EnlightenGAN: Deep light enhancement without paired supervision,” IEEE Trans. Image Process., vol. 30, pp. 2340–2349, 2021.
[16]
W. Wang, S. Zhao, J. Shen, S. C. Hoi, and A. Borji, “Salient object detection with pyramid attention and salient edges,” in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit., 2019, pp. 1448–1457.
[17]
Y. Zhang, J. Zhang, and X. Guo, “Kindling the darkness: A practical low-light image enhancer,” in Proc. 27th ACM Int. Conf. Multimedia, 2019, pp. 1632–1640.
[18]
Y. Zhang, X. Di, B. Zhang, and C. Wang, “Self-supervised image enhancement network: Training with low light images only,” 2020, arXiv:2002.11300.
[19]
M. Zhu, P. Pan, W. Chen, and Y. Yang, “Eemefn: Low-light image enhancement via edge-enhanced multi-exposure fusion network,” in Proc. 34th AAAI Conf. Artif. Intell., 2020, pp. 13106–13113.
[20]
C. Chen, Q. Chen, J. Xu, and V. Koltun, “Learning to see in the dark,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., 2018, pp. 3291–3300.
[21]
I. Goodfellow et al., “Generative adversarial nets,” in Proc. Adv. Neural Inf. Process. Syst., 2014, pp. 2672–2680.
[22]
L.-W. Wang, Z.-S. Liu, W.-C. Siu, and D. P. Lun, “Lightening network for low-light image enhancement,” IEEE Trans. Image Process., vol. 29, pp. 7984–7996, 2020.
[23]
M. Haris, G. Shakhnarovich, and N. Ukita, “Deep back-projection networks for super-resolution,” in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit., 2018, pp. 1664–1673.
[24]
K. Lu and L. Zhang, “TBEFN: A two-branch exposure-fusion network for low-light image enhancement,” IEEE Trans. Multimedia, vol. 23, pp. 4093–4105, 2021.
[25]
C. Guo et al., “Zero-reference deep curve estimation for low-light image enhancement,” in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit., 2020, pp. 1780–1789.
[26]
K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., 2016, pp. 770–778.
[27]
K. Simonyan and A. Zisserman, “Very deep convolutional networks for large-scale image recognition,” in Proc. 3rd Int. Conf. Learn. Representations, 2015, pp. 1–14.
[28]
C. Szegedy et al., “Going deeper with convolutions,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., 2015, pp. 1–9.
[29]
K. Kumar and D. D. Shrimankar, “Deep event learning boost-up approach: DELTA,” Multimedia Tools Appl., vol. 77, no. 20, pp. 26635–26655, 2018.
[30]
A. Vaswani et al., “Attention is all you need,” in Proc. 31st Int. Conf. Neural Inf. Process. Syst., 2017, pp. 6000–6010.
[31]
X. Xu, S. Wang, Z. Wang, X. Zhang, and R. Hu, “Exploring image enhancement for salient object detection in low light images,” ACM Trans. Multimedia Comput., Commun., Appl., vol. 17, no. 1s, pp. 1–19, Mar. 2021.
[32]
H. Zhang, Y. Dai, H. Li, and P. Koniusz, “Deep stacked hierarchical multi-patch network for image deblurring,” in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit., 2019, pp. 5978–5986.
[33]
S. Nah, T. Hyun Kim, and K. M. Lee, “Deep multi-scale convolutional neural network for dynamic scene deblurring,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., 2017, pp. 3883–3891.
[34]
X. Tao, H. Gao, X. Shen, J. Wang, and J. Jia, “Scale-recurrent network for deep image deblurring,” in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit., 2018, pp. 8174–8182.
[35]
X. Liu, Y. Ma, Z. Shi, and J. Chen, “Griddehazenet: Attention-based multi-scale network for image dehazing,” in Proc. IEEE/CVF Int. Conf. Comput. Vis., 2019, pp. 7314–7323.
[36]
A. Zhang, X. Jiang, B. Zhang, and X. Cao, “Multi-scale supervised attentive encoder-decoder network for crowd counting,” ACM Trans. Multimedia Comput., Commun., Appl., vol. 16, no. 1s, pp. 1–20, Mar. 2020.
[37]
S. Lim and W. Kim, “DSLR: Deep stacked Laplacian restorer for low-light image enhancement,” IEEE Trans. Multimedia, vol. 23, pp. 4272–4284, 2021.
[38]
M. Yang et al., “DPFL-Nets: Deep pyramid feature learning networks for multiscale change detection,” IEEE Trans. Neural Netw. Learn. Syst., vol. 33, no. 11, pp. 6402–6416, Nov. 2022.
[39]
A. Ranjan and M. J. Black, “Optical flow estimation using a spatial pyramid network,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., 2017, pp. 4161–4170.
[40]
D. Sun, X. Yang, M.-Y. Liu, and J. Kautz, “PWC-Net: CNNs for optical flow using pyramid, warping, and cost volume,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., 2018, pp. 8934–8943.
[41]
R. Xu, Z. Xiao, J. Huang, Y. Zhang, and Z. Xiong, “EDPN: Enhanced deep pyramid network for blurry image restoration,” in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit., 2021, pp. 414–423.
[42]
X. Fu, B. Liang, Y. Huang, X. Ding, and J. Paisley, “Lightweight pyramid networks for image deraining,” IEEE Trans. Neural Netw. Learn. Syst., vol. 31, no. 6, pp. 1794–1807, Jun. 2020.
[43]
F. Wang et al., “Residual attention network for image classification,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., 2017, pp. 3156–3164.
[44]
J. Hu, L. Shen, and G. Sun, “Squeeze-and-excitation networks,” in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit., 2018, pp. 7132–7141.
[45]
Y. Zhang et al., “Image super-resolution using very deep residual channel attention networks,” in Proc. Eur. Conf. Comput. Vis., 2018, pp. 286–301.
[46]
S. Woo, J. Park, J.-Y. Lee, and I. S. Kweon, “CBAM: Convolutional block attention module,” in Proc. Eur. Conf. Comput. Vis., 2018, pp. 3–19.
[47]
H. Lee, H.-E. Kim, and H. Nam, “SRM: A style-based recalibration module for convolutional neural networks,” in Proc. IEEE/CVF Int. Conf. Comput. Vis., 2019, pp. 1854–1862.
[48]
D. Zhang, J. Shao, and H. T. Shen, “Kernel attention network for single image super-resolution,” ACM Trans. Multimedia Comput., Commun., Appl., vol. 16, no. 3, pp. 1–15, Jul. 2020.
[49]
K. Dabov, A. Foi, V. Katkovnik, and K. Egiazarian, “Image denoising by sparse 3-D transform-domain collaborative filtering,” IEEE Trans. Image Process., vol. 16, no. 8, pp. 2080–2095, Aug. 2007.
[50]
A. Buades, B. Coll, and J.-M. Morel, “A non-local algorithm for image denoising,” in Proc. IEEE Comput. Soc. Conf. Comput. Vis. Pattern Recognit., 2005, vol. 2, pp. 60–65.
[51]
K. Zhang, W. Zuo, Y. Chen, D. Meng, and L. Zhang, “Beyond a Gaussian denoiser: Residual learning of deep CNN for image denoising,” IEEE Trans. Image Process., vol. 26, no. 7, pp. 3142–3155, Jul. 2017.
[52]
X. Mao, C. Shen, and Y.-B. Yang, “Image restoration using very deep convolutional encoder-decoder networks with symmetric skip connections,” in Proc. Adv. Neural Inf. Process. Syst., 2016, vol. 29, pp. 2810–2818.
[53]
M. Singh, M. C. Govil, E. S. Pilli, and S. K. Vipparthi, “SOD-CED: Salient object detection for noisy images using convolution encoder–decoder,” IET Comput. Vis., vol. 13, no. 6, pp. 578–587, 2019.
[54]
T. Huang, S. Li, X. Jia, H. Lu, and J. Liu, “Neighbor2neighbor: Self-supervised denoising from single noisy images,” in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit., 2021, pp. 14781–14790.
[55]
G. Ghiasi, T.-Y. Lin, and Q. V. Le, “NAS-FPN: Learning scalable feature pyramid architecture for object detection,” in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit., 2019, pp. 7036–7045.
[56]
Y.-C. Chen, C. Gao, E. Robb, and J.-B. Huang, “NAS-DIP: Learning deep image prior with neural architecture search,” in Proc. Eur. Conf. Comput. Vis., 2020, pp. 442–459.
[57]
Z. Daquan, Q. Hou, Y. Chen, J. Feng, and S. Yan, “Rethinking bottleneck structure for efficient mobile network design,” in Proc. Eur. Conf. Comput. Vis., Cham, Switzerland, 2020, pp. 680–697.
[58]
J. Cai, S. Gu, and L. Zhang, “Learning a deep single image contrast enhancer from multi-exposure images,” IEEE Trans. Image Process., vol. 27, no. 4, pp. 2049–2062, Apr. 2018.
[59]
M. Sandler, A. Howard, M. Zhu, A. Zhmoginov, and L.-C. Chen, “Mobilenetv2: Inverted residuals and linear bottlenecks,” in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit., 2018, pp. 4510–4520.
[60]
J. Johnson, A. Alahi, and L. Fei-Fei, “Perceptual losses for real-time style transfer and super-resolution,” in Proc. Eur. Conf. Comput. Vis., 2016, pp. 694–711.
[61]
L. I. Rudin, S. Osher, and E. Fatemi, “Nonlinear total variation based noise removal algorithms,” Physica D: Nonlinear Phenomena, vol. 60, no. 1/4, pp. 259–268, 1992.
[62]
A. Paszke et al., “Automatic differentiation in pytorch,” in Proc. Workshop Autodiff, 2017, pp. 1–4.
[63]
Y.-S. Chen, Y.-C. Wang, M.-H. Kao, and Y.-Y. Chuang, “Deep photo enhancer: Unpaired learning for image enhancement from photographs with GANs,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., 2018, pp. 6306–6314.
[64]
V. Bychkovsky, S. Paris, E. Chan, and F. Durand, “Learning photographic global tonal adjustment with a database of input/output image pairs,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., 2011, pp. 97–104.
[65]
D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” in Proc. 3rd Int. Conf. Learn. Representations, 2015, pp. 1–15.
[66]
X. Guo, Y. Li, and H. Ling, “LIME: Low-light image enhancement via illumination map estimation,” IEEE Trans. Image Process., vol. 26, no. 2, pp. 982–993, Feb. 2017.
[67]
Q. Zhang, Y. Nie, and W.-S. Zheng, “Dual illumination estimation for robust exposure correction,” Comput. Graph. Forum, vol. 38, no. 7. 2019, pp. 243–252.
[68]
B. Cai et al., “A joint intrinsic-extrinsic prior model for retinex,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., 2017, pp. 4000–4009.
[69]
Z. Wang, A. C. Bovik, H. R. Sheikh, and E. P. Simoncelli, “Image quality assessment: From error visibility to structural similarity,” IEEE Trans. Image Process., vol. 13, no. 4, pp. 600–612, Apr. 2004.
[70]
Z. Wang and Q. Li, “Information content weighting for perceptual image quality assessment,” IEEE Trans. Image Process., vol. 20, no. 5, pp. 1185–1198, May 2011.
[71]
H. R. Sheikh and A. C. Bovik, “Image information and visual quality,” IEEE Trans. Image Process., vol. 15, no. 2, pp. 430–444, Feb. 2006.
[72]
L. Zhang, L. Zhang, X. Mou, and D. Zhang, “FSIM: A feature similarity index for image quality assessment,” IEEE Trans. Image Process., vol. 20, no. 8, pp. 2378–2386, Aug. 2011.
[73]
L. Zhang, L. Zhang, X. Mou, and D. Zhang, “A comprehensive evaluation of full reference image quality assessment algorithms,” in Proc. IEEE 19th Int. Conf. Image Process., 2012, pp. 1477–1480.
[74]
Q. Wang et al., “ECA-Net: Efficient channel attention for deep convolutional neural networks,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., 2020, pp. 11531–11539.
[75]
Q. Hou, D. Zhou, and J. Feng, “Coordinate attention for efficient mobile network design,” in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit., 2021, pp. 13713–13722.
Index Terms
- SMNet: Synchronous Multi-Scale Low Light Enhancement Network With Local and Global Concern
Index terms have been assigned to the content through auto-classification.
Recommendations
Reversible data hiding based on global and local automatic contrast enhancement of low-light color images
AbstractCurrently, many scholars have investigated reversible data hiding (RDH) based on automatic contrast enhancement (ACE). For RDH based on ACE (ACERDH), various methods have been proposed on how to better improve image quality. Preserving brightness ...
Better Than Reference in Low-Light Image Enhancement: Conditional Re-Enhancement Network
Low-light images suffer from severe noise, low brightness, low contrast, etc. In previous researches, many image enhancement methods have been proposed, but few methods can deal with these problems simultaneously. In this paper, to solve these problems ...
Enhancement of low quality underwater image through integrated global and local contrast correction
A method of integration global and local contrast stretching is proposed to increase underwater image quality.The integrated modification of RGB and HSV color model is applied in the proposed method.Resultant image outperforms the current state-of-the-...
Comments
Information & Contributors
Information
Published In
1520-9210 © 2023 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See https://www.ieee.org/publications/rights/index.html for more information.
Publisher
IEEE Press
Publication History
Published: 08 March 2023
Qualifiers
- Research-article
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- View Citations1Total Citations
- 0Total Downloads
- Downloads (Last 12 months)0
- Downloads (Last 6 weeks)0
Reflects downloads up to 12 Nov 2024
Other Metrics
Citations
Cited By
View all- Wang YLu TYao YZhang YXiong Z(2024)Learning to Hallucinate Face in the DarkIEEE Transactions on Multimedia10.1109/TMM.2023.329480826(2314-2326)Online publication date: 1-Jan-2024
View Options
View options
Get Access
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in