Location via proxy:   [ UP ]  
[Report a bug]   [Manage cookies]                
skip to main content
10.1145/3474085.3475410acmconferencesArticle/Chapter ViewAbstractPublication PagesmmConference Proceedingsconference-collections
research-article

ReLLIE: Deep Reinforcement Learning for Customized Low-Light Image Enhancement

Published: 17 October 2021 Publication History

Abstract

Low-light image enhancement (LLIE) is a pervasive yet challenging problem, since: 1) low-light measurements may vary due to different imaging conditions in practice; 2) images can be enlightened subjectively according to diverse preference by each individual. To tackle these two challenges, this paper presents a novel deep reinforcement learning based method, dubbed ReLLIE, for customized low-light enhancement. ReLLIE models LLIE as a markov decision process, i.e., estimating the pixel-wise image-specific curves sequentially and recurrently. Given the reward computed from a set of carefully crafted non-reference loss functions, a lightweight network is proposed to estimate the curves for enlightening of a low-light image input. As ReLLIE learns a policy instead of one-one image translation, it can handle various low-light measurements and provide customized enhanced outputs by flexibly applying the policy different times. Furthermore, ReLLIE can enhance real-world images with hybrid corruptions, i.e., noise, by using a plug-and-play denoiser easily. Extensive experiments on various benchmarks demonstrate the advantages of ReLLIE, comparing to the state-of-the-art methods. (Code is available: https://github.com/GuoLanqing/ReLLIE.)

References

[1]
Mohammad Abdullah-Al-Wadud, Md Hasanul Kabir, M Ali Akber Dewan, and Oksam Chae. 2007. A dynamic histogram equalization for image contrast enhancement. IEEE Transactions on Consumer Electronics, Vol. 53, 2 (2007), 593--600.
[2]
Xueyang Fu, Delu Zeng, Yue Huang, Yinghao Liao, Xinghao Ding, and John Paisley. 2016. A fusion-based enhancing method for weakly illuminated images. Signal Processing, Vol. 129 (2016), 82--96.
[3]
Ryosuke Furuta, Naoto Inoue, and Toshihiko Yamasaki. 2019. Fully convolutional network with multi-step reinforcement learning for image processing. In Proceedings of the AAAI Conference on Artificial Intelligence. 3598--3605.
[4]
Michaël Gharbi, Jiawen Chen, Jonathan T Barron, Samuel W Hasinoff, and Frédo Durand. 2017. Deep bilateral learning for real-time image enhancement. ACM Transactions on Graphics (TOG), Vol. 36, 4 (2017), 118.
[5]
Shuhang Gu, Lei Zhang, W. Zuo, and Xiangchu Feng. 2014. Weighted Nuclear Norm Minimization with Application to Image Denoising. 2014 IEEE Conference on Computer Vision and Pattern Recognition (2014), 2862--2869.
[6]
Chunle Guo, Chongyi Li, Jichang Guo, Chen Change Loy, Junhui Hou, Sam Kwong, and Runmin Cong. 2020. Zero-Reference Deep Curve Estimation for Low-Light Image Enhancement. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 1780--1789.
[7]
Lanqing Guo, Zhiyuan Zha, Saiprasad Ravishankar, and Bihan Wen. 2021. Self-convolution: A highly-efficient operator for non-local image restoration. In ICASSP 2021--2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 1860--1864.
[8]
Xiaojie Guo, Yu Li, and Haibin Ling. 2016. LIME: Low-light image enhancement via illumination map estimation. IEEE Transactions on image processing, Vol. 26, 2 (2016), 982--993.
[9]
Yuanming Hu, Hao He, Chenxi Xu, Baoyuan Wang, and Stephen Lin. 2018. Exposure: A white-box photo post-processing framework. ACM Transactions on Graphics (TOG), Vol. 37, 2 (2018), 1--17.
[10]
Yifan Jiang, Xinyu Gong, Ding Liu, Yu Cheng, Chen Fang, Xiaohui Shen, Jianchao Yang, Pan Zhou, and Zhangyang Wang. 2019. Enlightengan: Deep light enhancement without paired supervision. arXiv preprint arXiv:1906.06972 (2019).
[11]
Aupendu Kar, Sobhan Kanti Dhara, Debashis Sen, and Prabir Kumar Biswas. 2021. Zero-shot Single Image Restoration through Controlled Perturbation of Koschmieder's Model. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).
[12]
Edwin H Land. 1977. The retinex theory of color vision. Scientific american, Vol. 237, 6 (1977), 108--129.
[13]
Chulwoo Lee, Chul Lee, and Chang-Su Kim. 2012. Contrast enhancement based on layered difference representation. In 2012 19th IEEE International Conference on Image Processing. IEEE, 965--968.
[14]
Chulwoo Lee, Chul Lee, and Chang-Su Kim. 2013. Contrast enhancement based on layered difference representation of 2D histograms. IEEE transactions on image processing, Vol. 22, 12 (2013), 5372--5384.
[15]
Ding Liu, Bihan Wen, Yuchen Fan, Chen Change Loy, and Thomas S Huang. 2018. Non-Local Recurrent Network for Image Restoration. In Advances in Neural Information Processing Systems, Vol. 31.
[16]
Kin Gwn Lore, Adedotun Akintayo, and Soumik Sarkar. 2017. LLNet: A deep autoencoder approach to natural low-light image enhancement. Pattern Recognition, Vol. 61 (2017), 650--662.
[17]
Tom Mertens, Jan Kautz, and Frank Van Reeth. 2009. Exposure fusion: A simple and practical alternative to high dynamic range photography. In Computer graphics forum, Vol. 28. Wiley Online Library, 161--171.
[18]
Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. 2016. Asynchronous methods for deep reinforcement learning. In International conference on machine learning. 1928--1937.
[19]
Jongchan Park, Joon-Young Lee, Donggeun Yoo, and In So Kweon. 2018. Distort-and-recover: Color enhancement using deep reinforcement learning. In Proceedings of the IEEE conference on computer vision and pattern recognition. 5928--5936.
[20]
Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. 2017. Automatic differentiation in pytorch. (2017).
[21]
Richard S Sutton and Andrew G Barto. 2018. Reinforcement learning: An introduction .MIT press.
[22]
Richard S Sutton, David McAllester, Satinder Singh, and Yishay Mansour. [n.d.]. Policy Gradient Methods for Reinforcement Learning with Function Approximation. In Advances in Neural Information Processing Systems. 1057--1063.
[23]
Shuhang Wang, Jin Zheng, Hai-Miao Hu, and Bo Li. 2013. Naturalness preserved enhancement algorithm for non-uniform illumination images. IEEE Transactions on Image Processing, Vol. 22, 9 (2013), 3538--3548.
[24]
Chen Wei, Wenjing Wang, Wenhan Yang, and Jiaying Liu. 2018. Deep retinex decomposition for low-light enhancement. arXiv preprint arXiv:1808.04560 (2018).
[25]
Wenhan Yang, Shiqi Wang, Yuming Fang, Yue Wang, and Jiaying Liu. 2020. From Fidelity to Perceptual Quality: A Semi-Supervised Approach for Low-Light Image Enhancement. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).
[26]
Ke Yu, Chao Dong, Liang Lin, and Chen Change Loy. 2018. Crafting a toolchain for image restoration by deep reinforcement learning. In Proceedings of the IEEE conference on computer vision and pattern recognition. 2443--2452.
[27]
Kai Zhang, Wangmeng Zuo, and Lei Zhang. 2018b. FFDNet: Toward a fast and flexible solution for CNN-based image denoising. IEEE Transactions on Image Processing, Vol. 27, 9 (2018), 4608--4622.
[28]
Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. 2018a. The Unreasonable Effectiveness of Deep Features as a Perceptual Metric. In CVPR.
[29]
Rongkai Zhang, Jiang Zhu, Zhiyuan Zha, Justin Dauwels, and Bihan Wen. 2021. R3L: Connecting Deep Reinforcement Learning to Recurrent Neural Networks for Image Denoising via Residual Recovery. arxiv: 2107.05318
[30]
Yonghua Zhang, Jiawan Zhang, and Xiaojie Guo. 2019. Kindling the darkness: A practical low-light image enhancer. In Proceedings of the 27th ACM International Conference on Multimedia. 1632--1640.
[31]
Ziqiang Zheng, Yang Wu, Xinran Han, and Jianbo Shi. 2020. ForkGAN: Seeing into the Rainy Night. In The IEEE European Conference on Computer Vision (ECCV).
[32]
Anqi Zhu, Lin Zhang, Ying Shen, Yong Ma, Shengjie Zhao, and Yicong Zhou. 2020. Zero-shot restoration of underexposed images via robust retinex decomposition. In 2020 IEEE International Conference on Multimedia and Expo (ICME). IEEE, 1--6.

Cited By

View all
  • (2024)Graph embedded low‐light image enhancement transformer based on federated learning for Internet of Vehicle under tunnel environmentComputational Intelligence10.1111/coin.1264840:2Online publication date: 11-Apr-2024
  • (2024)Fundus Image Enhancement via Semi-Supervised GAN and Anatomical Structure PreservationIEEE Transactions on Emerging Topics in Computational Intelligence10.1109/TETCI.2023.33013378:1(313-326)Online publication date: Feb-2024
  • (2024)An Improved CycleGAN-Based Model for Low-Light Image EnhancementIEEE Sensors Journal10.1109/JSEN.2023.329616724:14(21879-21892)Online publication date: 15-Jul-2024
  • Show More Cited By

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences
MM '21: Proceedings of the 29th ACM International Conference on Multimedia
October 2021
5796 pages
ISBN:9781450386517
DOI:10.1145/3474085
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 17 October 2021

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. deep reinforcement learning
  2. low-light image enhancement

Qualifiers

  • Research-article

Conference

MM '21
Sponsor:
MM '21: ACM Multimedia Conference
October 20 - 24, 2021
Virtual Event, China

Acceptance Rates

Overall Acceptance Rate 2,145 of 8,556 submissions, 25%

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)98
  • Downloads (Last 6 weeks)9
Reflects downloads up to 01 Nov 2024

Other Metrics

Citations

Cited By

View all
  • (2024)Graph embedded low‐light image enhancement transformer based on federated learning for Internet of Vehicle under tunnel environmentComputational Intelligence10.1111/coin.1264840:2Online publication date: 11-Apr-2024
  • (2024)Fundus Image Enhancement via Semi-Supervised GAN and Anatomical Structure PreservationIEEE Transactions on Emerging Topics in Computational Intelligence10.1109/TETCI.2023.33013378:1(313-326)Online publication date: Feb-2024
  • (2024)An Improved CycleGAN-Based Model for Low-Light Image EnhancementIEEE Sensors Journal10.1109/JSEN.2023.329616724:14(21879-21892)Online publication date: 15-Jul-2024
  • (2024)Equipping Diffusion Models with Differentiable Spatial Entropy for Low-Light Image Enhancement2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW)10.1109/CVPRW63382.2024.00661(6671-6681)Online publication date: 17-Jun-2024
  • (2024)Image enhancement with intensity transformation on embedding spaceCAAI Transactions on Intelligence Technology10.1049/cit2.122799:1(101-115)Online publication date: 18-Jan-2024
  • (2024)Revisiting coarse-to-fine strategy for low-light image enhancement with deep decomposition guided trainingComputer Vision and Image Understanding10.1016/j.cviu.2024.103952241:COnline publication date: 2-Jul-2024
  • (2024)PIE: Physics-Inspired Low-Light EnhancementInternational Journal of Computer Vision10.1007/s11263-024-01995-yOnline publication date: 25-Apr-2024
  • (2024)I3En: A Multi-Level Iterative Low-Light Enhancement Network Based on Sketch Prior GuidancePattern Recognition and Computer Vision10.1007/978-981-97-8692-3_6(69-83)Online publication date: 1-Nov-2024
  • (2023)Machine Learning Application for Evidence Image EnhancementHandbook of Research on Thrust Technologies’ Effect on Image Processing10.4018/978-1-6684-8618-4.ch003(25-38)Online publication date: 30-Jun-2023
  • (2023)CMID: Crossmodal Image Denoising via Pixel-Wise Deep Reinforcement LearningSensors10.3390/s2401004224:1(42)Online publication date: 20-Dec-2023
  • Show More Cited By

View Options

Get Access

Login options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Media

Figures

Other

Tables

Share

Share

Share this Publication link

Share on social media