research-article

Brighten-and-Colorize: A Decoupled Network for Customized Low-Light Image Enhancement

Authors:

Zhi JinAuthors Info & Claims

MM '23: Proceedings of the 31st ACM International Conference on Multimedia

Pages 8356 - 8366

https://doi.org/10.1145/3581783.3611907

Published: 27 October 2023 Publication History

Abstract

Low-Light Image Enhancement (LLIE) aims to improve the perceptual quality of an image captured in low-light conditions. Generally, a low-light image can be divided into lightness and chrominance components. Recent advances in this area mainly focus on the refinement of the lightness, while ignoring the role of chrominance. It easily leads to chromatic aberration and, to some extent, limits the diverse applications of chrominance in customized LLIE. In this work, a "brighten-and-colorize'' network (called BCNet), which introduces image colorization to LLIE, is proposed to address the above issues. BCNet can accomplish LLIE with accurate color and simultaneously enables customized enhancement with varying saturations and color styles based on user preferences. Specifically, BCNet regards LLIE as a multi-task learning problem: brightening and colorization. The brightening sub-task aligns with other conventional LLIE methods to get a well-lit lightness. The colorization sub-task is accomplished by regarding the chrominance of the low-light image as color guidance like the user-guide image colorization. Upon completion of model training, the color guidance (i.e., input low-light chrominance) can be simply manipulated by users to acquire customized results. This customized process is optional and, due to its decoupled nature, does not compromise the structural and detailed information of lightness. Extensive experiments on the commonly used LLIE datasets show that the proposed method achieves both State-Of-The-Art (SOTA) performance and user-friendly customization.

References

[1]

Vladimir Bychkovsky, Sylvain Paris, Eric Chan, and Frédo Durand. 2011. Learning photographic global tonal adjustment with a database of input/output image pairs. In IEEE Conf. Comput. Vis. Pattern Recog. IEEE, 97--104.

[2]

Antonin Chambolle. 2004. An algorithm for total variation minimization and applications. Journal of Mathematical Imaging and Vision, Vol. 20 (2004), 89--97.

Digital Library

[3]

Michaël Gharbi, Jiawen Chen, Jonathan T Barron, Samuel W Hasinoff, and Frédo Durand. 2017. Deep bilateral learning for real-time image enhancement. ACM Trans. Graph., Vol. 36, 4 (2017), 1--12.

Digital Library

[4]

Sergio Guadarrama, Ryan Dahl, David Bieber, Mohammad Norouzi, Jonathon Shlens, and Kevin Murphy. 2017. Pixcolor: Pixel recursive colorization. arXiv preprint arXiv:1705.07208 (2017).

[5]

Chunle Guo, Chongyi Li, Jichang Guo, Chen Change Loy, Junhui Hou, Sam Kwong, and Runmin Cong. 2020. Zero-reference deep curve estimation for low-light image enhancement. In IEEE Conf. Comput. Vis. Pattern Recog. 1780--1789.

[6]

Mingming He, Dongdong Chen, Jing Liao, Pedro V Sander, and Lu Yuan. 2018. Deep exemplar-based colorization. ACM Trans. Graph., Vol. 37, 4 (2018), 1--16.

Digital Library

[7]

Jie Huang, Yajing Liu, Feng Zhao, Keyu Yan, Jinghao Zhang, Yukun Huang, Man Zhou, and Zhiwei Xiong. 2022. Deep Fourier-Based Exposure Correction Network with Spatial-Frequency Interaction. In Eur. Conf. Comput. Vis. Springer, 163--180.

[8]

Yifan Jiang, Xinyu Gong, Ding Liu, Yu Cheng, Chen Fang, Xiaohui Shen, Jianchao Yang, Pan Zhou, and Zhangyang Wang. 2021. Enlightengan: Deep light enhancement without paired supervision. IEEE Trans. Image Process., Vol. 30 (2021), 2340--2349.

Digital Library

[9]

Zhi Jin, Muhammad Zafar Iqbal, Dmytro Bobkov, Wenbin Zou, Xia Li, and Eckehard Steinbach. 2019. A flexible deep CNN framework for image restoration. IEEE Trans. Multimedia, Vol. 22, 4 (2019), 1055--1068.

Digital Library

[10]

Zhi Jin, Muhammad Zafar Iqbal, Wenbin Zou, Xia Li, and Eckehard Steinbach. 2020. Dual-stream multi-path recursive residual network for JPEG image compression artifacts reduction. IEEE Trans. Circuit Syst. Video Technol., Vol. 31, 2 (2020), 467--479.

[11]

Daniel J Jobson, Zia-ur Rahman, and Glenn A Woodell. 1997. A multiscale retinex for bridging the gap between color images and the human observation of scenes. IEEE Trans. Image Process., Vol. 6, 7 (1997), 965--976.

Digital Library

[12]

Hanul Kim, Su-Min Choi, Chang-Su Kim, and Yeong Jun Koh. 2021. Representative color transform for image enhancement. In Int. Conf. Comput. Vis. 4459--4468.

[13]

Han-Ul Kim, Young Jun Koh, and Chang-Su Kim. 2020a. Global and local enhancement networks for paired and unpaired image enhancement. In Eur. Conf. Comput. Vis. Springer, 339--354.

Digital Library

[14]

Han-Ul Kim, Young Jun Koh, and Chang-Su Kim. 2020b. PieNet: Personalized image enhancement network. In Eur. Conf. Comput. Vis. Springer, 374--390.

Digital Library

[15]

Diederik P Kingma and Jimmy Ba. 2015. Adam: A Method for Stochastic Optimization. In Int. Conf. Learn. Represent.

[16]

Durk P Kingma and Prafulla Dhariwal. 2018. Glow: Generative flow with invertible 1x1 convolutions. Adv. Neural Inform. Process. Syst., Vol. 31 (2018).

[17]

Manoj Kumar, Dirk Weissenborn, and Nal Kalchbrenner. 2020. Colorization Transformer. In Int. Conf. Learn. Represent.

[18]

Wei-Sheng Lai, Jia-Bin Huang, Narendra Ahuja, and Ming-Hsuan Yang. 2018. Fast and accurate image super-resolution with deep laplacian pyramid networks. IEEE Trans. Pattern Anal. Mach. Intell., Vol. 41, 11 (2018), 2599--2613.

[19]

Chongyi Li, Chun-Le Guo, Man Zhou, Zhexin Liang, Shangchen Zhou, Ruicheng Feng, and Chen Change Loy. 2023. Embedding Fourier for Ultra-High-Definition Low-Light Image Enhancement. arXiv preprint arXiv:2302.11831 (2023).

[20]

Xiang Li, Wenhai Wang, Xiaolin Hu, and Jian Yang. 2019. Selective kernel networks. In IEEE Conf. Comput. Vis. Pattern Recog. 510--519.

[21]

Risheng Liu, Long Ma, Jiaao Zhang, Xin Fan, and Zhongxuan Luo. 2021. Retinex-inspired unrolling with cooperative prior architecture search for low-light image enhancement. In IEEE Conf. Comput. Vis. Pattern Recog. 10561--10570.

[22]

Kin Gwn Lore, Adedotun Akintayo, and Soumik Sarkar. 2017. LLNet: A deep autoencoder approach to natural low-light image enhancement. Pattern Recognition, Vol. 61 (2017), 650--662.

Digital Library

[23]

Peng Lu, Jinbei Yu, Xujun Peng, Zhaoran Zhao, and Xiaojie Wang. 2020. Gray2colornet: Transfer more colors from reference image. In ACM Int. Conf. Multimedia. 3210--3218.

Digital Library

[24]

Long Ma, Tengyu Ma, Risheng Liu, Xin Fan, and Zhongxuan Luo. 2022. Toward fast, flexible, and robust low-light image enhancement. In IEEE Conf. Comput. Vis. Pattern Recog. 5637--5646.

[25]

Sean Moran, Pierre Marza, Steven McDonagh, Sarah Parisot, and Gregory Slabaugh. 2020. Deeplpf: Deep local parametric filters for image enhancement. In IEEE Conf. Comput. Vis. Pattern Recog. 12826--12835.

[26]

Zhangkai Ni, Wenhan Yang, Shiqi Wang, Lin Ma, and Sam Kwong. 2020. Towards unsupervised deep image enhancement with generative adversarial network. IEEE Trans. Image Process., Vol. 29 (2020), 9140--9151.

[27]

Qingzhe Pan, Zhifu Zhao, Xuemei Xie, Jianan Li, Yuhan Cao, and Guangming Shi. 2021. View-normalized skeleton generation for action recognition. In ACM Int. Conf. Multimedia. 1875--1883.

Digital Library

[28]

Stephen M Pizer, E Philip Amburn, John D Austin, Robert Cromartie, Ari Geselowitz, Trey Greer, Bart ter Haar Romeny, John B Zimmerman, and Karel Zuiderveld. 1987. Adaptive histogram equalization and its variations. Computer vision, graphics, and image processing, Vol. 39, 3 (1987), 355--368.

[29]

Zia-ur Rahman, Daniel J Jobson, and Glenn A Woodell. 2004. Retinex processing for automatic image enhancement. Journal of Electronic imaging, Vol. 13, 1 (2004), 100--110.

[30]

Erik Reinhard, Michael Adhikhmin, Bruce Gooch, and Peter Shirley. 2001. Color transfer between images. IEEE Computer Graphics and Applications, Vol. 21, 5 (2001), 34--41.

Digital Library

[31]

Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. 2015. Faster r-cnn: Towards real-time object detection with region proposal networks. Adv. Neural Inform. Process. Syst., Vol. 28 (2015).

[32]

Karen Simonyan and Andrew Zisserman. 2014. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556 (2014).

[33]

Jheng-Wei Su, Hung-Kuo Chu, and Jia-Bin Huang. 2020. Instance-aware image colorization. In IEEE Conf. Comput. Vis. Pattern Recog. 7968--7977.

[34]

Xiaopeng Sun, Muxingzi Li, Tianyu He, and Lubin Fan. 2021. Enhance images as you like with unpaired learning. arXiv preprint arXiv:2110.01161 (2021).

[35]

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Adv. Neural Inform. Process. Syst., Vol. 30 (2017).

[36]

Haolin Wang, Jiawei Zhang, Ming Liu, Xiaohe Wu, and Wangmeng Zuo. 2022. Learning Diverse Tone Styles for Image Retouching. arXiv preprint arXiv:2207.05430 (2022).

[37]

Ruixing Wang, Qing Zhang, Chi-Wing Fu, Xiaoyong Shen, Wei-Shi Zheng, and Jiaya Jia. 2019. Underexposed photo enhancement using deep illumination estimation. In IEEE Conf. Comput. Vis. Pattern Recog. 6849--6857.

[38]

Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli. 2004. Image quality assessment: from error visibility to structural similarity. IEEE Trans. Image Process., Vol. 13, 4 (2004), 600--612.

Digital Library

[39]

Chen Wei, Wenjing Wang, Wenhan Yang, and Jiaying Liu. 2018. Deep retinex decomposition for low-light enhancement. In Brit. Mach. Vis. Conf.

[40]

Hongjun Wu, Haoran Qi, Jingzhou Luo, Yining Li, and Zhi Jin. 2022. A Lightweight Image Entropy-Based Divide-and-Conquer Network for Low-Light Image Enhancement. In Int. Conf. Multimedia and Expo. IEEE, 01-06.

[41]

Xiaogang Xu, Ruixing Wang, Chi-Wing Fu, and Jiaya Jia. 2022. SNR-Aware Low-Light Image Enhancement. In IEEE Conf. Comput. Vis. Pattern Recog. 17714--17724.

[42]

Bowen Yang, Chun Yang, Qi Liu, and Xu-Cheng Yin. 2019. Joint rotation-invariance face detection and alignment with angle-sensitivity cascaded networks. In ACM Int. Conf. Multimedia. 1473--1480.

Digital Library

[43]

Wenhan Yang, Wenjing Wang, Haofeng Huang, Shiqi Wang, and Jiaying Liu. 2021. Sparse gradient regularized deep retinex network for robust low-light image enhancement. IEEE Trans. Image Process., Vol. 30 (2021), 2072--2086.

[44]

Wang Yin, Peng Lu, Zhaoran Zhao, and Xujun Peng. 2021. Yes,“ Attention Is All You Need", for Exemplar based Colorization. In ACM Int. Conf. Multimedia. 2243--2251.

Digital Library

[45]

SW Zamir, A Arora, SH Khan, H Munawar, FS Khan, MH Yang, and L Shao. 2022. Learning Enriched Features for Fast Image Restoration and Enhancement. IEEE Trans. Pattern Anal. Mach. Intell. (2022).

[46]

Syed Waqas Zamir, Aditya Arora, Salman Khan, Munawar Hayat, Fahad Shahbaz Khan, Ming-Hsuan Yang, and Ling Shao. 2020. Learning enriched features for real image restoration and enhancement. In Eur. Conf. Comput. Vis. Springer, 492--511.

Digital Library

[47]

Rongkai Zhang, Lanqing Guo, Siyu Huang, and Bihan Wen. 2021a. ReLLIE: Deep reinforcement learning for customized low-light image enhancement. In ACM Int. Conf. Multimedia. 2429--2437.

Digital Library

[48]

Richard Zhang, Phillip Isola, and Alexei A Efros. 2016. Colorful image colorization. In Eur. Conf. Comput. Vis. Springer, 649--666.

[49]

Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. 2018. The unreasonable effectiveness of deep features as a perceptual metric. In IEEE Conf. Comput. Vis. Pattern Recog. 586--595.

[50]

Richard Yi Zhang, Jun Yan Zhu, Phillip Isola, Xinyang Geng, Angela S Lin, Tianhe Yu, and Alexei A Efros. 2017. Real-time user-guided image colorization with learned deep priors. ACM Trans. Graph., Vol. 36, 4 (2017), 119.

Digital Library

[51]

Yonghua Zhang, Xiaojie Guo, Jiayi Ma, Wei Liu, and Jiawan Zhang. 2021b. Beyond brightening low-light images. Int. J. Comput. Vis., Vol. 129, 4 (2021), 1013--1037.

Digital Library

[52]

Yonghua Zhang, Jiawan Zhang, and Xiaojie Guo. 2019. Kindling the darkness: A practical low-light image enhancer. In ACM Int. Conf. Multimedia. 1632--1640.

Digital Library

[53]

Zhao Zhang, Huan Zheng, Richang Hong, Mingliang Xu, Shuicheng Yan, and Meng Wang. 2022. Deep Color Consistent Network for Low-Light Image Enhancement. In IEEE Conf. Comput. Vis. Pattern Recog. 1899--1908.

[54]

Jiaojiao Zhao, Jungong Han, Ling Shao, and Cees GM Snoek. 2020. Pixelated semantic colorization. Int. J. Comput. Vis., Vol. 128, 4 (2020), 818--834.

Digital Library

[55]

Suiyi Zhao, Zhao Zhang, Richang Hong, Mingliang Xu, Yi Yang, and Meng Wang. 2022. FCL-GAN: A Lightweight and Real-Time Baseline for Unsupervised Blind Image Deblurring. arXiv preprint arXiv:2204.07820 (2022).

[56]

Suiyi Zhao, Zhao Zhang, Richang Hong, Mingliang Xu, Haijun Zhang, Meng Wang, and Shuicheng Yan. 2021. Unsupervised color retention network and new quantization metric for blind motion deblurring. TechRxiv Preprint (2021), 1--20.

[57]

Naishan Zheng, Jie Huang, Qi Zhu, Man Zhou, Feng Zhao, and Zheng-Jun Zha. 2022. Enhancement by Your Aesthetic: An Intelligible Unsupervised Personalized Enhancer for Low-Light Images. In ACM Int. Conf. Multimedia. 6521--6529.

Digital Library

Cited By

Li YXu RNiu YGuo WZhao T(2024)Perceptual Decoupling With Heterogeneous Auxiliary Tasks for Joint Low-Light Image Enhancement and DeblurringIEEE Transactions on Multimedia10.1109/TMM.2024.335563426(6663-6675)Online publication date: 2024
https://doi.org/10.1109/TMM.2024.3355634
Qiu YZhang KWang CLuo WLi HJin Z(2023)MB-TaylorFormer: Multi-branch Efficient Transformer Expanded by Taylor Formula for Image Dehazing2023 IEEE/CVF International Conference on Computer Vision (ICCV)10.1109/ICCV51070.2023.01176(12756-12767)Online publication date: 1-Oct-2023
https://doi.org/10.1109/ICCV51070.2023.01176

Index Terms

Brighten-and-Colorize: A Decoupled Network for Customized Low-Light Image Enhancement
1. Computing methodologies
  1. Artificial intelligence
    1. Computer vision
  2. Computer graphics
    1. Image manipulation
      1. Image processing

Recommendations

A Low-light Image Enhancement Method Balancing Brightness Boosting and Noise Smoothing
CSAI '22: Proceedings of the 2022 6th International Conference on Computer Science and Artificial Intelligence

Images captured in low-light or uneven lighting environments need to use low-light image enhancement techniques to boost brightness and contrast for high-quality improved results. Some existing low-light image enhancement methods over-enhance the image ...
Low-light image enhancement using inverted image normalized by atmospheric light
Highlights
- This paper presents a novel low-light image enhancement algorithm using the inverted image normalized by atmospheric light.
- A mathematical connection between the Retinex theory and the atmospheric scattering model by normalizing an ...
Abstract
Images captured under low-light conditions are characterized by problems, including low brightness, low contrast, and color distortion. Low-light image enhancement, aims to enhance visual effects for the benefit of subsequent image processing and ...
Low-light image enhancement based on GAN with attention mechanism and color Constancy
Abstract
Images captured in low-light often suffer from severe quality degraded problems, such as low contrast and color distortion, which make it intractable for further computer vision tasks. To solve the problems above, we proposed a trainable parallel ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

MM '23: Proceedings of the 31st ACM International Conference on Multimedia

October 2023

9913 pages

ISBN:9798400701085

DOI:10.1145/3581783

General Chairs:
Abdulmotaleb El Saddik
University of Ottawa, Canada & MBZUAI, UAE
,
Tao Mei
HiDream.ai, China
,
Rita Cucchiara
University of Modena and Reggio Emilia, Italy
,
Program Chairs:
Marco Bertini
University of Florence, Italy
,
Diana Patricia Tobon Vallejo
Unversidad de Medellin, Colombia
,
Pradeep K. Atrey
University at Albany, State University of New York, USA
,
M. Shamim Hossain
M. Shamim Hossain (King Saud University, KSA

Copyright © 2023 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

SIGMM: ACM Special Interest Group on Multimedia

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 27 October 2023

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

National Natural Science Foundation of China
National Natural Science Foundation of China the Sino-German Center

Conference

MM '23

Sponsor:

SIGMM

MM '23: The 31st ACM International Conference on Multimedia

October 29 - November 3, 2023

Ottawa ON, Canada

Acceptance Rates

Overall Acceptance Rate 995 of 4,171 submissions, 24%

Upcoming Conference

MM '24

Sponsor:
sigmm

The 32nd ACM International Conference on Multimedia

October 28 - November 1, 2024

Melbourne , VIC , Australia

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

2
Total Citations
View Citations
213
Total Downloads

Downloads (Last 12 months)213
Downloads (Last 6 weeks)9

Reflects downloads up to 09 Aug 2024

Other Metrics

View Author Metrics

Citations

Cited By

Li YXu RNiu YGuo WZhao T(2024)Perceptual Decoupling With Heterogeneous Auxiliary Tasks for Joint Low-Light Image Enhancement and DeblurringIEEE Transactions on Multimedia10.1109/TMM.2024.335563426(6663-6675)Online publication date: 2024
https://doi.org/10.1109/TMM.2024.3355634
Qiu YZhang KWang CLuo WLi HJin Z(2023)MB-TaylorFormer: Multi-branch Efficient Transformer Expanded by Taylor Formula for Image Dehazing2023 IEEE/CVF International Conference on Computer Vision (ICCV)10.1109/ICCV51070.2023.01176(12756-12767)Online publication date: 1-Oct-2023
https://doi.org/10.1109/ICCV51070.2023.01176

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents