research-article

Integrating Semantic Segmentation and Retinex Model for Low-Light Image Enhancement

Authors:

Jiaying LiuAuthors Info & Claims

MM '20: Proceedings of the 28th ACM International Conference on Multimedia

Pages 2317 - 2325

https://doi.org/10.1145/3394171.3413757

Published: 12 October 2020 Publication History

Abstract

Retinex model is widely adopted in various low-light image enhancement tasks. The basic idea of the Retinex theory is to decompose images into reflectance and illumination. The ill-posed decomposition is usually handled by hand-crafted constraints and priors. With the recently emerging deep-learning based approaches as tools, in this paper, we integrate the idea of Retinex decomposition and semantic information awareness. Based on the observation that various objects and backgrounds have different material, reflection and perspective attributes, regions of a single low-light image may require different adjustment and enhancement regarding contrast, illumination and noise. We propose an enhancement pipeline with three parts that effectively utilize the semantic layer information. Specifically, we extract the segmentation, reflectance as well as illumination layers, and concurrently enhance every separate region, i.e. sky, ground and objects for outdoor scenes. Extensive experiments on both synthetic data and real world images demonstrate the superiority of our method over current state-of-the-art low-light enhancement algorithms.

Supplementary Material

MP4 File (3394171.3413757.mp4)

We introduce our motivation, framework, and experimental results in this video. More can be found on: https://mm20-semanticreti.github.io/

Download
10.17 MB

References

[1]

M. Abdullah-Al-Wadud, M. H. Kabir, M. A. Akber Dewan, and O. Chae. 2007. A Dynamic Histogram Equalization for Image Contrast Enhancement. IEEE Trans. on Consumer Electronics, Vol. 53, 2 (May 2007), 593--600.

Digital Library

[2]

T. Arici, S. Dikbas, and Y. Altunbasak. 2009. A histogram modification framework and its application for image contrast enhancement. IEEE Trans. on Image Processing, Vol. 18, 9 (2009), 1921--1935.

Digital Library

[3]

G. Brostow, J. Shotton, J. Fauqueur, and R. Cipolla. 2008. Segmentation and Recognition Using Structure from Motion Point Clouds. In Proc. IEEE European Conf. Computer Vision. 44--57.

[4]

J. Cai, S. Gu, and L. Zhang. 2018. Learning a deep single image contrast enhancer from multi-exposure images. IEEE Trans. on Image Processing, Vol. 27, 4 (April 2018), 2049--2062.

[5]

J. Cai, W. Zuo, and L. Zhang. 2019. Extreme Channel Prior Embedded Network for Dynamic Scene Deblurring. arXiv, Vol. abs/1903.00763 (2019). arxiv: 1903.00763

[6]

C. Chen, Q. Chen, J. Xu, and V. Koltun. 2018. Learning to See in the Dark. In Proc. IEEE Int'l Conf. Computer Vision and Pattern Recognition.

[7]

M. Cordts, M. Omran, S. Ramos, T. Rehfeld, M. Enzweiler, R. Benenson, U. Franke, S. Roth, and B. Schiele. 2016. The Cityscapes Dataset for Semantic Urban Scene Understanding. In Proc. IEEE Int'l Conf. Computer Vision and Pattern Recognition.

[8]

K. Dabov, A. Foi, V. Katkovnik, and K. Egiazarian. 2007. Image Denoising by Sparse 3-D Transform-Domain Collaborative Filtering. IEEE Trans. on Image Processing, Vol. 16, 8 (Aug 2007), 2080--2095.

[9]

X. Fu, Y. Sun, M. LiWang, Y. Huang, X. Zhang, and X. Ding. 2014. A novel retinex based approach for image enhancement with illumination adjustment. In Proc. IEEE Int'l Conf. Acoustics, Speech, and Signal Processing. 1190--1194.

[10]

X. Fu, D. Zeng, Y. Huang, Y. Liao, X. Ding, and J. Paisley. 2016a. A fusion-based enhancing method for weakly illuminated images. Signal Processing, Vol. 129 (2016), 82 -- 96.

Digital Library

[11]

X. Fu, D. Zeng, Y. Huang, X. P. Zhang, and X. Ding. 2016b. A Weighted Variational Model for Simultaneous Reflectance and Illumination Estimation. In Proc. IEEE Int'l Conf. Computer Vision and Pattern Recognition. 2782--2790.

[12]

X. Guo, Y. Li, and H. Ling. 2017. LIME: Low-Light Image Enhancement via Illumination Map Estimation. IEEE Trans. on Image Processing, Vol. 26, 2 (Feb 2017), 982--993.

Digital Library

[13]

H. Ibrahim and N. S. Pik Kong. 2007. Brightness Preserving Dynamic Histogram Equalization for Image Contrast Enhancement. IEEE Trans. on Consumer Electronics, Vol. 53, 4 (Nov 2007), 1752--1758.

Digital Library

[14]

Y. Jiang, X. Gong, D. Liu, Y. Cheng, C. Fang, X. Shen, J. Yang, P. Zhou, and Z. Wang. 2019. EnlightenGAN: Deep Light Enhancement without Paired Supervision. arXiv e-prints (June 2019), arXiv:1906.06972. arxiv: 1906.06972

[15]

D. Jobson, Z. Rahman, and G. Woodell. 1997 a. A multiscale retinex for bridging the gap between color images and the human observation of scenes. IEEE Trans. on Image Processing, Vol. 6, 7 (Jul 1997), 965--976.

Digital Library

[16]

D. Jobson, Z. Rahman, and G. Woodell. 1997 b. Properties and performance of a center/surround retinex. IEEE Trans. on Image Processing, Vol. 6, 3 (Mar 1997), 451--462.

Digital Library

[17]

G. Kim, D. Kwon, and J. Kwon. 2019. Low-Lightgan: Low-Light Enhancement Via Advanced Generative Adversarial Network With Task-Driven Training. In Proc. IEEE Int'l Conf. Image Processing. 2811--2815.

[18]

E. Land. 1977. The retinex theory of color vision. Sci. Amer (1977), 108--128.

[19]

C. Lee, J. Kim, C. Lee, and C. Kim. 2014a. Optimized Brightness Compensation and Contrast Enhancement for Transmissive Liquid Crystal Displays. IEEE Trans. on Circuits and Systems for Video Technology, Vol. 24, 4 (April 2014), 576--590.

[20]

C. Lee, C. Lee, and C. Kim. 2013a. Contrast Enhancement Based on Layered Difference Representation of 2D Histograms. IEEE Trans. on Image Processing, Vol. 22, 12 (Dec 2013), 5372--5384.

Digital Library

[21]

C. Lee, J. Shih, C. Lien, and C. Han. 2013b. Adaptive multiscale retinex for image contrast enhancement. In Signal-Image Technology & Internet-Based Systems (SITIS), 2013 International Conference on. IEEE, 43--50.

[22]

J. Lee, C. Lee, J. Sim, and C. Kim. 2014b. Depth-guided adaptive contrast enhancement using 2D histograms. In Proc. IEEE Int'l Conf. Image Processing. 4527--4531.

[23]

L. Li, R. Wang, W. Wang, and W. Gao. 2015. A low-light image enhancement method for both denoising and contrast enlarging. In Proc. IEEE Int'l Conf. Image Processing. 3730--3734.

[24]

M. Li, J. Liu, W. Yang, X. Sun, and Z. Guo. 2018. Structure-Revealing Low-Light Image Enhancement Via Robust Retinex Model. IEEE Trans. on Image Processing, Vol. 27, 6 (June 2018), 2828--2841.

[25]

Y. Loh and C. Chan. 2019. Getting to know low-light images with the Exclusively Dark dataset. Computer Vision and Image Understanding, Vol. 178 (2019), 30 -- 42.

[26]

K. Lore, A. Akintayo, and S. Sarkar. 2017. LLNet: A deep autoencoder approach to natural low-light image enhancement. Pattern Recognition, Vol. 61 (2017), 650--662.

Digital Library

[27]

F. Lv, F. Lu, J. Wu, and C. Lim. 2018. MBLLEN: Low-light Image/Video Enhancement Using CNNs. In British Machine Vision Conference.

[28]

B. Mildenhall, J. Barron, J. Chen, D. Sharlet, R. Ng, and R. Carroll. 2018. Burst Denoising With Kernel Prediction Networks. In Proc. IEEE Int'l Conf. Computer Vision and Pattern Recognition. 2502--2510.

[29]

A. Mittal, R. Soundararajan, and A. Bovik. 2013. Making a "Completely Blind" Image Quality Analyzer. IEEE Signal Process. Lett., Vol. 20, 3 (2013), 209--212.

[30]

K. Nakai, Y. Hoshi, and A. Taguchi. 2013. Color image contrast enhacement method based on differential intensity/saturation gray-levels histograms. In International Symposium on Intelligent Signal Processing and Communications Systems. 445--449.

[31]

S. Pizer, R. Johnston, J. Ericksen, B. Yankaskas, and K. Muller. 1990. Contrast-limited adaptive histogram equalization: speed and effectiveness. In Proc. of Conf. on Visualization in Biomedical Computing. 337--345.

[32]

W. Ren, S. Liu, L. Ma, Q. Xu, X. Xu, X. Cao, J. Du, and M. Yang. 2019. Low-Light Image Enhancement via a Deep Hybrid Network. IEEE Trans. on Image Processing, Vol. 28, 9 (Sep. 2019), 4364--4375.

[33]

X. Ren, M. Li, W. Cheng, and J. Liu. 201 Joint Enhancement and Denoising Method via Sequential Decomposition. In IEEE Int'l Symposium on Circuits and Systems. 1--5.

[34]

Y. Ren, Z. Ying, T. Li, and G. Li. 2019. LECARM: Low-Light Image Enhancement Using the Camera Response ModelIEEE Trans. Circuits Syst. Video Techn., Vol. 29, 4 (2019), 968--981.

[35]

L. Shen, Z. Yue, F. Feng, Q. Chen, S. Liu, and J. Ma. 2017. MSR-net:Low-light Image Enhancement Using Deep Convolutional Network. ArXiv e-prints (November 2017). arxiv: cs.CV/1711.02488

[36]

L. Tao, C. Zhu, G. Xiang, Y. Li, H. Jia, and X. Xie. 2017. LLCNN: A convolutional neural network for low-light image enhancement. In Proc. IEEE Visual Communication and Image Processing. 1--4.

[37]

L. Wang, L. Xiao, H. Liu, and Z. Wei. 2014. Variational Bayesian Method for Retinex. IEEE Trans. on Image Processing, Vol. 23, 8 (Aug 2014), 3381--3396.

[38]

R. Wang, Q. Zhang, C. Fu, X. Shen, W. Zheng, and J. Jia. 2019. Underexposed Photo Enhancement Using Deep Illumination Estimation. In Proc. IEEE Int'l Conf. Computer Vision and Pattern Recognition.

[39]

S. Wang, J. Zheng, H. M. Hu, and B. Li. 2013. Naturalness Preserved Enhancement Algorithm for Non-Uniform Illumination Images. IEEE Trans. on Image Processing, Vol. 22, 9 (Sept 2013), 3538--3548.

Digital Library

[40]

W. Wang, C. Wei, W. Yang, and J. Liu. 2018. GLADNet: Low-Light Enhancement Network with Global Awareness. In Automatic Face & Gesture Recognition (FG 2018), 2018 13th IEEE International Conference. IEEE, 751--755.

[41]

Y. Wang, Y. Cao, Z. Zha, J. Zhang, Z. Xiong, W. Zhang, and F. Wu. 2019. Progressive Retinex: Mutually Reinforced Illumination-Noise Perception Network for Low Light Image Enhancement. In ACM Trans. Multimedia.

[42]

Z. Wang, A. C. Bovik, H. R. Sheikh, and E. P. Simoncelli. 2004. Image quality assessment: from error visibility to structural similarity. IEEE Trans. on Image Processing, Vol. 13, 4 (April 2004), 600--612.

Digital Library

[43]

C. Wei, W. Wang, W. Yang, and J. Liu. 2018. Deep Retinex Decomposition for Low-Light Enhancement. In British Machine Vision Conference.

[44]

X. Wu, X. Liu, K. Hiramatsu, and K. Kashino. 2017. Contrast-accumulated histogram equalization for image enhancement. In Proc. IEEE Int'l Conf. Image Processing. 3190--3194.

[45]

W. Yang, S. Wang, Y. Fang, Y. Wu, and J. Liu. 2020. From Fidelity to Perceptual Quality: A Semi-Supervised Approach for Low-Light Image Enhancement. In Proc. IEEE Int'l Conf. Computer Vision and Pattern Recognition.

[46]

Z. Ying, G. Li, and W. Gao. 2017. A Bio-Inspired Multi-Exposure Fusion Framework for Low-light Image Enhancement. ArXiv e-prints (November 2017). arxiv: cs.CV/1711.00591

[47]

Z. Ying, G. Li, Y. Ren, R. Wang, and W. Wang. 2017. A New Image Contrast Enhancement Algorithm Using Exposure Fusion Framework. In Proc. Int'l Conf. Computer Analysis of Images and Patterns. 36--46.

[48]

Y. Yuan, W. Yang, W. Ren, J. Liu, W. Scheirer, and Z. Wang. 2019. UG2+ Track 2: A Collective Benchmark Effort for Evaluating and Advancing Image Understanding in Poor Visibility Environments. arXiv e-prints (Apr 2019), arXiv:1904.04474.

[49]

X. Zhang, P. Shen, L. Luo, L. Zhang, and J. Song. 2012. Enhancement and noise reduction of very low light level images. In Proc. IEEE Int'l Conf. Pattern Recognition. 2034--2037.

[50]

Y. Zhang, Y. Tian, Y. Kong, B. Zhong, and Y. Fu. 2018. Residual Dense Network for Image Super-Resolution. In Proc. IEEE Int'l Conf. Computer Vision and Pattern Recognition.

[51]

Y. Zhang, J. Zhang, and X. Guo. 2019. Kindling the Darkness: A Practical Low-light Image Enhancer. In Proc. ACM Int'l Conf. Multimedia.

Cited By

Wang HCao JHuang J(2024)Adaptive Dual Aggregation Network with Normalizing Flows for Low-Light Image EnhancementEntropy10.3390/e2603018426:3(184)Online publication date: 22-Feb-2024
https://doi.org/10.3390/e26030184
Pan JLiu XBai YZhai DJiang JZhao D(2024)Illumination-Aware Low-Light Image Enhancement with Transformer and Auto-Knee CurveACM Transactions on Multimedia Computing, Communications, and Applications10.1145/366465320:8(1-23)Online publication date: 29-Jun-2024
https://dl.acm.org/doi/10.1145/3664653
Ye DChen BWang SKwong S(2024)CodedBGT: Code Bank-Guided Transformer for Low-Light Image EnhancementIEEE Transactions on Multimedia10.1109/TMM.2024.340066826(9880-9891)Online publication date: 2024
https://doi.org/10.1109/TMM.2024.3400668
Show More Cited By

Index Terms

Integrating Semantic Segmentation and Retinex Model for Low-Light Image Enhancement
1. Computing methodologies
  1. Artificial intelligence
    1. Computer vision
      1. Image and video acquisition
        Computational photography

Recommendations

CRetinex: A Progressive Color-Shift Aware Retinex Model for Low-Light Image Enhancement
Abstract
Low-light environments introduce various complex degradations into captured images. Retinex-based methods have demonstrated effective enhancement performance by decomposing an image into illumination and reflectance, allowing for selective ...
Low-light image enhancement using inverted image normalized by atmospheric light
Highlights
- This paper presents a novel low-light image enhancement algorithm using the inverted image normalized by atmospheric light.
- A mathematical connection between the Retinex theory and the atmospheric scattering model by normalizing an ...
Abstract
Images captured under low-light conditions are characterized by problems, including low brightness, low contrast, and color distortion. Low-light image enhancement, aims to enhance visual effects for the benefit of subsequent image processing and ...
A Low-light Image Enhancement Method Balancing Brightness Boosting and Noise Smoothing
CSAI '22: Proceedings of the 2022 6th International Conference on Computer Science and Artificial Intelligence

Images captured in low-light or uneven lighting environments need to use low-light image enhancement techniques to boost brightness and contrast for high-quality improved results. Some existing low-light image enhancement methods over-enhance the image ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

MM '20: Proceedings of the 28th ACM International Conference on Multimedia

October 2020

4889 pages

ISBN:9781450379885

DOI:10.1145/3394171

General Chairs:
Chang Wen Chen
Chinese University of Hong Kong, Shenzhen, China
,
Rita Cucchiara
UNIMORE, Italy
,
Xian-Sheng Hua
Alibaba Group, China
,
Program Chairs:
Guo-Jun Qi
Futurewei Technologies, USA
,
Elisa Ricci
UNITN & Fondazione Bruno Kessler, Italy
,
Zhengyou Zhang
Tencent, China
,
Roger Zimmermann
National University of Singapore, Singapore

Copyright © 2020 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGMM: ACM Special Interest Group on Multimedia

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 12 October 2020

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

National Natural Science Foundation of China
National Key R&D Program of China under Grand
Beijing Natural Science Foundation

Conference

MM '20

Sponsor:

SIGMM

MM '20: The 28th ACM International Conference on Multimedia

October 12 - 16, 2020

WA, Seattle, USA

Acceptance Rates

Overall Acceptance Rate 995 of 4,171 submissions, 24%

Upcoming Conference

MM '24

Sponsor:
sigmm

The 32nd ACM International Conference on Multimedia

October 28 - November 1, 2024

Melbourne , VIC , Australia

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

63
Total Citations
View Citations
1,160
Total Downloads

Downloads (Last 12 months)244
Downloads (Last 6 weeks)15

Reflects downloads up to 26 Sep 2024

Other Metrics

View Author Metrics

Citations

Cited By

Wang HCao JHuang J(2024)Adaptive Dual Aggregation Network with Normalizing Flows for Low-Light Image EnhancementEntropy10.3390/e2603018426:3(184)Online publication date: 22-Feb-2024
https://doi.org/10.3390/e26030184
Pan JLiu XBai YZhai DJiang JZhao D(2024)Illumination-Aware Low-Light Image Enhancement with Transformer and Auto-Knee CurveACM Transactions on Multimedia Computing, Communications, and Applications10.1145/366465320:8(1-23)Online publication date: 29-Jun-2024
https://dl.acm.org/doi/10.1145/3664653
Ye DChen BWang SKwong S(2024)CodedBGT: Code Bank-Guided Transformer for Low-Light Image EnhancementIEEE Transactions on Multimedia10.1109/TMM.2024.340066826(9880-9891)Online publication date: 2024
https://doi.org/10.1109/TMM.2024.3400668
Liu YMa YZhu ZCheng JChen X(2024)TransSea: Hybrid CNN–Transformer With Semantic Awareness for 3-D Brain Tumor SegmentationIEEE Transactions on Instrumentation and Measurement10.1109/TIM.2024.341313073(16-31)Online publication date: 2024
https://doi.org/10.1109/TIM.2024.3413130
Fan YWang YLiang DChen YXie HWang FLi JWei M(2024)Low-FaceNet: Face Recognition-Driven Low-Light Image EnhancementIEEE Transactions on Instrumentation and Measurement10.1109/TIM.2024.337223073(1-13)Online publication date: 2024
https://doi.org/10.1109/TIM.2024.3372230
He ZRan WLiu SLi KLu JXie CLiu YLu H(2024)Low-Light Image Enhancement with Multi-Scale Attention and Frequency-Domain OptimizationIEEE Transactions on Circuits and Systems for Video Technology10.1109/TCSVT.2023.3313348(1-1)Online publication date: 2024
https://doi.org/10.1109/TCSVT.2023.3313348
Xu HMa JYuan YZhang HTian XGuo X(2024)More Than Lightening: A Self-Supervised Low-Light Image Enhancement Method Capable for Multiple DegradationsIEEE/CAA Journal of Automatica Sinica10.1109/JAS.2024.12426311:3(622-637)Online publication date: Mar-2024
https://doi.org/10.1109/JAS.2024.124263
Hu JLi LLiu BCheng JLi JDai C(2024)A Semantic-Aware Approach to Image Deraining with Transformer2024 International Joint Conference on Neural Networks (IJCNN)10.1109/IJCNN60899.2024.10651129(1-8)Online publication date: 30-Jun-2024
https://doi.org/10.1109/IJCNN60899.2024.10651129
Guan JQi NZhu QChen L(2024)UTrCGAN:Uncertainty-Driven Cycle-Consistent Generative Adversarial Network for Low-Light Image Enhancement2024 IEEE International Conference on Image Processing (ICIP)10.1109/ICIP51287.2024.10648196(1473-1479)Online publication date: 27-Oct-2024
https://doi.org/10.1109/ICIP51287.2024.10648196
Huang ZLee TChang LChen YChiang PWu JChuang Y(2024)Semantic-Region Specific Lookup Tables for Image Enhancement Via Unpaired Learning2024 IEEE International Conference on Image Processing (ICIP)10.1109/ICIP51287.2024.10647940(1690-1696)Online publication date: 27-Oct-2024
https://doi.org/10.1109/ICIP51287.2024.10647940
Show More Cited By

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents