research-article

FastLLVE: Real-Time Low-Light Video Enhancement with Intensity-Aware Look-Up Table

Authors:

Xiaohong LiuAuthors Info & Claims

MM '23: Proceedings of the 31st ACM International Conference on Multimedia

Pages 8134 - 8144

https://doi.org/10.1145/3581783.3611933

Published: 27 October 2023 Publication History

Abstract

Low-Light Video Enhancement (LLVE) has received considerable attention in recent years. One of the critical requirements of LLVE is inter-frame brightness consistency, which is essential for maintaining the temporal coherence of the enhanced video. However, most existing single-image-based methods fail to address this issue, resulting in flickering effect that degrades the overall quality after enhancement. Moreover, 3D Convolution Neural Network (CNN)-based methods, which are designed for video to maintain inter-frame consistency, are computationally expensive, making them impractical for real-time applications. To address these issues, we propose an efficient pipeline named FastLLVE that leverages the Look-Up-Table (LUT) technique to maintain inter-frame brightness consistency effectively. Specifically, we design a learnable Intensity-Aware LUT (IA-LUT) module for adaptive enhancement, which addresses the low-dynamic problem in low-light scenarios. This enables FastLLVE to perform low-latency and low-complexity enhancement operations while maintaining high-quality results. Experimental results on benchmark datasets demonstrate that our method achieves the State-Of-The-Art (SOTA) performance in terms of both image quality and inter-frame brightness consistency. More importantly, our FastLLVE can process 1,080p videos at 50+ Frames Per Second (FPS), which is 2 X faster than SOTA CNN-based methods in inference time, making it a promising solution for real-time applications. The code is available at https://github.com/Wenhao-Li-777/FastLLVE.

References

[1]

Zohreh Azizi and C-C Jay Kuo. 2022. SALVE: Self-supervised Adaptive Low-light Video Enhancement. arXiv preprint arXiv:2212.11484 (2022).

[2]

Chen Chen, Qifeng Chen, Minh N Do, and Vladlen Koltun. 2019a. Seeing motion in the dark. In Proceedings of the IEEE/CVF International Conference on Computer Vision. 3185--3194.

[3]

Chang Chen, Zhiwei Xiong, Xinmei Tian, Zheng-Jun Zha, and Feng Wu. 2019b. Real-world image denoising with deep boosting. IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 42, 12 (2019), 3071--3087.

[4]

Gong Cheng, Peicheng Zhou, and Junwei Han. 2016. Learning rotation-invariant convolutional neural networks for object detection in VHR optical remote sensing images. IEEE Transactions on Geoscience and Remote Sensing, Vol. 54, 12 (2016), 7405--7415.

[5]

Xueyang Fu, Yinghao Liao, Delu Zeng, Yue Huang, Xiao-Ping Zhang, and Xinghao Ding. 2015. A probabilistic method for image enhancement with simultaneous illumination and reflectance estimation. IEEE Transactions on Image Processing, Vol. 24, 12 (2015), 4965--4977.

Digital Library

[6]

Xueyang Fu, Delu Zeng, Yue Huang, Xiao-Ping Zhang, and Xinghao Ding. 2016. A weighted variational model for simultaneous reflectance and illumination estimation. In Proceedings of the IEEE conference on computer vision and pattern recognition. 2782--2790.

[7]

Akhilesh Gotmare, Nitish Shirish Keskar, Caiming Xiong, and Richard Socher. 2018. A closer look at deep learning heuristics: Learning rate restarts, warmup and distillation. arXiv preprint arXiv:1810.13243 (2018).

[8]

Shan Huang, Xiaohong Liu, Tao Tan, Menghan Hu, Xiaoer Wei, Tingli Chen, and Bin Sheng. 2023. TransMRSR: Transformer-based Self-Distilled Generative Prior for Brain MRI Super-Resolution. arXiv preprint arXiv:2306.06669 (2023).

[9]

Haidi Ibrahim and Nicholas Sia Pik Kong. 2007. Brightness preserving dynamic histogram equalization for image contrast enhancement. IEEE Transactions on Consumer Electronics, Vol. 53, 4 (2007), 1752--1758.

Digital Library

[10]

Haiyang Jiang and Yinqiang Zheng. 2019. Learning to see moving objects in the dark. In Proceedings of the IEEE/CVF International Conference on Computer Vision. 7324--7333.

[11]

Hakki Can Karaimer and Michael S Brown. 2016. A software platform for manipulating the camera imaging pipeline. In Computer Vision-ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part I 14. Springer, 429--444.

[12]

Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014).

[13]

Wei-Sheng Lai, Jia-Bin Huang, Narendra Ahuja, and Ming-Hsuan Yang. 2017. Deep laplacian pyramid networks for fast and accurate super-resolution. In Proceedings of the IEEE conference on computer vision and pattern recognition. 624--632.

[14]

Wei-Sheng Lai, Jia-Bin Huang, Narendra Ahuja, and Ming-Hsuan Yang. 2018a. Fast and accurate image super-resolution with deep laplacian pyramid networks. IEEE transactions on pattern analysis and machine intelligence, Vol. 41, 11 (2018), 2599--2613.

[15]

Wei-Sheng Lai, Jia-Bin Huang, Oliver Wang, Eli Shechtman, Ersin Yumer, and Ming-Hsuan Yang. 2018b. Learning blind video temporal consistency. In Proceedings of the European conference on computer vision (ECCV). 170--185.

Digital Library

[16]

Edwin H Land. 1977. The retinex theory of color vision. Scientific american, Vol. 237, 6 (1977), 108--129.

[17]

Jaakko Lehtinen, Jacob Munkberg, Jon Hasselgren, Samuli Laine, Tero Karras, Miika Aittala, and Timo Aila. 2018. Noise2Noise: Learning image restoration without clean data. arXiv preprint arXiv:1803.04189 (2018).

[18]

Guofa Li, Yifan Yang, Xingda Qu, Dongpu Cao, and Keqiang Li. 2021. A deep learning based image enhancement approach for autonomous driving at night. Knowledge-Based Systems, Vol. 213 (2021), 106617.

Digital Library

[19]

Xiaohong Liu, Lei Chen, Wenyi Wang, and Jiying Zhao. 2018. Robust multi-frame super-resolution based on spatially weighted half-quadratic estimation and adaptive BTV regularization. IEEE Transactions on Image Processing, Vol. 27, 10 (2018), 4971--4986.

Digital Library

[20]

Xiaohong Liu, Lingshi Kong, Yang Zhou, Jiying Zhao, and Jun Chen. 2020. End-to-end trainable video super-resolution based on a new mechanism for implicit motion estimation and compensation. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision. 2416--2425.

[21]

Xiaohong Liu, Kangdi Shi, Zhe Wang, and Jun Chen. 2021. Exploit camera raw data for video super-resolution via hidden Markov model inference. IEEE Transactions on Image Processing, Vol. 30 (2021), 2127--2140.

Digital Library

[22]

Feifan Lv, Yu Li, and Feng Lu. 2021. Attention guided low-light image enhancement with a large scale low-light simulation dataset. International Journal of Computer Vision, Vol. 129, 7 (2021), 2175--2193.

Digital Library

[23]

Feifan Lv, Feng Lu, Jianhua Wu, and Chongsoon Lim. 2018. MBLLEN: Low-Light Image/Video Enhancement Using CNNs. In BMVC, Vol. 220. 4.

[24]

Xiaojiao Mao, Chunhua Shen, and Yu-Bin Yang. 2016. Image restoration using very deep convolutional encoder-decoder networks with symmetric skip connections. Advances in neural information processing systems, Vol. 29 (2016).

[25]

Nick Moran, Dan Schmidt, Yu Zhong, and Patrick Coady. 2020b. Noisier2noise: Learning to denoise from unpaired noisy data. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 12064--12072.

[26]

Sean Moran, Pierre Marza, Steven McDonagh, Sarah Parisot, and Gregory Slabaugh. 2020a. Deeplpf: Deep local parametric filters for image enhancement. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 12826--12835.

[27]

Keita Nakai, Yoshikatsu Hoshi, and Akira Taguchi. 2013. Color image contrast enhacement method based on differential intensity/saturation gray-levels histograms. In 2013 International Symposium on Intelligent Signal Processing and Communication Systems. IEEE, 445--449.

[28]

Jinwang Pan, Deming Zhai, Yuanchao Bai, Junjun Jiang, Debin Zhao, and Xianming Liu. 2022. ChebyLighter: Optimal Curve Estimation for Low-light Image Enhancement. In Proceedings of the 30th ACM International Conference on Multimedia. 1358--1366.

Digital Library

[29]

Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. 2019. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, Vol. 32 (2019).

[30]

Haoyu Ren, Mostafa El-Khamy, and Jungwon Lee. 2019. Dn-resnet: Efficient deep residual network for image denoising. In Computer Vision-ACCV 2018: 14th Asian Conference on Computer Vision, Perth, Australia, December 2-6, 2018, Revised Selected Papers, Part V 14. Springer, 215--230.

[31]

Sourav Samanta, Amartya Mukherjee, Amira S Ashour, Nilanjan Dey, Jo ao Manuel RS Tavares, Wahiba Ben Abdessalem Karâa, Redha Taiar, Ahmad Taher Azar, and Aboul Ella Hassanien. 2018. Log transform based optimal image enhancement using firefly algorithm for autonomous mini unmanned aerial vehicle: An application of aerial photography. International Journal of Image and Graphics, Vol. 18, 04 (2018), 1850019.

[32]

Zhihao Shi, Xiaohong Liu, Chengqi Li, Linhui Dai, Jun Chen, Timothy N Davidson, and Jiying Zhao. 2021a. Learning for unconstrained space-time video super-resolution. IEEE Transactions on Broadcasting, Vol. 68, 2 (2021), 345--358.

[33]

Zhihao Shi, Xiaohong Liu, Kangdi Shi, Linhui Dai, and Jun Chen. 2021b. Video frame interpolation via generalized deformable convolution. IEEE transactions on multimedia, Vol. 24 (2021), 426--439.

[34]

Zhihao Shi, Xiangyu Xu, Xiaohong Liu, Jun Chen, and Ming-Hsuan Yang. 2022. Video frame interpolation transformer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 17482--17491.

[35]

Matias Tassano, Julie Delon, and Thomas Veit. 2020. Fastdvdnet: Towards real-time deep video denoising without flow estimation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 1354--1363.

[36]

Dmitry Ulyanov, Andrea Vedaldi, and Victor Lempitsky. 2016. Instance normalization: The missing ingredient for fast stylization. arXiv preprint arXiv:1607.08022 (2016).

[37]

Qing Wang and Rabab K Ward. 2007. Fast image/video contrast enhancement based on weighted thresholded histogram equalization. IEEE transactions on Consumer Electronics, Vol. 53, 2 (2007), 757--764.

Digital Library

[38]

Ruixing Wang, Xiaogang Xu, Chi-Wing Fu, Jiangbo Lu, Bei Yu, and Jiaya Jia. 2021b. Seeing dynamic scene in the dark: A high-quality video dataset with mechatronic alignment. In Proceedings of the IEEE/CVF International Conference on Computer Vision. 9700--9709.

[39]

Ruixing Wang, Qing Zhang, Chi-Wing Fu, Xiaoyong Shen, Wei-Shi Zheng, and Jiaya Jia. 2019. Underexposed photo enhancement using deep illumination estimation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 6849--6857.

[40]

Shuhang Wang, Jin Zheng, Hai-Miao Hu, and Bo Li. 2013. Naturalness preserved enhancement algorithm for non-uniform illumination images. IEEE transactions on image processing, Vol. 22, 9 (2013), 3538--3548.

Digital Library

[41]

Tao Wang, Yong Li, Jingyang Peng, Yipeng Ma, Xian Wang, Fenglong Song, and Youliang Yan. 2021a. Real-time image enhancer via learnable spatial-aware 3d lookup tables. In Proceedings of the IEEE/CVF International Conference on Computer Vision. 2471--2480.

[42]

Guangyang Wu, Xiaohong Liu, Kunming Luo, Xi Liu, Qingqing Zheng, Shuaicheng Liu, Xinyang Jiang, Guangtao Zhai, and Wenyi Wang. 2023. AccFlow: Backward Accumulation for Long-Range Optical Flow. In Proceedings of the IEEE/CVF International Conference on Computer Vision.

[43]

Bing Xu, Naiyan Wang, Tianqi Chen, and Mu Li. 2015. Empirical evaluation of rectified activations in convolutional network. arXiv preprint arXiv:1505.00853 (2015).

[44]

Canqian Yang, Meiguang Jin, Xu Jia, Yi Xu, and Ying Chen. 2022a. AdaInt: learning adaptive intervals for 3D lookup tables on real-time image enhancement. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 17522--17531.

[45]

Canqian Yang, Meiguang Jin, Yi Xu, Rui Zhang, Ying Chen, and Huaida Liu. 2022b. SepLUT: Separable Image-Adaptive Lookup Tables for Real-Time Image Enhancement. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XVIII. Springer, 201--217.

[46]

Meifang Yang, Xin Nie, and Ryan Wen Liu. 2019. Coarse-to-fine luminance estimation for low-light image enhancement in maritime video surveillance. In 2019 IEEE Intelligent Transportation Systems Conference (ITSC). IEEE, 299--304.

Digital Library

[47]

Wenhan Yang, Shiqi Wang, Yuming Fang, Yue Wang, and Jiaying Liu. 2020. From fidelity to perceptual quality: A semi-supervised approach for low-light image enhancement. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 3063--3072.

[48]

Guanghao Yin, Zefan Qu, Xinyang Jiang, Shan Jiang, Zhenhua Han, Ningxin Zheng, Xiaohong Liu, Huan Yang, Yuqing Yang, Dongsheng Li, and Lili Qiu. 2023. Online Video Streaming Super-Resolution with Adaptive Look-Up Table Fusion. arXiv preprint arXiv:2303.00334 (2023).

[49]

Matthew D Zeiler and Rob Fergus. 2014. Visualizing and understanding convolutional networks. In Computer Vision--ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part I 13. Springer, 818--833.

[50]

Hui Zeng, Jianrui Cai, Lida Li, Zisheng Cao, and Lei Zhang. 2020. Learning image-adaptive 3d lookup tables for high performance photo enhancement in real-time. IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 44, 4 (2020), 2058--2073.

[51]

Fan Zhang, Yu Li, Shaodi You, and Ying Fu. 2021. Learning temporal consistency for low light video enhancement from single images. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 4967--4976.

[52]

Kai Zhang, Wangmeng Zuo, Yunjin Chen, Deyu Meng, and Lei Zhang. 2017. Beyond a gaussian denoiser: Residual learning of deep cnn for image denoising. IEEE transactions on image processing, Vol. 26, 7 (2017), 3142--3155.

Digital Library

[53]

Shangchen Zhou, Chongyi Li, and Chen Change Loy. 2022. Lednet: Joint low-light enhancement and deblurring in the dark. In European Conference on Computer Vision. Springer, 573--589.

Digital Library

Cited By

Fu KPeng YZhang ZXu QLiu XWang JZhai G(2024)AttentionLUT: Attention Fusion-Based Canonical Polyadic LUT for Real-Time Image EnhancementICASSP 2024 - 2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)10.1109/ICASSP48485.2024.10445905(3255-3259)Online publication date: 14-Apr-2024
https://doi.org/10.1109/ICASSP48485.2024.10445905
Wu GLiu XJia JCui XZhai G(2024)Text2QR: Harmonizing Aesthetic Customization and Scanning Robustness for Text-Guided QR Code Generation2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)10.1109/CVPR52733.2024.00808(8456-8465)Online publication date: 16-Jun-2024
https://doi.org/10.1109/CVPR52733.2024.00808
Wu GTao XLi CWang WLiu XZheng Q(2024)Perception-Oriented Video Frame Interpolation via Asymmetric Blending2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)10.1109/CVPR52733.2024.00266(2753-2762)Online publication date: 16-Jun-2024
https://doi.org/10.1109/CVPR52733.2024.00266

Index Terms

FastLLVE: Real-Time Low-Light Video Enhancement with Intensity-Aware Look-Up Table
1. Computing methodologies
  1. Artificial intelligence
    1. Computer vision
      1. Computer vision problems
        Reconstruction

Recommendations

Light-VQA: A Multi-Dimensional Quality Assessment Model for Low-Light Video Enhancement
MM '23: Proceedings of the 31st ACM International Conference on Multimedia

Recently, Users Generated Content (UGC) videos becomes ubiquitous in our daily lives. However, due to the limitations of photographic equipments and techniques, UGC videos often contain various degradations, in which one of the most visually unfavorable ...
Better Than Reference in Low-Light Image Enhancement: Conditional Re-Enhancement Network
Low-light images suffer from severe noise, low brightness, low contrast, etc. In previous researches, many image enhancement methods have been proposed, but few methods can deal with these problems simultaneously. In this paper, to solve these problems ...
Retinex Based Flicker-Free Low-Light Video Enhancement
Pattern Recognition and Computer Vision
Abstract
Videos captured in low light environment tend to be poor visual effect. To get better visual experience, a video enhancement algorithm based on improved center-surrounded Retinex and optical flow is proposed in this paper, which contains intra-...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

MM '23: Proceedings of the 31st ACM International Conference on Multimedia

October 2023

9913 pages

ISBN:9798400701085

DOI:10.1145/3581783

General Chairs:
Abdulmotaleb El Saddik
University of Ottawa, Canada & MBZUAI, UAE
,
Tao Mei
HiDream.ai, China
,
Rita Cucchiara
University of Modena and Reggio Emilia, Italy
,
Program Chairs:
Marco Bertini
University of Florence, Italy
,
Diana Patricia Tobon Vallejo
Unversidad de Medellin, Colombia
,
Pradeep K. Atrey
University at Albany, State University of New York, USA
,
M. Shamim Hossain
M. Shamim Hossain (King Saud University, KSA

Copyright © 2023 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

SIGMM: ACM Special Interest Group on Multimedia

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 27 October 2023

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

Shanghai Pujiang Program

Conference

MM '23

Sponsor:

SIGMM

MM '23: The 31st ACM International Conference on Multimedia

October 29 - November 3, 2023

Ottawa ON, Canada

Acceptance Rates

Overall Acceptance Rate 995 of 4,171 submissions, 24%

Upcoming Conference

MM '24

Sponsor:
sigmm

The 32nd ACM International Conference on Multimedia

October 28 - November 1, 2024

Melbourne , VIC , Australia

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

3
Total Citations
View Citations
180
Total Downloads

Downloads (Last 12 months)180
Downloads (Last 6 weeks)7

Reflects downloads up to 15 Oct 2024

Other Metrics

View Author Metrics

Citations

Cited By

Fu KPeng YZhang ZXu QLiu XWang JZhai G(2024)AttentionLUT: Attention Fusion-Based Canonical Polyadic LUT for Real-Time Image EnhancementICASSP 2024 - 2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)10.1109/ICASSP48485.2024.10445905(3255-3259)Online publication date: 14-Apr-2024
https://doi.org/10.1109/ICASSP48485.2024.10445905
Wu GLiu XJia JCui XZhai G(2024)Text2QR: Harmonizing Aesthetic Customization and Scanning Robustness for Text-Guided QR Code Generation2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)10.1109/CVPR52733.2024.00808(8456-8465)Online publication date: 16-Jun-2024
https://doi.org/10.1109/CVPR52733.2024.00808
Wu GTao XLi CWang WLiu XZheng Q(2024)Perception-Oriented Video Frame Interpolation via Asymmetric Blending2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)10.1109/CVPR52733.2024.00266(2753-2762)Online publication date: 16-Jun-2024
https://doi.org/10.1109/CVPR52733.2024.00266
Wen SLi W(2024)An unsupervised low-light video enhancement network based on inter-frame consistencySignal, Image and Video Processing10.1007/s11760-024-03439-z18:11(7909-7920)Online publication date: 1-Aug-2024
https://doi.org/10.1007/s11760-024-03439-z

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents