Cited By
View all- Zhou FZheng ZQiu G(2024)Removing Banding Artifacts in HDR Videos Generated From Inverse Tone MappingIEEE Transactions on Broadcasting10.1109/TBC.2024.339429770:2(753-762)Online publication date: Jun-2024
Video Inverse Tone Mapping Network with Luma and Chroma Mapping
Pages 1383 - 1391
Abstract
\beginabstract With the popularity of consumer high dynamic range (HDR) display devices, video inverse tone mapping (iTM) has become a research hotspot. However, existing methods are designed based on a perceptual non-uniformity color space (e.g., RGB and YC_BC_R), resulting in limited quality of HDR video rendered by these methods. Considering the two key factors involved in the video iTM task: luma and chroma, in this paper, we design an IC_TC_P color space based video iTM model, which reproduces high quality HDR video by processing luma and chroma information. Benefitting from the decorrelated perception of luma and chroma in the IC_TC_P color space, two global mapping networks (INet and TPNet) are developed to enhance the luma and chroma pixels, respectively. However, luma and chroma mapping in the iTM task may be affected by color appearance phenomena. Thus, a luma-chroma adaptation transform network (LCATNet) is proposed to process the luma and chroma pixels affected by color appearance phenomena, which can complement the local details to the globally enhanced luma and chroma pixels. In the LCATNet, either the luma mapping or the chroma mapping is adaptively adjusted according to both the luma and the chroma information. Besides, benefitting from the perceptually consistent property of the IC_T C_P color space, the same pixel errors can draw equal model attentions during the training. Thus, the proposed model can correctly render luma and chroma information without highlighting special regions or designing special training losses. Extensive experimental results demonstrate the effectiveness of the proposed model. \endabstract
Supplementary Material
A brief exposition of the paper.
- Download
- 47.64 MB
References
[1]
Gaofeng Cao, Fei Zhou, Kanglin Liu, and Liu Bozhi. 2021. A brightness-adaptive kernel prediction network for inverse tone mapping. Neurocomputing 464 (2021), 1--14.
[2]
Gaofeng Cao, Fei Zhou, Kanglin Liu, Anjie Wang, and Leidong Fan. 2023. A Decoupled Kernel Prediction Network Guided by Soft Mask for Single Image HDR Reconstruction. ACM Transactions on Multimedia Computing, Communications, and Applications 19, 2s (2023), 1--23.
[3]
Gaofeng Cao, Fei Zhou, Han Yan, Anjie Wang, and Leidong Fan. 2022. KPN-MFI: A Kernel Prediction Network with Multi-frame Interaction for Video Inverse Tone Mapping. In Proceedings of the Thirty-First International Joint Conference onArtificial Intelligence, IJCAI-22. International Joint Conferences on Artificial Intelligence Organization, 806--812.
[4]
Guanying Chen, Chaofeng Chen, Shi Guo, Zhetong Liang, Kwan-Yee K Wong, and Lei Zhang. 2021. Hdr video reconstruction: A coarse-to-fine network and a real-world benchmark dataset. In Proceedings of the IEEE/CVF International Conference on Computer Vision. 2502--2511.
[5]
Xiangyu Chen, Yihao Liu, Zhengwen Zhang, Yu Qiao, and Chao Dong. 2021. HDRUnet: Single Image HDR Reconstruction with Denoising and Dequantization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 354--363.
[6]
Xiangyu Chen, Zhengwen Zhang, Jimmy S. Ren, Lynhoo Tian, Yu Qiao, and Chao Dong. 2021. A New Journey from SDRTV to HDRTV. In 2021 IEEE/CVF International Conference on Computer Vision, ICCV 2021, Montreal, QC, Canada, October 10--17, 2021. IEEE, 4480--4489.
[7]
Min Dai and Ai-Mei Huang. 2013. Study on panel sharpening in different color spaces. In Applications of Digital Image Processing XXXVI, Vol. 8856. International Society for Optics and Photonics, SPIE, 88560H.
[8]
Dolby. 2016. ICtCp Dolby White Paper. Dolby Vision.
[9]
Chao Dong, Chen Change Loy, Kaiming He, and Xiaoou Tang. 2014. Learning a Deep Convolutional Network for Image Super-Resolution. In Computer Vision - ECCV 2014 - 13th European Conference, Zurich, Switzerland, September 6--12, 2014, Proceedings, Part IV (Lecture Notes in Computer Science, Vol. 8692). Springer, 184--199.
[10]
Mark D. Fairchild and Garrett M. Johnson. 2004. The iCAM Framework for Image Appearance, Image Differences, and Image Quality. Journal of Electronic Imaging 13, 1 (2004), 126--138.
[11]
Xavier Glorot and Y. Bengio. 2010. Understanding the difficulty of training deep feedforward neural networks. Journal of Machine Learning Research 9 (01 2010), 249--256.
[12]
Cheng Guo, Leidong Fan, Ziyu Xue, and Xiuhua Jiang. 2023. Learning a Practical SDR-to-HDRTV Up-Conversion Using New Dataset and Degradation Models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 22231--22241.
[13]
Gang He, Kepeng Xu, Li Xu, Chang Wu, Ming Sun, Xing Wen, and Yu-Wing Tai. 2022. SDRTV-to-HDRTV via Hierarchical Dynamic Context Feature Mapping. In MM '22: The 30th ACM International Conference on Multimedia, Lisboa, Portugal, October 10 - 14, 2022. ACM, 2890--2898.
[14]
Gang He, Kepeng Xu, Li Xu, Chang Wu, Ming Sun, Xing Wen, and Yu-Wing Tai. 2022. SDRTV-to-HDRTV via Hierarchical Dynamic Context Feature Mapping. In MM '22: The 30th ACM International Conference on Multimedia, Lisboa, Portugal, October 10 - 14, 2022. ACM, 2890--2898.
[15]
Jingwen He, Yihao Liu, Yu Qiao, and Chao Dong. 2020. Conditional Sequential Modulation for Efficient Global Image Retouching. In Computer Vision - ECCV 2020 - 16th European Conference, Glasgow, UK, August 23--28, 2020, Proceedings, Part XIII (Lecture Notes in Computer Science, Vol. 12358). 679--695.
[16]
Robert William Gainer Hunt. 2005. The reproduction of colour. Wiley-IS&T Series in Imaging Science and Technology. 342 pages.
[17]
ITU-R. 2015. Colour conversion from Recommendation ITU-R BT.709 to Recommendation ITU-R BT.2020. ITU-R Rec, BT.2087 (2015).
[18]
ITU-R. 2015. Parameter values for the HDTV standards for production and international programme exchange. ITU-R Rec, BT.709--6 (2015).
[19]
ITU-R. 2015. Parameter values for ultra-high definition television systems for production and international programme exchange. ITU-R Rec, BT.2020--2 (2015).
[20]
ITU-R. 2020. High dynamic range television for production and international programme exchangee. ITU-R Rec, BT.2390--8 (2020).
[21]
ITU-R. 2021. Methods for conversion of high dynamic range content to standard dynamic range content and vice-versa. ITU-R Rec, BT.2446--1 (2021).
[22]
ITU-R BT.2124. 2019. Objective Metric for the Assessment of the Potential Visibility of Colour Differences in Television. (2019).
[23]
Max Jaderberg, Karen Simonyan, Andrew Zisserman, et al. 2015. Spatial transformer networks. Advances in neural information processing systems 28 (2015).
[24]
Sing Bing Kang, Matthew Uyttendaele, Simon Winder, and Richard Szeliski. 2003. High Dynamic Range Video. ACM Transactions on Graphics 22, 3 (2003), 319--325.
[25]
Soo Ye Kim, Dae-Eun Kim, and Munchurl Kim. 2019. ITM-CNN: Learning the inverse tone mapping from low dynamic range video to high dynamic range displays using convolutional neural networks. In Computer Vision--ACCV 2018: 14th Asian Conference on Computer Vision, Perth, Australia, December 2--6, 2018, Revised Selected Papers, Part III 14. Springer, 395--409.
[26]
Soo Ye Kim, Jihyong Oh, and Munchurl Kim. 2019. Deep SR-ITM: Joint Learning of Super-Resolution and Inverse Tone-Mapping for 4K UHD HDR Applications. In 2019 IEEE/CVF International Conference on Computer Vision, ICCV 2019, Seoul, Korea (South), October 27 - November 2, 2019. IEEE, 3116--3125.
[27]
Soo Ye Kim, Jihyong Oh, and Munchurl Kim. 2020. JSI-GAN: GAN-Based Joint Super-Resolution and Inverse Tone-Mapping with Pixel-Wise Task-Specific Filters for UHD HDR Video. In The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI, New York, NY, USA, February 7--12, 2020. AAAI Press, 11287-- 11295.
[28]
Diederik P. Kingma and Jimmy Ba. 2015. Adam: A Method for Stochastic Optimization. In 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7--9, 2015, Conference Track Proceedings.
[29]
Min Lin, Qiang Chen, and Shuicheng Yan. 2014. Network In Network. In 2nd International Conference on Learning Representations, ICLR 2014, Banff, AB, Canada, April 14--16, 2014, Conference Track Proceedings.
[30]
David L MacAdam. 1942. Visual sensitivities to color difference in daylight. Journal of the Optical Society of America 32, 5 (1942), 247--274.
[31]
David L MacAdam. 1942. Visual sensitivities to color differences in daylight. Josa 32, 5 (1942), 247--274.
[32]
Rafat Mantiuk, Kil Joong Kim, Allan G Rempel, and Wolfgang Heidrich. 2011. HDR-VDP-2: A Calibrated Visual Metric for Visibility and Quality Predictions in all Luminance Conditions. ACM Transactions on Graphics 30, 4 (2011), 1--14.
[33]
Pedram Mohammadi, Mahsa T Pourazad, and Panos Nasiopoulos. 2020. A perception-based inverse tone mapping operator for high dynamic range video applications. IEEE Transactions on Circuits and Systems for Video Technology 31, 5 (2020), 1711--1723.
[34]
Erik Reinhard, GregWard, Sumanta N. Pattanaik, Paul E. Debevec, andWolfgang Heidrich. 2010. High Dynamic Range Imaging - Acquisition, Display, and Image- Based Lighting (2. ed.). Academic Press.
[35]
Marcel Santana Santos, Tsang Ing Ren, and Nima Khademi Kalantari. 2020. Single Image HDR Reconstruction Using a CNN With Masked Features and Perceptual Loss. ACM Transactions on Graphics 39, 4 (2020), 1--10.
[36]
Tong Shao, Deming Zhai, Junjun Jiang, and Xianming Liu. 2022. Hybrid Conditional Deep Inverse Tone Mapping. In MM '22: The 30th ACM International Conference on Multimedia, Lisboa, Portugal, October 10 - 14, 2022. ACM, 1016--1024.
[37]
SMPTE Standard. 2014. High dynamic range electro-optical transfer function of mastering reference displays. SMPTE ST 2084, 2014 (2014), 1--14.
[38]
Wilhelm von Bezold. 1873. Über das Gesetz der Farbenmischung und die physiologischen Grundfarben. Annalen der Physik 226, 10 (1873), 221--247.
[39]
Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli. 2004. Image Quality Assessment: from Error Visibility to Structural Similarity. IEEE Transactions on Image Processing 13, 4 (2004), 600--612.
[40]
Sanghyun Woo, Jongchan Park, Joon-Young Lee, and In So Kweon. 2018. CBAM: Convolutional Block Attention Module. In Proceedings of the European Conference on Computer Vision (ECCV).
[41]
Gang Xu, Qibin Hou, Le Zhang, and Ming-Ming Cheng. 2022. FMNet: Frequency-Aware Modulation Network for SDR-to-HDR Translation. In Proceedings of the 30th ACM International Conference on Multimedia. 6425--6435.
Index Terms
- Video Inverse Tone Mapping Network with Luma and Chroma Mapping
Recommendations
A framework for inverse tone mapping
AbstractIn recent years many tone mapping operators (TMOs) have been presented in order to display high dynamic range images (HDRI) on typical display devices. TMOs compress the luminance range while trying to maintain contrast. The inverse of tone ...
Lighting condition adaptive tone mapping method
SIGGRAPH '18: ACM SIGGRAPH 2018 PostersWe propose an adaptive tone mapping method for displaying HDR images according to ambient light conditions. To compensate the loss of perceived luminance in brighter viewing conditions, we enhance the HDR image by an algorithm based on the Naka-Rushton ...
Zonal brightness coherency for video tone mapping
Tone Mapping Operators (TMOs) compress High Dynamic Range (HDR) contents to address Low Dynamic Range (LDR) displays. While many solutions have been designed over the last decade, only few of them can cope with video sequences. Indeed, these TMOs tone ...
Comments
Information & Contributors
Information
Published In
October 2023
9913 pages
ISBN:9798400701085
DOI:10.1145/3581783
- General Chairs:
- Abdulmotaleb El Saddik,
- Tao Mei,
- Rita Cucchiara,
- Program Chairs:
- Marco Bertini,
- Diana Patricia Tobon Vallejo,
- Pradeep K. Atrey,
- M. Shamim Hossain
Copyright © 2023 ACM.
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].
Sponsors
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 27 October 2023
Check for updates
Author Tags
Qualifiers
- Research-article
Funding Sources
- Guangdong Basic and Applied Basic Research Foundation
- the National Natural Science Foundation of China
- Shenzhen R&D Program
Conference
MM '23
Sponsor:
MM '23: The 31st ACM International Conference on Multimedia
October 29 - November 3, 2023
Ottawa ON, Canada
Acceptance Rates
Overall Acceptance Rate 995 of 4,171 submissions, 24%
Upcoming Conference
MM '24
- Sponsor:
- sigmm
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- View Citations1Total Citations
- 143Total Downloads
- Downloads (Last 12 months)143
- Downloads (Last 6 weeks)11
Reflects downloads up to 11 Aug 2024
Other Metrics
Citations
Cited By
View all- Zhou FZheng ZQiu G(2024)Removing Banding Artifacts in HDR Videos Generated From Inverse Tone MappingIEEE Transactions on Broadcasting10.1109/TBC.2024.339429770:2(753-762)Online publication date: Jun-2024
View Options
Get Access
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in