Location via proxy:   [ UP ]  
[Report a bug]   [Manage cookies]                
skip to main content
research-article

A Decoupled Kernel Prediction Network Guided by Soft Mask for Single Image HDR Reconstruction

Published: 17 February 2023 Publication History
  • Get Citation Alerts
  • Abstract

    Recent works on single image high dynamic range (HDR) reconstruction fail to hallucinate plausible textures, resulting in information missing and artifacts in large-scale under/over-exposed regions. In this article, a decoupled kernel prediction network is proposed to infer an HDR image from a low dynamic range (LDR) image. Specifically, we first adopt a simple module to generate a preliminary result, which can precisely estimate well-exposed HDR regions. Meanwhile, an encoder-decoder backbone network with a soft mask guidance module is presented to predict pixel-wise kernels, which is further convolved with the preliminary result to obtain the final HDR output. Instead of traditional kernels, our predicted kernels are decoupled along the spatial and channel dimensions. The advantages of our method are threefold at least. First, our model is guided by the soft mask so that it can focus on the most relevant information for under/over-exposed regions. Second, pixel-wise kernels are able to adaptively solve the different degradations for differently exposed regions. Third, decoupled kernels can avoid information redundancy across channels and reduce the solution space of our model. Thus, our method is able to hallucinate fine details in the under/over-exposed regions and renders visually pleasing results. Extensive experiments demonstrate that our model outperforms state-of-the-art ones.

    Supplementary Material

    3550277-supp (3550277-supp.pdf)
    Supplementary material

    References

    [1]
    Akhil K. A and C. V. Jiji. 2021. Single image HDR synthesis using a densely connected dilated ConvNet. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, CVPR Workshops 2021, virtual, June 19–25, 2021. Computer Vision Foundation/IEEE, 526–531.
    [2]
    Tunç Ozan Aydin, Rafal Mantiuk, and Hans-Peter Seidel. 2008. Extending quality metrics to full dynamic range images. In Proceedings of the Human Vision and Electronic Imaging XIII (Proceedings of SPIE). San Jose, 6806–10.
    [3]
    Steve Bako, Thijs Vogels, Brian McWilliams, Mark Meyer, Jan Novák, Alex Harvill, Pradeep Sen, Tony DeRose, and Fabrice Rousselle. 2017. Kernel-predicting convolutional networks for denoising Monte Carlo renderings. ACM Transactions on Graphics 36, 4 (2017), 97:1–97:14.
    [4]
    Francesco Banterle, Alessandro Artusi, Kurt Debattista, and Alan Chalmers. 2017. Advanced High Dynamic Range Imaging (2nd Edition). AK Peters (CRC Press), Natick, MA.
    [5]
    Francesco Banterle, Kurt Debattista, Alessandro Artusi, Sumanta N. Pattanaik, Karol Myszkowski, Patrick Ledda, and Alan Chalmers. 2009. High dynamic range imaging and low dynamic range expansion for generating HDR content. Computer Graphics Forum 28, 8 (2009), 2343–2367.
    [6]
    Francesco Banterle, Patrick Ledda, Kurt Debattista, Alan Chalmers, and Marina Bloj. 2007. A framework for inverse tone mapping. Visual Computer 23, 7 (2007), 467–478.
    [7]
    Vladimir Bychkovsky, Sylvain Paris, Eric Chan, and Frédo Durand. 2011. Learning photographic global tonal adjustment with a database of input/output image pairs. In Proceedings of the 24th IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2011. IEEE Computer Society, 97–104.
    [8]
    Gaofeng Cao, Fei Zhou, Kanglin Liu, and Bozhi Liu. 2021. A brightness-adaptive kernel prediction network for inverse tone mapping. Neurocomputing 464 (2021), 1–14.
    [9]
    Guannan Chen, Lijie Zhang, Mengdi Sun, Yan Gao, Pablo Navarrete Michelini, and Yanhong Wu. 2021. Single-image HDR reconstruction with task-specific network based on channel adaptive RDN. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, CVPR Workshops 2021, virtual, June 19–25, 2021. Computer Vision Foundation/IEEE, 398–403.
    [10]
    Xiangyu Chen, Yihao Liu, Zhengwen Zhang, Yu Qiao, and Chao Dong. 2021. HDRUNet: Single image HDR reconstruction with denoising and dequantization. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, CVPR Workshops 2021, virtual, June 19-25, 2021. Computer Vision Foundation/IEEE, 354–363.
    [11]
    Duc-Tien Dang-Nguyen, Cecilia Pasquini, Valentina Conotter, and Giulia Boato. 2015. RAISE: A raw images dataset for digital image forensics. In Proceedings of the 6th ACM Multimedia Systems Conference, MMSys 2015. ACM, 219–224.
    [12]
    Paul E. Debevec and Jitendra Malik. 1997. Recovering high dynamic range radiance maps from photographs. In Proceedings of the 24th Annual Conference on Computer Graphics and Interactive Techniques, SIGGRAPH 1997. ACM, 369–378.
    [13]
    Gabriel Eilertsen, Saghi Hajisharif, Param Hanji, Apostolia Tsirikoglou, Rafal K. Mantiuk, and Jonas Unger. 2021. How to cheat with metrics in single-image HDR reconstruction. In Proceedings of the IEEE/CVF International Conference on Computer Vision Workshops, ICCVW 2021. IEEE, 3981–3990.
    [14]
    Gabriel Eilertsen, Joel Kronander, Gyorgy Denes, Rafal K. Mantiuk, and Jonas Unger. 2017. HDR image reconstruction from a single exposure using deep CNNs. ACM Transactions on Graphics 36, 6 (2017), 178:1–178:15.
    [15]
    Yuki Endo, Yoshihiro Kanamori, and Jun Mitani. 2017. Deep reverse tone mapping. ACM Transactions on Graphics 36, 6 (2017), 177:1–177:10.
    [16]
    Michael D. Grossberg and Shree K. Nayar. 2003. What is the space of camera response functions?. In Proceedings of the 2003 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR 2003). IEEE Computer Society, 602–612.
    [17]
    Jie Hu, Li Shen, and Gang Sun. 2018. Squeeze-and-excitation networks. In Proceedings of the 2018 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2018. Computer Vision Foundation/IEEE Computer Society, 7132–7141.
    [18]
    Wei Hu, Mozhdeh Seifi, and Erik Reinhard. 2018. Over- and under-exposure reconstruction of a single plenoptic capture. ACM Transactions on Multimedia Computing, Communications, and Applications 14, 2 (2018), 52:1–52:21.
    [19]
    Yongqing Huo, Fan Yang, Le Dong, and Vincent Brost. 2014. Physiological inverse tone mapping based on retina response. Visual Computer 30, 5 (2014), 507–517.
    [20]
    Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A. Efros. 2017. Image-to-image translation with conditional adversarial networks. In Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017. IEEE Computer Society, 5967–5976.
    [21]
    Zeeshan Khan, Mukul Khanna, and Shanmuganathan Raman. 2019. FHDR: HDR image reconstruction from a single LDR image using feedback network. In Proceedings of the 2019 IEEE Global Conference on Signal and Information Processing, GlobalSIP 2019. IEEE, 1–5.
    [22]
    Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In Proceedings of the 3rd International Conference on Learning Representations, ICLR 2015, Conference Track Proceedings.
    [23]
    Yuma Kinoshita, Sayaka Shiota, and Hitoshi Kiya. 2017. Reinhard’s global operator based inverse tone mapping with one parameter. In Proceedings of the 8th International Workshop on Signal Design and Its Applications in Communications, IWSDA 2017. IEEE, 49–53.
    [24]
    Rafael Pacheco Kovaleski and Manuel Menezes de Oliveira Neto. 2014. High-quality reverse tone mapping for a wide range of exposures. In Proceedings of the 27th SIBGRAPI Conference on Graphics, Patterns and Images, SIBGRAPI 2014. IEEE Computer Society, 49–56.
    [25]
    Hayden Landis. 2002. Production-ready global illumination. Siggraph Course Notes 16, 2002 (2002), 11.
    [26]
    Christian Ledig, Lucas Theis, Ferenc Huszar, Jose Caballero, Andrew Cunningham, Alejandro Acosta, Andrew P. Aitken, Alykhan Tejani, Johannes Totz, Zehan Wang, and Wenzhe Shi. 2017. Photo-realistic single image super-resolution using a generative adversarial network. In Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017. IEEE Computer Society, 105–114.
    [27]
    Yu-Lun Liu, Wei-Sheng Lai, Yu-Sheng Chen, Yi-Lung Kao, Ming-Hsuan Yang, Yung-Yu Chuang, and Jia-Bin Huang. 2020. Single-image HDR reconstruction by learning to reverse the camera pipeline. In Proceedings of the 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2020, Seattle. Computer Vision Foundation/IEEE, 1648–1657.
    [28]
    Rafal Mantiuk, Kil Joong Kim, Allan G. Rempel, and Wolfgang Heidrich. 2011. HDR-VDP-2: A calibrated visual metric for visibility and quality predictions in all luminance conditions. ACM Transactions on Graphics 30, 4 (2011), 40.
    [29]
    Demetris Marnerides, Thomas Bashford-Rogers, Jonathan Hatchett, and Kurt Debattista. 2018. ExpandNet: A deep convolutional neural network for high dynamic range expansion from low dynamic range content. Computer Graphics Forum 37, 2 (2018), 37–49.
    [30]
    Simon Niklaus, Long Mai, and Feng Liu. 2017. Video frame interpolation via adaptive separable convolution. In Proceedings of the IEEE International Conference on Computer Vision, ICCV 2017. IEEE Computer Society, 261–270.
    [31]
    Erik Reinhard and Kate Devlin. 2005. Dynamic range reduction inspired by photoreceptor physiology. IEEE Transactions on Visualization and Computer Graphics 11, 1 (2005), 13–24.
    [32]
    Allan G. Rempel, Matthew Trentacoste, Helge Seetzen, H. David Young, Wolfgang Heidrich, Lorne Whitehead, and Greg Ward. 2007. Ldr2Hdr: On-the-fly reverse tone mapping of legacy video and photographs. ACM Transactions on Graphics 26, 3 (2007), 39.
    [33]
    Olaf Ronneberger, Philipp Fischer, and Thomas Brox. 2015. U-net: Convolutional networks for biomedical image segmentation. In Proceedings of the Medical Image Computing and Computer-Assisted Intervention—MICCAI 2015-18th International Conference Munich, Proceedings, Part III (Lecture Notes in Computer Science), Vol. 9351. Springer, 234–241.
    [34]
    Marcel Santana Santos, Tsang Ing Ren, and Nima Khademi Kalantari. 2020. Single image HDR reconstruction using a CNN with masked features and perceptual loss. ACM Transactions on Graphics 39, 4 (2020), 1–10.
    [35]
    Xintao Wang, Ke Yu, Chao Dong, and Chen Change Loy. 2018. Recovering realistic texture in image super-resolution by deep spatial feature transform. In Proceedings of the 2018 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2018. Computer Vision Foundation/IEEE Computer Society, 606–615.
    [36]
    Zhou Wang, Alan C. Bovik, Hamid R. Sheikh, and Eero P. Simoncelli. 2004. Image quality assessment: From error visibility to structural similarity. IEEE Transactions on Image Processing 13, 4 (2004), 600–612.
    [37]
    Zhou Wang, Eero P. Simoncelli, and Alan C. Bovik. 2003. Multiscale structural similarity for image quality assessment. In Proceedings of the 37th Asilomar Conference on Signals, Systems & Computers, 2003, Vol. 2. IEEE, 1398–1402.
    [38]
    Sanghyun Woo, Jongchan Park, Joon-Young Lee, and In So Kweon. 2018. CBAM: Convolutional block attention module. In Proceedings of the Computer Vision - ECCV 2018-15th European Conference, Proceedings, Part VII (Lecture Notes in Computer Science), Vol. 11211. Springer, 3–19.
    [39]
    Zhihao Xia, Federico Perazzi, Michaël Gharbi, Kalyan Sunkavalli, and Ayan Chakrabarti. 2020. Basis prediction networks for effective burst denoising with large kernels. In Proceedings of the 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2020. Computer Vision Foundation/IEEE, 11841–11850.
    [40]
    Xin Xu, Shiqin Wang, Zheng Wang, Xiaolong Zhang, and Ruimin Hu. 2021. Exploring image enhancement for salient object detection in low light images. ACM Transactions on Multimedia Computing, Communications, and Applications 17, 1s (2021), 1–19.
    [41]
    Qingsen Yan, Dong Gong, Qinfeng Shi, Anton van den Hengel, Chunhua Shen, Ian D. Reid, and Yanning Zhang. 2019. Attention-guided network for ghost-free high dynamic range imaging. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019. Computer Vision Foundation/IEEE, 1751–1760.
    [42]
    Qingsen Yan, Dong Gong, Qinfeng (Javen) Shi, Anton van den Hengel, Chunhua Shen, Ian D. Reid, and Yanning Zhang. 2022. Dual-attention-guided network for ghost-free high dynamic range imaging. International Journal of Computer Vision 130, 1 (2022), 76–94.
    [43]
    Qingsen Yan, Dong Gong, Qinfeng (Javen) Shi, Anton van den Hengel, Jinqiu Sun, Yu Zhu, and Yanning Zhang. 2022. High dynamic range imaging via gradient-aware context aggregation network. Pattern Recognition 122 (2022), 108342.
    [44]
    Qingsen Yan, Lei Zhang, Yu Liu, Yu Zhu, Jinqiu Sun, Qinfeng Shi, and Yanning Zhang. 2020. Deep HDR imaging via A non-local network. IEEE Transactions on Image Processing 29 (2020), 4308–4322.
    [45]
    Dongyang Zhang, Jie Shao, and Heng Tao Shen. 2020. Kernel attention network for single image super-resolution. ACM Transactions on Multimedia Computing, Communications, and Applications 16, 3 (2020), 90:1–90:15.
    [46]
    Han Zhang, Ian J. Goodfellow, Dimitris N. Metaxas, and Augustus Odena. 2019. Self-attention generative adversarial networks. In Proceedings of the 36th International Conference on Machine Learning, ICML (Proceedings of Machine Learning Research), Vol. 97. PMLR, 7354–7363.
    [47]
    Zhuoran Zheng, Wenqi Ren, Xiaochun Cao, Tao Wang, and Xiuyi Jia. 2021. Ultra-high-definition image HDR reconstruction via collaborative bilateral learning. In Proceedings of the 2021 IEEE/CVF International Conference on Computer Vision, ICCV 2021. IEEE, 4429–4438.

    Cited By

    View all
    • (2024)UHeadProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36435518:1(1-28)Online publication date: 6-Mar-2024
    • (2024)Mixing and Matching Elements for Intelligent Fashion Design: A Generative Adversarial Network With Structure and Texture DisentanglementIEEE Transactions on Consumer Electronics10.1109/TCE.2023.332957470:1(358-370)Online publication date: Feb-2024
    • (2024)High Dynamic Range Imaging via Visual Attention ModulesIEEE Access10.1109/ACCESS.2024.338609612(50911-50924)Online publication date: 2024
    • Show More Cited By

    Index Terms

    1. A Decoupled Kernel Prediction Network Guided by Soft Mask for Single Image HDR Reconstruction

      Recommendations

      Comments

      Information & Contributors

      Information

      Published In

      cover image ACM Transactions on Multimedia Computing, Communications, and Applications
      ACM Transactions on Multimedia Computing, Communications, and Applications  Volume 19, Issue 2s
      April 2023
      545 pages
      ISSN:1551-6857
      EISSN:1551-6865
      DOI:10.1145/3572861
      • Editor:
      • Abdulmotaleb El Saddik
      Issue’s Table of Contents

      Publisher

      Association for Computing Machinery

      New York, NY, United States

      Publication History

      Published: 17 February 2023
      Online AM: 22 July 2022
      Accepted: 19 July 2022
      Revised: 06 June 2022
      Received: 28 February 2022
      Published in TOMM Volume 19, Issue 2s

      Permissions

      Request permissions for this article.

      Check for updates

      Author Tags

      1. High dynamic range image
      2. inverse tone mapping
      3. kernel prediction network
      4. image reconstruction

      Qualifiers

      • Research-article

      Funding Sources

      • Guangdong Basic and Applied Basic Research Foundation
      • National Natural Science Foundation of China

      Contributors

      Other Metrics

      Bibliometrics & Citations

      Bibliometrics

      Article Metrics

      • Downloads (Last 12 months)223
      • Downloads (Last 6 weeks)23
      Reflects downloads up to 11 Aug 2024

      Other Metrics

      Citations

      Cited By

      View all
      • (2024)UHeadProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36435518:1(1-28)Online publication date: 6-Mar-2024
      • (2024)Mixing and Matching Elements for Intelligent Fashion Design: A Generative Adversarial Network With Structure and Texture DisentanglementIEEE Transactions on Consumer Electronics10.1109/TCE.2023.332957470:1(358-370)Online publication date: Feb-2024
      • (2024)High Dynamic Range Imaging via Visual Attention ModulesIEEE Access10.1109/ACCESS.2024.338609612(50911-50924)Online publication date: 2024
      • (2024)LπCETIET Information Security10.1049/2024/26347442024Online publication date: 12-Jun-2024
      • (2024)DEUNet: Dual-encoder UNet for simultaneous denoising and reconstruction of single HDR imageComputers & Graphics10.1016/j.cag.2024.01.009119(103882)Online publication date: Apr-2024
      • (2023)Exploiting Light Polarization for Deep HDR Imaging from a Single ExposureSensors10.3390/s2312537023:12(5370)Online publication date: 6-Jun-2023
      • (2023)Learning Event Guided High Dynamic Range Video Reconstruction2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)10.1109/CVPR52729.2023.01338(13924-13934)Online publication date: Jun-2023
      • (2023)ArtHDR-Net: Perceptually Realistic and Accurate HDR Content Creation2023 Asia Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA ASC)10.1109/APSIPAASC58517.2023.10317568(806-812)Online publication date: 31-Oct-2023

      View Options

      Get Access

      Login options

      Full Access

      View options

      PDF

      View or Download as a PDF file.

      PDF

      eReader

      View online with eReader.

      eReader

      Full Text

      View this article in Full Text.

      Full Text

      HTML Format

      View this article in HTML Format.

      HTML Format

      Media

      Figures

      Other

      Tables

      Share

      Share

      Share this Publication link

      Share on social media