research-article

HDR-cGAN: single LDR to HDR image translation using conditional GAN

Authors:

Prarabdh Raipurkar,

Shanmuganathan RamanAuthors Info & Claims

ICVGIP '21: Proceedings of the Twelfth Indian Conference on Computer Vision, Graphics and Image Processing

Article No.: 17, Pages 1 - 9

https://doi.org/10.1145/3490035.3490275

Published: 19 December 2021 Publication History

Abstract

The prime goal of digital imaging techniques is to reproduce the realistic appearance of a scene. Low Dynamic Range (LDR) cameras are incapable of representing the wide dynamic range of the real-world scene. The captured images turn out to be either too dark (underexposed) or too bright (overexposed). Specifically, saturation in overexposed regions makes the task of reconstructing a High Dynamic Range (HDR) image from single LDR image challenging. In this paper, we propose a deep learning based approach to recover details in the saturated areas while reconstructing the HDR image. We formulate this problem as an image-to-image (I2I) translation task. To this end, we present a novel conditional GAN (cGAN) based framework trained in an end-to-end fashion over the HDR-REAL and HDR-SYNTH datasets. Our framework uses an overexposed mask obtained from a pre-trained segmentation model to facilitate the hallucination task of adding details in the saturated regions. We demonstrate the effectiveness of the proposed method by performing an extensive quantitative and qualitative comparison with several state-of-the-art single-image HDR reconstruction techniques.

References

[1]

2017. Photomatix. https://www.hdrsoft.com/.

[2]

Md Zahangir Alom, Mahmudul Hasan, Chris Yakopcic, Tarek M Taha, and Vijayan K Asari. 2018. Recurrent residual convolutional neural network based on u-net (r2u-net) for medical image segmentation. arXiv preprint arXiv:1802.06955 (2018).

[3]

Francesco Banterle, Patrick Ledda, Kurt Debattista, and Alan Chalmers. 2006. Inverse tone mapping. In Proceedings of the 4th International Conference on Computer Graphics and Interactive Techniques in Australasia and Southeast Asia 2006, Kuala Lumpur, Malaysia, November 29 - December 2, 2006, Y. T. Lee, Siti Mariyam Hj. Shamsuddin, Diego Gutierrez, and Norhaida Mohd. Suaib (Eds.). ACM, 349--356.

Digital Library

[4]

Michael S Brown and SJ Kim. 2019. Understanding the in-camera image processing pipeline for computer vision. In IEEE International Conference on Computer Vision (ICCV)-Tutorial, Vol. 3.

[5]

Liang-Chieh Chen, George Papandreou, Florian Schroff, and Hartwig Adam. 2017. Rethinking atrous convolution for semantic image segmentation. arXiv preprint arXiv:1706.05587 (2017).

[6]

Paul E. Debevec and Jitendra Malik. 1997. Recovering High Dynamic Range Radiance Maps from Photographs. In Proceedings of the 24th Annual Conference on Computer Graphics and Interactive Techniques (SIGGRAPH '97). ACM Press/Addison-Wesley Publishing Co., USA, 369--378.

Digital Library

[7]

Gabriel Eilertsen, Joel Kronander, Gyorgy Denes, Rafał K Mantiuk, and Jonas Unger. 2017. HDR image reconstruction from a single exposure using deep CNNs. ACM transactions on graphics (TOG) 36, 6 (2017), 1--15.

Digital Library

[8]

Yuki Endo, Yoshihiro Kanamori, and Jun Mitani. 2017. Deep Reverse Tone Mapping. ACM Trans. Graph. 36, 6, Article 177 (Nov. 2017), 10 pages.

Digital Library

[9]

Mark D Fairchild. 2007. The HDR photographic survey. In Color and imaging conference, Vol. 2007. Society for Imaging Science and Technology, 233--238.

[10]

B. Funt and Lilong Shi. 2010. The Rehabilitation of MaxRGB. In Color Imaging Conference.

[11]

Brian Funt and Lilong Shi. 2010. The effect of exposure on MaxRGB color constancy. In Human Vision and Electronic Imaging XV, Bernice E. Rogowitz and Thrasyvoulos N. Pappas (Eds.), Vol. 7527. International Society for Optics and Photonics, SPIE, 282 -- 288.

[12]

Ian J Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. 2014. Generative adversarial networks. arXiv preprint arXiv:1406.2661 (2014).

Digital Library

[13]

M.D. Grossberg and S.K. Nayar. 2004. Modeling the space of camera response functions. IEEE Transactions on Pattern Analysis and Machine Intelligence 26, 10 (2004), 1272--1282.

Digital Library

[14]

Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A Efros. 2017. Image-to-Image Translation with Conditional Adversarial Networks. CVPR (2017).

[15]

Darshita Jain and Shanmuganathan Raman. 2021. Deep over and Under Exposed Region Detection. In Computer Vision and Image Processing. Springer Singapore, Singapore, 34--45.

[16]

Justin Johnson, Alexandre Alahi, and Li Fei-Fei. 2016. Perceptual losses for real-time style transfer and super-resolution. In European conference on computer vision. Springer, 694--711.

[17]

Nima Khademi Kalantari and Ravi Ramamoorthi. 2017. Deep High Dynamic Range Imaging of Dynamic Scenes. ACM Transactions on Graphics (Proceedings of SIGGRAPH 2017) 36, 4 (2017).

Digital Library

[18]

Nima Khademi Kalantari and Ravi Ramamoorthi. 2019. Deep HDR Video from Sequences with Alternating Exposures. Computer Graphics Forum 38, 2 (2019), 193--205.

[19]

Hakki Can Karaimer and Michael S. Brown. 2016. A Software Platform for Manipulating the Camera Imaging Pipeline. In European Conference on Computer Vision (ECCV).

[20]

Zeeshan Khan, Mukul Khanna, and Shanmuganathan Raman. 2019. FHDR: HDR Image Reconstruction from a Single LDR Image using Feedback Network. In 2019 IEEE Global Conference on Signal and Information Processing (GlobalSIP). 1--5.

[21]

Seon Joo Kim, Hai Ting Lin, Zheng Lu, Sabine Süsstrunk, Stephen Lin, and Michael S. Brown. 2012. A new in-camera imaging model for color computer vision and its application. IEEE Transactions on Pattern Analysis and Machine Intelligence 34, 12 (2012), 2289--2302.

Digital Library

[22]

Diederik Kingma and Jimmy Ba. 2014. Adam: A Method for Stochastic Optimization. International Conference on Learning Representations (12 2014).

[23]

Siyeong Lee, Gwon Hwan An, and Suk-Ju Kang. 2018. Deep Chain HDRI: Reconstructing a High Dynamic Range Image from a Single Low Dynamic Range Image. IEEE Access 6 (2018), 49913--49924.

[24]

Siyeong Lee, Gwon Hwan An, and Suk-Ju Kang. 2018. Deep recursive hdri: Inverse tone mapping using generative adversarial networks. In Proceedings of the European Conference on Computer Vision (ECCV). 596--611.

Digital Library

[25]

LeeJunHyun. 2018. Image_Segmentation. https://github.com/LeeJunHyun/Image_Segmentation.

[26]

Jinghui Li and Peiyu Fang. 2019. Hdrnet: Single-image-based hdr reconstruction using channel attention cnn. In Proceedings of the 2019 4th International Conference on Multimedia Systems and Signal Processing. 119--124.

Digital Library

[27]

Ming Liang and Xiaolin Hu. 2015. Recurrent convolutional neural network for object recognition. In 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 3367--3375.

[28]

Yu-Lun Liu, Wei-Sheng Lai, Yu-Sheng Chen, Yi-Lung Kao, Ming-Hsuan Yang, Yung-Yu Chuang, and Jia-Bin Huang. 2020. Single-image hdr reconstruction by learning to reverse the camera pipeline. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 1651--1660.

[29]

S. Mann and R. W. Picard. 1995. On Being 'Undigital' With Digital Cameras: Extending Dynamic Range By Combining Differently Exposed Pictures. In PROCEEDINGS OF IS&T. 442--448.

[30]

Demetris Marnerides, Thomas Bashford-Rogers, Jonathan Hatchett, and Kurt Debattista. 2018. Expandnet: A deep convolutional neural network for high dynamic range expansion from low dynamic range content. In Computer Graphics Forum, Vol. 37. Wiley Online Library, 37--49.

[31]

Tom Mertens, Jan Kautz, and Frank Van Reeth. 2007. Exposure Fusion (PG '07). IEEE Computer Society, USA, 382--390.

Digital Library

[32]

Mehdi Mirza and Simon Osindero. 2014. Conditional generative adversarial nets. arXiv preprint arXiv:1411.1784 (2014).

[33]

Kenta Moriwaki, Ryota Yoshihashi, Rei Kawakami, Shaodi You, and Takeshi Naemura. 2018. Hybrid loss for learning single-image-based HDR reconstruction. arXiv preprint arXiv:1812.07134 (2018).

[34]

Manish Narwaria, Rafal Mantiuk, Matthieu Perreira Da Silva, and Patrick Le Callet. 2015. HDR-VDP-2.2: A calibrated method for objective quality prediction of high-dynamic range and standard images. Journal of Electronic Imaging 24 (01 2015), 010501.

[35]

S.K. Nayar and T. Mitsunaga. 2000. High dynamic range imaging: spatially varying pixel exposures. In Proceedings IEEE Conference on Computer Vision and Pattern Recognition. CVPR 2000 (Cat. No.PR00662), Vol. 1. 472--479 vol.1.

[36]

Hiromi Nemoto, Pavel Korshunov, Philippe Hanhart, and Touradj Ebrahimi. 2015. Visual attention in LDR and HDR images. In 9th International Workshop on Video Processing and Quality Metrics for Consumer Electronics (VPQM).

[37]

Yuzhen Niu, Jianbin Wu, Wenxi Liu, W. Guo, and Rynson W. H. Lau. 2021. HDR-GAN: HDR Image Reconstruction From Multi-Exposed LDR Images With Large Motions. IEEE Transactions on Image Processing 30 (2021), 3885--3896.

[38]

Ozan Oktay, Jo Schlemper, Loic Le Folgoc, Matthew Lee, Mattias Heinrich, Kazunari Misawa, Kensaku Mori, Steven McDonagh, Nils Y Hammerla, Bernhard Kainz, et al. 2018. Attention u-net: Learning where to look for the pancreas. arXiv preprint arXiv:1804.03999 (2018).

[39]

Yingxue Pang, Jianxin Lin, Tao Qin, and Zhibo Chen. 2021. Image-to-Image Translation: Methods and Applications. arXiv preprint arXiv:2101.08629 (2021).

[40]

Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. 2019. Pytorch: An imperative style, high-performance deep learning library. arXiv preprint arXiv:1912.01703 (2019).

Digital Library

[41]

Erik Reinhard, Wolfgang Heidrich, Paul Debevec, Sumanta Pattanaik, Greg Ward, and Karol Myszkowski. 2010. High dynamic range imaging: acquisition, display, and image-based lighting. Morgan Kaufmann.

Digital Library

[42]

Olaf Ronneberger, Philipp Fischer, and Thomas Brox. 2015. U-net: Convolutional networks for biomedical image segmentation. In International Conference on Medical image computing and computer-assisted intervention. Springer, 234--241.

[43]

Marcel Santana Santos, Ing Ren Tsang, and N. Kalantari. 2020. Single image HDR reconstruction using a CNN with masked features and perceptual loss. ACM Transactions on Graphics (TOG) 39 (2020), 80:1 -- 80:10.

Digital Library

[44]

Pradeep Sen, Nima Khademi Kalantari, Maziar Yaesoubi, Soheil Darabi, Dan B. Goldman, and Eli Shechtman. 2012. Robust Patch-Based Hdr Reconstruction of Dynamic Scenes. ACM Trans. Graph. 31, 6, Article 203 (Nov. 2012), 11 pages.

Digital Library

[45]

Karen Simonyan and Andrew Zisserman. 2014. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556 (2014).

[46]

J. Tumblin, A. Agrawal, and R. Raskar. 2005. Why I want a gradient camera. In 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05), Vol. 1. 103--110 vol. 1.

Digital Library

[47]

Ziyu Wan, Bo Zhang, Dongdong Chen, Pan Zhang, Dong Chen, Jing Liao, and Fang Wen. 2020. Bringing Old Photos Back to Life. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2747--2757.

[48]

Zhou Wang, A.C. Bovik, H.R. Sheikh, and E.P. Simoncelli. 2004. Image quality assessment: from error visibility to structural similarity. IEEE Transactions on Image Processing 13, 4 (2004), 600--612.

Digital Library

[49]

Greg Ward et al. 2006. High dynamic range image encodings. (2006).

[50]

Shangzhe Wu, Jiarui Xu, Yu-Wing Tai, and Chi-Keung Tang. 2018. Deep High Dynamic Range Imaging with Large Foreground Motions. In Computer Vision - ECCV 2018, Vittorio Ferrari, Martial Hebert, Cristian Sminchisescu, and Yair Weiss (Eds.). Springer International Publishing, Cham, 120--135.

[51]

Feng Xiao, Jeffrey M DiCarlo, Peter B Catrysse, and Brian A Wandell. 2002. High dynamic range imaging of natural scenes. In Color and imaging conference, Vol. 2002. Society for Imaging Science and Technology, 337--342.

[52]

Qingsen Yan, Dong Gong, Qinfeng Shi, A. V. Hengel, Chunhua Shen, I. Reid, and Y. Zhang. 2019. Attention-Guided Network for Ghost-Free High Dynamic Range Imaging. 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2019), 1751--1760.

[53]

Qingsen Yan, Jinqiu Sun, Haisen Li, Yu Zhu, and Yanning Zhang. 2017. High dynamic range imaging by sparse representation. Neurocomputing 269 (2017), 160--169.

Digital Library

[54]

Xin Yang, Ke Xu, Yibing Song, Qiang Zhang, Xiaopeng Wei, and Rynson W.H. Lau. 2018. Image Correction via Deep Reciprocating HDR Transformation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[55]

Bolei Zhou, Agata Lapedriza, Jianxiong Xiao, Antonio Torralba, and Aude Oliva. 2014. Learning Deep Features for Scene Recognition using Places Database. In Advances in Neural Information Processing Systems, Z. Ghahramani, M. Welling, C. Cortes, N. Lawrence, and K. Q. Weinberger (Eds.), Vol. 27. Curran Associates, Inc. https://proceedings.neurips.cc/paper/2014/file/3fe94a002317b5f9259f82690aeea4cd-Paper.pdf

Digital Library

Cited By

Barua HMg TPramanick PSarkar C(2024)Enabling Social Robots to Perceive and Join Socially Interacting Groups Using F-formation: A Comprehensive OverviewACM Transactions on Human-Robot Interaction10.1145/368207213:4(1-48)Online publication date: 23-Oct-2024
https://dl.acm.org/doi/10.1145/3682072
Nayak AVenugopala PAshwini B(2024)A Systematic Review on Generative Adversarial Network (GAN): Challenges and Future DirectionsArchives of Computational Methods in Engineering10.1007/s11831-024-10119-131:8(4739-4772)Online publication date: 14-May-2024
https://doi.org/10.1007/s11831-024-10119-1
Di Maro AAzpiroz IBiain XLongo GOlaizola I(2023)Overcoming Adverse Conditions in Rescue Scenarios: A Deep Learning and Image Processing ApproachApplied Sciences10.3390/app1309549913:9(5499)Online publication date: 28-Apr-2023
https://doi.org/10.3390/app13095499
Show More Cited By

Index Terms

HDR-cGAN: single LDR to HDR image translation using conditional GAN
1. Computing methodologies
  1. Computer graphics
    1. Image manipulation
      1. Computational photography
      2. Image processing

Recommendations

Single image HDR reconstruction using a CNN with masked features and perceptual loss

Digital cameras can only capture a limited range of real-world scenes' luminance, producing images with saturated pixels. Existing single image high dynamic range (HDR) reconstruction methods attempt to expand the range of luminance, but are not able to ...
HDR-LFNet: Inverse tone mapping using fusion network
Abstract
To capture the real-world luminance values, High Dynamic Range (HDR) image processing has been developed. HDR images have a richer content than the widely-used Standard Dynamic Range (SDR) images, and are used in a number of situations, e.g. in ...
Graphical abstract

Display Omitted
Highlights
- We propose a new Inverse Tone Mapping Operator to create High Dynamic Range images.
- Convolutional Neural Network are used to fuse together existing methods.
- We performed a subjective study to evaluate our method.
- The network’s ...
Deep Recursive HDRI: Inverse Tone Mapping Using Generative Adversarial Networks
Computer Vision – ECCV 2018
Abstract
High dynamic range images contain luminance information of the physical world and provide more realistic experience than conventional low dynamic range images. Because most images have a low dynamic range, recovering the lost dynamic range from a ...

Comments

Information & Contributors

Information

Published In

cover image ACM Other conferences

ICVGIP '21: Proceedings of the Twelfth Indian Conference on Computer Vision, Graphics and Image Processing

December 2021

428 pages

ISBN:9781450375962

DOI:10.1145/3490035

General Chairs:
Rama Chellappa
Johns Hopkins University
,
Santanu Chaudhury
IIT Jodhpur
,
Program Chairs:
Chetan Arora
IIT Delhi
,
Parag Chaudhuri
IIT Bombay
,
Subhransu Maji
University of Massachusetts, Amherst

Copyright © 2021 ACM.

Publication rights licensed to ACM. ACM acknowledges that this contribution was authored or co-authored by an employee, contractor or affiliate of a national government. As such, the Government retains a nonexclusive, royalty-free right to publish or reproduce this article, or to allow others to do so, for Government purposes only.

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 19 December 2021

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

Science and Engineering Research Board (SERB)

Conference

ICVGIP '21

ICVGIP '21: Indian Conference on Computer Vision, Graphics and Image Processing

December 19 - 22, 2021

Jodhpur, India

Acceptance Rates

Overall Acceptance Rate 95 of 286 submissions, 33%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

6
Total Citations
View Citations
148
Total Downloads

Downloads (Last 12 months)43
Downloads (Last 6 weeks)8

Reflects downloads up to 25 Dec 2024

Other Metrics

View Author Metrics

Citations

Cited By

Barua HMg TPramanick PSarkar C(2024)Enabling Social Robots to Perceive and Join Socially Interacting Groups Using F-formation: A Comprehensive OverviewACM Transactions on Human-Robot Interaction10.1145/368207213:4(1-48)Online publication date: 23-Oct-2024
https://dl.acm.org/doi/10.1145/3682072
Nayak AVenugopala PAshwini B(2024)A Systematic Review on Generative Adversarial Network (GAN): Challenges and Future DirectionsArchives of Computational Methods in Engineering10.1007/s11831-024-10119-131:8(4739-4772)Online publication date: 14-May-2024
https://doi.org/10.1007/s11831-024-10119-1
Di Maro AAzpiroz IBiain XLongo GOlaizola I(2023)Overcoming Adverse Conditions in Rescue Scenarios: A Deep Learning and Image Processing ApproachApplied Sciences10.3390/app1309549913:9(5499)Online publication date: 28-Apr-2023
https://doi.org/10.3390/app13095499
Dalal DVashishtha GSingh PRaman S(2023)Single Image LDR to HDR Conversion Using Conditional Diffusion2023 IEEE International Conference on Image Processing (ICIP)10.1109/ICIP49359.2023.10222821(3533-3537)Online publication date: 8-Oct-2023
https://doi.org/10.1109/ICIP49359.2023.10222821
Guo BLin C(2023)Single-Image HDR Reconstruction Based on Two-Stage GAN Structure2023 IEEE International Conference on Image Processing (ICIP)10.1109/ICIP49359.2023.10222156(91-95)Online publication date: 8-Oct-2023
https://doi.org/10.1109/ICIP49359.2023.10222156
Barua HKrishnasamy GWong KStefanov KDhall A(2023)ArtHDR-Net: Perceptually Realistic and Accurate HDR Content Creation2023 Asia Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA ASC)10.1109/APSIPAASC58517.2023.10317568(806-812)Online publication date: 31-Oct-2023
https://doi.org/10.1109/APSIPAASC58517.2023.10317568

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents