research-article

Deep Learning Thermal Image Translation for Night Vision Perception

Authors:

Erik BlaschAuthors Info & Claims

ACM Transactions on Intelligent Systems and Technology (TIST), Volume 12, Issue 1

Article No.: 9, Pages 1 - 18

https://doi.org/10.1145/3426239

Published: 22 December 2020 Publication History

Abstract

Context enhancement is critical for the environmental perception in night vision applications, especially for the dark night situation without sufficient illumination. In this article, we propose a thermal image translation method, which can translate thermal/infrared (IR) images into color visible (VI) images, called IR2VI. The IR2VI consists of two cascaded steps: translation from nighttime thermal IR images to gray-scale visible images (GVI), which is called IR-GVI; and the translation from GVI to color visible images (CVI), which is known as GVI-CVI in this article. For the first step, we develop the Texture-Net, a novel unsupervised image translation neural network based on generative adversarial networks. Texture-Net can learn the intrinsic characteristics from the GVI and integrate them into the IR image. In comparison with the state-of-the-art unsupervised image translation methods, the proposed Texture-Net is able to address some common challenges, e.g., incorrect mapping and lack of fine details, with a structure connection module and a region-of-interest focal loss. For the second step, we investigated the state-of-the-art gray-scale image colorization methods and integrate the deep convolutional neural network into the IR2VI framework. The results of the comprehensive evaluation experiments demonstrate the effectiveness of the proposed IR2VI image translation method. This solution will contribute to the environmental perception and understanding in varied night vision applications.

References

[1]

SENSIAC. 2008. Military Sensing Information Analysis Center. Retrieved from https://www.sensiac.org/external/products/list_databases/.

[2]

Gaurav Bhatnagar and Zheng Liu. 2015. A novel image fusion framework for night-vision navigation and surveillance. Signal Image Video Process. 9, 1 (2015), 165--175.

[3]

Rick S. Blum and Zheng Liu. 2005. Multi-sensor Image Fusion and Its Applications. CRC Press.

[4]

Huiwen Chang, Ohad Fried, Yiming Liu, Stephen DiVerdi, and Adam Finkelstein. 2015. Palette-based photo recoloring. ACM Trans. Graph. 34, 4 (2015), 139.

Digital Library

[5]

Qifeng Chen and Vladlen Koltun. 2017. Photographic image synthesis with cascaded refinement networks. In Proceedings of the IEEE International Conference on Computer Vision (ICCV’17).

[6]

Zezhou Cheng, Qingxiong Yang, and Bin Sheng. 2015. Deep colorization. In Proceedings of the IEEE International Conference on Computer Vision. 415--423.

Digital Library

[7]

Alex Yong-Sang Chia, Shaojie Zhuo, Raj Kumar Gupta, Yu-Wing Tai, Siu-Yeung Cho, Ping Tan, and Stephen Lin. 2011. Semantic colorization with internet images. In ACM Transactions on Graphics, Vol. 30. ACM, 156.

[8]

Taeko Chijiiwa, Tatsuro Ishibashi, and Hajime Inomata. 1990. Histological study of choroidal melanocytes in animals with tapetum lucidum cellulosum. Graefe’s Arch. Clin. Exper. Ophthalmol. 228, 2 (1990), 161--168.

[9]

Yunjey Choi, Minje Choi, Munyoung Kim, Jung-Woo Ha, Sunghun Kim, and Jaegul Choo. 2017. StarGAN: Unified generative adversarial networks for multi-domain image-to-image translation. Retrieved from httpas://Arxiv:1711.09020.

[10]

Serge O. Dumoulin, Steven C. Dakin, and Robert F. Hess. 2008. Sparsely distributed contours dominate extra-striate responses to complex scenes. Neuroimage 42, 2 (2008), 890--901.

[11]

Lucas Rodes-Guirao Federico Baldassarre, and Diego Gonzalez-Morin. 2017. Deep-koalarization: Image colorization using CNNs and inception-resnet-v2. Retrieved from https://ArXiv:1712.03400.

[12]

Salvador Gabarda and Gabriel Cristóbal. 2007. Blind image quality assessment through anisotropy. JOSA A 24, 12 (2007), B42--B51.

[13]

Ross Girshick. 2015. Fast R-CNN. In Proceedings of the IEEE International Conference on Computer Vision (ICCV’15). 1440--1448.

[14]

Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. 2014. Generative adversarial nets. In Advances in Neural Information Processing Systems. MIT Press, 2672--2680.

Digital Library

[15]

Raj Kumar Gupta, Alex Yong-Sang Chia, Deepu Rajan, Ee Sin Ng, and Huang Zhiyong. 2012. Image colorization using similar images. In Proceedings of the 20th ACM International Conference on Multimedia (ACMMM’12). ACM, 369--378.

Digital Library

[16]

Tomer Hamam, Yedidyah Dordek, and Deborah Cohen. 2012. Single-band infrared texture-based image colorization. In Proceedings of the IEEE 27th Convention of Electrical 8 Electronics Engineers in Israel (IEEEI’12). IEEE, 1--5.

[17]

Aaron Hertzmann, Charles E. Jacobs, Nuria Oliver, Brian Curless, and David H. Salesin. 2001. Image analogies. In Proceedings of the 28th Annual Conference on Computer Graphics and Interactive Techniques (SIGGRAPH’01). ACM, 327--340.

[18]

Jonathan Huang, Vivek Rathod, Chen Sun, Menglong Zhu, Anoop Korattikara, Alireza Fathi, Ian Fischer, Zbigniew Wojna, Yang Song, Sergio Guadarrama, et al. 2017. Speed/accuracy trade-offs for modern convolutional object detectors. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR’17).

[19]

Yi-Chin Huang, Yi-Shin Tung, Jun-Cheng Chen, Sung-Wen Wang, and Ja-Ling Wu. 2005. An adaptive edge detection based colorization algorithm and its applications. In Proceedings of the 13th Annual ACM International Conference on Multimedia. ACM, 351--354.

Digital Library

[20]

Satoshi Iizuka, Edgar Simo-Serra, and Hiroshi Ishikawa. 2016. Let there be color!: Joint end-to-end learning of global and local image priors for automatic image colorization with simultaneous classification. ACM Trans. Graph. 35, 4 (2016), 110.

Digital Library

[21]

Revital Irony, Daniel Cohen-Or, and Dani Lischinski. 2005. Colorization by example. In Proceedings of the 16th Eurographics Conference on Rendering Techniques (EGSR’05). Eurographics Association, Aire-la-Ville, Switzerland, 201--210.

[22]

M. Jeong, B. C. Ko, and J. Y. Nam. 2017. Early detection of sudden pedestrian crossing for safe driving during summer nights. IEEE Trans. Circ. Syst. Video Technol. 27, 6 (June 2017), 1368--1380.

Digital Library

[23]

Xin Jin, Qian Jiang, Shaowen Yao, Dongming Zhou, Rencan Nie, Jinjin Hai, and Kangjian He. 2017. A survey of infrared and visual image fusion methods. Infrared Phys. Technol. 85 (2017), 478--501.

[24]

Justin Johnson, Alexandre Alahi, and Li Fei-Fei. 2016. Perceptual losses for real-time style transfer and super-resolution. In Proceedings of the European Conference on Computer Vision (ECCV’16). Springer, 694--711.

[25]

Zhu Jun-Yan, Park Taesung, Isola Phillip, and A. Efros Alexei.2017. Unpaired image-to-image translation using cycle-consistent adversarial networks. In Proceedings of the IEEE International Conference on Computer Vision (ICCV’17). 2242--2251.

[26]

He Kaiming, Zhang Xiangyu, Ren Shaoqing, and Sun Jian. 2016. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR’16). 770--778.

[27]

Junho Kim, Minjae Kim, Hyeonwoo Kang, and Kwang Hee Lee. 2020. U-GAT-IT: Unsupervised generative attentional networks with adaptive layer-instance normalization for image-to-image translation. In Proceedings of the International Conference on Learning Representations. Retrieved from https://openreview.net/forum?id=BJlZ5ySKPH.

[28]

Diederik P. Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. CoRR abs/1412.6980.

[29]

Gustav Larsson, Michael Maire, and Gregory Shakhnarovich. 2016. Learning representations for automatic colorization. In Proceedings of the European Conference on Computer Vision. Springer, 577--593.

[30]

Anat Levin, Dani Lischinski, and Yair Weiss. 2004. Colorization using optimization. In ACM Transactions on Graphics, Vol. 23. ACM, 689--694.

Digital Library

[31]

Matthias Limmer and Hendrik P. A. Lensch. 2016. Infrared colorization using deep convolutional neural networks. In Proceedings of the IEEE International Conference on Machine Learning and Applications (ICMLA’16). IEEE, 61--68.

[32]

Ming-Yu Liu, Thomas Breuel, and Jan Kautz. 2017. Unsupervised image-to-image translation networks. In Advances in Neural Information Processing Systems. MIT Press, 700--708.

Digital Library

[33]

Shuo Liu and Zheng Liu. 2017. Multi-channel CNN-based object detection for enhanced situation awareness. In Proceedings of the 9th NATO Military Sensing Symposium on Sensors and Electronics Technology (SET’17).

[34]

X. Liu, J. v. d. Weijer, and A. D. Bagdanov. 2017. RankIQA: Learning from rankings for no-reference image quality assessment. In Proceedings of the IEEE International Conference on Computer Vision (ICCV’17). 1040--1049.

[35]

Zheng Liu, Erik Blasch, Zhiyun Xue, Jiying Zhao, Robert Laganiere, and Wei Wu. 2012. Objective assessment of multiresolution image fusion algorithms for context enhancement in night vision: A comparative study. IEEE Trans. Pattern Anal. Mach. Intell. 34, 1 (2012), 94--109.

Digital Library

[36]

Jiayi Ma, Chen Chen, Chang Li, and Jun Huang. 2016. Infrared and visible image fusion via gradient transfer and total variation minimization. Info. Fusion 31 (2016), 100--109.

Digital Library

[37]

Jiayi Ma, Pengwei Liang, Wei Yu, Chen Chen, Xiaojie Guo, Jia Wu, and Junjun Jiang. 2020. Infrared and visible image fusion via detail preserving adversarial learning. Info. Fusion 54 (2020), 85--98.

[38]

Jiayi Ma, Yong Ma, and Chang Li. 2019. Infrared and visible image fusion methods and applications: A survey. Info. Fusion 45 (2019), 153--178.

[39]

Jiayi Ma, Wei Yu, Pengwei Liang, Chang Li, and Junjun Jiang. 2019. FusionGAN: A generative adversarial network for infrared and visible image fusion. Info. Fusion 48 (2019), 11--26.

[40]

Bangalore S. Manjunath, J.-R. Ohm, Vinod V. Vasudevan, and Akio Yamada. 2001. Color and texture descriptors. IEEE Trans. Circ. Syst. Video Technol. 11, 6 (2001), 703--715.

Digital Library

[41]

Anish Mittal, Anush Krishna Moorthy, and Alan Conrad Bovik. 2012. No-reference image quality assessment in the spatial domain. IEEE Trans. Image Process. 21, 12 (2012), 4695--4708.

Digital Library

[42]

Anish Mittal, Rajiv Soundararajan, and Alan C. Bovik. 2013. Making a âcompletely blindâ image quality analyzer. IEEE Signal Process. Lett. 20, 3 (2013), 209--212.

[43]

Serafim Opricovic and Gwo-Hshiung Tzeng. 2004. Compromise solution by MCDM methods: A comparative analysis of VIKOR and TOPSIS. Eur. J. Operation. Res. 156, 2 (2004), 445--455.

[44]

G. Paschos. 2001. Perceptually uniform color spaces for color texture analysis: An empirical evaluation. IEEE Trans. Image Process. 10, 6 (June 2001), 932--937.

Digital Library

[45]

Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. 2017. Automatic differentiation in PyTorch. In Proceedings of the Advances in Neural Information Processing Workshop (NIPS’17).

[46]

L. Suarez Patricia, D. Sappa Angel, and X. Vintimilla Boris. 2017. Infrared image colorization based on a triplet DCGAN architecture. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW’17). 212--217.

[47]

Isola Phillip, Zhu Jun-Yan, Zhou Tinghui, and A. Efros Alexei. 2017. Image-to-image translation with conditional adversarial networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR’17). 5967--5976.

[48]

Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. 2015. ImageNet large scale visual recognition challenge. Int. J. Comput. Vision 115, 3 (2015), 211--252.

Digital Library

[49]

Michele A. Saad, Alan C. Bovik, and Christophe Charrier. 2012. Blind image quality assessment: A natural scene statistics approach in the DCT domain. IEEE Trans. Image Process. 21, 8 (2012), 3339--3352.

Digital Library

[50]

Faranak Shamsafar, Hadi Seyedarabi, and Ali Aghagolzadeh. 2014. Fusing the information in visible light and near-infrared images for iris recognition. Mach. Vision Appl. 25, 4 (2014), 881--899.

Digital Library

[51]

B. Sheng, H. Sun, M. Magnor, and P. Li. 2014. Video colorization using parallel optimization in feature space. IEEE Trans. Circ. Syst. Video Technol. 24, 3 (Mar. 2014), 407--417.

Digital Library

[52]

C. H. Son and X. P. Zhang. 2017. Near-infrared fusion via color regularization for haze and color distortion removals. IEEE Trans. Circ. Syst. Video Technol. (2017), 1--1.

[53]

Patricia L. Suárez, Angel D. Sappa, and Boris X. Vintimilla. 2017. Learning to colorize infrared images. In Proceedings of the International Conference on Practical Applications of Agents and Multi-Agent Systems. Springer, 164--172.

[54]

John M. Sullivan. 1999. Assessing the potential benefit of adaptive headlighting using crash databases. Technical report. University of Michigan, Ann Arbor, Transportation Research Institute.

[55]

Christian Szegedy, Sergey Ioffe, Vincent Vanhoucke, and Alexander A. Alemi. 2017. Inception-v4, inception-resnet and the impact of residual connections on learning.

Digital Library

[56]

Hossein Talebi and Peyman Milanfar. 2018. Nima: Neural image assessment. IEEE Trans. Image Process. 27, 8 (2018), 3998--4011.

[57]

Anwaar Ulhaq, Xiaoxia Yin, Jing He, and Yanchun Zhang. 2016. FACE: Fully automated context enhancement for night-time video sequences. J. Vis. Commun. Image Represent. 40 (2016), 682--693.

Digital Library

[58]

Ting-Chun Wang, Ming-Yu Liu, Jun-Yan Zhu, Andrew Tao, Jan Kautz, and Bryan Catanzaro. 2017. High-resolution image synthesis and semantic manipulation with conditional GANs. Retrieved from https://Arxiv:1711.11585.

[59]

Tomihisa Welsh, Michael Ashikhmin, and Klaus Mueller. 2002. Transferring color to greyscale images. In ACM Transactions on Graphics, Vol. 21. ACM, 277--280.

Digital Library

[60]

Han Xu, Pengwei Liang, Wei Yu, Junjun Jiang, and Jiayi Ma. 2019. Learning a generative model for fusing infrared and visible images via conditional generative adversarial network with dual discriminators. In Proceedings of the International Joint Conference on Artificial Intelligence. 3954--3960.

[61]

Richard Zhang, Phillip Isola, and Alexei A. Efros. 2016. Colorful image colorization. In Proceedings of the European Conference on Computer Vision. Springer, 649--666.

[62]

Richard Zhang, Jun-Yan Zhu, Phillip Isola, Xinyang Geng, Angela S. Lin, Tianhe Yu, and Alexei A. Efros. 2017. Real-time user-guided image colorization with learned deep priors. Retrieved from https://Arxiv:1705.02999.

[63]

Yufeng Zheng, Erik P. Blasch, and Zheng Liu. 2018. Multispectral Image Fusion and Night Vision Colorization. International Society for Optics and Photonics.

[64]

Yufeng Zheng, Wenjie Dong, and Erik P. Blasch. 2012. Qualitative and quantitative comparisons of multispectral night vision colorization techniques. Optic. Eng. 51, 8 (2012), 087004.

[65]

Bolei Zhou, Agata Lapedriza, Jianxiong Xiao, Antonio Torralba, and Aude Oliva. 2014. Learning deep features for scene recognition using places database. In Advances in Neural Information Processing Systems. MIT Press, 487--495.

Digital Library

[66]

Zhiqiang Zhou, Mingjie Dong, Xiaozhu Xie, and Zhifeng Gao. 2016. Fusion of infrared and visible images for night-vision context enhancement. Appl. Optics 55, 23 (2016), 6480--6490.

Cited By

Blasch ESchrader PChen GWei SChen YKhan SAved AMunir A(2024)Dynamic Digital Twins for Situation AwarenessNAECON 2024 - IEEE National Aerospace and Electronics Conference10.1109/NAECON61878.2024.10670654(433-440)Online publication date: 15-Jul-2024
https://doi.org/10.1109/NAECON61878.2024.10670654
Blasch E(2024)Digital Twins for Cognitive Situation Awareness2024 IEEE Conference on Cognitive and Computational Aspects of Situation Management (CogSIMA)10.1109/CogSIMA61085.2024.10553721(63-70)Online publication date: 7-May-2024
https://doi.org/10.1109/CogSIMA61085.2024.10553721
Liu HPeng SZhu LChang YZhou HYan L(2024)Seeing Motion at Nighttime with an Event Camera2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)10.1109/CVPR52733.2024.02423(25648-25658)Online publication date: 16-Jun-2024
https://doi.org/10.1109/CVPR52733.2024.02423
Show More Cited By

Index Terms

Deep Learning Thermal Image Translation for Night Vision Perception
1. Computing methodologies
  1. Artificial intelligence
    1. Computer vision
      1. Computer vision representations
        Image representations
  2. Machine learning
    1. Machine learning approaches
      1. Neural networks

Recommendations

Objective Assessment of Multiresolution Image Fusion Algorithms for Context Enhancement in Night Vision: A Comparative Study

Comparison of image processing techniques is critically important in deciding which algorithm, method, or metric to use for enhanced image assessment. Image fusion is a popular choice for various image enhancement applications such as overlay of two ...
Color night vision: opponent processing in the fusion of visible and IR imagery
Physical model and image translation fused network for single-image dehazing
Highlights
- An end-to-end physical model and image translate fused network is proposed for single image dehazing. The proposed network contains a common feature ...
Abstract
The visibility and contrast of images captured in adverse weather such as haze or fog degrade dramatically, which further hinders the accomplishment of high-level computer vision tasks such as object detection and semantic segmentation ...

Comments

Information & Contributors

Information

Published In

cover image ACM Transactions on Intelligent Systems and Technology

ACM Transactions on Intelligent Systems and Technology Volume 12, Issue 1

Regular Papers

February 2021

280 pages

ISSN:2157-6904

EISSN:2157-6912

DOI:10.1145/3436534

Editor:
Yu Zheng
JD Digits, China

Issue’s Table of Contents

Copyright © 2020 ACM.

© 2020 Association for Computing Machinery. ACM acknowledges that this contribution was authored or co-authored by an employee, contractor or affiliate of the United States government. As such, the United States Government retains a nonexclusive, royalty-free right to publish or reproduce this article, or to allow others to do so, for Government purposes only.

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 22 December 2020

Accepted: 01 September 2020

Revised: 01 May 2020

Received: 01 September 2019

Published in TIST Volume 12, Issue 1

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

23
Total Citations
View Citations
696
Total Downloads

Downloads (Last 12 months)127
Downloads (Last 6 weeks)12

Reflects downloads up to 12 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Blasch ESchrader PChen GWei SChen YKhan SAved AMunir A(2024)Dynamic Digital Twins for Situation AwarenessNAECON 2024 - IEEE National Aerospace and Electronics Conference10.1109/NAECON61878.2024.10670654(433-440)Online publication date: 15-Jul-2024
https://doi.org/10.1109/NAECON61878.2024.10670654
Blasch E(2024)Digital Twins for Cognitive Situation Awareness2024 IEEE Conference on Cognitive and Computational Aspects of Situation Management (CogSIMA)10.1109/CogSIMA61085.2024.10553721(63-70)Online publication date: 7-May-2024
https://doi.org/10.1109/CogSIMA61085.2024.10553721
Liu HPeng SZhu LChang YZhou HYan L(2024)Seeing Motion at Nighttime with an Event Camera2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)10.1109/CVPR52733.2024.02423(25648-25658)Online publication date: 16-Jun-2024
https://doi.org/10.1109/CVPR52733.2024.02423
Bin JRogers SLiu Z(2024)Vision Fourier transformer empowered multi-modal imaging system for ethane leakage detectionInformation Fusion10.1016/j.inffus.2024.102266106(102266)Online publication date: Jun-2024
https://doi.org/10.1016/j.inffus.2024.102266
Stančić IKuzmanić Skelin AMusić JCecić M(2023)The Development of a Cost-Effective Imaging Device Based on Thermographic TechnologySensors10.3390/s2310458223:10(4582)Online publication date: 9-May-2023
https://doi.org/10.3390/s23104582
Das AMah MHunt JTalghader J(2023)Thermodynamically limited uncooled infrared detector using an ultra-low mass perforated subwavelength absorberOptica10.1364/OPTICA.48976110:8(1018)Online publication date: 27-Jul-2023
https://doi.org/10.1364/OPTICA.489761
Bin JBahrami ZRahman CDu SRogers SLiu Z(2023)Foreground Fusion-Based Liquefied Natural Gas Leak Detection Framework From Surveillance Thermal ImagingIEEE Transactions on Emerging Topics in Computational Intelligence10.1109/TETCI.2022.32148267:4(1151-1162)Online publication date: Aug-2023
https://doi.org/10.1109/TETCI.2022.3214826
Zou YYan CFu Y(2023)RawHDR: High Dynamic Range Image Reconstruction from a Single Raw Image2023 IEEE/CVF International Conference on Computer Vision (ICCV)10.1109/ICCV51070.2023.01133(12300-12310)Online publication date: 1-Oct-2023
https://doi.org/10.1109/ICCV51070.2023.01133
Chen YZhan WJiang YZhu DGuo RXu X(2023)DDGAN: Dense Residual Module and Dual-stream Attention-Guided Generative Adversarial Network for colorizing near-infrared imagesInfrared Physics & Technology10.1016/j.infrared.2023.104822133(104822)Online publication date: Sep-2023
https://doi.org/10.1016/j.infrared.2023.104822
Song WZhai WGao MLi QChehri AJeon G(2023)Multiscale aggregation and illumination‐aware attention network for infrared and visible image fusionConcurrency and Computation: Practice and Experience10.1002/cpe.771236:10Online publication date: 25-Apr-2023
https://doi.org/10.1002/cpe.7712
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

HTML Format

View this article in HTML Format.

Media

Figures

Other

Tables

View Issue’s Table of Contents