Abstract
Widely used optical remote sensing images are often contaminated by clouds. The missing or cloud-contaminated data leads to incorrect predictions by the downstream machine learning tasks. However, the availability of multi-sensor remote sensing imagery has great potential for improving imputation under clouds. Existing cloud imputation methods could generally preserve the spatial structure in the imputed regions, however, the spectral distribution does not match the target image due to differences in sensor characteristics and temporal differences. In this paper, we present a novel deep learning-based multi-sensor imputation technique inspired by the computer vision-based style transfer. The proposed deep learning framework consists of two modules: (i) cluster-based attentional instance normalization (CAIN), and (ii) adaptive instance normalization (AdaIN). The combined module, CAINA, exploits the style information from cloud-free regions. These regions (land cover) were obtained through clustering to reduce the style differences between the target and predicted image patches. We have conducted extensive experiments and made comparisons against the state-of-the-art methods using a benchmark dataset with images from Landsat-8 and Sentinel-2 satellites. Our experiments show that the proposed CAINA is at least 24.49% better on MSE and 18.38% better on cloud MSE as compared to state-of-the-art methods.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Cresson, R., Ienco, D., Gaetano, R., Ose, K., Minh, D.H.T.: Optical image gap filling using deep convolutional autoencoder from optical and radar images. In: IGARSS 2019–2019 IEEE International Geoscience and Remote Sensing Symposium, pp. 218–221. IEEE (2019)
Cresson, R., et al.: Comparison of convolutional neural networks for cloudy optical images reconstruction from single or multitemporal joint SAR and optical images. arXiv preprint arXiv:2204.00424 (2022)
Dumoulin, V., Shlens, J., Kudlur, M.: A learned representation for artistic style. arXiv preprint arXiv:1610.07629 (2016)
Ebel, P., Meraner, A., Schmitt, M., Zhu, X.X.: Multisensor data fusion for cloud removal in global and all-season sentinel-2 imagery. IEEE Trans. Geosci. Remote Sens. 59(7), 5866–5878 (2020)
Ebel, P., Xu, Y., Schmitt, M., Zhu, X.X.: SEN12MS-CR-TS: a remote-sensing data set for multimodal multitemporal cloud removal. IEEE Trans. Geosci. Remote Sens. 60, 1–14 (2022)
Gatys, L.A., Ecker, A.S., Bethge, M.: Image style transfer using convolutional neural networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2414–2423 (2016)
Hore, A., Ziou, D.: Image quality metrics: PSNR vs. SSIM. In: 2010 20th International Conference on Pattern Recognition, pp. 2366–2369. IEEE (2010)
Huang, X., Belongie, S.: Arbitrary style transfer in real-time with adaptive instance normalization. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 1501–1510 (2017)
Kang, S.H., Choi, Y., Choi, J.Y.: Restoration of missing patterns on satellite infrared sea surface temperature images due to cloud coverage using deep generative inpainting network. J. Mar. Sci. Eng. 9(3), 310 (2021)
King, M.D., Platnick, S., Menzel, W.P., Ackerman, S.A., Hubanks, P.A.: Spatial and temporal distribution of clouds observed by MODIS onboard the terra and aqua satellites. IEEE Trans. Geosci. Remote Sens. 51(7), 3826–3852 (2013)
Kruse, F.A., et al.: The spectral image processing system (SIPS)–interactive visualization and analysis of imaging spectrometer data. Remote Sens. Environ. 44(2–3), 145–163 (1993)
Li, X., Guo, Q., Lin, D., Li, P., Feng, W., Wang, S.: MISF: multi-level interactive siamese filtering for high-fidelity image inpainting. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1869–1878 (2022)
Ma, W., et al.: A novel adaptive hybrid fusion network for multiresolution remote sensing images classification. IEEE Trans. Geosci. Remote Sens. 60, 1–17 (2021)
Meraner, A., Ebel, P., Zhu, X.X., Schmitt, M.: Cloud removal in sentinel-2 imagery using a deep residual neural network and SAR-optical data fusion. ISPRS J. Photogramm. Remote. Sens. 166, 333–346 (2020)
Park, T., Efros, A.A., Zhang, R., Zhu, J.-Y.: Contrastive learning for unpaired image-to-image translation. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, J.-M. (eds.) ECCV 2020. LNCS, vol. 12354, pp. 319–345. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-58545-7_19
Qu, J., Shi, Y., Xie, W., Li, Y., Wu, X., Du, Q.: MSSL: hyperspectral and panchromatic images fusion via multiresolution spatial-spectral feature learning networks. IEEE Trans. Geosci. Remote Sens. 60, 1–13 (2021)
Requena-Mesa, C., Benson, V., Reichstein, M., Runge, J., Denzler, J.: Earthnet 2021: a large-scale dataset and challenge for earth surface forecasting as a guided video prediction task. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1132–1142 (2021)
Ronneberger, O., Fischer, P., Brox, T.: U-Net: convolutional networks for biomedical image segmentation. In: Navab, N., Hornegger, J., Wells, W.M., Frangi, A.F. (eds.) MICCAI 2015. LNCS, vol. 9351, pp. 234–241. Springer, Cham (2015). https://doi.org/10.1007/978-3-319-24574-4_28
Roy, D.P., et al.: Multi-temporal MODIS-Landsat data fusion for relative radiometric normalization, gap filling, and prediction of landsat data. Remote Sens. Environ. 112(6), 3112–3130 (2008)
Rudner, T.G., et al.: Multi3Net: segmenting flooded buildings via fusion of multiresolution, multisensor, and multitemporal satellite imagery. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 33, pp. 702–709 (2019)
Singh, P., Komodakis, N.: Cloud-Gan: cloud removal for sentinel-2 imagery using a cyclic consistent generative adversarial networks. In: IGARSS 2018–2018 IEEE International Geoscience and Remote Sensing Symposium, pp. 1772–1775. IEEE (2018)
Stock, A., et al.: Comparison of cloud-filling algorithms for marine satellite data. Remote Sens. 12(20), 3313 (2020)
Sun, Z., Zhou, W., Ding, C., Xia, M.: Multi-resolution transformer network for building and road segmentation of remote sensing image. ISPRS Int. J. Geo Inf. 11(3), 165 (2022)
Ulyanov, D., Vedaldi, A., Lempitsky, V.: Improved texture networks: maximizing quality and diversity in feed-forward stylization and texture synthesis. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 6924–6932 (2017)
Varshney, D., Persello, C., Gupta, P.K., Nikam, B.R.: Multiresolution fully convolutional networks to detect clouds and snow through optical satellite images. arXiv preprint arXiv:2201.02350 (2022)
Wang, L., Weng, L., Xia, M., Liu, J., Lin, H.: Multi-resolution supervision network with an adaptive weighted loss for desert segmentation. Remote Sens. 13(11), 2054 (2021)
Wang, L., Zhang, C., Li, R., Duan, C., Meng, X., Atkinson, P.M.: Scale-aware neural network for semantic segmentation of multi-resolution remote sensing images. Remote Sens. 13(24), 5015 (2021)
Wang, Z., Bovik, A.C., Sheikh, H.R., Simoncelli, E.P.: Image quality assessment: from error visibility to structural similarity. IEEE Trans. Image Process. 13(4), 600–612 (2004)
Weiss, D.J., Atkinson, P.M., Bhatt, S., Mappin, B., Hay, S.I., Gething, P.W.: An effective approach for gap-filling continental scale remotely sensed time-series. ISPRS J. Photogramm. Remote. Sens. 98, 106–118 (2014)
Yadav, J., Sharma, M.: A review of k-mean algorithm. Int. J. Eng. Trends Technol. 4(7), 2972–2976 (2013)
Yang, X., Zhao, Y., Vatsavai, R.R.: Deep residual network with multi-image attention for imputing under clouds in satellite imagery. In: 2022 27th International Conference on Pattern Recognition (ICPR). IEEE (2022)
Yu, W., Zhang, X., Pun, M.O., Liu, M.: A hybrid model-based and data-driven approach for cloud removal in satellite imagery using multi-scale distortion-aware networks. In: 2021 IEEE International Geoscience and Remote Sensing Symposium IGARSS, pp. 7160–7163. IEEE (2021)
Zhang, Q., Yuan, Q., Zeng, C., Li, X., Wei, Y.: Missing data reconstruction in remote sensing image with a unified spatial-temporal-spectral deep convolutional neural network. IEEE Trans. Geosci. Remote Sens. 56(8), 4274–4288 (2018)
Zhao, Y., Yang, X., Vatsavai, R.R.: Multi-stream deep residual network for cloud imputation using multi-resolution remote sensing imagery. In: 2022 21st IEEE International Conference on Machine Learning and Applications (ICMLA), pp. 97–104. IEEE (2022)
Zhao, Y., Shen, S., Hu, J., Li, Y., Pan, J.: Cloud removal using multimodal GAN with adversarial consistency loss. IEEE Geosci. Remote Sens. Lett. 19, 1–5 (2021)
Zhu, H., Ma, W., Li, L., Jiao, L., Yang, S., Hou, B.: A dual-branch attention fusion deep network for multiresolution remote-sensing image classification. Inf. Fusion 58, 116–131 (2020)
Zhu, J.Y., Park, T., Isola, P., Efros, A.A.: Unpaired image-to-image translation using cycle-consistent adversarial networks. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 2223–2232 (2017)
Acknowledgments
This research is based upon work supported in part by the Office of the Director of National Intelligence (ODNI), Intelligence Advanced Research Projects Activity (IARPA), via Contract #2021-21040700001. The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies, either expressed or implied, of ODNI, IARPA, or the U.S. Government. The U.S. Government is authorized to reproduce and distribute reprints for governmental purposes notwithstanding any copyright annotation therein. We would like to thank Benjamin Raskob at ARA for useful feedback on this project.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Ethical Statement
Ethical Statement
Our proposed method improves cloud imputation performance. Remote sensing imagery has been widely used in applications ranging from land-use land-cover mapping to national security. By improving the imputation performance, we are directly improving the downstream applications such as assessing damages due to natural disasters, forest fires, and climate impacts. Our work does not have direct ethical implications or adverse impacts on humans.
Rights and permissions
Copyright information
© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Zhao, Y., Yang, X., Vatsavai, R.R. (2023). Cloud Imputation for Multi-sensor Remote Sensing Imagery with Style Transfer. In: De Francisci Morales, G., Perlich, C., Ruchansky, N., Kourtellis, N., Baralis, E., Bonchi, F. (eds) Machine Learning and Knowledge Discovery in Databases: Applied Data Science and Demo Track. ECML PKDD 2023. Lecture Notes in Computer Science(), vol 14175. Springer, Cham. https://doi.org/10.1007/978-3-031-43430-3_3
Download citation
DOI: https://doi.org/10.1007/978-3-031-43430-3_3
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-43429-7
Online ISBN: 978-3-031-43430-3
eBook Packages: Computer ScienceComputer Science (R0)
