Location via proxy:   [ UP ]  
[Report a bug]   [Manage cookies]                
skip to main content
10.1145/3474085.3475600acmconferencesArticle/Chapter ViewAbstractPublication PagesmmConference Proceedingsconference-collections
research-article

SSconv: Explicit Spectral-to-Spatial Convolution for Pansharpening

Published: 17 October 2021 Publication History

Abstract

Pansharpening aims to fuse a high spatial resolution panchromatic (PAN) image and a low resolution multispectral (LR-MS) image to obtain a multispectral image with the same spatial resolution as the PAN image. Thanks to the flexible structure of convolution neural networks (CNNs), they have been successfully applied to the problem of pansharpening. However, most of the existing methods only simply feed the up-sampled LR-MS into the CNNs and ignore the spatial distortion caused by direct up-sampling. In this paper, we propose an explicit spectral-to-spatial convolution (SSconv) that aggregates spectral features into the spatial domain to perform the up-sampling operation, which can get better performance than the direct up-sampling. Furthermore, SSconv is embedded into a multiscale U-shaped convolution neural network (MUCNN) for fully utilizing the multispectral information of involved images. In particular, multiscale injection branch and mixed loss on cross-scale levels are employed to fuse pixel-wise image information. Benefiting from the distortion-free property of SSconv, the proposed MUCNN can generate state-of-the-art performance with a simple structure, both on reduced-resolution and full-resolution datasets acquired from WorldView-3 and GaoFen-2. Please find the code from the project page.

Supplementary Material

MP4 File (MM21-fp2371.mp4)
The presentation video, briefly talks about the background, related works, the method we proposed, and the results.

References

[1]
Bruno Aiazzi, Luciano Alparone, Stefano Baronti, and Andrea Garzelli. 2002. Context-driven fusion of high spatial and spectral resolution images based on oversampled multiresolution analysis. IEEE Transactions on geoscience and remote sensing, Vol. 40, 10 (2002), 2300--2312.
[2]
B Aiazzi, L Alparone, S Baronti, A Garzelli, and M Selva. 2006. MTF-tailored multiscale fusion of high-resolution MS and Pan imagery. Photogrammetric Engineering & Remote Sensing, Vol. 72, 5 (2006), 591--596.
[3]
Luciano Alparone, Lucien Wald, Jocelyn Chanussot, Claire Thomas, Paolo Gamba, and Lori Mann Bruce. 2007 a. Comparison of pansharpening algorithms: Outcome of the 2006 GRS-S data-fusion contest. IEEE Transactions on Geoscience and Remote Sensing, Vol. 45, 10 (2007), 3012--3021.
[4]
Luciano Alparone, Lucien Wald, Jocelyn Chanussot, Claire Thomas, Paolo Gamba, and Lori Mann Bruce. 2007 b. Comparison of pansharpening algorithms: Outcome of the 2006 GRS-S data-fusion contest. IEEE Transactions on Geoscience and Remote Sensing, Vol. 45, 10 (2007), 3012--3021.
[5]
Jaewan Choi, Kiyun Yu, and Yongil Kim. 2010. A new adaptive component-substitution-based satellite image fusion by using partial replacement. IEEE Transactions on Geoscience and Remote Sensing, Vol. 49, 1 (2010), 295--309.
[6]
özgün Ćiçek, Ahmed Abdulkadir, Soeren S Lienkamp, Thomas Brox, and Olaf Ronneberger. 2016. 3D U-Net: learning dense volumetric segmentation from sparse annotation. In International conference on medical image computing and computer-assisted intervention. Springer, 424--432.
[7]
Mauro Dalla Mura, Saurabh Prasad, Fabio Pacifici, Paulo Gamba, Jocelyn Chanussot, and Jón Atli Benediktsson. 2015. Challenges and opportunities of multimodality and data fusion in remote sensing. Proc. IEEE, Vol. 103, 9 (2015), 1585--1601.
[8]
Liang-Jian Deng, Gemine Vivone, Cheng Jin, and Jocelyn Chanussot. 2020. Detail Injection-Based Deep Convolutional Neural Networks for Pansharpening. IEEE Transactions on Geoscience and Remote Sensing (2020).
[9]
Xueyang Fu, Wu Wang, Yue Huang, Xinghao Ding, and John Paisley. 2020. Deep multiscale detail networks for multiband spectral image sharpening. IEEE Transactions on Neural Networks and Learning Systems (2020).
[10]
Andrea Garzelli and Filippo Nencini. 2009. Hypercomplex quality assessment of multi/hyperspectral images. IEEE Geoscience and Remote Sensing Letters, Vol. 6, 4 (2009), 662--665.
[11]
Andrea Garzelli, Filippo Nencini, and Luca Capobianco. 2007. Optimal MMSE pan sharpening of very high resolution multispectral images. IEEE Transactions on Geoscience and Remote Sensing, Vol. 46, 1 (2007), 228--236.
[12]
Kaiming He and Jian Sun. 2015. Convolutional neural networks at constrained time cost. In Proceedings of the IEEE conference on computer vision and pattern recognition. 5353--5360.
[13]
Lin He, Yizhou Rao, Jun Li, Jocelyn Chanussot, Antonio Plaza, Jiawei Zhu, and Bo Li. 2019. Pansharpening via detail injection based convolutional neural networks. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, Vol. 12, 4 (2019), 1188--1204.
[14]
Zheng Hui, Xinbo Gao, Yunchu Yang, and Xiumei Wang. 2019. Lightweight image super-resolution with information multi-distillation network. In Proceedings of the 27th ACM International Conference on Multimedia. 2024--2032.
[15]
Menghui Jiang, Huanfeng Shen, Jie Li, Qiangqiang Yuan, and Liangpei Zhang. 2020. A differential information residual convolutional neural network for pansharpening. ISPRS Journal of Photogrammetry and Remote Sensing, Vol. 163 (2020), 257--271.
[16]
Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014).
[17]
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. 2012. Imagenet classification with deep convolutional neural networks. Advances in neural information processing systems, Vol. 25 (2012), 1097--1105.
[18]
Craig A Laben and Bernard V Brower. 2000. Process for enhancing the spatial resolution of multispectral imagery using pan-sharpening. US Patent 6,011,875.
[19]
JG Liu. 2000. Smoothing filter-based intensity modulation: A spectral preserve image fusion technique for improving spatial details. International Journal of Remote Sensing, Vol. 21, 18 (2000), 3461--3472.
[20]
Jonathan Long, Evan Shelhamer, and Trevor Darrell. 2015. Fully convolutional networks for semantic segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition. 3431--3440.
[21]
Giuseppe Masi, Davide Cozzolino, Luisa Verdoliva, and Giuseppe Scarpa. 2016. Pansharpening by convolutional neural networks. Remote Sensing, Vol. 8, 7 (2016), 594.
[22]
Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida. 2018. Spectral normalization for generative adversarial networks. arXiv preprint arXiv:1802.05957 (2018).
[23]
Ozan Oktay, Jo Schlemper, Loic Le Folgoc, Matthew Lee, Mattias Heinrich, Kazunari Misawa, Kensaku Mori, Steven McDonagh, Nils Y Hammerla, Bernhard Kainz, et almbox. 2018. Attention u-net: Learning where to look for the pancreas. arXiv preprint arXiv:1804.03999 (2018).
[24]
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. 2015. U-net: Convolutional networks for biomedical image segmentation. In International Conference on Medical image computing and computer-assisted intervention. Springer, 234--241.
[25]
Wenzhe Shi, Jose Caballero, Ferenc Huszár, Johannes Totz, Andrew P Aitken, Rob Bishop, Daniel Rueckert, and Zehan Wang. 2016. Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network. In Proceedings of the IEEE conference on computer vision and pattern recognition. 1874--1883.
[26]
Karasawa Takumi, Kohei Watanabe, Qishen Ha, Antonio Tejero-De-Pablos, Yoshitaka Ushiku, and Tatsuya Harada. 2017. Multispectral object detection for autonomous vehicles. In Proceedings of the on Thematic Workshops of ACM Multimedia 2017. 35--43.
[27]
Claire Thomas, Thierry Ranchin, Lucien Wald, and Jocelyn Chanussot. 2008. Synthesis of multispectral images to high spatial resolution: A critical review of fusion methods based on remote sensing physics. IEEE Transactions on Geoscience and Remote Sensing, Vol. 46, 5 (2008), 1301--1312.
[28]
Gemine Vivone. 2019. Robust band-dependent spatial-detail approaches for panchromatic sharpening. IEEE transactions on Geoscience and Remote Sensing, Vol. 57, 9 (2019), 6421--6433.
[29]
Gemine Vivone, Luciano Alparone, Jocelyn Chanussot, Mauro Dalla Mura, Andrea Garzelli, Giorgio A Licciardi, Rocco Restaino, and Lucien Wald. 2014a. A critical comparison among pansharpening algorithms. IEEE Transactions on Geoscience and Remote Sensing, Vol. 53, 5 (2014), 2565--2586.
[30]
Gemine Vivone, Luciano Alparone, Jocelyn Chanussot, Mauro Dalla Mura, Andrea Garzelli, Giorgio A Licciardi, Rocco Restaino, and Lucien Wald. 2014b. A critical comparison among pansharpening algorithms. IEEE Transactions on Geoscience and Remote Sensing, Vol. 53, 5 (2014), 2565--2586.
[31]
Gemine Vivone, Mauro Dalla Mura, Andrea Garzelli, Rocco Restaino, Giuseppe Scarpa, Magnus Orn Ulfarsson, Luciano Alparone, and Jocelyn Chanussot. 2020. A New Benchmark Based on Recent Advances in Multispectral Pansharpening: Revisiting pansharpening with classical and emerging pansharpening methods. IEEE Geoscience and Remote Sensing Magazine (2020).
[32]
Gemine Vivone, Rocco Restaino, Mauro Dalla Mura, Giorgio Licciardi, and Jocelyn Chanussot. 2013. Contrast and error-based fusion schemes for multispectral image pansharpening. IEEE Geoscience and Remote Sensing Letters, Vol. 11, 5 (2013), 930--934.
[33]
Lucien Wald. 2002. Data fusion: definitions and architectures: fusion of images of different spatial resolutions. Presses des MINES.
[34]
Zhou Wang and Alan C Bovik. 2002. A universal image quality index. IEEE signal processing letters, Vol. 9, 3 (2002), 81--84.
[35]
Shuang Wu, Guanrui Wang, Pei Tang, Feng Chen, and Luping Shi. 2019. Convolution with even-sized kernels and symmetric padding. arXiv preprint arXiv:1903.08385 (2019).
[36]
Junfeng Yang, Xueyang Fu, Yuwen Hu, Yue Huang, Xinghao Ding, and John Paisley. 2017. PanNet: A deep network architecture for pan-sharpening. In Proceedings of the IEEE international conference on computer vision. 5449--5457.
[37]
Roberta H Yuhas, Alexander FH Goetz, and Joe W Boardman. 1992. Discrimination among semi-arid landscape endmembers using the spectral angle mapper (SAM) algorithm. In Proc. Summaries 3rd Annu. JPL Airborne Geosci. Workshop, Vol. 1. 147--149.
[38]
Matthew D Zeiler and Rob Fergus. 2014. Visualizing and understanding convolutional networks. In European conference on computer vision. Springer, 818--833.
[39]
Huanrong Zhang, Zhi Jin, Xiaojun Tan, and Xiying Li. 2020. Towards Lighter and Faster: Learning Wavelets Progressively for Image Super-Resolution. In Proceedings of the 28th ACM International Conference on Multimedia. 2113--2121.
[40]
J Zhou, DL Civco, and JA Silander. 1998. A wavelet transform method to merge Landsat TM and SPOT panchromatic data. International journal of remote sensing, Vol. 19, 4 (1998), 743--757.
[41]
Qiang Zhou, Shifeng Chen, Jianzhuang Liu, and Xiaoou Tang. 2011. Edge-preserving single image super-resolution. In Proceedings of the 19th ACM International Conference on Multimedia. 1037--1040.
[42]
Zongwei Zhou, Md Mahfuzur Rahman Siddiquee, Nima Tajbakhsh, and Jianming Liang. 2018. Unet: A nested u-net architecture for medical image segmentation. In Deep learning in medical image analysis and multimodal learning for clinical decision support. Springer, 3--11.

Cited By

View all
  • (2025)Fully-Connected Transformer for Multi-Source Image FusionIEEE Transactions on Pattern Analysis and Machine Intelligence10.1109/TPAMI.2024.352336447:3(2071-2088)Online publication date: Mar-2025
  • (2024)Conditional Skipping Mamba Network for Pan-SharpeningSymmetry10.3390/sym1612168116:12(1681)Online publication date: 19-Dec-2024
  • (2024)DPDU-Net: Double Prior Deep Unrolling Network for PansharpeningRemote Sensing10.3390/rs1612214116:12(2141)Online publication date: 13-Jun-2024
  • Show More Cited By

Index Terms

  1. SSconv: Explicit Spectral-to-Spatial Convolution for Pansharpening

    Recommendations

    Comments

    Information & Contributors

    Information

    Published In

    cover image ACM Conferences
    MM '21: Proceedings of the 29th ACM International Conference on Multimedia
    October 2021
    5796 pages
    ISBN:9781450386517
    DOI:10.1145/3474085
    Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

    Sponsors

    Publisher

    Association for Computing Machinery

    New York, NY, United States

    Publication History

    Published: 17 October 2021

    Permissions

    Request permissions for this article.

    Check for updates

    Author Tags

    1. convolution neural networks
    2. multiscale
    3. pansharpening
    4. spectral-to-spatial

    Qualifiers

    • Research-article

    Funding Sources

    Conference

    MM '21
    Sponsor:
    MM '21: ACM Multimedia Conference
    October 20 - 24, 2021
    Virtual Event, China

    Acceptance Rates

    Overall Acceptance Rate 2,145 of 8,556 submissions, 25%

    Contributors

    Other Metrics

    Bibliometrics & Citations

    Bibliometrics

    Article Metrics

    • Downloads (Last 12 months)60
    • Downloads (Last 6 weeks)4
    Reflects downloads up to 17 Feb 2025

    Other Metrics

    Citations

    Cited By

    View all
    • (2025)Fully-Connected Transformer for Multi-Source Image FusionIEEE Transactions on Pattern Analysis and Machine Intelligence10.1109/TPAMI.2024.352336447:3(2071-2088)Online publication date: Mar-2025
    • (2024)Conditional Skipping Mamba Network for Pan-SharpeningSymmetry10.3390/sym1612168116:12(1681)Online publication date: 19-Dec-2024
    • (2024)DPDU-Net: Double Prior Deep Unrolling Network for PansharpeningRemote Sensing10.3390/rs1612214116:12(2141)Online publication date: 13-Jun-2024
    • (2024)Multi-Frequency Spectral–Spatial Interactive Enhancement Fusion Network for Pan-SharpeningElectronics10.3390/electronics1314280213:14(2802)Online publication date: 16-Jul-2024
    • (2024)Deep learning-based spectral image super-resolution: a surveyJournal of Image and Graphics10.11834/jig.23074729:8(2113-2136)Online publication date: 2024
    • (2024)HINER: Neural Representation for Hyperspectral ImageProceedings of the 32nd ACM International Conference on Multimedia10.1145/3664647.3681643(9837-9846)Online publication date: 28-Oct-2024
    • (2024)DMF-Net: A Dual Remote Sensing Image Fusion Network Based on Multiscale Convolutional Dense Connectivity With Performance MeasureIEEE Transactions on Instrumentation and Measurement10.1109/TIM.2024.337081673(1-15)Online publication date: 2024
    • (2024)FusionMamba: Efficient Remote Sensing Image Fusion With State Space ModelIEEE Transactions on Geoscience and Remote Sensing10.1109/TGRS.2024.349607362(1-16)Online publication date: 2024
    • (2024) INF 3 : Implicit Neural Feature Fusion Function for Multispectral and Hyperspectral Image Fusion IEEE Transactions on Computational Imaging10.1109/TCI.2024.348856910(1547-1558)Online publication date: 2024
    • (2024)Multiscale Bilateral Attention Fusion Network for PansharpeningIEEE Transactions on Artificial Intelligence10.1109/TAI.2024.34183785:11(5828-5843)Online publication date: Nov-2024
    • Show More Cited By

    View Options

    Login options

    View options

    PDF

    View or Download as a PDF file.

    PDF

    eReader

    View online with eReader.

    eReader

    Figures

    Tables

    Media

    Share

    Share

    Share this Publication link

    Share on social media