Abstract
Omnidirectional images (ODIs) are commonly used in real-world visual tasks, and high-resolution ODIs help improve the performance of related visual tasks. Most existing super-resolution methods for ODIs use end-to-end learning strategies, resulting in inferior realness of generated images and a lack of effective out-of-domain generalization capabilities in training methods. Image generation methods represented by diffusion model provide strong priors for visual tasks and have been proven to be effectively applied to image restoration tasks. Leveraging the image priors of the Stable Diffusion (SD) model, we achieve omnidirectional image Super Resolution with both fidelity and realness, dubbed as OmniSSR. Firstly, we transform the equirectangular projection (ERP) images into tangent projection (TP) images, whose distribution approximates the planar image domain. Then, we use SD to iteratively sample initial high-resolution results. At each denoising iteration, we further correct and update the initial results using the proposed Octadecaplex Tangent Information Interaction (OTII) and Gradient Decomposition (GD) technique to ensure better consistency. Finally, the TP images are transformed back to obtain the final high-resolution results. Our method is zero-shot, requiring no training or fine-tuning. Experiments of our method on two benchmark datasets demonstrate the effectiveness of our proposed method.
R. Li and X. Sheng—Equal contributor.
This work was supported in part by National Science Foundation of China (No. 62372016) and Shenzhen Research Project Grant (No. JSGGZD20220822095800001).
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
Notes
- 1.
This claim will be further illustrated in subsequent experiments.
- 2.
- 3.
Code is at https://github.com/LiRunyi2001/OmniSSR.
References
An, H., Zhang, X.: Perception-oriented omnidirectional image super-resolution based on transformer network. In: Proceedings of the IEEE International Conference on Image Processing (ICIP) (2023)
Arican, Z., Frossard, P.: Joint registration and super-resolution with omnidirectional images. IEEE Trans. Image Process. (TIP) 20, 3151–3162 (2011)
Chan, K.C., Xu, X., Wang, X., Gu, J., Loy, C.C.: GLEAN: generative latent bank for image super-resolution and beyond. IEEE Trans. Pattern Anal. Mach. Intell. (TPAMI) 45, 3154–3468 (2022)
Chan, S.H., Wang, X., Elgendy, O.A.: Plug-and-play ADMM for image restoration: Fixed-point convergence and applications. IEEE Trans. Comput. Imaging 3, 84–98 (2016)
Chen, Y., Liu, S., Wang, X.: Learning continuous image representation with local implicit image function. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2021)
Chen, Z., Zhang, Y., Gu, J., Kong, L., Yang, X., Yu, F.: Dual aggregation transformer for image super-resolution. In: Proceedings of the IEEE/CVF international conference on computer vision (ICCV) (2023)
Cheng, M., et al.: Hybrid transformer and CNN attention network for stereo image super-resolution. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2023)
Chong, M., Yanze, W., Xintao, W., Chao, D., Jian, Z., Ying, S.: Metric learning based interactive modulation for real-world super-resolution. In: Proceedings of the European Conference on Computer Vision (ECCV) (2022)
Chung, H., Kim, J., Mccann, M.T., Klasky, M.L., Ye, J.C.: Diffusion posterior sampling for general noisy inverse problems. arXiv preprint arXiv:2209.14687 (2022)
Chung, H., Sim, B., Ye, J.C.: Improving diffusion models for inverse problems using manifold constraints. In: Proceedings of the Advances in Neural Information Processing Systems (NeurIPS) (2022)
Coxeter, H.S.M.: Introduction to Geometry. Wiley (1961)
Daras, G., Dean, J., Jalal, A., Dimakis, A.: Intermediate layer optimization for inverse problems using deep generative models. In: Proceedings of the International Conference on Machine Learning (ICML) (2021)
Deng, X., Wang, H., Xu, M., Guo, Y., Song, Y., Yang, L.: LAU-Net: latitude adaptive upscaling network for omnidirectional image super-resolution. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2021)
Deng, X., Wang, H., Xu, M., Li, L., Wang, Z.: Omnidirectional image super-resolution via latitude adaptive network. IEEE Trans. Multimedia (TMM) 25, 4108–4120 (2022)
Ding, K., Ma, K., Wang, S., Simoncelli, E.P.: Image quality assessment: unifying structure and texture similarity. IEEE Trans. Pattern Anal. Mach. Intell. (TPAMI) 44, 2567–2581 (2020)
Dong, C., Loy, C.C., He, K., Tang, X.: Image super-resolution using deep convolutional networks. IEEE Trans. Pattern Anal. Mach. Intell. (TPAMI) 38, 295–307 (2015)
Duan, H., Zhai, G., Min, X., Zhu, Y., Fang, Y., Yang, X.: Perceptual quality assessment of omnidirectional images. In: Proceedings of the IEEE International Symposium on Circuits and Systems (ISCAS) (2018)
Fakour-Sevom, V., Guldogan, E., Kämäräinen, J.K.: 360 panorama super-resolution using deep convolutional networks. In: Proceedings of the International Conference on Computer Vision Theory and Applications (VISAPP) (2018)
Fei, B., et al.: Generative diffusion prior for unified image restoration and enhancement. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2023)
Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., Bengio, Y.: Generative adversarial nets. In: Proceedings of the Advances in Neural Information Processing Systems (NeurIPS) (2014)
Guo, L., et al.: Cas-DiffCom: cascaded diffusion model for infant longitudinal super-resolution 3d medical image completion. arXiv preprint arXiv:2402.13776 (2024)
Heusel, M., Ramsauer, H., Unterthiner, T., Nessler, B., Hochreiter, S.: GANs trained by a two time-scale update rule converge to a local NASH equilibrium. In: Proceedings of the Advances in Neural Information Processing Systems (NeurIPS) (2017)
Ho, J., Jain, A., Abbeel, P.: Denoising diffusion probabilistic models. In: Proceedings of the Advances in Neural Information Processing Systems (NeurIPS) (2020)
Jiang, Y., Chan, K.C., Wang, X., Loy, C.C., Liu, Z.: Reference-based image and video super-resolution via \(c^2\)-matching. IEEE Trans. Pattern Anal. Mach. Intell. (TPAMI) 45, 8874–8887 (2022)
Kawar, B., Elad, M., Ermon, S., Song, J.: Denoising diffusion restoration models. In: Proceedings of the ICLR Workshop on Deep Generative Models for Highly Structured Data (ICLRW) (2022)
Kerbl, B., Kopanas, G., Leimkühler, T., Drettakis, G.: 3D Gaussian splatting for real-time radiance field rendering. ACM Trans. Graphics (TOG) 42, 1–14 (2023)
Li, W., Chen, B., Zhang, J.: D3C2-Net: dual-domain deep convolutional coding network for compressive sensing. IEEE Trans. Circ. Syst. Video Technol. (TCSVT), 1 (2024)
Li, Y., Guo, Y., Yan, Z., Huang, X., Duan, Y., Ren, L.: OmniFusion: 360 monocular depth estimation via geometry-aware fusion. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2022)
Liang, J., Cao, J., Sun, G., Zhang, K., Van Gool, L., Timofte, R.: SwinIR: image restoration using Swin transformer. In: Proceedings of the IEEE/CVF International Conference on Computer Vision Workshops (ICCVW) (2021)
Liu, J., Wang, Q., Fan, H., Wang, Y., Tang, Y., Qu, L.: Residual denoising diffusion models. arXiv preprint arXiv:2308.13712 (2023)
Lu, Z., Li, J., Liu, H., Huang, C., Zhang, L., Zeng, T.: Transformer for single image super-resolution. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) (2022)
Lugmayr, A., Danelljan, M., Timofte, R.: NTIRE 2020 challenge on real-world image super-resolution: Methods and results. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) (2020)
Menon, S., Damian, A., Hu, S., Ravi, N., Rudin, C.: PULSE: self-supervised photo upsampling via latent space exploration of generative models. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2020)
Mittal, A., Soundararajan, R., Bovik, A.C.: Making a “completely blind’’ image quality analyzer. IEEE Signal Process. Lett. (SPL) 20, 209–212 (2013)
Nishiyama, A., Ikehata, S., Aizawa, K.: 360\(^\circ \) single image super resolution via distortion-aware network and distorted perspective images. In: Proceedings of the IEEE International Conference on Image Processing (ICIP) (2021)
Ozcinar, C., Rana, A., Smolic, A.: Super-resolution of omnidirectional images using adversarial learning. In: Proceedings of the IEEE International Workshop on Multimedia Signal Processing (MMSPW) (2019)
Pan, X., Zhan, X., Dai, B., Lin, D., Loy, C.C., Luo, P.: Exploiting deep generative prior for versatile image restoration and manipulation. IEEE Trans. Pattern Anal. Mach. Intell. (TPAMI) 44, 7474–7489 (2021)
Rombach, R., Blattmann, A., Lorenz, D., Esser, P., Ommer, B.: High-resolution image synthesis with latent diffusion models. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2022)
Rout, L., Raoof, N., Daras, G., Caramanis, C., Dimakis, A., Shakkottai, S.: Solving linear inverse problems provably via posterior sampling with latent diffusion models. In: Proceedings of the Advances in Neural Information Processing Systems (NeurIPS) (2023)
Saharia, C., Ho, J., Chan, W., Salimans, T., Fleet, D.J., Norouzi, M.: Image super-resolution via iterative refinement. IEEE Trans. Pattern Anal. Mach. Intell. (TPAMI) 45, 4713–4726 (2022)
Schönbein, M., Geiger, A.: Omnidirectional 3D reconstruction in augmented Manhattan worlds. In: Proceedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) (2014)
Song, J., et al.: Loss-guided diffusion models for plug-and-play controllable generation. In: Proceedings of the International Conference on Machine Learning (ICML) (2023)
Song, Y., Sohl-Dickstein, J., Kingma, D.P., Kumar, A., Ermon, S., Poole, B.: Score-based generative modeling through stochastic differential equations. In: Proceedings of the International Conference on Learning Representations (ICLR) (2020)
Sun, X., et al.: OPDN: omnidirectional position-aware deformable network for omnidirectional image super-resolution. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) (2023)
Sun, Y., Lu, A., Yu, L.: Weighted-to-spherically-uniform quality evaluation for omnidirectional video. IEEE Signal Process. Lett. (SPL) 24, 1408–1412 (2017)
Vaswani, A., et al.: Attention is all you need. In: Proceedings of the Advances in Neural Information Processing Systems (NeurIPS) (2017)
Wang, J., Yue, Z., Zhou, S., Chan, K., Loy, C.: Exploiting diffusion prior for real-world image super-resolution. arXiv preprint arXiv:2305.07015 (2023)
Wang, Q., Li, W., Mou, C., Cheng, X., Zhang, J.: 360DVD: controllable panorama video generation with 360-degree video diffusion model. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2024)
Wang, X., Xie, L., Dong, C., Shan, Y.: Real-ESRGAN: training real-world blind super-resolution with pure synthetic data. In: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) (2021)
Wang, X., et al.: ESRGAN: enhanced super-resolution generative adversarial networks. In: Leal-Taixé, L., Roth, S. (eds.) ECCV 2018. LNCS, vol. 11133, pp. 63–79. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-11021-5_5
Wang, Y., Yu, J., Zhang, J.: Zero-shot image restoration using denoising diffusion null-space model. arXiv preprint arXiv:2212.00490 (2022)
Xia, B., et al.: DiffIR: efficient diffusion model for image restoration. In: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) (2023)
Xiao, J., Ehinger, K.A., Oliva, A., Torralba, A.: Recognizing scene viewpoint using panoramic place representation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2012)
Yagi, Y.: Omnidirectional sensing and its applications. IEICE Transactions on Information and Systems (TOIS) (1999)
Yamazawa, K., Yagi, Y., Yachida, M.: Omnidirectional imaging with hyperboloidal projection. In: Proceedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) (1993)
Yang, S., Zhou, Y., Liu, Z., Loy, C.C.: Rerender a video: Zero-shot text-guided video-to-video translation. In: Proceedings of the SIGGRAPH Asia 2023 Conference Papers (SIGGRAPH Asia) (2023)
Yang, S., Mou, C., Yu, J., Wang, Y., Meng, X., Zhang, J.: Neural video fields editing. arXiv preprint arXiv:2312.08882 (2023)
Yinhuai, W., Yujie, H., Jiwen, Y., Jian, Z.: Gan prior based null-space learning for consistent super-resolution. In: Proceedings of the AAAI Conference on Artificial Intelligence (AAAI) (2023)
Yoon, Y., Chung, I., Wang, L., Yoon, K.J.: SphereSR: 360deg image super-resolution with arbitrary projection via continuous spherical image representation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2022)
Yu, F., Wang, X., Cao, M., Li, G., Shan, Y., Dong, C.: OSRT: omnidirectional image super-resolution with distortion-aware transformer. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2023)
Yu, J., Zhang, X., Xu, Y., Zhang, J.: Cross: Diffusion model makes controllable, robust and secure image steganography. In: Proceedings of the Advances in Neural Information Processing Systems (NeurIPS) (2023)
Yue, Z., Wang, J., Loy, C.C.: ResShift: efficient diffusion model for image super-resolution by residual shifting. In: Advances in Neural Information Processing Systems (NeurIPS) (2023)
Zhang, K., Liang, J., Van Gool, L., Timofte, R.: Designing a practical degradation model for deep blind image super-resolution. In: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) (2021)
Zhang, R., Isola, P., Efros, A.A., Shechtman, E., Wang, O.: The unreasonable effectiveness of deep features as a perceptual metric. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2018)
Zhang, W., et al.: Real-world image super-resolution as multi-task learning. In: Advances in Neural Information Processing Systems (NeurIPS) (2023)
Zhang, X., Zhang, Y., Xiong, R., Sun, Q., Zhang, J.: HerosNet: hyperspectral explicable reconstruction and optimal sampling deep network for snapshot compressive imaging. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2022)
Zhou, Y., Yu, M., Ma, H., Shao, H., Jiang, G.: Weighted-to-spherically-uniform SSIM objective quality evaluation for panoramic video. In: Proceedings of the IEEE International Conference on Signal Processing (ICSP) (2018)
Zongying, L., et al.: TaxDiff: taxonomic-guided diffusion model for protein sequence generation. arXiv preprint arXiv:2402.17156 (2024)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
1 Electronic supplementary material
Below is the link to the electronic supplementary material.
Rights and permissions
Copyright information
© 2025 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Li, R., Sheng, X., Li, W., Zhang, J. (2025). OmniSSR: Zero-Shot Omnidirectional Image Super-Resolution Using Stable Diffusion Model. In: Leonardis, A., Ricci, E., Roth, S., Russakovsky, O., Sattler, T., Varol, G. (eds) Computer Vision – ECCV 2024. ECCV 2024. Lecture Notes in Computer Science, vol 15089. Springer, Cham. https://doi.org/10.1007/978-3-031-72751-1_12
Download citation
DOI: https://doi.org/10.1007/978-3-031-72751-1_12
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-72750-4
Online ISBN: 978-3-031-72751-1
eBook Packages: Computer ScienceComputer Science (R0)