Location via proxy:   [ UP ]  
[Report a bug]   [Manage cookies]                
Skip to main content
Springer Nature Link
Log in

Neural Poisson Solver: A Universal and Continuous Framework for Natural Signal Blending

  • Conference paper
  • First Online:
Computer Vision – ECCV 2024 (ECCV 2024)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 15138))

Included in the following conference series:

  • 74 Accesses

Abstract

Implicit Neural Representation (INR) has become a popular method for representing visual signals (e.g., 2D images and 3D scenes), demonstrating promising results in various downstream applications. Given its potential as a medium for visual signals, exploring the development of a neural blending method that utilizes INRs is a natural progression. Neural blending involves merging two INRs to create a new INR that encapsulates information from both original representations. A direct approach involves applying traditional image editing methods to the INR rendering process. However, this method often results in blending distortions, artifacts, and color shifts, primarily due to the discretization of the underlying pixel grid and the introduction of boundary conditions for solving variational problems. To tackle this issue, we introduce the Neural Poisson Solver, a plug-and-play and universally applicable framework across different signal dimensions for blending visual signals represented by INRs. Our Neural Poisson Solver offers a variational problem-solving approach based on the continuous Poisson equation, demonstrating exceptional performance across various domains. Specifically, we propose a gradient-guided neural solver to represent the solution process of the variational problem, refining the target signal to achieve natural blending results. We also develop a Poisson equation-based loss and optimization scheme to train our solver, ensuring it effectively blends the input INR scenes while preserving their inherent structure and semantic content. The lack of dependence on additional prior knowledge makes our method easily adaptable to various task categories, highlighting its versatility. Comprehensive experimental results validate the robustness of our approach across multiple dimensions and blending tasks. Project: https://ep1phany05.github.io/NeuralPoissonSolver-website/.

D. Wu and H. Zhu—Contributed equally to this work.

This work was supported by NSFC under Grant 62025108 and Tencent Rhino-Bird Joint Research Program RBFR2024009.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 74.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 89.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

References

  1. Avrahami, O., Lischinski, D., Fried, O.: Blended diffusion for text-driven editing of natural images. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 18208–18218 (2022)

    Google Scholar 

  2. Benaim, S., Warburg, F., Christensen, P.E., Belongie, S.: Volumetric disentanglement for 3D scene manipulation. In: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 8667–8677 (2024)

    Google Scholar 

  3. Chen, Y., et al.: UPST-nerf: universal photorealistic style transfer of neural radiance fields for 3D scene. arXiv preprint arXiv:2208.07059 (2022)

  4. Chen, Y., Lu, L., Karniadakis, G.E., Dal Negro, L.: Physics-informed neural networks for inverse problems in nano-optics and metamaterials. Opt. Express 28(8), 11618–11633 (2020)

    Article  Google Scholar 

  5. Chiang, P.Z., Tsai, M.S., Tseng, H.Y., Lai, W.S., Chiu, W.C.: Stylizing 3D scene via implicit representation and hypernetwork. In: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 1475–1484 (2022)

    Google Scholar 

  6. Dessein, A., Smith, W.A., Wilson, R.C., Hancock, E.R.: Seamless texture stitching on a 3D mesh by poisson blending in patches. In: 2014 IEEE International Conference on Image Processing (ICIP), pp. 2031–2035. IEEE (2014)

    Google Scholar 

  7. Elder, J.H., Goldberg, R.M.: Image editing in the contour domain. In: Proceedings of 1998 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (Cat. No. 98CB36231), pp. 374–381. IEEE (1998)

    Google Scholar 

  8. Fan, Z., Jiang, Y., Wang, P., Gong, X., Xu, D., Wang, Z.: Unified implicit neural stylization. In: Avidan, S., Brostow, G., Cissé, M., Farinella, G.M., Hassner, T. (eds.) ECCV 2022. LNCS, vol. 13675, pp. 636–654. Springer, Cham (2022). https://doi.org/10.1007/978-3-031-19784-0_37

    Chapter  Google Scholar 

  9. Fattal, R., Lischinski, D., Werman, M.: Gradient domain high dynamic range compression. In: Seminal Graphics Papers: Pushing the Boundaries, vol. 2, pp. 671–678 (2023)

    Google Scholar 

  10. Gao, R., et al.: Objectfolder 2.0: a multisensory object dataset for sim2real transfer. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10598–10608 (2022)

    Google Scholar 

  11. Gong, B., Wang, Y., Han, X., Dou, Q.: Recolornerf: layer decomposed radiance field for efficient color editing of 3D scenes. arXiv preprint arXiv:2301.07958 (2023)

  12. Gordon, O., Avrahami, O., Lischinski, D.: Blended-nerf: zero-shot object generation and blending in existing neural radiance fields. arXiv preprint arXiv:2306.12760 (2023)

  13. Huang, Y.H., He, Y., Yuan, Y.J., Lai, Y.K., Gao, L.: Stylizednerf: consistent 3D scene stylization as stylized nerf via 2D-3D mutual learning. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 18342–18352 (2022)

    Google Scholar 

  14. Jain, A., Mildenhall, B., Barron, J.T., Abbeel, P., Poole, B.: Zero-shot text-guided object generation with dream fields. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 867–876 (2022)

    Google Scholar 

  15. Karniadakis, G.E., Kevrekidis, I.G., Lu, L., Perdikaris, P., Wang, S., Yang, L.: Physics-informed machine learning. Nat. Rev. Phys. 3(6), 422–440 (2021)

    Article  Google Scholar 

  16. Kingma, D.P., Ba, J.: Adam: a method for stochastic optimization (2017)

    Google Scholar 

  17. Kobayashi, S., Matsumoto, E., Sitzmann, V.: Decomposing nerf for editing via feature field distillation. In: Advances in Neural Information Processing Systems, vol. 35, pp. 23311–23330 (2022)

    Google Scholar 

  18. Lewis, J.: Lifting detail from darkness. In: SIGGRAPH (2001)

    Google Scholar 

  19. Liu, R., Sun, Y., Zhu, J., Tian, L., Kamilov, U.S.: Recovery of continuous 3D refractive index maps from discrete intensity-only measurements using neural fields. Nat. Mach. Intell. 4(9), 781–791 (2022)

    Article  Google Scholar 

  20. Liu, S., Zhang, X., Zhang, Z., Zhang, R., Zhu, J.Y., Russell, B.: Editing conditional radiance fields. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 5773–5783 (2021)

    Google Scholar 

  21. Lu, S., Liu, Y., Kong, A.W.K.: TF-icon: diffusion-based training-free cross-domain image composition. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 2294–2305 (2023)

    Google Scholar 

  22. Ma, Z., et al.: Cardiacfield: computational echocardiography for universal screening (2023)

    Google Scholar 

  23. Mildenhall, B., Srinivasan, P.P., Tancik, M., Barron, J.T., Ramamoorthi, R., Ng, R.: Nerf: representing scenes as neural radiance fields for view synthesis. Commun. ACM 65(1), 99–106 (2021)

    Article  Google Scholar 

  24. Morel, J.M., Petro, A.B., Sbert, C.: Fourier implementation of poisson image editing. Pattern Recogn. Lett. 33(3), 342–348 (2012)

    Article  Google Scholar 

  25. Park, J.J., Florence, P., Straub, J., Newcombe, R., Lovegrove, S.: Deepsdf: learning continuous signed distance functions for shape representation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 165–174 (2019)

    Google Scholar 

  26. Pérez, P., Gangnet, M., Blake, A.: Poisson image editing. ACM Trans. Graph. (TOG) 22(3), 313–318 (2003)

    Article  Google Scholar 

  27. Raissi, M., Perdikaris, P., Karniadakis, G.E.: Physics-informed neural networks: a deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations. J. Comput. Phys. 378, 686–707 (2019)

    Article  MathSciNet  Google Scholar 

  28. Raissi, M., Yazdani, A., Karniadakis, G.E.: Hidden fluid mechanics: learning velocity and pressure fields from flow visualizations. Science 367(6481), 1026–1030 (2020)

    Article  MathSciNet  Google Scholar 

  29. Rocchini, C., Cignoni, P., Montani, C., Scopigno, R.: Multiple textures stitching and blending on 3D objects. In: Lischinski, D., Larson, G.W. (eds.) EGSR 1999. E, pp. 119–130. Springer, Vienna (1999). https://doi.org/10.1007/978-3-7091-6809-7_12

    Chapter  Google Scholar 

  30. Rückert, D., Wang, Y., Li, R., Idoughi, R., Heidrich, W.: Neat: neural adaptive tomography. ACM Trans. Graph. (TOG) 41(4), 1–13 (2022)

    Google Scholar 

  31. Shen, S., et al.: Non-line-of-sight imaging via neural transient fields. IEEE Trans. Pattern Anal. Mach. Intell. 43(7), 2257–2268 (2021)

    Article  Google Scholar 

  32. Sitzmann, V., Martel, J., Bergman, A., Lindell, D., Wetzstein, G.: Implicit neural representations with periodic activation functions. In: Advances in Neural Information Processing Systems, vol. 33, pp. 7462–7473 (2020)

    Google Scholar 

  33. Sitzmann, V., Rezchikov, S., Freeman, B., Tenenbaum, J., Durand, F.: Light field networks: neural scene representations with single-evaluation rendering. In: Advances in Neural Information Processing Systems, vol. 34, pp. 19313–19325 (2021)

    Google Scholar 

  34. Strümpler, Y., Postels, J., Yang, R., Gool, L.V., Tombari, F.: Implicit neural representations for image compression. In: Avidan, S., Brostow, G., Cissé, M., Farinella, G.M., Hassner, T. (eds.) ECCV 2022. LNCS, vol. 13686, pp. 74–91. Springer, Cham (2022). https://doi.org/10.1007/978-3-031-19809-0_5

    Chapter  Google Scholar 

  35. Tancik, M., et al.: Fourier features let networks learn high frequency functions in low dimensional domains. In: Advances in Neural Information Processing Systems, vol. 33, pp. 7537–7547 (2020)

    Google Scholar 

  36. Tewari, A., et al.: Advances in neural rendering. In: Computer Graphics Forum, vol. 41, pp. 703–735. Wiley Online Library (2022)

    Google Scholar 

  37. Wang, C., Chai, M., He, M., Chen, D., Liao, J.: Clip-nerf: text-and-image driven manipulation of neural radiance fields. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3835–3844 (2022)

    Google Scholar 

  38. Wu, Q., Tan, J., Xu, K.: Palettenerf: palette-based color editing for nerfs. arXiv preprint arXiv:2212.12871 (2022)

  39. Xu, J., et al.: Nesvor: implicit neural representation for slice-to-volume reconstruction in MRI. IEEE Trans. Med. Imaging 42(6), 1707–1719 (2023)

    Article  Google Scholar 

  40. Yuan, Y.J., Sun, Y.T., Lai, Y.K., Ma, Y., Jia, R., Gao, L.: Nerf-editing: geometry editing of neural radiance fields. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 18353–18364 (2022)

    Google Scholar 

  41. Zhang, L., Wen, T., Shi, J.: Deep image blending. In: The IEEE Winter Conference on Applications of Computer Vision, pp. 231–240 (2020)

    Google Scholar 

  42. Zhou, H., et al.: Fourier ptychographic microscopy image stack reconstruction using implicit neural representations. Optica 10(12), 1679–1687 (2023)

    Article  Google Scholar 

  43. Zhu, H., Liu, Z., Zhou, Y., Ma, Z., Cao, X.: DNF: diffractive neural field for lensless microscopic imaging. Opt. Express 30(11), 18168–18178 (2022)

    Article  Google Scholar 

  44. Zhu, H., et al.: Disorder-invariant implicit neural representation. arXiv preprint arXiv:2304.00837 (2023)

  45. Zhu, J., Zhu, H., Zhang, Q., Zhu, F., Ma, Z., Cao, X.: Pyramid nerf: frequency guided fast radiance field optimization. Int. J. Comput. Vis. 1–16 (2023)

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Nanjing University, Nanjing, China

    Delong Wu, Hao Zhu, Zhan Ma & Xun Cao

  2. Tencent AI Lab, Shenzhen, China

    Qi Zhang

  3. China Astronaut Research and Training Center, Beijing, China

    You Li

Authors
  1. Delong Wu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Hao Zhu
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Qi Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. You Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Zhan Ma
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Xun Cao
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding authors

Correspondence to Zhan Ma or Xun Cao .

Editor information

Editors and Affiliations

  1. University of Birmingham, Birmingham, UK

    Aleš Leonardis

  2. University of Trento, Trento, Italy

    Elisa Ricci

  3. Technical University of Darmstadt, Darmstadt, Hessen, Germany

    Stefan Roth

  4. Princeton University, Palo Alto, CA, USA

    Olga Russakovsky

  5. Czech Technical University in Prague, Prague, Czech Republic

    Torsten Sattler

  6. École des Ponts ParisTech, Marne-la-Vallée, France

    Gül Varol

Rights and permissions

Reprints and permissions

Copyright information

© 2025 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Wu, D., Zhu, H., Zhang, Q., Li, Y., Ma, Z., Cao, X. (2025). Neural Poisson Solver: A Universal and Continuous Framework for Natural Signal Blending. In: Leonardis, A., Ricci, E., Roth, S., Russakovsky, O., Sattler, T., Varol, G. (eds) Computer Vision – ECCV 2024. ECCV 2024. Lecture Notes in Computer Science, vol 15138. Springer, Cham. https://doi.org/10.1007/978-3-031-72989-8_15

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-72989-8_15

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-72988-1

  • Online ISBN: 978-3-031-72989-8

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation