Location via proxy:   [ UP ]  
[Report a bug]   [Manage cookies]                
Skip to main content
Springer Nature Link
Log in

Rendering real-world unbounded scenes with cars by learning positional bias

  • Original article
  • Published:
The Visual Computer Aims and scope Submit manuscript

Abstract

In real-world unbounded outdoor scenes with cars, there are various specular reflections caused by the surrounding environment appearing on the reflective surfaces of cars. Background regions of unbounded scenes encode inherent ambiguity of rendering, and specular reflections on cars violates the multi-view consistency. NeRF++ struggles in these scenes because of the enormous ambiguity. To deal with the challenges of rendering unbounded scenes with cars, we present a novel module to strengthen the capability of the basic model in this task. We propose to learn the positional bias between sampled points along a camera ray and target points along the incident light by multi-layer perceptrons to reconstitute the input points and view direction with regularization constraints for physical rendering. Considering the variety of materials and textures in unbounded scenes, we implicitly separate learned foreground colors into two components, diffuse and specular colors, to acquire smooth results. Our module improves basic models by 2.5% on average SSIM in our extensive experiments, produces more photo-realistic novel views of real-world unbounded scenes than other compared methods, and achieves the physical color editing of cars.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12

Similar content being viewed by others

Explore related subjects

Discover the latest articles, news and stories from top researchers in related subjects.

Data availability

We used two common datasets in this work: CO3D [15] (https://ai.facebook.com/datasets/CO3D-dataset/), IBR [18] (https://gitlab.inria.fr/sibr/projects/semantic-reflections/semantic_reflections/). and Tanks and Temples datasets [9] (https://www.tanksandtemples.org/).

Notes

  1. https://github.com/yenchenlin/nerf-pytorch.

  2. https://github.com/isl-org/StableViewSynthesis.

  3. https://github.com/Totoro97/NeuS.

  4. https://github.com/Kai-46/nerfplusplus.

  5. https://github.com/google-research/multinerf.

References

  1. Barron, J.T., Mildenhall, B., Tancik, M., et al.: (2021a) Mip-nerf: A multiscale representation for anti-aliasing neural radiance fields. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 5855–5864

  2. Barron, J.T., Mildenhall, B., Verbin, D., et al.: (2021b) Mip-nerf 360: unbounded anti-aliased neural radiance fields. arXiv preprint arXiv:2111.12077

  3. Bemana, M., Myszkowski, K., Revall Frisvad, J., et al.: (2022) Eikonal fields for refractive novel-view synthesis. In: ACM SIGGRAPH 2022 Conference Proceedings, pp. 1–9

  4. Boss, M., Braun, R., Jampani, V., et al.: (2021) Nerd: neural reflectance decomposition from image collections. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 12684–12694

  5. Firmino, A., Frisvad, J.R., Jensen, H.W.: Progressive Denoising of Monte Carlo Rendered Images. In: Computer Graphics Forum, pp. 1–11. Wiley (2022)

    Google Scholar 

  6. Guo, Y.C., Kang, D., Bao, L., et al.: (2021) Nerfren: neural radiance fields with reflections. arXiv preprint arXiv:2111.15234

  7. Immel, D.S., Cohen, M.F., Greenberg, D.P.: A radiosity method for non-diffuse environments. ACM Siggraph. Comput. Graph. 20(4), 133–142 (1986)

    Article  Google Scholar 

  8. Kajiya, J.T.: The rendering equation. In: Proceedings of the 13th annual conference on Computer graphics and interactive techniques, pp. 143–150 (1986)

  9. Knapitsch, A., Park, J., Zhou, Q.Y., et al.: Tanks and temples: benchmarking large-scale scene reconstruction. ACM Trans. Graph. (ToG) 36(4), 1–13 (2017)

    Article  Google Scholar 

  10. Mildenhall, B., Srinivasan, P.P., Tancik, M., et al.: Nerf: representing scenes as neural radiance fields for view synthesis. In: European Conference on Computer Vision, pp. 405–421. Springer, (2020)

  11. Park, K., Sinha, U., Barron, J.T., et al.: Nerfies: deformable neural radiance fields. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 5865–5874 (2021)

  12. Phong, B.T.: Illumination for computer generated pictures. Commun. ACM 18(6), 311–317 (1975)

    Article  Google Scholar 

  13. Pumarola, A., Corona, E., Pons-Moll, G., et al.: D-nerf: neural radiance fields for dynamic scenes. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10318–10327 (2021)

  14. Qiu, J., Zhu, Y., Jiang, P.T., et al.: Rdnerf: relative depth guided nerf for dense free view synthesis. Vis. Comput. (2023). https://doi.org/10.1007/s00371-023-02863-5

    Article  Google Scholar 

  15. Reizenstein, J., Shapovalov, R., Henzler, P., et al.: Common objects in 3d: large-scale learning and evaluation of real-life 3d category reconstruction. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 10901–10911 (2021)

  16. Riegler, G., Koltun, V.: Free view synthesis. In: European Conference on Computer Vision, pp 623–640 . Springer (2020)

  17. Riegler, G., Koltun, V.: Stable view synthesis. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12216–12225 (2021)

  18. Rodriguez, S., Prakash, S., Hedman, P., et al.: Image-based rendering of cars using semantic labels and approximate reflection flow. Proc. ACM Comput. Graph. Interact. Tech. 3 (2020)

  19. Schonberger, J.L., Frahm, J.M.: Structure-from-motion revisited. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4104–4113 (2016)

  20. Sinha, S.N., Kopf, J., Goesele, M., et al.: Image-based rendering for scenes with reflections. ACM Trans. Graph. (TOG) 31(4), 1–10 (2012)

    Article  Google Scholar 

  21. Srinivasan, P.P., Deng, B., Zhang, X., et al.: Nerv: neural reflectance and visibility fields for relighting and view synthesis. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7495–7504 (2021)

  22. Teed, Z., Deng, J.: Raft: recurrent all-pairs field transforms for optical flow. In: Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part II 16, pp. 402–419. Springer (2020)

  23. Vaswani, A., Shazeer, N., Parmar, N., et al.: Attention is all you need. In: Advances in neural information processing systems 30 (2017)

  24. Verbin, D., Hedman, P., Mildenhall, B., et al.: Ref-nerf: structured view-dependent appearance for neural radiance fields. arXiv preprint arXiv:2112.03907 (2021)

  25. Vicini, D., Adler, D., Novák, J., et al.: Denoising Deep Monte Carlo Renderings. In: Computer Graphics Forum, pp. 316–327. Wiley (2019)

    Google Scholar 

  26. Wang, P., Liu, L., Liu, Y., et al.: Neus: learning neural implicit surfaces by volume rendering for multi-view reconstruction. arXiv preprint arXiv:2106.10689 (2021a)

  27. Wang, Z., Wang, L., Zhao, F., et al.: Mirrornerf: one-shot neural portrait radiance field from multi-mirror catadioptric imaging. In: 2021 IEEE International Conference on Computational Photography (ICCP), IEEE, pp. 1–12 (2021b)

  28. Wu, H., Hu, Z., Li, L., et al.: Nefii: Inverse rendering for reflectance decomposition with near-field indirect illumination. arXiv preprint arXiv:2303.16617 (2023)

  29. Xu, J., Wu, X., Zhu, Z., et al.: Scalable image-based indoor scene rendering with reflections. ACM Trans. Graph. (TOG) 40(4), 1–14 (2021)

    Article  Google Scholar 

  30. Yariv, L., Kasten, Y., Moran, D., et al.: Multiview neural surface reconstruction by disentangling geometry and appearance. Adv. Neural Inf. Process. Syst. 33, 2492–2502 (2020)

    Google Scholar 

  31. Zhang, J., Yang, G., Tulsiani, S., et al.: Ners: neural reflectance surfaces for sparse-view 3d reconstruction in the wild. Adv. Neural Inf. Process. Syst. 34, 29835–29847 (2021)

    Google Scholar 

  32. Zhang, K., Riegler, G., Snavely, N., et al.: Nerf++: analyzing and improving neural radiance fields. arXiv preprint arXiv:2010.07492 (2020)

  33. Zhang, R., Isola, P., Efros, A.A., et al.: The unreasonable effectiveness of deep features as a perceptual metric. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 586–595 (2018)

  34. Zhang, X., Srinivasan, P.P., Deng, B., et al.: Nerfactor: neural factorization of shape and reflectance under an unknown illumination. ACM Trans. Graph. (TOG) 40(6), 1–18 (2021)

    Article  Google Scholar 

  35. Zhang, Y., Sun, J., He, X., et al.: (2022) Modeling indirect illumination for inverse rendering. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 18643–18652

Download references

Funding

This work is supported by the National Key Research and Development Program of China Grant (No.2018AAA0100400), NSFC (No.61922046) and NSFC (No.62132012).

Author information

Authors and Affiliations

  1. TMCC, College of Computer Science, Nankai University, Tianjin, 300000, China

    Jiaxiong Qiu, Ze-Xin Yin, Ming-Ming Cheng & Bo Ren

Authors
  1. Jiaxiong Qiu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Ze-Xin Yin
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Ming-Ming Cheng
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Bo Ren
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

J-XQ contributed to conceiving, designing the analysis and writing; Z-XY performed data collection; BR performed writing—review and editing; and M-MC performed supervision.

Corresponding author

Correspondence to Bo Ren.

Ethics declarations

Conflict of interest

The authors declare that they have no conflict of interest.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Qiu, J., Yin, ZX., Cheng, MM. et al. Rendering real-world unbounded scenes with cars by learning positional bias. Vis Comput 40, 4085–4098 (2024). https://doi.org/10.1007/s00371-023-03070-y

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00371-023-03070-y

Keywords

Search

Navigation