Refracting Once is Enough: Neural Radiance Fields for Novel-View Synthesis of Real Refractive Objects
Authors: 
Xiaoqian Liang, Jianji Wang, Yuanliang Lu, Xubin Duan, Xichun Liu, Nanning ZhengAuthors Info & Claims
Xiaoqian Liang, Jianji Wang, Yuanliang Lu, Xubin Duan, Xichun Liu, Nanning ZhengAuthors Info & ClaimsPages 694 - 703
Abstract
Neural Radiance Fields (NeRF) have shown promise in novel view synthesis, but it still face challenges when applied to refractive objects. The presence of refraction disrupts multiview consistency, often resulting in renderings that are either blurred or distorted. Recent methods alleviate this challenge by introducing external supervision, such as mask images and Index of Refraction. However,acquiring such information is often impractical,limiting the application of NeRF-like models to complex scenes with refracting elementsand yielding unsatisfactory results. To address these limitations, we introduce RoseNeRF (Refracting once is enough for NeRF), a novel method that simplifies the complex interaction of rays within objects to a single refraction event. We design the refraction network that efficiently maps a ray in the 4D light field to its refracted counterpart, better modeling curved ray paths. Furthermore, we introduce a regularization strategy to ensure the reversibility of optical paths, which is anchored in physical world theorems. To help it easier for the network to learn the highly view-dependent appearance of refractive objects, we also propose novel density decoding strategies. Our method is designed for seamless integration into most NeRF-like frameworks and has demonstrated state-of-the-art performance without any additional information on both the Eikonal Fields' dataset and Shiny dataset.
References
[1]
Chong Bao, Yinda Zhang, Bangbang Yang, Tianxing Fan, Zesong Yang, Hujun Bao, Guofeng Zhang, and Zhaopeng Cui. 2023. Sine: Semantic-driven image-based nerf editing with prior-guided editing field. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 20919--20929.
[2]
Jonathan T Barron, Ben Mildenhall, Matthew Tancik, Peter Hedman, Ricardo Martin-Brualla, and Pratul P Srinivasan. 2021. Mip-nerf: A multiscale representation for anti-aliasing neural radiance fields. In Proceedings of the IEEE/CVF International Conference on Computer Vision. 5855--5864.
[3]
Jonathan T Barron, Ben Mildenhall, Dor Verbin, Pratul P Srinivasan, and Peter Hedman. 2022. Mip-nerf 360: Unbounded anti-aliased neural radiance fields. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 5470--5479.
[4]
Jonathan T Barron, Ben Mildenhall, Dor Verbin, Pratul P Srinivasan, and Peter Hedman. 2023. Zip-NeRF: Anti-aliased grid-based neural radiance fields. arXiv preprint arXiv:2304.06706 (2023).
[5]
Mojtaba Bemana, Karol Myszkowski, Jeppe Revall Frisvad, Hans-Peter Seidel, and Tobias Ritschel. 2022. Eikonal fields for refractive novel-view synthesis. In ACM SIGGRAPH 2022 Conference Proceedings. 1--9.
[6]
Mark Boss, Raphael Braun, Varun Jampani, Jonathan T Barron, Ce Liu, and Hendrik Lensch. 2021. Nerd: Neural reflectance decomposition from image collections. In Proceedings of the IEEE/CVF International Conference on Computer Vision. 12684--12694.
[7]
Chris Buehler, Michael Bosse, Leonard McMillan, Steven Gortler, and Michael Cohen. 2023. Unstructured lumigraph rendering. In Seminal Graphics Papers: Pushing the Boundaries, Volume 2. 497--504.
[8]
Ang Cao and Justin Johnson. 2023. Hexplane: A fast representation for dynamic scenes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 130--141.
[9]
Anpei Chen, Zexiang Xu, Andreas Geiger, Jingyi Yu, and Hao Su. 2022. Tensorf: Tensorial radiance fields. In European Conference on Computer Vision. Springer, 333--350.
[10]
Tongbo Chen, Michael Goesele, and H-P Seidel. 2006. Mesostructure from specularity. In 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'06), Vol. 2. IEEE, 1825--1832.
[11]
Zhiqin Chen, Thomas Funkhouser, Peter Hedman, and Andrea Tagliasacchi. 2023. Mobilenerf: Exploiting the polygon rasterization pipeline for efficient neural field rendering on mobile architectures. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 16569--16578.
[12]
Zhangkaiwen Chu. 2022. Neural Raidance Fields with Refractions. https://github.com/ZhangkaiwenChu/Neural-Radiance-Fields-with-Refractions
[13]
Yung-Yu Chuang, Douglas E Zongker, Joel Hindorff, Brian Curless, David H Salesin, and Richard Szeliski. 2000. Environment matting extensions: Towards higher accuracy and real-time capture. In Proceedings of the 27th annual conference on Computer graphics and interactive techniques. 121--130.
[14]
Paul E Debevec, Camillo J Taylor, and Jitendra Malik. 2023. Modeling and rendering architecture from photographs: A hybrid geometry-and image-based approach. In Seminal Graphics Papers: Pushing the Boundaries, Volume 2. 465--474.
[15]
Weijian Deng, Dylan Campbell, Chunyi Sun, Shubham Kanitkar, Matthew Shaffer, and Stephen Gould. 2024. Ray Deformation Networks for Novel View Synthesis of Refractive Objects. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV). 3118--3128.
[16]
John Flynn, Michael Broxton, Paul Debevec, Matthew DuVall, Graham Fyffe, Ryan Overbeck, Noah Snavely, and Richard Tucker. 2019. Deepview: View synthesis with learned gradient descent. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2367--2376.
[17]
Sara Fridovich-Keil, Giacomo Meanti, Frederik Rahbæk Warburg, Benjamin Recht, and Angjoo Kanazawa. 2023. K-planes: Explicit radiance fields in space, time, and appearance. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 12479--12488.
[18]
Taku Fujitomi, Ken Sakurada, Ryuhei Hamaguchi, Hidehiko Shishido, Masaki Onishi, and Yoshinari Kameda. 2022. LB-NERF: light bending neural radiance fields for transparent medium. In 2022 IEEE International Conference on Image Processing (ICIP). IEEE, 2142--2146.
[19]
Steven J Gortler, Radek Grzeszczuk, Richard Szeliski, and Michael F Cohen. 2023. The lumigraph. In Seminal Graphics Papers: Pushing the Boundaries, Volume 2. 453--464.
[20]
Kai Han, Kwan-Yee K Wong, and Miaomiao Liu. 2018. Dense reconstruction of transparent objects by altering incident light paths through refraction. International Journal of Computer Vision, Vol. 126 (2018), 460--475.
[21]
Ayaan Haque, Matthew Tancik, Alexei A Efros, Aleksander Holynski, and Angjoo Kanazawa. 2023. Instruct-nerf2nerf: Editing 3d scenes with instructions. arXiv preprint arXiv:2303.12789 (2023).
[22]
Peter Hedman, Julien Philip, True Price, Jan-Michael Frahm, George Drettakis, and Gabriel Brostow. 2018. Deep blending for free-viewpoint image-based rendering. ACM Transactions on Graphics (ToG), Vol. 37, 6 (2018), 1--15.
[23]
Cong Phuoc Huynh, Antonio Robles-Kelly, and Edwin Hancock. 2010. Shape and refractive index recovery from single-view polarisation images. In 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. 1229--1236. https://doi.org/10.1109/CVPR.2010.5539828
[24]
Ivo Ihrke, Gernot Ziegler, Art Tevs, Christian Theobalt, Marcus Magnor, and Hans-Peter Seidel. 2007. Eikonal rendering: Efficient light transport in refractive objects. ACM Transactions on Graphics (TOG), Vol. 26, 3 (2007), 59--es.
[25]
Abhishek Kar, Christian H"ane, and Jitendra Malik. 2017. Learning a multi-view stereo machine. Advances in neural information processing systems, Vol. 30 (2017).
[26]
Wooseok Kim, Taiki Fukiage, and Takeshi Oishi. 2023. REF $^ 2$-NeRF: Reflection and Refraction aware Neural Radiance Field. arXiv preprint arXiv:2311.17116 (2023).
[27]
Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014).
[28]
Kiriakos N Kutulakos and Steven M Seitz. 2000. A theory of shape by space carving. International journal of computer vision, Vol. 38 (2000), 199--218.
[29]
Marc Levoy and Pat Hanrahan. 2023. Light field rendering. In Seminal Graphics Papers: Pushing the Boundaries, Volume 2. 441--452.
[30]
Zhengqin Li, Yu-Ying Yeh, and Manmohan Chandraker. 2020. Through the Looking Glass: Neural 3D Reconstruction of Transparent Shapes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).
[31]
Yong Liu, Hang Dong, Boyang Liang, Songwei Liu, Qingji Dong, Kai Chen, Fangmin Chen, Lean Fu, and Fei Wang. 2023. Unfolding Once is Enough: A Deployment-Friendly Transformer Unit for Super-Resolution. In Proceedings of the 31st ACM International Conference on Multimedia. 7952--7960.
[32]
Stephen Lombardi, Tomas Simon, Jason Saragih, Gabriel Schwartz, Andreas Lehrmann, and Yaser Sheikh. 2019. Neural volumes: Learning dynamic renderable volumes from images. arXiv preprint arXiv:1906.07751 (2019).
[33]
Li Ma, Xiaoyu Li, Jing Liao, Qi Zhang, Xuan Wang, Jue Wang, and Pedro V Sander. 2022. Deblur-nerf: Neural radiance fields from blurry images. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 12861--12870.
[34]
Nelson Max. 1995. Optical models for direct volume rendering. IEEE Transactions on Visualization and Computer Graphics, Vol. 1, 2 (1995), 99--108.
[35]
Ben Mildenhall, Pratul P Srinivasan, Rodrigo Ortiz-Cayon, Nima Khademi Kalantari, Ravi Ramamoorthi, Ren Ng, and Abhishek Kar. 2019. Local light field fusion: Practical view synthesis with prescriptive sampling guidelines. ACM Transactions on Graphics (TOG), Vol. 38, 4 (2019), 1--14.
[36]
Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. 2021. Nerf: Representing scenes as neural radiance fields for view synthesis. Commun. ACM, Vol. 65, 1 (2021), 99--106.
[37]
Nigel J. W. Morris and Kiriakos N. Kutulakos. 2007. Reconstructing the Surface of Inhomogeneous Transparent Scenes by Scatter-Trace Photography. In 2007 IEEE 11th International Conference on Computer Vision. 1--8. https://doi.org/10.1109/ICCV.2007.4408882
[38]
Thomas Müller, Alex Evans, Christoph Schied, and Alexander Keller. 2022. Instant neural graphics primitives with a multiresolution hash encoding. ACM Transactions on Graphics (ToG), Vol. 41, 4 (2022), 1--15.
[39]
Michael Niemeyer, Jonathan T Barron, Ben Mildenhall, Mehdi SM Sajjadi, Andreas Geiger, and Noha Radwan. 2022. Regnerf: Regularizing neural radiance fields for view synthesis from sparse inputs. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 5480--5490.
[40]
Jen-I Pan, Jheng-Wei Su, Kai-Wen Hsiao, Ting-Yu Yen, and Hung-Kuo Chu. 2022. Sampling Neural Radiance Fields for Refractive Objects. In SIGGRAPH Asia 2022 Technical Communications (Daegu, Republic of Korea) (SA '22). Association for Computing Machinery, New York, NY, USA, Article 5, bibinfonumpages4 pages. https://doi.org/10.1145/3550340.3564234
[41]
Keunhong Park, Utkarsh Sinha, Jonathan T Barron, Sofien Bouaziz, Dan B Goldman, Steven M Seitz, and Ricardo Martin-Brualla. 2021. Nerfies: Deformable neural radiance fields. In Proceedings of the IEEE/CVF International Conference on Computer Vision. 5865--5874.
[42]
Eric Penner and Li Zhang. 2017. Soft 3d reconstruction for view synthesis. ACM Transactions on Graphics (TOG), Vol. 36, 6 (2017), 1--11.
[43]
Albert Pumarola, Enric Corona, Gerard Pons-Moll, and Francesc Moreno-Noguer. 2021. D-nerf: Neural radiance fields for dynamic scenes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 10318--10327.
[44]
Yiming Qian, Minglun Gong, and Yee Hong Yang. 2016. 3D Reconstruction of Transparent Objects With Position-Normal Consistency. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR).
[45]
Gernot Riegler and Vladlen Koltun. 2020. Free view synthesis. In Computer Vision--ECCV 2020: 16th European Conference, Glasgow, UK, August 23--28, 2020, Proceedings, Part XIX 16. Springer, 623--640.
[46]
Pratul P Srinivasan, Boyang Deng, Xiuming Zhang, Matthew Tancik, Ben Mildenhall, and Jonathan T Barron. 2021. Nerv: Neural reflectance and visibility fields for relighting and view synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 7495--7504.
[47]
Matthew Tancik, Ethan Weber, Evonne Ng, Ruilong Li, Brent Yi, Terrance Wang, Alexander Kristoffersen, Jake Austin, Kamyar Salahi, Abhik Ahuja, et al. 2023. Nerfstudio: A modular framework for neural radiance field development. In ACM SIGGRAPH 2023 Conference Proceedings. 1--12.
[48]
Borislav Trifonov, Derek Bradley, and Wolfgang Heidrich. 2006. Tomographic Reconstruction of Transparent Objects. In ACM SIGGRAPH 2006 Sketches (Boston, Massachusetts) (SIGGRAPH '06). Association for Computing Machinery, New York, NY, USA, 55--es. https://doi.org/10.1145/1179849.1179918
[49]
Dor Verbin, Peter Hedman, Ben Mildenhall, Todd Zickler, Jonathan T. Barron, and Pratul P. Srinivasan. 2022. Ref-NeRF: Structured View-Dependent Appearance for Neural Radiance Fields. CVPR (2022).
[50]
Jianyi Wang, Zongsheng Yue, Shangchen Zhou, Kelvin C. K. Chan, and Chen Change Loy. 2023 c. Exploiting Diffusion Prior for Real-World Image Super-Resolution. arxiv: 2305.07015 [cs.CV]
[51]
Yuze Wang, Junyi Wang, Yansong Qu, and Yue Qi. 2023 a. RIP-NeRF: Learning Rotation-Invariant Point-based Neural Radiance Field for Fine-grained Editing and Compositing. In Proceedings of the 2023 ACM International Conference on Multimedia Retrieval. 125--134.
[52]
Zhou Wang, A.C. Bovik, H.R. Sheikh, and E.P. Simoncelli. 2004. Image quality assessment: from error visibility to structural similarity. IEEE Transactions on Image Processing, Vol. 13, 4 (2004), 600--612. https://doi.org/10.1109/TIP.2003.819861
[53]
Ziyu Wang, Wei Yang, Junming Cao, Qiang Hu, Lan Xu, Junqing Yu, and Jingyi Yu. 2023 b. NeReF: Neural Refractive Field for Fluid Surface Reconstruction and Rendering. In 2023 IEEE International Conference on Computational Photography (ICCP). IEEE, 1--11.
[54]
Frederik Warburg*, Ethan Weber*, Matthew Tancik, Aleksander Ho?y'ski, and Angjoo Kanazawa. 2023. Nerfbusters: Removing Ghostly Artifacts from Casually Captured NeRFs. In International Conference on Computer Vision (ICCV).
[55]
Gordon Wetzstein, David Roodnick, Wolfgang Heidrich, and Ramesh Raskar. 2011. Refractive shape from light field distortion. In 2011 International Conference on Computer Vision. IEEE, 1180--1186.
[56]
Ydo Wexler, Andrew Fitzgibbon, and Andrew Zisserman. 2002. Image-based environment matting. In Proceedings, Eurographics Workshop on Rendering. 289--299.
[57]
Suttisak Wizadwongsa, Pakkapon Phongthawee, Jiraphon Yenphraphai, and Supasorn Suwajanakorn. 2021. NeX: Real-time View Synthesis with Neural Basis Expansion. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR).
[58]
Jiamin Xu, Zihan Zhu, Hujun Bao, and Weiwei Xu. 2022. Hybrid Mesh-neural Representation for 3D Transparent Object Reconstruction. arXiv preprint arXiv:2203.12613 (2022).
[59]
Ze-Xin Yin, Jiaxiong Qiu, Ming-Ming Cheng, and Bo Ren. 2023. Multi-Space Neural Radiance Fields. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 12407--12416.
[60]
Yifan Zhan, Shohei Nobuhara, Ko Nishino, and Yinqiang Zheng. 2023. NeRFrac: Neural Radiance Fields through Refractive Surface. In Proceedings of the IEEE/CVF International Conference on Computer Vision. 18402--18412.
[61]
Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. 2018. The Unreasonable Effectiveness of Deep Features as a Perceptual Metric. In CVPR.
[62]
Tinghui Zhou, Richard Tucker, John Flynn, Graham Fyffe, and Noah Snavely. 2018. Stereo magnification: Learning view synthesis using multiplane images. arXiv preprint arXiv:1805.09817 (2018).
[63]
Douglas E Zongker, Dawn M Werner, Brian Curless, and David H Salesin. 2023. Environment matting and compositing. In Seminal Graphics Papers: Pushing the Boundaries, Volume 2. 537--546.
Index Terms
- Refracting Once is Enough: Neural Radiance Fields for Novel-View Synthesis of Real Refractive Objects
Recommendations
Deep Reflectance Volumes: Relightable Reconstructions from Multi-view Photometric Images
Computer Vision – ECCV 2020AbstractWe present a deep learning approach to reconstruct scene appearance from unstructured images captured under collocated point lighting. At the heart of Deep Reflectance Volumes is a novel volumetric scene representation consisting of opacity, ...
Neural Light Transport for Relighting and View Synthesis
The light transport (LT) of a scene describes how it appears under different lighting conditions from different viewing directions, and complete knowledge of a scene’s LT enables the synthesis of novel views under arbitrary lighting. In this article, we ...
NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis
Computer Vision – ECCV 2020AbstractWe present a method that achieves state-of-the-art results for synthesizing novel views of complex scenes by optimizing an underlying continuous volumetric scene function using a sparse set of input views. Our algorithm represents a scene using a ...
Comments
Information & Contributors
Information
Published In
May 2024
1379 pages
ISBN:9798400706196
DOI:10.1145/3652583
- General Chairs:
- Cathal Gurrin,
- Rachada Kongkachandra,
- Klaus Schoeffmann,
- Program Chairs:
- Duc-Tien Dang-Nguyen,
- Luca Rossetto,
- Shin'ichi Satoh,
- Liting Zhou
Copyright © 2024 ACM.
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].
Sponsors
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 07 June 2024
Check for updates
Author Tags
Qualifiers
- Research-article
Funding Sources
Conference
Acceptance Rates
Overall Acceptance Rate 254 of 830 submissions, 31%
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 130Total Downloads
- Downloads (Last 12 months)130
- Downloads (Last 6 weeks)16
Reflects downloads up to 10 Nov 2024
Other Metrics
Citations
View Options
Get Access
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in