Location via proxy:   [ UP ]  
[Report a bug]   [Manage cookies]                
skip to main content
research-article
Open access

Soft 3D reconstruction for view synthesis

Published: 20 November 2017 Publication History

Abstract

We present a novel algorithm for view synthesis that utilizes a soft 3D reconstruction to improve quality, continuity and robustness. Our main contribution is the formulation of a soft 3D representation that preserves depth uncertainty through each stage of 3D reconstruction and rendering. We show that this representation is beneficial throughout the view synthesis pipeline. During view synthesis, it provides a soft model of scene geometry that provides continuity across synthesized views and robustness to depth uncertainty. During 3D reconstruction, the same robust estimates of scene visibility can be applied iteratively to improve depth estimation around object edges. Our algorithm is based entirely on O(1) filters, making it conducive to acceleration and it works with structured or unstructured sets of input views. We compare with recent classical and learning-based algorithms on plenoptic lightfields, wide baseline captures, and lightfield videos produced from camera arrays.

References

[1]
Robert Anderson, David Gallup, Jonathan T. Barron, Janne Kontkanen, Noah Snavely, Carlos Hernández, Sameer Agarwal, and Steven M. Seitz. 2016. Jump: Virtual Reality Video. ACM Trans. Graph. 35, 6, Article 198 (Nov. 2016), 198:1--198:13 pages.
[2]
Chris Buehler, Michael Bosse, Leonard McMillan, Steven Gortler, and Michael Cohen. 2001. Unstructured Lumigraph Rendering. In Proceedings of the 28th Annual Conference on Computer Graphics and Interactive Techniques (SIGGRAPH '01). ACM, New York, NY, USA, 425--432.
[3]
Brian K Cabral. 2016. Introducing Facebook Surround 360: An open, high-quality 3D-360 video capture system. (2016). https://code.facebook.com/posts/1755691291326688
[4]
Gaurav Chaurasia, Sylvain Duchene, Sorkine-Hornung, and Olga Drettakis George. 2013. Depth Synthesis and Local Warps for Plausible Image-based Navigation. ACM Trans. Graph. 32, 3, Article 30 (July 2013), 30:1--30:12 pages.
[5]
Gaurav Chaurasia, Olga Sorkine-Hornung, and George Drettakis. 2011. Silhouette-Aware Warping for Image-Based Rendering. Computer Graphics Forum 30, 4 (2011).
[6]
Shenchang Eric Chen and Lance Williams. 1993. View Interpolation for Image Synthesis. In Proceedings of the 20th Annual Conference on Computer Graphics and Interactive Techniques (SIGGRAPH '93). ACM, 279--288.
[7]
Paul E. Debevec. 1996. Modeling and Rendering Architecture from Photographs. Ph.D. Dissertation. University of California at Berkeley, Computer Science Division, Berkeley CA.
[8]
Pedro F. Felzenszwalb and Daniel P. Huttenlocher. 2006. Efficient Belief Propagation for Early Vision. Int. J. Comput. Vision 70, 1 (Oct. 2006), 41--54.
[9]
John Flynn, Ivan Neulander, James Philbin, and Noah Snavely. 2016. DeepStereo: Learning to Predict New Views From the World's Imagery. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR).
[10]
Yasutaka Furukawa and Carlos Hernández. 2015. Multi-View Stereo: A Tutorial. Foundations and Trends in Computer Graphics and Vision 9, 1--2 (2015), 1--148.
[11]
Yoav HaCohen, Eli Shechtman, Dan B. Goldman, and Dani Lischinski. 2011. Nonrigid Dense Correspondence with Applications for Image Enhancement. In ACM SIGGRAPH 2011 Papers (SIGGRAPH '11). ACM, New York, NY, USA, Article 70, 70:1--70:10 pages.
[12]
Samuel W. Hasinoff, Sing Bing Kang, and Richard Szeliski. 2006. Boundary matting for view synthesis. Computer Vision and Image Understanding 103, 1 (2006), 22--32.
[13]
Kaiming He, Jian Sun, and Xiaoou Tang. 2010. Guided Image Filtering. In Proceedings of the 11th European Conference on Computer Vision: Part I (ECCV'10). Springer-Verlag, Berlin, Heidelberg, 1--14.
[14]
Asmaa Hosni, Michael Bleyer, Christoph Rhemann, Margrit Gelautz, and Carsten Rother. 2011. Real-time Local Stereo Matching Using Guided Image Filtering. In Proceedings of the IEEE International Conference on Multimedia and Expo (ICME 2011). 1--6. Vortrag: IEEE International Conference on Multimedia and Expo (ICME), Barcelona, Spain; 2011-07-11 -- 2011-07-15.
[15]
Hae-Gon Jeon, Jaesik Park, Gyeongmin Choe, Jinsun Park, Yunsu Bok, Yu-Wing Tai, and In So Kweon. 2015. Accurate Depth Map Estimation From a Lenslet Light Field Camera. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR).
[16]
Nima Khademi Kalantari, Ting-Chun Wang, and Ravi Ramamoorthi. 2016. Learning-Based View Synthesis for Light Field Cameras. ACM Transactions on Graphics (Proceedings of SIGGRAPH Asia 2016) 35, 6 (2016).
[17]
Sing Bing Kang, Richard Szeliski, and Jinxiang Chai. 2001. Handling Occlusions in Dense Multi-view Stereo. In CVPR (1). IEEE Computer Society, 103--110.
[18]
V. Kolmogorov and R. Zabih. 2001. Computing visual correspondence with occlusions using graph cuts. In Proceedings Eighth IEEE International Conference on Computer Vision. ICCV 2001, Vol. 2. 508--515 vol.2.
[19]
Johannes Kopf, Fabian Langguth, Daniel Scharstein, Richard Szeliski, and Michael Goesele. 2013. Image-based Rendering in the Gradient Domain. ACM Trans. Graph. 32, 6, Article 199 (Nov. 2013), 199:1--199:9 pages.
[20]
Tom Lokovic and Eric Veach. 2000. Deep Shadow Maps. In Proceedings of the 27th Annual Conference on Computer Graphics and Interactive Techniques (SIGGRAPH '00). ACM Press/Addison-Wesley Publishing Co., New York, NY, USA, 385--392.
[21]
Jiangbo Lu, Hongsheng Yang, Dongbo Min, and Minh N. Do. 2013. Patch Match Filter: Efficient Edge-Aware Filtering Meets Randomized Search for Fast Correspondence Field Estimation. In Proceedings of the 2013 IEEE Conference on Computer Vision and Pattern Recognition (CVPR '13). IEEE Computer Society, Washington, DC, USA, 1854--1861.
[22]
Ziyang Ma, Kaiming He, Yichen Wei, Jian Sun, and Enhua Wu. 2013. Constant Time Weighted Median Filtering for Stereo Matching and Beyond. In Proceedings of the 2013 IEEE International Conference on Computer Vision (ICCV '13). IEEE Computer Society, Washington, DC, USA, 49--56.
[23]
Leonard McMillan and Gary Bishop. 1995. Plenoptic Modeling: An Image-based Rendering System. In Proceedings of the 22Nd Annual Conference on Computer Graphics and Interactive Techniques (SIGGRAPH '95). ACM, New York, NY, USA, 39--46.
[24]
William T. Reeves, David H. Salesin, and Robert L. Cook. 1987. Rendering Antialiased Shadows with Depth Maps. In Proceedings of the 14th Annual Conference on Computer Graphics and Interactive Techniques (SIGGRAPH '87). ACM, New York, NY, USA, 283--291.
[25]
Sudipta Sinha, Drew Steedly, and Rick Szeliski. 2009. Piecewise Planar Stereo for Image-based Rendering, In Twelfth IEEE International Conference on Computer Vision (ICCV 2009).
[26]
Sudipta N. Sinha, Johannes Kopf, Michael Goesele, Daniel Scharstein, and Richard Szeliski. 2012. Image-based Rendering for Scenes with Reflections. ACM Trans. Graph. 31, 4, Article 100 (July 2012), 100:1--100:10 pages.
[27]
Noah Snavely, Steven M. Seitz, and Richard Szeliski. 2006. Photo Tourism: Exploring Photo Collections in 3D. In ACM SIGGRAPH 2006 Papers (SIGGRAPH '06). ACM, New York, NY, USA, 835--846.
[28]
C. Strecha, R. Fransens, and L. Van Gool. 2004. Wide-baseline stereo from multiple views: A probabilistic account. In Proceedings of the 2004 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2004. CVPR 2004., Vol. 1. I-552-I-559 Vol.1.
[29]
Jian Sun, Nan-Ning Zheng, and Heung-Yeung Shum. 2003. Stereo Matching Using Belief Propagation. IEEE Trans. Pattern Anal. Mach. Intell. 25, 7 (July 2003), 787--800.
[30]
Rick Szeliski and Polina Golland. 1999. Stereo Matching with Transparency and Matting. International Journal of Computer Vision 32/1 (May 1999), 45âĂŞ61.
[31]
Michael W. Tao, Sunil Hadap, Jitendra Malik, and Ravi Ramamoorthi. 2013. Depth from Combining Defocus and Correspondence Using light-Field Cameras. International Conference on Computer Vision (ICCV).
[32]
Ting-Chun Wang, Alexei Efros, and Ravi Ramamoorthi. 2015. Occlusion-aware depth estimation using light-field cameras. In Proceedings of the IEEE International Conference on Computer Vision (ICCV).
[33]
Zhou Wang, A. C. Bovik, H. R. Sheikh, and E. P. Simoncelli. 2004. Image Quality Assessment: From Error Visibility to Structural Similarity. Trans. Img. Proc. 13, 4 (April 2004), 600--612.
[34]
Sven Wanner and Bastian Goldluecke. 2014. Variational light field analysis for disparity estimation and super-resolution. 36 (2014), 606--619.
[35]
F. L. Zhang, J. Wang, E. Shechtman, Z. Y. Zhou, J. X. Shi, and S. M. Hu. 2016. PlenoPatch: Patch-based Plenoptic Image Manipulation. IEEE Transactions on Visualization and Computer Graphics PP, 99 (2016), 1--1.
[36]
Enliang Zheng, Enrique Dunn, Vladimir Jojic, and Jan-Michael Frahm. 2014. PatchMatch Based Joint View Selection and Depthmap Estimation. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

Cited By

View all
  • (2025)ProbIBR: Fast Image-Based Rendering With Learned Probability-Guided SamplingIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2024.337215231:3(1888-1901)Online publication date: 1-Mar-2025
  • (2024)光场表征及其分辨率提升技术:文献综述及最新进展(特邀)Infrared and Laser Engineering10.3788/IRLA2024034753:9(20240347)Online publication date: 2024
  • (2024)A Virtual View Acquisition Technique for Complex Scenes of Monocular Images Based on Layered Depth ImagesApplied Sciences10.3390/app14221055714:22(10557)Online publication date: 15-Nov-2024
  • Show More Cited By

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Transactions on Graphics
ACM Transactions on Graphics  Volume 36, Issue 6
December 2017
973 pages
ISSN:0730-0301
EISSN:1557-7368
DOI:10.1145/3130800
Issue’s Table of Contents
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 20 November 2017
Published in TOG Volume 36, Issue 6

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. 3D reconstruction
  2. view synthesis

Qualifiers

  • Research-article

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)534
  • Downloads (Last 6 weeks)68
Reflects downloads up to 17 Feb 2025

Other Metrics

Citations

Cited By

View all
  • (2025)ProbIBR: Fast Image-Based Rendering With Learned Probability-Guided SamplingIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2024.337215231:3(1888-1901)Online publication date: 1-Mar-2025
  • (2024)光场表征及其分辨率提升技术:文献综述及最新进展(特邀)Infrared and Laser Engineering10.3788/IRLA2024034753:9(20240347)Online publication date: 2024
  • (2024)A Virtual View Acquisition Technique for Complex Scenes of Monocular Images Based on Layered Depth ImagesApplied Sciences10.3390/app14221055714:22(10557)Online publication date: 15-Nov-2024
  • (2024)Enhancing endoscopic scene reconstruction with color-aware inverse rendering through neural SDF and radiance fieldsBiomedical Optics Express10.1364/BOE.52161215:6(3914)Online publication date: 23-May-2024
  • (2024)Virtual viewpoint image synthesis using neural radiance fields with depth information supervisionJournal of Image and Graphics10.11834/jig.22118829:7(2035-2045)Online publication date: 2024
  • (2024)Superpixel-based Efficient Sampling for Learning Neural Fields from Large InputProceedings of the 32nd ACM International Conference on Multimedia10.1145/3664647.3681299(10421-10430)Online publication date: 28-Oct-2024
  • (2024)Refracting Once is Enough: Neural Radiance Fields for Novel-View Synthesis of Real Refractive ObjectsProceedings of the 2024 International Conference on Multimedia Retrieval10.1145/3652583.3658000(694-703)Online publication date: 30-May-2024
  • (2024)Factorized Motion Fields for Fast Sparse Input Dynamic View SynthesisACM SIGGRAPH 2024 Conference Papers10.1145/3641519.3657498(1-12)Online publication date: 13-Jul-2024
  • (2024)4D-Rotor Gaussian Splatting: Towards Efficient Novel View Synthesis for Dynamic ScenesACM SIGGRAPH 2024 Conference Papers10.1145/3641519.3657463(1-11)Online publication date: 13-Jul-2024
  • (2024)TRIPS: Trilinear Point Splatting for Real‐Time Radiance Field RenderingComputer Graphics Forum10.1111/cgf.1501243:2Online publication date: 30-Apr-2024
  • Show More Cited By

View Options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Login options

Full Access

Figures

Tables

Media

Share

Share

Share this Publication link

Share on social media