research-article

A new free viewpoint video dataset and DIBR benchmark

Authors:

Li SongAuthors Info & Claims

MMSys '22: Proceedings of the 13th ACM Multimedia Systems Conference

Pages 265 - 271

https://doi.org/10.1145/3524273.3532897

Published: 05 August 2022 Publication History

Abstract

Free viewpoint video (FVV) has drawn great attention in recent years, which provides viewers with strong interactive and immersive experience. Despite the developments made, further progress of FVV research is limited by existing datasets that mostly have too few number of camera views, or static scenes. To overcome the limitations, in this paper, we present a new dynamic RGB-D video dataset with up to 12 views. Our dataset consists of 13 groups of dynamic video sequences that are taken at the same scene, and a group of video sequences of the empty scene. Each group has 12 HD video sequences taken by synchronized cameras and 12 correspondingly estimated depth video sequences. Moreover, we also introduce a FVV synthesis benchmark on the basis of depth image based rendering (DIBR) to help researchers validate their data-driven methods. We hope our work will inspire more FVV synthesis methods with enhanced robustness, improved performance and deeper understanding.

References

[1]

Henrik Aanæs, Rasmus Ramsbøl Jensen, George Vogiatzis, Engin Tola, and Anders Bjorholm Dahl. 2016. Large-scale data for multiple-view stereopsis. International Journal of Computer Vision 120, 2 (2016), 153--168.

Digital Library

[2]

Aayush Bansal, Minh Vo, Yaser Sheikh, Deva Ramanan, and Srinivasa Narasimhan. 2020. 4d visualization of dynamic events from unconstrained multi-view videos. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 5366--5375.

[3]

M Domański, A Dziembowski, M Kurc, A Łuczak, D Mieloch, J Siast, O Stankiewicz, and K Wegner. 2015. Poznan University of Technology test multiview video sequences acquired with circular camera arrangement'Poznan Team'and'Poznan Blocks' sequences. ISO/IEC JTC1/SC29/WG11, Doc. MPEG M 35846 (2015).

[4]

Marek Domaski, T. Grajek, K. Klimaszewski, M. Kurc, and K. Wegner. 2009. Contribution Poznań Multiview Video Test Sequences and Camera Parameters. (2009).

[5]

M Domański, A. Dziembowski, T. Grajek, A. Grzelka, and K. Wegner. 2015. [FTV AHG] Video and depth multiview test sequences acquired with circular camera arrangement - "Poznan Service" and "Poznan People". (2015).

[6]

M Domański, A. Dziembowski, A. Grzelka, D. Mieloch, and K. Wegner. 2016. Multiview test video sequences for free navigation exploration obtained using pairs of cameras. In ISO/IEC JTC1/SC29/WG11 MPEG2016/ m38247.

[7]

Christoph Fehn. 2004. Depth-image-based rendering (DIBR), compression, and transmission for a new approach on 3D-TV. In Stereoscopic Displays and Virtual Reality Systems XI, Vol. 5291. International Society for Optics and Photonics, 93--104.

[8]

John Flynn, Michael Broxton, Paul Debevec, Matthew DuVall, Graham Fyffe, Ryan Overbeck, Noah Snavely, and Richard Tucker. 2019. Deepview: View synthesis with learned gradient descent. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2367--2376.

[9]

Arno Knapitsch, Jaesik Park, Qian-Yi Zhou, and Vladlen Koltun. 2017. Tanks and Temples: Benchmarking Large-Scale Scene Reconstruction. ACM Transactions on Graphics 36, 4 (2017).

Digital Library

[10]

Chuen-Chien Lee, Ali Tabatabai, and Kenji Tashiro. 2015. Free viewpoint video (FVV) survey and future research direction. APSIPA Transactions on Signal and Information Processing 4 (2015).

[11]

Chuen-Chien Lee, Ali Tabatabai, and Kenji Tashiro. 2015. Free viewpoint video (FVV) survey and future research direction. APSIPA Transactions on Signal and Information Processing 4 (2015).

[12]

Qinbo Li and Nima Khademi Kalantari. 2020. Synthesizing light field from a single image with variable MPI and two network fusion. ACM Trans. Graph. 39, 6 (2020), 229--1.

Digital Library

[13]

Kai-En Lin, Lei Xiao, Feng Liu, Guowei Yang, and Ravi Ramamoorthi. 2021. Deep 3D Mask Volume for View Synthesis of Dynamic Scenes. In Proceedings of the IEEE/CVF International Conference on Computer Vision. 1749--1758.

[14]

Ben Mildenhall, Pratul P Srinivasan, Rodrigo Ortiz-Cayon, Nima Khademi Kalantari, Ravi Ramamoorthi, Ren Ng, and Abhishek Kar. 2019. Local light field fusion: Practical view synthesis with prescriptive sampling guidelines. ACM Transactions on Graphics (TOG) 38, 4 (2019), 1--14.

Digital Library

[15]

OpenCV. 2022. Camera Calibration. https://docs.opencv.org/3.4/dc/dbb/tutorial_py_calibration.html.

[16]

Néill O'Dwyer, Jan Ondřej, Rafael Pagés, Konstantinos Amplianitis, and Aljoša Smolić. 2018. Jonathan Swift: augmented reality application for Trinity library's long room. In International Conference on Interactive Digital Storytelling. Springer, 348--351.

Digital Library

[17]

Rafael Pagés, Konstantinos Amplianitis, David Monaghan, Jan Ondřej, and Aljosa Smolić. 2018. Affordable content creation for free-viewpoint video and VR/AR applications. Journal of Visual Communication and Image Representation 53 (2018), 192--201.

[18]

Gernot Riegler and Vladlen Koltun. 2020. Free view synthesis. In European Conference on Computer Vision. Springer, 623--640.

Digital Library

[19]

Johannes Lutz Schönberger and Jan-Michael Frahm. 2016. Structure-from-Motion Revisited. In Conference on Computer Vision and Pattern Recognition (CVPR).

[20]

Johannes Lutz Schönberger, Enliang Zheng, Marc Pollefeys, and Jan-Michael Frahm. 2016. Pixelwise View Selection for Unstructured Multi-View Stereo. In European Conference on Computer Vision (ECCV).

[21]

Thomas Schops, Johannes L Schonberger, Silvano Galliani, Torsten Sattler, Konrad Schindler, Marc Pollefeys, and Andreas Geiger. 2017. A multi-view stereo benchmark with high-resolution images and multi-camera videos. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 3260--3269.

[22]

Jangwoo Son, Serhan Gül, Gurdeep Singh Bhullar, Gabriel Hege, Wieland Morgenstern, Anna Hilsmann, Thomas Ebner, Sven Bliedung, Peter Eisert, Thomas Schierl, et al. 2020. Split Rendering for Mixed Reality: Interactive Volumetric Video in Action. In SIGGRAPH Asia 2020 XR. 1--3.

[23]

Olgierd Stankiewicz, Marek Domański, Adrian Dziembowski, Adam Grzelka, Dawid Mieloch, and Jarosław Samelak. 2018. A free-viewpoint television system for horizontal virtual navigation. IEEE Transactions on Multimedia 20, 8 (2018), 2182--2195.

[24]

Yanru Wang, Zhihao Huang, Hao Zhu, Wei Li, Xun Cao, and Ruigang Yang. 2020. Interactive free-viewpoint video generation. Virtual Reality & Intelligent Hardware 2, 3 (2020), 247--260.

[25]

Wenqi Xian, Jia-Bin Huang, Johannes Kopf, and Changil Kim. 2021. Space-time neural irradiance fields for free-viewpoint video. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 9421--9431.

[26]

Jae Shin Yoon, Kihwan Kim, Orazio Gallo, Hyun Soo Park, and Jan Kautz. 2020. Novel view synthesis of dynamic scenes with globally coherent depths from a monocular camera. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 5336--5345.

[27]

Liang Zhang, Wa James Tam, and Demin Wang. 2004. Stereoscopic image generation based on depth images. In 2004 International Conference on Image Processing, 2004. ICIP'04., Vol. 5. IEEE, 2993--2996.

[28]

Tinghui Zhou, Richard Tucker, John Flynn, Graham Fyffe, and Noah Snavely. 2018. Stereo magnification: Learning view synthesis using multiplane images. arXiv preprint arXiv:1805.09817 (2018).

[29]

C Lawrence Zitnick, Sing Bing Kang, Matthew Uyttendaele, Simon Winder, and Richard Szeliski. 2004. High-quality video view interpolation using a layered representation. ACM transactions on graphics (TOG) 23, 3 (2004), 600--608.

Cited By

Guo SHu JZhou KWang JSong LXie RZhang W(2024)Real-Time Free Viewpoint Video Synthesis System Based on DIBR and a Depth Estimation NetworkIEEE Transactions on Multimedia10.1109/TMM.2024.335563926(6701-6716)Online publication date: 18-Jan-2024
https://dl.acm.org/doi/10.1109/TMM.2024.3355639
Lu JLu BXu JSong LZhang W(2024)A Priority Aware Free Viewpoint Video Transmit Scheme Based on QUIC2024 IEEE International Symposium on Broadband Multimedia Systems and Broadcasting (BMSB)10.1109/BMSB62888.2024.10608226(1-6)Online publication date: 19-Jun-2024
https://doi.org/10.1109/BMSB62888.2024.10608226

Index Terms

A new free viewpoint video dataset and DIBR benchmark
1. Computer systems organization
  1. Dependable and fault-tolerant systems and networks
    1. Redundancy
  2. Embedded and cyber-physical systems
    1. Robotics
2. Networks
  1. Network properties
    1. Network reliability

Recommendations

Real-Time Layered View Synthesis for Free-Viewpoint Video from Unreliable Depth Information
MMVE '23: Proceedings of the 15th International Workshop on Immersive Mixed and Virtual Environment Systems

In this work we present a novel approach for the generation in real time of synthetic views for free-viewpoint video. Our system is based on purely passive stereo cameras which, under the constraints of real-time operation yield unreliable depth maps, ...
Free viewpoint video generation for walk-through experience using image-based rendering
MM '08: Proceedings of the 16th ACM international conference on Multimedia

This paper presents a novel method to represent a real 3D world using IBR (Image-based Rendering) technology. The major achievement is realization of "walk-through" experience, in which audiences can see the scene of sport games as if they were the ...
Free-viewpoint video rendering for mobile devices
MIRAGE '13: Proceedings of the 6th International Conference on Computer Vision / Computer Graphics Collaboration Techniques and Applications

Free-viewpoint video renderers (FVVR) allow a user to view captured video footage from any position and direction. Despite the obvious appeal of such systems, they have yet to make a major impact on digital entertainment. Current FVVR implementations ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

MMSys '22: Proceedings of the 13th ACM Multimedia Systems Conference

June 2022

432 pages

ISBN:9781450392839

DOI:10.1145/3524273

General Chairs:
Niall Murray
Technological University of the Shannon: Midlands Midwest
,
Gwendal Simon
Synamedia
,
Mylene Farias
University of Brasilia
,
Program Chairs:
Irene Viola
Centrum Wiskunde & Informatica
,
Mario Montagud
i2CAT Foundation & University of Valencia

Copyright © 2022 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGMM: ACM Special Interest Group on Multimedia

In-Cooperation

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 05 August 2022

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

Shanghai Key Laboratory of Digital Media Processing and Transmissions
111 project
National Key R&D Project of China
MoE-China Mobile Research Fund Project

Conference

MMSys '22

Sponsor:

SIGMM

MMSys '22: 13th ACM Multimedia Systems Conference

June 14 - 17, 2022

Athlone, Ireland

Acceptance Rates

Overall Acceptance Rate 176 of 530 submissions, 33%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

2
Total Citations
View Citations
212
Total Downloads

Downloads (Last 12 months)43
Downloads (Last 6 weeks)2

Reflects downloads up to 01 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Guo SHu JZhou KWang JSong LXie RZhang W(2024)Real-Time Free Viewpoint Video Synthesis System Based on DIBR and a Depth Estimation NetworkIEEE Transactions on Multimedia10.1109/TMM.2024.335563926(6701-6716)Online publication date: 18-Jan-2024
https://dl.acm.org/doi/10.1109/TMM.2024.3355639
Lu JLu BXu JSong LZhang W(2024)A Priority Aware Free Viewpoint Video Transmit Scheme Based on QUIC2024 IEEE International Symposium on Broadband Multimedia Systems and Broadcasting (BMSB)10.1109/BMSB62888.2024.10608226(1-6)Online publication date: 19-Jun-2024
https://doi.org/10.1109/BMSB62888.2024.10608226

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents