research-article

Stereoview to Multiview Conversion Architecture for Auto-Stereoscopic 3D Displays

Authors:

Huan-Kai Tseng,

Jar-Ferr YangAuthors Info & Claims

IEEE Transactions on Circuits and Systems for Video Technology, Volume 28, Issue 11

Pages 3274 - 3287

https://doi.org/10.1109/TCSVT.2017.2732061

Published: 01 November 2018 Publication History

Abstract

In this paper, a stereoview to multiview conversion system, which includes stereo matching and depth image-based rendering (DIBR) hardware designs, is proposed. To achieve an efficient architecture, the proposed stereo matching algorithm simply generates the raw matching costs and aggregates cost based on 1D iterative aggregation schemes. For the DIBR architecture, an inpainting-based method is used to find the most similar patch from the background, according to depth information. The simulation results show that the designed architecture achieves an averaged peak signal-to-noise ratio of 30.2 dB and structure similarity of 0.94 for the tested images. The hardware design for the proposed 2D to 3D conversion system operates at a maximum clock frequency of 160.2 MHz for outputting 1080p (<inline-formula> <tex-math notation="LaTeX">$1920 \times 1080$ </tex-math></inline-formula>) video at 60 frames per second.

References

[1]

C. Fehn, K. Hopf, and B. Quante, “Key technologies for an advanced 3D TV system,” Proc. SPIE, vol. 5599, pp. 66–80, Dec. 2004.

[2]

C. Fehn, R. de la Barre, and S. Pastoor, “Interactive 3-DTV—Concepts and key technologies,” Proc. IEEE, vol. 94, no. 3, pp. 524–538, Mar. 2006.

[3]

O. Stankiewicz, K. Wegner, M. Tanimoto, and M. Domański, Enhanced Depth Estimation Reference Software (DERS) for Free-Viewpoint Television, document ISO/IEC JTC 1/SC 29/WG11, M31518, Oct. 2013.

[4]

O. Stankiewicz, K. Wegner, M. Tanimoto, and M. Domański, Enhanced View Synthesis Reference Software (VSRS) for Free-Viewpoint Television, document ISO/IEC JTC 1/SC 29/WG11, M31520, Oct. 2013.

[5]

D. Kim, D. Min, and K. Sohn, “A stereoscopic video generation method using stereoscopic display characterization and motion analysis,” IEEE Trans. Broadcast., vol. 54, no. 2, pp. 188–197, Jun. 2008.

[6]

J. Lee, S. Yoo, C. Kim, and B. Vasudev, “Estimating scene-oriented pseudo depth with pictorial depth cues,” IEEE Trans. Broadcast., vol. 59, no. 2, pp. 238–250, Jun. 2013.

[7]

M. T. Pourazad, P. Nasiopoulos, and R. K. Ward, “An H.264-based scheme for 2D to 3D video conversion,” IEEE Trans. Consum. Electron., vol. 55, no. 2, pp. 742–748, May 2009.

Digital Library

[8]

Y. Feng, J. Ren, and J. Jiang, “Object-based 2D-to-3D video conversion for effective stereoscopic content generation in 3D-TV applications,” IEEE Trans. Broadcast., vol. 57, no. 2, pp. 500–509, Jun. 2011.

[9]

K.-J. Yoon and I. S. Kweon, “Adaptive support-weight approach for correspondence search,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 28, no. 4, pp. 650–656, Apr. 2006.

Digital Library

[10]

T. Kanade and M. Okutomi, “A stereo matching algorithm with an adaptive window: Theory and experiment,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 16, no. 9, pp. 920–932, Sep. 1994.

Digital Library

[11]

Y. Boykov, O. Veksler, and R. Zabih, “A variable window approach to early vision,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 20, no. 12, pp. 1283–1294, Dec. 1998.

Digital Library

[12]

O. Veksler, “Stereo correspondence with compact windows via minimum ratio cycle,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 24, no. 12, pp. 1654–1660, Dec. 2002.

Digital Library

[13]

K. Zhang, J. Lu, and G. Lafruit, “Cross-based local stereo matching using orthogonal integral images,” IEEE Trans. Circuits Syst. Video Technol., vol. 19, no. 7, pp. 1073–1079, Jul. 2009.

Digital Library

[14]

Q. Yang, D. Li, L. Wang, and M. Zhang, “Full-image guided filtering for fast stereo matching,” IEEE Signal Process. Lett., vol. 20, no. 3, pp. 237–240, Mar. 2013.

[15]

C. C. Pham and J. W. Jeon, “Domain transformation-based efficient cost aggregation for local stereo matching,” IEEE Trans. Circuits Syst. Video Technol., vol. 23, no. 7, pp. 1119–1130, Jul. 2013.

Digital Library

[16]

M. C. Yang, J. Y. Liu, Y. C. Yang, and K. H. Chen, “A quality-improved stereo matching by using incrementally-averaging orthogonally-scanned edge detection,” in Proc. 3D Syst. Appl. (3DSA), Hsinchu, Taiwan, 2012, pp. 489–491.

[17]

N. Y.-C. Chang, T.-H. Tsai, B.-H. Hsu, Y.-C. Chen, and T.-S. Chang, “Algorithm and architecture of disparity estimation with mini-census adaptive support weight,” IEEE Trans. Circuits Syst. Video Technol., vol. 20, no. 6, pp. 792–805, Jun. 2010.

Digital Library

[18]

S. Jinet al., “FPGA design and implementation of a real-time stereo vision system,” IEEE Trans. Circuits Syst. Video Technol., vol. 20, no. 1, pp. 15–26, Jan. 2010.

Digital Library

[19]

K. Ambrosch and W. Kubinger, “Accurate hardware-based stereo vision,” Comput. Vis. Image Understand., vol. 114, no. 11, pp. 1303–1316, Nov. 2010.

Digital Library

[20]

D.-W. Yang, L.-C. Chu, C.-W. Chen, J. Wang, and M.-D. Shieh, “Depth-reliability-based stereo-matching algorithm and its vlsi architecture design,” IEEE Trans. Circuits Syst. Video Technol., vol. 25, no. 6, pp. 1038–1050, Jun. 2015.

[21]

K. Ambrosch, M. Humenberger, W. Kubinger, and A. Steininger, “SAD-based stereo matching using FPGAs,” in Embedded Computer Vision, B. Kisacanin, S. Bhattacharyya, S. Chai, Eds. London, U.K.: Springer, 2009, pp. 121–138.

[22]

C. Georgoulas and I. Andreadis, “A real-time occlusion aware hardware structure for disparity map computation,” in Proc. Int. Conf. Image Anal. Process. (ICIAP), Vietri sul Mare, Italy, Sep. 2009, pp. 721–730.

[23]

A. Darabiha, W. J. MacLean, and J. Rose, “Reconfigurable hardware implementation of a phase-correlation stereoalgorithm,” Mach. Vis. Appl., vol. 17, no. 2, pp. 116–132, May 2006.

Digital Library

[24]

C. Ttofis and T. Theocharides, “Towards accurate hardware stereo correspondence: A real-time FPGA implementation of a segmentation-based adaptive support weight algorithm,” in Proc. IEEE Conf. Design, Autom. Test. (DATE), Dresden, Germany, Mar. 2012, pp. 703–708.

[25]

W. Q. Wang, J. Yan, N. Xu, Y. Wang, and F.-H. Hsu, “Real-time high-quality stereo vision system in FPGA,” IEEE Trans. Circuits Syst. Video Technol., vol. 25, no. 10, pp. 1696–1708, Oct. 2015.

Digital Library

[26]

A. Aysu, M. Sayinta, and C. Cigla, “Low cost FPGA design and implementation of a stereo matching system for 3D-TV applications,” in Proc. IEEE Conf. Very Large Scale Integr. (VLSI-SoC), Istanbul, Turkey, Oct. 2013, pp. 204–209.

[27]

P. C. Kuo, P. L. Chu, B.-D. Liu, and J. F. Yang, “A stereo matching algorithm with fast disparity propagation under homogeneous texture detection,” in Proc. Int. Conf. 3D Syst. Appl. (3DSA), Osaka, Japan., Jun. 2013, pp. 1–4.

[28]

H. W. Ho, P. C. Kuo, C. C. Tien, B. D. Liu, and J. F. Yang, “A fast stereo matching algorithm with simple edge detection,” in Proc. VLSI Design/CAD Symp. (VLSI/CAD), Taichung, Taiwan, Aug. 2014, pp. 1500–1505.

[29]

C. Fehn, “Depth-image-based rendering (DIBR), compression, and transmission for a new approach on 3D-TV,” Proc. SPIE, vol. 5291, pp. 93–104, Jan. 2004.

[30]

Y.-C. Fan and T.-C. Chi, “The novel non-hole-filling approach of depth image based rendering,” in Proc. IEEE 3DTV Conf. (3DTV-CON), Istanbul, Turkey, May 2008, pp. 325–328.

[31]

C. Fehn, “A 3D-TV approach using depth-image-based rendering (DIBR),” in Proc. Vis., Imag., Image Process. (VIIP), Benalmádena, Spain, Sep. 2003, pp. 482–487.

[32]

M. Solh and G. AlRegib, “Hierarchical hole-filling(HHF): Depth image based rendering without depth map filtering for 3D-TV,” in Proc. IEEE Int. Workshop Multimedia Signal Process. (MMSP), Saint-Malo, France, Oct. 2010, pp. 87–92.

[33]

M. S. Ko, D. W. Kim, D. L. Jones, and J. Yoo, “A new common-hole filling algorithm for arbitrary view synthesis,” in Proc. 3D Syst. Appl. (3DSA), Hsinchu, Taiwan, 2012, pp. 242–245.

[34]

T. Tezuka, M. P. Tehrani, K. Suzuki, K. Takahashi, and T. Fujii, “View synthesis using superpixel based inpainting capable of occlusion handling and hole filling,” in Proc. Picture Coding Symp. (PCS), Cairns, QLD, Australia, May 2015, pp. 124–128.

[35]

S. P. Lu, J. Hanca, A. Munteanu, and P. Schelkens, “Depth-based view synthesis using pixel-level image inpainting,” in Proc. Int. Conf. Digit. Signal Process. (DSP), Fira, Greek, Jul. 2013, pp. 1–6.

[36]

H. G. Kim and Y. M. Ro, “Multiview stereoscopic video hole filling considering spatiotemporal consistency and binocular symmetry for synthesized 3D Video,” IEEE Trans. Circuits Syst. Video Technol., vol. 27, no. 7, pp. 1435–1449, Jul. 2017.

Digital Library

[37]

M. Köppel, K. Müller, and T. Wiegand, “Filling disocclusions in extrapolated virtual views using hybrid texture synthesis,” IEEE Trans. Broadcast., vol. 62, no. 2, pp. 457–469, Jun. 2016.

[38]

I. Daribo and H. Saito, “A novel inpainting-based layered depth video for 3DTV,” IEEE Trans. Broadcast., vol. 57, no. 2, pp. 533–541, Jun. 2011.

[39]

P. Ndjiki-Nyaet al., “Depth image-based rendering with advanced texture synthesis for 3-D video,” IEEE Trans. Multimedia, vol. 13, no. 3, pp. 453–465, Jun. 2011.

Digital Library

[40]

A. Criminisi, P. Pérez, and K. Toyama, “Region filling and object removal by exemplar-based image inpainting,” IEEE Trans. Image Process., vol. 13, no. 9, pp. 1200–1212, Sep. 2004.

Digital Library

[41]

H. Shan, W.-D. Chien, H.-M. Wang, and J.-F. Yang, “A homography-based inpainting algorithm for effective depth image based rendering,” in Proc. IEEE Int. Conf. Image Process. (ICIP), Paris, France, Oct. 2014, pp. 5412–5416.

[42]

S. Reel, G. Cheung, P. Wong, and L. S. Dooley, “Joint texture-depth pixel inpainting of disocclusion holes in virtual view synthesis,” in Proc. Asia–Pacific Signal Inf. Process. Assoc. Annu. Summit Conf. (APSIPA ASC), Kaohsiung, Taiwan, Oct. 2013, pp. 1–7.

[43]

T.-C. Yang, P.-C. Kuo, B.-D. Liu, and J.-F. Yang, “Depth image-based rendering with edge-oriented hole filling for multiview synthesis,” in Proc. Int. Conf. Commun. Circuit Syst. (ICCCAS), Taichung, Taiwan, Aug. 2012, pp. 50–53.

[44]

P. J. Lee and C.-H. Su, “Nongeometric distortion smoothing approach for depth map preprocessing,” IEEE Trans. Multimedia, vol. 13, no. 2, pp. 246–254, Apr. 2011.

Digital Library

[45]

L. Zhang and W. J. Tam, “Stereoscopic image generation based on depth images for 3DTV,” IEEE Trans. Broadcast., vol. 51, no. 2, pp. 191–199, Jun. 2005.

[46]

W. J. Tam, G. Alain, L. Zhang, T. Martin, and R. Renaud, “Smoothing depth maps for improved steroscopic image quality,” Proc. SPIE, vol. 5599, pp. 162–172, Dec. 2004.

[47]

C. Cheng, J. Liu, H. Yuan, X. Yang, and W. Liu, “A DIBR method based on inverse mapping and depth-aided image inpainting,” in Proc. IEEE China Summit Int. Conf. Signal Inf. Process. (ChinaSIP), Beijing, China, Jul. 2013, pp. 518–522.

[48]

I. Ahn and C. Kim, “Depth-based disocclusion filling for virtual view synthesis,” in Proc. IEEE Int. Conf. Multimedia Expo Workshops (ICMEW), Melbourne, VIC, Australia, Jul. 2012, pp. 109–114.

[49]

K. Luo, D.-X. Li, Y.-M. Feng, and M. Zhang, “Depth-aided inpainting for disocclusion restoration of multi-view images using depth-image-based rendering,” J. Zhejiang Univ.-Sci. A, vol. 10, no. 12, pp. 1738–1749, Dec. 2009.

[50]

L. Ma, L. Do, and P. H. N. de With, “Depth-guided inpainting algorithm for free-viewpoint video,” in Proc. IEEE Int. Conf. Image Process. (ICIP), Orlando, FL, USA, Jul. 2012, pp. 1721–1724.

[51]

O. L. Meur, J. Gautier, and C. Guillemot, “Examplar-based inpainting based on local geometry,” in Proc. IEEE Int. Conf. Image Process. (ICIP), Brussels, Belgium, Sep. 2011, pp. 3401–3404.

[52]

P.-C. Kuo, J.-M. Lin, B.-D. Liu, and J.-F. Yang, “Inpainting-based multi-view synthesis algorithms and its GPU accelerated implementation,” in Proc. Int. Conf. Inf. Commun. Circuits Syst. (ICICS), Tainan, Taiwan, Dec. 2013, pp. 1–4.

[53]

P.-C. Kuo, J.-M. Lin, B.-D. Liu, and J.-F. Yang, “High efficiency depth image-based rendering with simplified inpainting-based hole filling,” Multidimensional Syst. Signal Process., vol. 27, no. 3, pp. 623–645, Jul. 2016.

Digital Library

[54]

D. Scharstein, R. Szeliski, and R. Zabih, “A taxonomy and evaluation of dense two-frame stereo correspondence algorithms,” in Proc. IEEE Workshop Stereo Multi-Baseline Vis. (SMBV), Kauai, HI, USA, Dec. 2001, pp. 131–140.

[55]

D. Scharstein and R. Szeliski, “High-accuracy stereo depth maps using structured light,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR), Madison, WI, USA, Jun. 2003, pp. 195–202.

[56]

D. Scharstein and C. Pal, “Learning conditional random fields for stereo,” in Proc. IEEE Conf. Comput. Vis. Pattern Recogn. (CVPR), Minneapolis, MN, USA, Jun. 2007, pp. 1–8.

[57]

Middlebury Stereo Vision Page. [Online]. Available: http://vision.middlebury.edu/stereo

[58]

M. Schaffner, F. K. Gürkaynak, P. Greisen, H. Kaeslin, L. Benini, and A. Smolic, “Hybrid ASIC/FPGA system for fully automatic stereo-to-multiview conversion using IDW,” IEEE Trans. Circuits Syst. Video Technol., vol. 26, no. 11, pp. 2093–2108, Nov. 2015.

[59]

N. Stefanoski, O. Wang, M. Lang, P. Greisen, S. Heinzle, and A. Smolic, “Automatic view synthesis by image-domain-warping,” IEEE Trans. Image Process., vol. 22, no. 9, pp. 3329–3341, Sep. 2013.

Digital Library

[60]

S.-J. Yao, P.-F. Jin, H. Fu, L.-H. Wang, D.-X. Li, and M. Zhang, “Real-time 3DTV system for autostereoscopic displays,” in Proc. Int. Conf. Audio Lang. Image Process. (ICALIP), Shanghai, China, Jul. 2012, pp. 621–626.

[61]

A. Akin, R. Capoccia, J. Narinx, J. Masur, A. Schmid, and Y. Leblebici, “Real-time free viewpoint synthesis using three-camera disparity estimation hardware,” in Proc. IEEE Int. Symp. Circuits Syst., Lisbon, Portugal, May 2015, pp. 2525–2528.

Cited By

Chan DChiu TWu X(2022)A causality-attentive stereo matching method for shape-preserved depth mapMultidimensional Systems and Signal Processing10.1007/s11045-022-00838-833:4(1203-1219)Online publication date: 1-Dec-2022
https://dl.acm.org/doi/10.1007/s11045-022-00838-8
Huang CHuang YChan DYang J(2020)Shape-reserved stereo matching with segment-based cost aggregation and dual-path refinementJournal on Image and Video Processing10.1186/s13640-020-00525-32020:1Online publication date: 7-Sep-2020
https://dl.acm.org/doi/10.1186/s13640-020-00525-3

Index Terms

Stereoview to Multiview Conversion Architecture for Auto-Stereoscopic 3D Displays
1. Computing methodologies
  1. Artificial intelligence
    1. Computer vision
      1. Image and video acquisition
  2. Computer graphics
2. Hardware

Index terms have been assigned to the content through auto-classification.

Recommendations

Interactive Stereoscopic Video Conversion

This paper presents a system of converting conventional monocular videos to stereoscopic ones. In the system, an input monocular video is firstly segmented into shots so as to reduce operations on similar frames. An automatic depth estimation method is ...
An interactive system of stereoscopic video conversion
MM '12: Proceedings of the 20th ACM international conference on Multimedia

With the recent booming of 3DTV industry, more and more stereoscopic videos are demanded by the market. This paper presents a system of converting conventional monocular videos to stereoscopic ones. In this system, an input video is firstly segmented ...
Key-frame-based depth propagation for semi-automatic stereoscopic video conversion

A key-frame-based bi-directional depth propagation algorithm.Multi-pass error correction to prevent depth artifacts from being further propagated.Better results against prior works in depth map quality and synthesized stereo views. In this paper, we ...

Comments

Information & Contributors

Information

Published In

cover image IEEE Transactions on Circuits and Systems for Video Technology

IEEE Transactions on Circuits and Systems for Video Technology Volume 28, Issue 11

Nov. 2018

221 pages

ISSN:1051-8215

Issue’s Table of Contents

1051-8215 © 2017 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.

Publisher

IEEE Press

Publication History

Published: 01 November 2018

Qualifiers

Research-article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

2
Total Citations
View Citations
0
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 01 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Chan DChiu TWu X(2022)A causality-attentive stereo matching method for shape-preserved depth mapMultidimensional Systems and Signal Processing10.1007/s11045-022-00838-833:4(1203-1219)Online publication date: 1-Dec-2022
https://dl.acm.org/doi/10.1007/s11045-022-00838-8
Huang CHuang YChan DYang J(2020)Shape-reserved stereo matching with segment-based cost aggregation and dual-path refinementJournal on Image and Video Processing10.1186/s13640-020-00525-32020:1Online publication date: 7-Sep-2020
https://dl.acm.org/doi/10.1186/s13640-020-00525-3

View Options

View options

Media

Figures

Other

Tables

View Issue’s Table of Contents