research-article

Visual Comfort for Stereoscopic 3D by Using Motion Sensors on 3D Mobile Devices

Author:

Chung-Hua ChuAuthors Info & Claims

ACM Transactions on Multimedia Computing, Communications, and Applications (TOMM), Volume 12, Issue 1s

Article No.: 14, Pages 1 - 20

https://doi.org/10.1145/2808211

Published: 21 October 2015 Publication History

Abstract

Advanced 3D mobile devices attract a lot of attentions for 3D visualization nowadays. Stereoscopic images and video taken from the 3D mobile devices are uncomfortable for 3D viewing experiences due to the limited hardware for stereoscopic 3D stabilization. The existing stereoscopic 3D stabilization methods are computationally inefficient for the 3D mobile devices. In this article, we point out that this critical issue deteriorates the 3D viewing experiences on the 3D mobile devices. To improve visual comfort, we propose an efficient and effective algorithm to stabilize the stereoscopic images and video for the 3D mobile devices. To rectify the video jitter, we use the gyroscope and accelerometer embedded on the mobile devices to obtain the geometry information of the cameras. Using a different method than video-content-based motion estimation, our algorithm based on the gyroscope and acceleration data can achieve higher accuracy to effectively stabilize the video. Therefore, our approach is robust in video stabilization even under poor lighting and substantial foreground motion. Our algorithm outperforms previous approaches in not only smaller running time but also the better comfort of the stereoscopic 3D visualization for the 3D mobile devices.

References

[1]

M. Abramowitz and I. A. Stegun. 1972. Handbook of Mathematical Functions. Dover Publications, New York, 72--89.

[2]

R. S. Allison. 2007. Analysis of the influence of vertical disparities arising in toed-in stereoscopic cameras. J. Imag. Sci. Technol. 51, 4, 317--327.

[3]

Pravin Bhat, C. Lawrence Zitnick, Noah Snavely, Aseem Agrawala, Michael Cohen, Brian Curless, and Sing Bing Kang. 2007. Using photographs to enhance videos of a static scene. In Proceedings of the 18^th Eurographics Conference on Rendering Techniques (EGSR'07). Eurographics Association, Aire-la-Ville, Switzerland, 327--338.

Digital Library

[4]

Piotr Didyk, Tobias Ritschel, Elmar Eisemann, Karol Myszkowski, and Hans-Peter Seidel. 2011. A perceptual model for disparity. ACM Trans. Graphics 30, 4 (2011), Article 96.

Digital Library

[5]

L. Falkenhagen. 1994. Depth estimation from stereoscopic image pairs assuming piecewise continuous surfaces. In Image Processing for Broadcast and Video Production, Springer, 115--127.

[6]

M. A. Fischler, and R. C. Bolles. 1981. Random sample consensus: A paradigm for model fitting with applications to image analysis and automated cartography. Comm. ACM 24, 6, 381--395.

Digital Library

[7]

Simon Heinzle, Pierre Greisen, David Gallup, Christine Chen, Daniel Saner, Aljoscha Smolic, Andreas Burg, Wojciech Matusik, and Markus Gross. 2011. Computational stereo camera system with programmable control loop. ACM Trans. Graphics 30, 4 (2011), Article 94.

Digital Library

[8]

T. S. Huang and A. N. Netravali. 1994. Motion and structure from feature correspondences: A review. Proc. IEEE 82, 2 (1994), 252--268.

[9]

S. Jain and U. Neumann. 2006. Real-time camera pose and focal length estimation. In Proceedings of the IEEE International Conference on Pattern Recognition. IEEE, 551--555.

Digital Library

[10]

S. B. Kang. 1999. A survey of image-based rendering techniques. Ph.D dissertation, University of North Carolina at Chapel Hill. In VideoMetrics, SPIE, 2--16.

[11]

Bahadir Karasulu and Sendar Korukoglu. 2013. Performance Evaluation Software: Moving Object Detection and Tracking in Videos. SpringerBriefs in Computer Science, Springer, 63--70.

Digital Library

[12]

Frank L. Kooi and Alexander Toet. 2004. Visual comfort of binocular and 3D displays. Displays 25, 2--3 (2004), 99--108.

[13]

Marc Lambooij, Wijnand IJsselsteijn, Marten Fortuin, and Ingrid Heynderickx. 2009. Visual discomfort and visual fatigue of stereoscopic displays: A review. J. Imag. Sci. Technol. 53, 3 (2009), 030201-1--030201-14.

[14]

Manuel Lang, Alexander, Hornung, Oliver Wang, Steven Poulakos, Aljoscha Smolic, and Markus Gross. 2010. Nonlinear disparity mapping for stereoscopic 3D. ACM Trans. Graphics 29, 4 (2010), Article 75.

Digital Library

[15]

Jeehong Lee, Kyu-yeol Chae, and S. Ji. 2012. The 3D video processing method in the stereoscopic camera for mobile devices. In Proceedings of the IEEE International Conference on Emerging Signal Processing Applications (ESPA). IEEE, 139--142.

[16]

Ken-Yi Lee, Yung-Yu Chuang, Bing-Yu Chen, and Ming Ouhyoung. 2009. Video stabilization using robust feature trajectories. In Proceedings of the IEEE 12^th International Conference on Computer Vision. IEEE, 1397--1404.

[17]

K. Levenberg. 1944. A method for the solution of certain non-linear problems in least squares. Quart. Appl. Math. 2. 164--168.

[18]

Chun-Wei Liu, Tz-Huan Huang, Ming-Hsu Chang, Ken-Yi Lee, Chia-Kai Liang, and Yung-Yu Chuang. 2011. 3D cinematography principles and their applications to stereoscopic media processing. In Proceeings of the 19^th ACM International Conference on Multimedia (MM'11). ACM, New York, 253--262.

Digital Library

[19]

Feng Liu, Michael Gleicher, Hailin Jin, and Aseem Agarwala. 2009. Content-preserving warps for 3D video stabilization. ACM Trans. Graphics 28, 3 (2009), Article 44.

Digital Library

[20]

Wan-Yen Lo., Jeroen van Baar, Claude Knaus, Matthias Zwicker, and Markus Gross. 2010. Stereoscopic 3D copy & paste. ACM Trans. Graphics 29, 6 (2010), Article 147.

Digital Library

[21]

S. Mangiat and J. Gibson. 2012. Disparity remapping for handheld 3D video communications. In Proceedings of the 2012 IEEE International Conference on Emerging Signal Processing Applications (ESPA). IEEE, 147--150.

[22]

Wojciech Matusik and Hanspeter Pfister. 2004. 3D TV: A scalable for real-time acquisition, transmission, and autostereoscopic display of dynamic scenes. ACM Trans. Graphics 23, 3 (August 2004), 814--824.

Digital Library

[23]

L. McMillan. 1997. An image-based approach on three-dimensional computer graphics. Ph.D dissertation. University of North Carolina at Chapel Hill, Chapel Hill, N.C.

Digital Library

[24]

P. Mendapara. A. Baradarani, and Q. M. J. Wu. 2010. An efficient depth map estimation technique using complex wavelets. In Proceedings of the IEEE International Conference on Multimedia and Expo (ICME). IEEE, 1409--1414.

[25]

C. Morimoto and R. Chellapa. 1998. Evaluation of image stabilization algorithms. In Proceedings of the IEEE Internationsl Conference on Acoustics, Speech and Signal Processing. Vol. 5, IEEE, 2789--2792.

[26]

Nguyen Ho Quoc Phuong, Hee-Jun Kang, Young-Soo Suh, and Young-Sik Ro. 2009. A DCM based orientation estimation algorithm with an inertial measurement unit and a magnetic compass. J. Univ. Comput. Sci. 15, 4, 859--876.

[27]

V. Rabaud and S. Belongie. 2006. Counting crowded moving objects. In Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR).

Digital Library

[28]

Rahul Raguram, Jan-Michael Frahm and Marc Pollefeys. 2008. A comparative analysis of RANSAC techniques leading to adaptive real-time random sample consensus. In Proceedings of the 10^th European Conference on Computer Vision, Part II (Computer Vision -- ECCV 2008). Lecture Notes in Computer Science, vol. 5303, Springer, Berlin, Heidelberg, 500--513.

Digital Library

[29]

N. Ritter, R. Owens, J. Cooper, R. H. Eikelboom, and P. P. Van Saarloos. 1999. Registration of stereo and temporal images of the retina. IEEE Trans. Med. Imag. 18, 5, 404--418.

[30]

A. Sabatini. 2006. Quaternion-based extended Kalman filter for determining orientation by inertial and magnetic sensing. IEEE Trans. Biomed. Engin. 53, 7.

[31]

D. Scharstein and R. Szeliski. 2002. A taxonomy and evaluation of dense two-frame stereo correspondence algorithms. Int. J. Comput. Vis. 47, 1, 7--42.

Digital Library

[32]

Jonathan Shade, Steven Gortler, Li-wei He, and Richard Szeliski. 1998. Layered depth images. In Proceedings of the 25^th Annual Conference on Computer Graphics and Interactive Techniques (SIGGRAPH'98). ACM, New York, 231--242.

Digital Library

[33]

J. Shi and C. Tomasi. 1994. Good features to track. In Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR).

[34]

Takashi Shibata, Joohwan Kim, David M. Hoffman, and Martin S. Banks. 2011. The zone of comfort: Predicting visual discomfort with stereo displays. J. Vis. 11, 8--11.

[35]

B. M. Smith, L. Zhang, H. Jin, and A. Agarwala. 2009. Light field video stabilization. In Proceedings of the IEEE 12^th International Conference on Computer Vision. IEEE, 341--348.

[36]

Noah Snavely, Steven M. Seitz, and Richard Szeliski. 2006. Photo tourism: Exploring photo collections in 3D. In Proceedings of ACM SIGGRAPH 2006 (SIGGRAPH'06). ACM, New York, 835--846.

Digital Library

[37]

Filippo Speranza, Wa J. Tam, Ron Renaud, and Namho Hur. 2006. Effect of disparity and motion on visual comfort of stereoscopic images. In Proceedings of the SPIE Stereoscopic Displays and Virtual Reality Systems XIII, Vol. 6055, 94--103.

[38]

Y. S. Suh. 2010. Orientation estimation using a quarternion-based Kalman filter with adative estimation acceleration. IEEE Trans. Instrum. Measure. 59, 12, 3296--3305.

[39]

Geng Sun and Nick Holliman. 2009. Evaluating methods for controlling depth perception in stereoscopic cinematography. Proc. SPIE, vol. 7237, Stereoscopic Displays and Applications XX, 72370I (2009).

[40]

Wa James Tam, F. Speranza, S. Yano, K. Shimono, and H. Ono. 2011. Stereoscopic 3D-TV: Visual comfort. IEEE Trans. Broadcast. 57, 2, 335--346.

[41]

Wa James Tam and L. Zhang. 2006. 3D-TV content generation: 2D-to-3D conversion. In Proceedings of the IEEE International Conference on Multimedia and Expo. IEEE, 1869--1872.

[42]

C. Tomasi, and R. Manduchi. 1998. Blateral filtering for gray and color images. In Proceedings of the IEEE International Conference on Computer Vision. IEEE, 839--846.

Digital Library

[43]

N. Uchida, T. Shibahara, T. Aoki, H. Nakajima, and K. Kobayashi. 2005. 3D face recognition using passive stereo vision. In Proceedings of the IEEE International Conference on Image Processing (ICIP'05). IEEE, 950--953.

[44]

Chiao Wang and Alexander A. Sawchuk. 2008. Disparity manipulation for stereo images and video. Proc. SPIE, vol. 6803, Stereoscopic Displays and Applications XIX, 68031E (February 29, 2008).

[45]

J. M. Wang, H. P. Chou, S. W. Chen, and C. S. Fuh. 2009. Video stabilization for a hand-held camera based on 3D motion model. In Proceedings of the 16^th IEEE International Conference on Image Processing (ICIP). IEEE, 3477--3480.

Digital Library

[46]

O. Wang, M. Lang, M. Frei, A. Hornung, A. Smolic, and M. Gross. 2011. Stereobrush: Interactive 2D to 3D conversion using discontinuous warps. In Proceedings of the EUROGRAPHICS Symposium on Sketch-Based Interfaces and Modeling. 47--54.

Digital Library

[47]

L. Zhang and W. J. Tam. 2005. Stereoscopic image generation based on depth images for 3D TV. IEEE Trans. Broadcast. 51, 2, 191--199.

Cited By

Wang JLing QLi P(2023)Robust Video Stabilization based on Motion DecompositionACM Transactions on Multimedia Computing, Communications, and Applications10.1145/358049819:5(1-24)Online publication date: 16-Mar-2023
https://dl.acm.org/doi/10.1145/3580498
Yue GHou CZhou T(2019)Subtitle Region Selection of S3D Images in Consideration of Visual Discomfort and Viewing HabitACM Transactions on Multimedia Computing, Communications, and Applications10.1145/332519715:3(1-16)Online publication date: 20-Aug-2019
https://dl.acm.org/doi/10.1145/3325197
Cánovas ATaha MLloret JTomas J(2019)A cognitive network management system to improve QoE in stereoscopic IPTV serviceInternational Journal of Communication Systems10.1002/dac.399232:12Online publication date: 17-May-2019
https://doi.org/10.1002/dac.3992
Show More Cited By

Index Terms

Visual Comfort for Stereoscopic 3D by Using Motion Sensors on 3D Mobile Devices
1. Computing methodologies
  1. Artificial intelligence
    1. Computer vision
      1. Computer vision problems
      2. Computer vision tasks
        Scene understanding

Recommendations

Perceptual Tolerance to Stereoscopic 3D Image Distortion

An intriguing aspect of picture perception is the viewer’s tolerance to variation in viewing position, perspective, and display size. These factors are also present in stereoscopic media, where there are additional parameters associated with the camera ...
3D Elastic Control for Mobile Devices

To increase the input space of mobile devices, the authors developed a proof-of-concept 3D elastic controller that easily adapts to mobile devices. This embedded device improves the completion of high-level interaction tasks such as visualization of ...
Investigating mobile stereoscopic 3D touchscreen interaction
OzCHI '13: Proceedings of the 25th Australian Computer-Human Interaction Conference: Augmentation, Application, Innovation, Collaboration

3D output is no longer limited to large screens in cinemas or living rooms. Nowadays more and more mobile devices are equipped with autostereoscopic 3D (S3D) touchscreens. As a consequence interaction with 3D content now also happens whilst users are on ...

Comments

Information & Contributors

Information

Published In

cover image ACM Transactions on Multimedia Computing, Communications, and Applications

ACM Transactions on Multimedia Computing, Communications, and Applications Volume 12, Issue 1s

Special Issue on Smartphone-Based Interactive Technologies, Systems, and Applications and Special Issue on Extended Best Papers from ACM Multimedia 2014

October 2015

317 pages

ISSN:1551-6857

EISSN:1551-6865

DOI:10.1145/2837676

Editor:
Ralf Steinmetz
Technische Universität Darmstadt, Germany

Issue’s Table of Contents

Copyright © 2015 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 21 October 2015

Accepted: 01 June 2015

Revised: 01 April 2015

Received: 01 January 2015

Published in TOMM Volume 12, Issue 1s

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed

Funding Sources

National Science Council of Taiwan, R.O.C

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

5
Total Citations
View Citations
266
Total Downloads

Downloads (Last 12 months)4
Downloads (Last 6 weeks)0

Reflects downloads up to

Other Metrics

View Author Metrics

Citations

Cited By

Wang JLing QLi P(2023)Robust Video Stabilization based on Motion DecompositionACM Transactions on Multimedia Computing, Communications, and Applications10.1145/358049819:5(1-24)Online publication date: 16-Mar-2023
https://dl.acm.org/doi/10.1145/3580498
Yue GHou CZhou T(2019)Subtitle Region Selection of S3D Images in Consideration of Visual Discomfort and Viewing HabitACM Transactions on Multimedia Computing, Communications, and Applications10.1145/332519715:3(1-16)Online publication date: 20-Aug-2019
https://dl.acm.org/doi/10.1145/3325197
Cánovas ATaha MLloret JTomas J(2019)A cognitive network management system to improve QoE in stereoscopic IPTV serviceInternational Journal of Communication Systems10.1002/dac.399232:12Online publication date: 17-May-2019
https://doi.org/10.1002/dac.3992
Chu C(2018)Image Deblur for 3D Sensing Mobile Devices2018 IEEE International Conference on Multimedia and Expo (ICME)10.1109/ICME.2018.8486596(1-6)Online publication date: Jul-2018
https://doi.org/10.1109/ICME.2018.8486596
Chu CTai C(2017)Camera Pose Trace Based on Motion Sensor in Mobile Devices2017 Conference on Technologies and Applications of Artificial Intelligence (TAAI)10.1109/TAAI.2017.44(5-8)Online publication date: Dec-2017
https://doi.org/10.1109/TAAI.2017.44

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Issue’s Table of Contents