A Survey on 360° Images and Videos in Mixed Reality: Algorithms and Applications
Pages 473 - 491
Abstract
Mixed reality technologies provide real-time and immersive experiences, which bring tremendous opportunities in entertainment, education, and enriched experiences that are not directly accessible owing to safety or cost. The research in this field has been in the spotlight in the last few years as the metaverse went viral. The recently emerging omnidirectional video streams, i.e., 360° videos, provide an affordable way to capture and present dynamic real-world scenes. In the last decade, fueled by the rapid development of artificial intelligence and computational photography technologies, the research interests in mixed reality systems using 360° videos with richer and more realistic experiences are dramatically increased to unlock the true potential of the metaverse. In this survey, we cover recent research aimed at addressing the above issues in the 360° image and video processing technologies and applications for mixed reality. The survey summarizes the contributions of the recent research and describes potential future research directions about 360° media in the field of mixed reality.
References
[1]
Friston S, Ritschel T, Steed A. Perceptual rasterization for head-mounted display image synthesis. ACM Trans. Graphics, 2019, 38(4): Article No. 97.
[2]
Tursun O T, Arabadzhiyska-Koleva E, Wernikowski M, Mantiuk R, Seidel H P, Myszkowski K, Didyk P. Luminance-contrast-aware foveated rendering. ACM Trans. Graphics, 2019, 38(4): Article No. 98.
[3]
Schroers C, Bazin J C, Sorkine-Hornung A. An omnistereoscopic video pipeline for capture and display of real-world VR. ACM Trans. Graphics, 2018, 37(3): Article No. 37.
[4]
Matzen K, Cohen M F, Evans B, Kopf J, Szeliski R. Low-cost 360 stereo photography and video capture. ACM Trans. Graphics, 2017, 36(4): Article No. 148.
[5]
Habermann M, Xu W P, Zollhöfer M, Pons-Moll G, Theobalt C. LiveCap: Real-time human performance capture from monocular video. ACM Trans. Graphics, 2019, 38(2): Article No. 14.
[6]
Xu W P, Chatterjee A, Zollhöfer M, Rhodin H, Mehta D, Seidel H P, Theobalt C. MonoPerfCap: Human performance capture from monocular video. ACM Trans. Graphics, 2018, 37(2): Article No. 27.
[7]
Kopf J. 360° video stabilization. ACM Trans. Graphics, 2016, 35(6): Article No. 195.
[8]
Tang C Z, Wang O, Liu F, Tan P. Joint stabilization and direction of 360° videos. ACM Trans. Graphics, 2019, 38(2): Article No. 18.
[9]
Silva R M A, Feijó B, Gomes P B, Frensh T, Monteiro D. Real time 360° video stitching and streaming. In Proc. the ACM SIGGRAPH 2016 Posters, Jul. 2016, Article No. 70.
[10]
Li Y H, Barnes C, Huang K, Zhang F L. Deep 360° optical flow estimation based on multi-projection fusion. In Proc. the 17th European Conference on Computer Vision, Oct. 2022, pp.336–352.
[11]
Jung R, Lee A S J, Ashtari A, Bazin J C. Deep360Up: A deep learning-based approach for automatic VR image upright adjustment. In Proc. the 2019 IEEE Conference on Virtual Reality and 3D User Interfaces (VR), Mar. 2019.
[12]
Li D Z Y, Langlois T R, Zheng C X. Scene-aware audio for 360° videos. ACM Trans. Graphics, 2018, 37(4): Article No. 111.
[13]
Rhee T, Petikam L, Allen B, and Chalmers A MR360: Mixed reality rendering for 360° panoramic videos IEEE Trans. Visualization and Computer Graphics 2017 23 4 1379-1388
[14]
Kang K, Cho S. Interactive and automatic navigation for 360° video playback. ACM Trans. Graphics, 2019, 38(4): Article No. 108.
[15]
Rees D W. Panoramic television viewing system. United States Patent, No. 3505465, 1970.
[16]
Yagi Y, Kawato S, and Tsuji S Real-time omnidirectional image sensor (COPIS) for vision-guided navigation IEEE Trans. Robotics and Automation 1994 10 1 11-22
[17]
Gledhill D, Tian GY, Taylor D, and Clarke D Panoramic imaging—A review Computers & Graphics 2003 27 3 435-445
[18]
Yagi Y and Yachida M Real-time omnidirectional image sensors International Journal of Computer Vision 2004 58 3 173-207
[19]
Debevec P. Image-based lighting. In Proc. the ACM SIGGRAPH 2005 Courses, Jul. 2005, Article No. 3-es.
[20]
Tarini M, Hormann K, Cignoni P, and Montani C Poly-Cube-maps ACM Trans. Graphics 2004 23 3 853-860
[21]
McMillan L, Bishop G. Plenoptic modeling: An image-based rendering system. In Proc. the 22nd Annual Conference on Computer Graphics and Interactive Techniques, Aug. 1995, pp.39–46.
[22]
Hilbert D, Cohn-Vossen S. Geometry and the Imagination, Volume 87. Providence: American Mathematical Soc., 2021.
[23]
Nadeem S, Su ZY, Zeng W, Kaufman A, and Gu XF Spherical parameterization balancing angle and area distortions IEEE Trans. Visualization and Computer Graphics 2017 23 6 1663-1676
[24]
Poranne R, Tarini M, Huber S, Panozzo D, Sorkine-Hornung O. Autocuts: Simultaneous distortion and cut optimization for UV mapping. ACM Trans. Graphics, 2017, 36(6): Article No. 215.
[25]
Jin Y L, Liu J H, Wang F X, Cui S G. Where are you looking? A large-scale dataset of head and gaze behavior for 360-degree videos and a pilot study. In Proc. the 30th ACM International Conference on Multimedia, Oct. 2022, pp.1025–1034.
[26]
David E J, Gutiérrez J, Coutrot A, Da Silva M P, Le Callet P. A dataset of head and eye movements for 360° videos. In Proc. the 9th ACM Multimedia Systems Conference, Jun. 2018, pp.432–437.
[27]
Armeni I, Sax S, Zamir A R, Savarese S. Joint 2D-3D-semantic data for indoor scene understanding. arXiv: 1702.01105, 2017. https://arxiv.org/abs/1702.01105, Jul. 2023.
[28]
Coors B, Condurache A P, Geiger A. SphereNet: Learning spherical representations for detection and classification in omnidirectional images. In Proc. the 15th European Conference on Computer Vision, Sept. 2018, pp.525–541.
[29]
Zhao Q, Zhu C, Dai F, Ma Y K, Jin G Q, Zhang Y D. Distortion-aware CNNs for spherical images. In Proc. the 27th International Joint Conference on Artificial Intelligence, Jul. 2018, pp.1198–1204.
[30]
Eder M, Shvets M, Lim J, Frahm J M. Tangent images for mitigating spherical distortion. In Proc. the 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Jun. 2020, pp.12423–12431.
[31]
Yuan M Z, Christian R. 360° optical flow using tangent images. In Proc. the 32nd British Machine Vision Conference, Nov. 2021.
[32]
Lee Y, Jeong J, Yun J, Cho W, Yoon K J. SpherePHD: Applying CNNs on a spherical PolyHeDron representation of 360° images. In Proc. the 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Jun. 2019, pp.9173–9181.
[33]
Zhang C, Liwicki S, Smith W, Cipolla R. Orientation-aware semantic segmentation on icosahedron spheres. In Proc. the 2019 IEEE/CVF International Conference on Computer Vision, Oct. 27–Nov. 2, 2019, pp.3532–3540.
[34]
Yoon Y, Chung I, Wang L, Yoon K J. SphereSR: 360° image super-resolution with arbitrary projection via continuous spherical image representation. In Proc. the 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Jun. 2022, pp.5667–5676.
[35]
Wu G, Shi YH, Sun XY, Wang J, and Yin BC SMSIR: Spherical measure based spherical image representation IEEE Trans. Image Processing 2021 30 6377-6391
[36]
Li J S, Wen Z Y, Li S H, Zhao Y K, Guo B C, Wen J T. Novel tile segmentation scheme for omnidirectional video. In Proc. the 2016 IEEE International Conference on Image Processing (ICIP), Sept. 2016, pp.370–374.
[37]
Cheng H T, Chao C H, Dong J D, Wen H K, Liu T L, Sun M. Cube padding for weakly-supervised saliency prediction in 360° videos. In Proc. the 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Jun. 2018, pp.1420–1429.
[38]
Monroy R, Lutz S, Chalasani T, and Smolic A SalNet360: Saliency maps for omni-directional images with CNN Signal Processing: Image Communication 2018 69 26-34
[39]
Wang F E, Yeh Y H, Tsai Y H, Chiu W C, Sun M. Bi-Fuse++: Self-supervised and efficient bi-projection fusion for 360° depth estimation. IEEE Trans. Pattern Analysis and Machine Intelligence, 2023, 45(5): 5448–5460.
[40]
Wang F E, Yeh Y H, Sun M, Chiu W C, Tsai Y H. Bi-Fuse: Monocular 360 depth estimation via bi-projection fusion. In Proc. the 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Jun. 2020, pp.459–468.
[41]
Li Y Y, Guo Y L, Yan Z X, Huang X Y, Duan Y, Ren L. OmniFusion: 360 monocular depth estimation via geometry-aware fusion. In Proc. the 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Jun. 2022, pp.2791–2800.
[42]
Sun C, Sun M, Chen H T. HoHoNet: 360 indoor holistic understanding with latent horizontal features. In Proc. the 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Jun. 2021, pp.2573–2582.
[43]
Yang KL, Hu XX, Fang YC, Wang KW, and Stiefelhagen R Omnisupervised omnidirectional semantic segmentation IEEE Trans. Intelligent Transportation Systems 2022 23 2 1184-1199
[44]
Zhang J M, Yang K L, Ma C X, Reiß S, Peng K Y, Stiefelhagen R. Bending reality: Distortion-aware transformers for adapting to panoramic semantic segmentation. In Proc. the 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Jun. 2022, pp.16896–16906.
[45]
Defferrard M, Milani M, Gusset F, Perraudin N. Deep-Sphere: A graph-based spherical CNN. In Proc. the 8th International Conference on Learning Representations, Apr. 2019.
[46]
Ros G, Sellart L, Materzynska J, Vazquez D, Lopez A M. The Synthia dataset: A large collection of synthetic images for semantic segmentation of urban scenes. In Proc. the 2016 IEEE Conference on Computer Vision and Pattern Recognition, Jun. 2016, pp.3234–3243.
[47]
Yang KL, Hu XX, Bergasa LM, Romera E, and Wang KW PASS: Panoramic annular semantic segmentation IEEE Trans. Intelligent Transportation Systems 2020 21 10 4171-4185
[48]
Su Y C, Grauman K. Learning spherical convolution for fast features from 360° imagery. In Proc. the 31st International Conference on Neural Information Processing Systems, Dec. 2017, pp.529–539.
[49]
Zhao P Y, You A S, Zhang Y X, Liu J Y, Bian K G, Tong Y H. Spherical criteria for fast and accurate 360° object detection. In Proc. the 34th AAAI Conference on Artificial Intelligence, Feb. 2020, pp.12959–12966.
[50]
Cao M, Ikehata S, Aizawa K. Field-of-view IoU for object detection in 360° images. arXiv: 2202.03176, 2022. https://arxiv.org/abs/2202.03176, Jul. 2023.
[51]
Chou S H, Sun C, Chang W Y, Hsu W T, Sun M, Fu J D. 360-indoor: Towards learning real-world objects in 360° indoor equirectangular images. In Proc. the 2020 IEEE Winter Conference on Applications of Computer Vision, Mar. 2020, pp.834–842.
[52]
Guerrero-Viu J, Fernandez-Labrador C, Demonceaux C, Guerrero J J. What’s in my room? Object recognition on indoor panoramic images. In Proc. the 2020 IEEE International Conference on Robotics and Automation (ICRA), May 2020, pp.567–573.
[53]
Zhang Z H, Xu Y Y, Yu J Y, Gao S H. Saliency detection in 360° videos. In Proc. the 15th European Conference on Computer Vision, Sept. 2018, pp.504–520.
[54]
Ma GX, Li S, Chen CLZ, Hao AM, and Qin H Stage-wise salient object detection in 360° omnidirectional image via object-level semantical saliency ranking IEEE Trans. Visualization and Computer Graphics 2020 26 12 3535-3545
[55]
Hu H N, Lin Y C, Liu M Y, Cheng H T, Chang Y J, Sun M. Deep 360 pilot: Learning a deep agent for piloting through 360° sports videos. In Proc. the 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Jul. 2017, pp.1396–1405.
[56]
Dahou Y, Tliba M, McGuinness K, O’Connor N. ATSal: An attention based architecture for saliency prediction in 360° videos. In Proc. the 26th International Conference on Pattern Recognition, Jan. 2021, pp.305–320.
[57]
Qiao ML, Xu M, Wang ZL, and Borji A Viewport-dependent saliency prediction in 360° video IEEE Trans. Multimedia 2021 23 748-760
[58]
Chao FY, Zhang L, Hamidouche W, and Déforges O A multi-FoV viewport-based visual saliency model using adaptive weighting losses for 360° images IEEE Trans. Multimedia 2021 23 1811-1826
[59]
Zhang Y, Chao F Y, Hamidouche W, Deforges O. PAVSOD: A new task towards panoramic audiovisual saliency detection. ACM Trans. Multimedia Computing, Communications, and Applications, 2023, 19(3): Article No. 101.
[60]
Jiang HL, Sheng Z, Zhu SY, Dong ZL, and Huang R Uni-Fuse: Unidirectional fusion for 360° panorama depth estimation IEEE Robotics and Automation Letters 2021 6 2 1519-1526
[61]
Zhuang C Q, Lu Z D, Wang Y Q, Xiao J, Wang Y. ACDNet: Adaptively combined dilated convolution for monocular panorama depth estimation. In Proc. the 36th AAAI Conference on Artificial Intelligence, Feb. 22–Mar. 1, 2022, pp.3653–3661.
[62]
Feng Q, Shum H P H, Morishima S. 360 depth estimation in the wild—The depth360 dataset and the SegFuse network. In Proc. the 2022 IEEE Conference on Virtual Reality and 3D User Interfaces (VR), Mar. 2022, pp.664–673.
[63]
Rey-Area M, Yuan M Z, Richardt C. 360MonoDepth: High-resolution 360° monocular depth estimation. In Proc. the 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Jun. 2022, pp.3752–3762.
[64]
Serrano A, Kim I, Chen ZL, DiVerdi S, Gutierrez D, Hertzmann A, and Masia B Motion parallax for 360° RGBD video IEEE Trans. Visualization and Computer Graphics 2019 25 5 1817-1827
[65]
Won C, Ryu J, Lim J. SweepNet: Wide-baseline omnidirectional depth estimation. In Proc. the 2019 International Conference on Robotics and Automation, May 2019, pp.6073–6079.
[66]
Wang N H, Solarte B, Tsai Y H, Chiu W C, Sun M. 360SD-net: 360° stereo depth estimation with learnable cost volume. In Proc. the 2020 IEEE International Conference on Robotics and Automation (ICRA), May 2020, pp.582–588.
[67]
Teed Z, Deng J. RAFT: Recurrent all-pairs field transforms for optical flow. arXiv: 2003.12039, 2020. https://arxiv.org/abs/2003.12039, Jul. 2023.
[68]
Sun D Q, Yang X D, Liu M Y, Kautz J. PWC-net: CNNs for optical flow using pyramid, warping, and cost volume. In Proc. the 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Jun. 2018, pp.8934–8943.
[69]
Bhandari K, Zong Z L, Yan Y. Revisiting optical flow estimation in 360 videos. In Proc. the 25th International Conference on Pattern Recognition (ICPR), Jan. 2021, pp.8196–8203.
[70]
Martin D, Serrano A, Bergman AW, Wetzstein G, and Masia B ScanGAN360: A generative model of realistic scanpaths for 360° images IEEE Trans. Visualization and Computer Graphics 2022 28 5 2003-2013
[71]
Yu M, Lakshman H, Girod B. A framework to evaluate omnidirectional video coding schemes. In Proc. the 2015 IEEE International Symposium on Mixed and Augmented Reality, Sept. 29–Oct. 3, 2015, pp.31–36.
[72]
Wang S B, Yang S S, Li H L, Zhang X D, Zhou C, Xu C R, Qian F, Wang N B, Xu Z B. SalientVR: Saliency-driven mobile 360-degree video streaming with gaze information. In Proc. the 28th Annual International Conference on Mobile Computing and Networking, Oct. 2022, pp.542–555.
[73]
Xu Y Y, Dong Y B, Wu J R, Sun Z Z, Shi Z R, Yu J Y, Gao S H. Gaze prediction in dynamic 360° immersive videos. In Proc. the 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Jun. 2018, pp.5333–5342.
[74]
Yang L, Xu M, Guo Y C, Deng X, Gao F Y, Guan Z Y. Hierarchical Bayesian LSTM for head trajectory prediction on omnidirectional images. IEEE Trans. Pattern Analysis and Machine Intelligence, 2022, 44(11): 7563–7580.
[75]
Rondón M, Sassatelli L, Aparicio-Pardo R, and Precioso F TRACK: A new method from a re-examination of deep architectures for head motion prediction in 360° videos. IEEE Trans. Pattern Analysis and Machine Intelligence 2022 44 9 5681-5699
[76]
Assens M, Giro-i-Nieto X, McGuinness K, O’Connor N E. SaltiNet: Scan-path prediction on 360 degree images using saliency volumes. In Proc. the 2017 IEEE International Conference on Computer Vision Workshops, Oct. 2017, pp.2331–2338.
[77]
de Belen R A J, Bednarz T, Sowmya A. ScanpathNet: A recurrent mixture density network for scanpath prediction. In Proc. the 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Jun. 2022, pp.5006–5016.
[78]
Griffin R, Langlotz T, Zollmann S. 6DIVE: 6 degrees-of-freedom immersive video editor. Frontiers in Virtual Reality, 2021, 2: 676895.
[79]
Xu M, Li C, Zhang SY, and Le Callet P State-of-the-art in 360° video/image processing: Perception, assessment and compression IEEE Journal of Selected Topics in Signal Processing 2020 14 1 5-26
[80]
Barron J T, Mildenhall B, Verbin D, Srinivasan P P, Hedman P. Mip-NeRF 360: Unbounded anti-aliased neural radiance fields. In Proc. the 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Jun. 2022, pp.5460–5469.
[81]
Huang J W, Chen Z L, Ceylan D, Jin H L. 6-DOF VR videos with a single 360-camera. In Proc. the 2017 IEEE Virtual Reality (VR), Mar. 2017, pp.37–44.
[82]
Baker L, Mills S, Zollmann S, Ventura J. CasualStereo: Casual capture of stereo panoramas with spherical structure-from-motion. In Proc. the 2020 IEEE Conference on Virtual Reality and 3D User Interfaces (VR), Mar. 2020, pp.782–790.
[83]
Chen R S, Zhang F L, Finnie S, Chalmers A, Rhee T. Casual 6-DoF: Free-viewpoint panorama using a handheld 360° camera. IEEE Trans. Visualization and Computer Graphics, 2022: 1.
[84]
Waidhofer J, Gadgil R, Dickson A, Zollmann S, Ventura J. PanoSynthVR: Toward light-weight 360-degree view synthesis from a single panoramic input. In Proc. the 2022 IEEE International Symposium on Mixed and Augmented Reality (ISMAR), Oct. 2022, pp.584–592.
[85]
DiVerdi S, Wither J, Hollerer T. Envisor: Online environment map construction for mixed reality. In Proc. the 2008 IEEE Virtual Reality Conference, Mar. 2008, pp.19–26.
[86]
Park J, Park H, Yoon SE, and Woo W Physically-inspired deep light estimation from a homogeneous-material object for mixed reality lighting IEEE Trans. Visualization and Computer Graphics 2020 26 5 2002-2011
[87]
Georgoulis S, Rematas K, Ritschel T, Gavves E, Fritz M, Van Gool L, Tuytelaars T. Reflectance and natural illumination from single-material specular objects using deep learning. IEEE Trans. Pattern Analysis and Machine Intelligence, 2018, 40(8): 1932–1947.
[88]
Wei X, Chen G J, Dong Y, Lin S, Tong X. Object-based illumination estimation with rendering-aware neural networks. In Proc. the 16th European Conference on Computer Vision, Aug. 2020, pp.380–396.
[89]
Hold-Geoffroy Y, Sunkavalli K, Hadap S, Gambaretto E, Lalonde J F. Deep outdoor illumination estimation. In Proc. the 2017 IEEE Conference on Computer Vision and Pattern Recognition, Jul. 2017, pp.2373–2382.
[90]
Zhang J S, Sunkavalli K, Hold-Geoffroy Y, Hadap S, Eisenman J, Lalonde J F. All-weather deep outdoor lighting estimation. In Proc. the 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Jun. 2019, pp.10150–10158.
[91]
Hold-Geoffroy Y, Athawale A, Lalonde J F. Deep sky modeling for single image outdoor lighting estimation. In Proc. the 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Jun. 2019, pp.6920–6928.
[92]
Yu P P, Guo J, Huang F, Zhou C, Che H W, Ling X, Guo Y W. Hierarchical disentangled representation learning for outdoor illumination estimation and editing. In Proc. the 2021 IEEE/CVF International Conference on Computer Vision, Oct. 2021, pp.15293–15302.
[93]
Zhu Y J, Zhang Y D, Li S, Shi B X. Spatially-varying outdoor lighting estimation from intrinsics. In Proc. the 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Jun. 2021, pp.12829–12837.
[94]
Tang J J, Zhu Y J, Wang H Y, Chan J H, Li S, Shi B X. Estimating spatially-varying lighting in urban scenes with disentangled representation. In Proc. the 17th European Conference on Computer Vision, Oct. 2022, pp.454–469.
[95]
Gardner M A, Sunkavalli K, Yumer E, Shen X H, Gambaretto E, Gagné C, Lalonde J F. Learning to predict indoor illumination from a single image. ACM Trans. Graphics, 2017, 36(6): Article No. 176.
[96]
Song S R, Funkhouser T. Neural Illumination: Lighting prediction for indoor environments. In Proc. the 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Jun. 2019, pp.6911–6919.
[97]
Srinivasan P P, Mildenhall B, Tancik M, Barron J T, Tucker R, Snavely N. Lighthouse: Predicting lighting volumes for spatially-coherent illumination. In Proc. the 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Jun. 2020, pp.8077–8086.
[98]
Chalmers A, Zhao JH, Medeiros D, and Rhee T Reconstructing reflection maps using a stacked-CNN for mixed reality rendering IEEE Trans. Visualization and Computer Graphics 2021 27 10 4073-4084
[99]
Zhao JH, Chalmers A, and Rhee T Adaptive light estimation using dynamic filtering for diverse lighting conditions IEEE Trans. Visualization and Computer Graphics 2021 27 11 4097-4106
[100]
Zhan F N, Zhang C G, Yu Y C, Chang Y, Lu S J, Ma F Y, Xie X S. EMLight: Lighting estimation via spherical distribution approximation. In Proc. the 35th AAAI Conference on Artificial Intelligence, Feb. 2021, pp.3287–3295.
[101]
Zhan F N, Yu Y C, Wu R L, Zhang C G, Lu S J, Shao L, Ma F Y, Xie X S. GMLight: Lighting estimation via geometric distribution approximation. arXiv: 2102.10244, 2021. https://arxiv.org/abs/2102.10244v1, Jul. 2023.
[102]
Xu J P, Zuo C Y, Zhang F L, Wang M. Rendering-aware HDR environment map prediction from a single image. In Proc. the 36th AAAI Conference on Artificial Intelligence, Feb. 22–Mar. 1, 2022, pp.2857–2865.
[103]
Akimoto N, Matsuo Y, Aoki Y. Diverse plausible 360-degree image outpainting for efficient 3DCG background creation. In Proc. the 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Jun. 2022, pp.11431–11440.
[104]
Somanath G, Kurz D. HDR environment map estimation for real-time augmented reality. In Proc. the 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Jun. 2021, pp.11293–11301.
[105]
Wagner D, Mulloni A, Langlotz T, Schmalstieg D. Real-time panoramic mapping and tracking on mobile phones. In Proc. the 2010 IEEE Virtual Reality Conference (VR), Mar. 2010, pp.211–218.
[106]
Gauglitz S, Sweeney C, Ventura J, Turk M, Hlerer T. Live tracking and mapping from both general and rotation-only camera motion. In Proc. the 2012 IEEE International Symposium on Mixed and Augmented Reality (ISMAR), Nov. 2012, pp.13–22.
[107]
Gauglitz S, Sweeney C, Ventura J, Turk M, and Hollerer T Model estimation and selection towards unconstrained real-time tracking and mapping IEEE Trans. Visualization and Computer Graphics 2014 20 6 825-838
[108]
Pirchheim C, Schmalstieg D, Reitmayr G. Handling pure camera rotation in keyframe-based SLAM. In Proc. the 2013 IEEE International Symposium on Mixed and Augmented Reality (ISMAR), Oct. 2013, pp.229–238.
[109]
Baker L, Ventura J, Zollmann S, Mills S, Langlotz T. SPLAT: Spherical localization and tracking in large spaces. In Proc. the 2020 IEEE Conference on Virtual Reality and 3D User Interfaces (VR), Mar. 2020, pp.809–817.
[110]
Pan Q, Arth C, Reitmayr G, Rosten E, Drummond T. Rapid scene reconstruction on mobile phones from panoramic images. In Proc. the 10th IEEE International Symposium on Mixed and Augmented Reality, Oct. 2011, pp.55–64.
[111]
Arth C, Klopschitz M, Reitmayr G, Schmalstieg D. Real-time self-localization from panoramic images on mobile devices. In Proc. the 10th IEEE International Symposium on Mixed and Augmented Reality, Oct. 2011, pp.37–46.
[112]
Reinisch G, Arth C, Schmalstieg D. Panoramic mapping on a mobile phone GPU. In Proc. the 2013 IEEE International Symposium on Mixed and Augmented Reality (ISMAR), Oct. 2013, pp.291–292.
[113]
Pece F, Steptoe W, Wanner F, Julier S, Weyrich T, Kautz J, Steed A. PanoInserts: Mobile spatial teleconferencing. In Proc. the 2013 SIGCHI Conference on Human Factors in Computing Systems, Apr. 2013, pp.1319–1328.
[114]
Speicher M, Cao J C, Yu A, Zhang H H, Nebeling M. 360Anywhere: Mobile ad-hoc collaboration in any environment using 360 video and augmented reality. Proceedings of the ACM on Human-Computer Interaction, 2018, 2(EICS): 9.
[115]
Piumsomboon T, Lee G A, Irlitti A, Ens B, Thomas B H, Billinghurst M. On the shoulder of the giant: A multi-scale mixed reality collaboration with 360 video sharing and tangible interaction. In Proc. the 2019 CHI Conference on Human Factors in Computing Systems, May 2019, Article No. 228.
[116]
Teo T, Lawrence L, Lee G A, Billinghurst M, Adcock M. Mixed reality remote collaboration combining 360 video and 3D reconstruction. In Proc. the 2019 CHI Conference on Human Factors in Computing Systems, May 2019, Article No. 201.
[117]
Wang P, Bai X L, Billinghurst M, Zhang S S, Zhang X Y, Wang S X, He W P, Yan Y X, Ji H Y. AR/MR remote collaboration on physical tasks: A review. Robotics and Computer-Integrated Manufacturing, 2021, 72: 102071.
[118]
Nebeling M, Madier K. 360proto: Making interactive virtual reality & augmented reality prototypes from paper. In Proc. the 2019 CHI Conference on Human Factors in Computing Systems, May 2019, Article No. 596.
[119]
Zhu Z, Martin RR, and Hu SM Panorama completion for street views Computational Visual Media 2015 1 1 49-57
[120]
He K M, Sun J. Image completion approaches using the statistics of similar patches. IEEE Trans. Pattern Analysis and Machine Intelligence, 2014, 36(12): 2423–2435.
[121]
Xu BB, Pathak S, Fujii H, Yamashita A, and Asama H Spatio-temporal video completion in spherical image sequences IEEE Robotics and Automation Letters 2017 2 4 2032-2039
[122]
Zhao Q, Wan L, Feng W, Zhang J W, Wong T T. 360 panorama cloning on sphere. arXiv: 1709.01638, 2017. https://arxiv.org/abs/1709.01638, Jul. 2023.
[123]
Huang K, Zhang F L, Zhao J H, Li Y H, Dodgson N. 360° stereo image composition with depth adaption. arXiv: 2212.10062, 2022. https://arxiv.org/abs/2212.10062, Jul. 2023.
[124]
Jung J, Kim B, Lee JY, Kim B, and Lee S Robust upright adjustment of 360 spherical panoramas The Visual Computer 2017 33 6 737-747
[125]
Zhang Y, Zhang FL, Lai YK, and Zhu Z Efficient propagation of sparse edits on 360° panoramas Computers & Graphics 2021 96 61-70
[126]
Zhang Y, Zhang FL, Zhu Z, Wang LD, and Jin Y Fast edit propagation for 360 degree panoramas using function interpolation IEEE Access 2022 10 43882-43894
[127]
Wong K M. View-adaptive asymmetric image detail enhancement for 360-degree stereoscopic VR content. In Proc. the 2022 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops (VRW), Mar. 2022, pp.23–26.
[128]
Wang M, Li Y J, Zhang W X, Richardt C, Hu S M. Transitioning360: Content-aware NFOV virtual camera paths for 360° video playback. In Proc. the 2020 IEEE International Symposium on Mixed and Augmented Reality (ISMAR), Nov. 2020, pp.185–194.
[129]
Li Y J, Shi J C, Zhang F L, Wang M. Bullet comments for 360°video. In Proc. the 2022 IEEE Conference on Virtual Reality and 3D User Interfaces (VR), Mar. 2022.
Recommendations
Mixed Reality MIDI Keyboard Demonstration
AM '17: Proceedings of the 12th International Audio Mostly Conference on Augmented and Participatory Sound and Music ExperiencesThe Mixed Reality MIDI Keyboard is a prototype designed to augment virtual reality experiences through the inclusion of a physical interface which aligns the user's senses with the virtual environment. It also serves as a platform on which the uses of ...
A survey of visual, mixed, and augmented reality gaming
Theoretical and Practical Computer Applications in EntertainmentVisual mixed and augmented realities have historically been applied to the gaming application domain. This article provides a survey of visual mixed and augmented reality gaming in both the academic and commercial contexts. There is an exploration of ...
Headset removal for virtual and mixed reality
SIGGRAPH '17: ACM SIGGRAPH 2017 TalksVirtual Reality (VR) has advanced significantly in recent years and allows users to explore novel environments (both real and imaginary), play games, and engage with media in a way that is unprecedentedly immersive. However, compared to physical reality,...
Comments
Information & Contributors
Information
Published In
© Institute of Computing Technology, Chinese Academy of Sciences 2023.
Publisher
Springer-Verlag
Berlin, Heidelberg
Publication History
Published: 30 May 2023
Accepted: 24 May 2023
Received: 06 March 2023
Author Tags
Qualifiers
- Research-article
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 0Total Downloads
- Downloads (Last 12 months)0
- Downloads (Last 6 weeks)0
Reflects downloads up to 10 Nov 2024
Other Metrics
Citations
View Options
View options
Get Access
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in