short-paper

A saliency dataset for 360-degree videos

Authors:

Zhisheng YanAuthors Info & Claims

MMSys '19: Proceedings of the 10th ACM Multimedia Systems Conference

Pages 279 - 284

https://doi.org/10.1145/3304109.3325820

Published: 18 June 2019 Publication History

Abstract

Despite the increasing popularity, realizing 360-degree videos in everyday applications is still challenging. Considering the unique viewing behavior in head-mounted display (HMD), understanding the saliency of 360-degree videos becomes the key to various 360-degree video research. Unfortunately, existing saliency datasets are either irrelevant to 360-degree videos or too small to support saliency modeling. In this paper, we introduce a large saliency dataset for 360-degree videos with 50,654 saliency maps from 24 diverse videos. The dataset is created by a new methodology supported by psychology studies in HMD viewing. We describe an open-source software implementing this methodology that can generate saliency maps from any head tracking data. Evaluation of the dataset shows that the generated saliency is highly correlated with the actual user fixation and that the saliency data can provide useful insight on user attention in 360-degree video viewing. The dataset and the program used to extract saliency are both made publicly available to facilitate future research.

References

[1]

Ana De Abreu, Cagri Ozcinar, and Aljosa Smolic. 2017. Look Around You: Saliency Maps for Omnidirectional Images in VR Applications. In IEEE International Conference on Quality of Multimedia Experience (QoMEX).

[2]

Ali Borji and Laurent Itti. 2015. CAT2000: A Large Scale Fixation Dataset for Boosting Saliency Research. In arXiv:1505.03581 {cs.CV}.

[3]

Zoya Bylinskii, Tilke Judd, Aude Oliva, Antonio Torralba, and Fredo Durand. 2018. What Do Different Evaluation Metrics Tell Us About Saliency Models? IEEE Transactions on Pattern Analysis and Machine Intelligence PP (March 2018), 1--1.

[4]

Kai-Yueh Chang, Tyng-Luh Liu, Hwann-Tzong Chen, and Shang-Hong Lai. 2011. Fusing generic objectness and visual saliency for salient object detection. In IEEE International Conference on Computer Vision.

Digital Library

[5]

Xavier Corbillon, Francesca De Simone, and Gwendal Simon. 2017. 360-Degree Video Head Movement Dataset. In Proceedings of the 8th ACM on Multimedia Systems Conference (MMSys'17).

Digital Library

[6]

Xavier Corbillon, Gwendal Simon, Alisa Devlic, and Jacob Chakareski. 2017. Viewport-adaptive Navigable 360-degree Video Delivery. In IEEE International Conference on Communications (ICC).

[7]

Marcella Cornia, Lorenzo Baraldi, Giuseppe Serra, and Rita Cucchiara. 2016. A deep Multi-level Network for Saliency Prediction. In IEEE International Conference on Pattern Recognition (ICPR).

[8]

Erwan J. David, Jesus Gutierrez, Antoine Coutrot, Matthieu Perreira Da Silva, and Patrick Le Callet. 2018. A dataset of head and eye movements for 360-degree videos. In roceedings of the 9th ACM on Multimedia Systems Conference (MMSys'18).

[9]

Ching-Ling Fan, Jean Lee, Wen-Chih Lo, Chun-Ying Huang, Kuan-Ta Chen, and Cheng-Hsin Hsu. 2017. Fixation Prediction for 360 Video Streaming in Head-Mounted Virtual Reality. In ACM Workshop on Network and Operating Systems Support for Digital Audio and Video (NOSSDAV).

Digital Library

[10]

Yu Fang, Ryoichi Nakashima, Kazumichi Matsumiya, Ichiro Kuriki, and Satoshi Shioiri. 2015. Eye-head coordination for visual cognitive processing. In PLoS ONE 10(3): e0121035.

[11]

John M Findlay and Iain D Gilchrist. 2008. Active Vision: The Psychology of Looking and Seeing. Oxford Scholarship Online.

[12]

Hadi Hadizadeh and Ivan V. Bajic. 2013. Saliency-Aware Video Compression. In IEEE Transactions on Image Processing.

Digital Library

[13]

Jian He, Mubashir Adnan Qureshi, Lili Qiu, Jin Li, Feng Li, and Lei Han. 2018. Rubiks: Practical 360-Degree Streaming for Smartphones. In Proceedings of the 16th Annual International Conference on Mobile Systems, Applications, and Services (MobiSys'18).

Digital Library

[14]

Xun Huang, Chengyao Shen, Xavier Boix, and Qi Zhao. 2015. Salicon: Reducing the semantic gap in saliency prediction by adapting deep neural networks. In ICCV.

Digital Library

[15]

Mordor Intelligence. 2018. MS Windows NT Kernel Description. https://www.mordorintelligence.com/industry-reports/virtual-reality-market

[16]

Tilke Judd, Krista Ehinger, Fredo Durand, and Antonio Torralba. 2009. Learning to predict where humans look. In International Conference on Computer Vision (ICCV).

[17]

Wolf Kienzle, Bernhard Scholkopf, Felix Wichmann, and Matthias Franz. 2007. How to Find Interesting Locations in Video: A Spatiotemporal Interest Point Detector Learned from Human Eye Movements. In Hamprecht F.A., Schnorr C., Jahne B. (eds) Pattern Recognition. DAGM 2007. Lecture Notes in Computer Science.

Digital Library

[18]

Christof Koch and Shimon Ullman. 1985. Shifts in selective visual attention: towards the underlying neural circuitry. In Synthese Library (Studies in Epistemology, Logic, Methodology, and Philosophy of Science). Springer.

[19]

Srinivas S. S. Kruthiventi, Vennela Gudisa, Jaley H. Dholakiya, and R. Venkatesh Babu. 2016. Saliency Unified: A Deep Architecture for simultaneous Eye Fixation Prediction and Salient Object Segmentation. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[20]

Wen-Chih Lo, Ching-Ling Fan, and Jean Lee. 2017. 360-degree Video Viewing Dataset in Head-Mounted Virtual Reality. In Proceedings of the 8th ACM on Multimedia Systems Conference (MMSys'17).

Digital Library

[21]

Anh Nguyen, Zhisheng Yan, and Klara Nahrstedt. 2018. Your Attention is Unique: Detecting 360-Degree Video Saliency in Head-Mounted Display for Head Movement Prediction. In ACM Multimedia Conference for 2018 (ACMMM2018).

Digital Library

[22]

Junting Pan, Elisa Sayrol, Xavier G. Nieto, Kevin McGuinness, and Noel E. O'Connor. 2016. Shallow and Deep Convolutional Networks for Saliency Prediction. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[23]

Feng Qian, Bo Han, Lusheng Ji, and Vijay Gopalakrishnan. 2016. Optimizing 360 video delivery over cellular networks. In Proceedings of the 5th Workshop on All Things Cellular Operations, Applications and Challenges - ATC '16.

Digital Library

[24]

Yashas Rai, Jesus Gutierrez, and Patrick Le Callet. 2017. A Dataset of Head and Eye Movements for 360 Degree Images. In roceedings of the 8th ACM on Multimedia Systems Conference (MMSys'17).

[25]

Vidya Setlurand, Saeko Takagi, Ramesh Raskar, Michael Gleicher, and Bruce Gooch. 2005. Automatic image retargeting. In Proceedings of the 4th international conference on Mobile and ubiquitous multimedia (MUM'05).

Digital Library

[26]

Evgeniy Upenik and Touradj Ebrahimi. 2017. A Simple Method to Obtain Visual Attention Data in Head Mounted Virtual Reality. In IEEE International Conference on Multimedia & Expo Workshops (ICMEW).

[27]

Chenglei Wu, Zhihao Tan, and Zhi Wang. 2017. 360-degree Video Viewing Dataset in Head-Mounted Virtual Reality. In Proceedings of the 8th ACM on Multimedia Systems Conference (MMSys'17).

[28]

Chengxin Yan, Nong Sang, Tianxu, and Zhangb. 2003. Local entropy-based transition region extraction and thresholding. In Pattern Recognition Letters.

Digital Library

[29]

Zhisheng Yan, Chen Song, Feng Lin, and Wenyao Xu. 2018. Exploring Eye Adaptation in Head-Mounted Display for Energy Efficient Smartphone Virtual Reality. In Proceedings of the 19th International Workshop on Mobile Computing Systems & Applications (HotMobile'18).

Digital Library

Cited By

Xu YDu JWang JNing YZhou SCao Y(2024)Panonut360Proceedings of the 15th ACM Multimedia Systems Conference10.1145/3625468.3652176(319-325)Online publication date: 15-Apr-2024
https://dl.acm.org/doi/10.1145/3625468.3652176
Wang MChen XYang XPeng SZhao YXu MXu C(2024)CoLive: Edge-Assisted Clustered Learning Framework for Viewport Prediction in 360$^{\circ }$ Live StreamingIEEE Transactions on Multimedia10.1109/TMM.2023.333011226(5078-5091)Online publication date: 2024
https://doi.org/10.1109/TMM.2023.3330112
Agrahari Baniya ALee TEklund PAryal S(2024)Omnidirectional Video Super-Resolution Using Deep LearningIEEE Transactions on Multimedia10.1109/TMM.2023.326729426(540-554)Online publication date: 1-Jan-2024
https://dl.acm.org/doi/10.1109/TMM.2023.3267294
Show More Cited By

Index Terms

A saliency dataset for 360-degree videos
1. Information systems
  1. Information systems applications
    1. Multimedia information systems
      1. Multimedia databases

Recommendations

360-Degree Video Head Movement Dataset
MMSys'17: Proceedings of the 8th ACM on Multimedia Systems Conference

While Virtual Reality applications are increasingly attracting the attention of developers and business analysts, the behaviour of users watching 360-degree (i.e. omnidirectional) videos has not been thoroughly studied yet. This paper introduces a ...
Understanding How Virtual 360-Degree Videos Generate Behavioral Intention to Visit Projected Destinations
Cross-Cultural Design
Abstract
Existing virtual reality (VR) tourism research tends to emphasize VR headsets due to a higher level of immersion. However, there is a lack of discussion over the latent impact that virtual 360-degree videos as a basic form of virtual technology ...
Toward Visual Behavior and Attention Understanding for Augmented 360 Degree Videos
Augmented reality (AR) overlays digital content onto reality. In an AR system, correct and precise estimations of user visual fixations and head movements can enhance the quality of experience by allocating more computational resources for analyzing, ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

MMSys '19: Proceedings of the 10th ACM Multimedia Systems Conference

June 2019

374 pages

ISBN:9781450362979

DOI:10.1145/3304109

General Chair:
Michael Zink
University of Massachusetts Amherst
,
Program Chairs:
Laura Toni
University College London
,
Ali C. Begen
Özyeğin University/Networked Media

Copyright © 2019 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGMM: ACM Special Interest Group on Multimedia

In-Cooperation

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 18 June 2019

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Badges

Author Tags

Qualifiers

Short-paper

Conference

MMSys '19

Sponsor:

SIGMM

MMSys '19: 10th ACM Multimedia Systems Conference

June 18 - 21, 2019

Massachusetts, Amherst

Acceptance Rates

MMSys '19 Paper Acceptance Rate 40 of 82 submissions, 49%;

Overall Acceptance Rate 176 of 530 submissions, 33%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

25
Total Citations
View Citations
565
Total Downloads

Downloads (Last 12 months)74
Downloads (Last 6 weeks)4

Reflects downloads up to 11 Aug 2024

Other Metrics

View Author Metrics

Citations

Cited By

Xu YDu JWang JNing YZhou SCao Y(2024)Panonut360Proceedings of the 15th ACM Multimedia Systems Conference10.1145/3625468.3652176(319-325)Online publication date: 15-Apr-2024
https://dl.acm.org/doi/10.1145/3625468.3652176
Wang MChen XYang XPeng SZhao YXu MXu C(2024)CoLive: Edge-Assisted Clustered Learning Framework for Viewport Prediction in 360$^{\circ }$ Live StreamingIEEE Transactions on Multimedia10.1109/TMM.2023.333011226(5078-5091)Online publication date: 2024
https://doi.org/10.1109/TMM.2023.3330112
Agrahari Baniya ALee TEklund PAryal S(2024)Omnidirectional Video Super-Resolution Using Deep LearningIEEE Transactions on Multimedia10.1109/TMM.2023.326729426(540-554)Online publication date: 1-Jan-2024
https://dl.acm.org/doi/10.1109/TMM.2023.3267294
Tliba MZhou XViola ICesar PChetouani AValenzise GDufaux F(2024)Enhancing Immersive Experiences through 3D Point Cloud Analysis: A Novel Framework for Applying 2D Visual Saliency Models to 3D Point Clouds2024 16th International Conference on Quality of Multimedia Experience (QoMEX)10.1109/QoMEX61742.2024.10598254(307-313)Online publication date: 18-Jun-2024
https://doi.org/10.1109/QoMEX61742.2024.10598254
Kotseruba ITsotsos J(2024)Data Limitations for Modeling Top-Down Effects on Drivers’ Attention2024 IEEE Intelligent Vehicles Symposium (IV)10.1109/IV55156.2024.10588528(1345-1352)Online publication date: 2-Jun-2024
https://doi.org/10.1109/IV55156.2024.10588528
Nguyen AYan Z(2023)Enhancing 360 Video Streaming through Salient Content in Head-Mounted DisplaysSensors10.3390/s2308401623:8(4016)Online publication date: 15-Apr-2023
https://doi.org/10.3390/s23084016
Chen XWang MXu CZhao YShujie YJiang KLi QZhong LMuntean G(2023)FedLive: A Federated Transmission Framework for Panoramic Livecast With Reinforced Variational InferenceIEEE Transactions on Multimedia10.1109/TMM.2023.323732525(8471-8486)Online publication date: 1-Jan-2023
https://dl.acm.org/doi/10.1109/TMM.2023.3237325
Zhou XViola IAlexiou EJansen JCesar P(2023)QAVA-DPC: Eye-Tracking Based Quality Assessment and Visual Attention Dataset for Dynamic Point Cloud in 6 DoF2023 IEEE International Symposium on Mixed and Augmented Reality (ISMAR)10.1109/ISMAR59233.2023.00021(69-78)Online publication date: 16-Oct-2023
https://doi.org/10.1109/ISMAR59233.2023.00021
Zhang YWu MTian BLi JChen BZhou QNahrstedt K(2023)SAVG360: Saliency-aware Viewport-guidance-enabled 360-video Streaming System2023 IEEE International Symposium on Multimedia (ISM)10.1109/ISM59092.2023.00011(36-43)Online publication date: 11-Dec-2023
https://doi.org/10.1109/ISM59092.2023.00011
Lin CChen YWang YChou YYang YLin NLiao W(2023)Adaptive Wireless Streaming for Tile-based 360-degree Video2023 International Conference on Consumer Electronics - Taiwan (ICCE-Taiwan)10.1109/ICCE-Taiwan58799.2023.10226708(775-776)Online publication date: 17-Jul-2023
https://doi.org/10.1109/ICCE-Taiwan58799.2023.10226708
Show More Cited By

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents