Known-Item Search in Video: An Eye Tracking-Based Study
Authors: 
Lucas Joos, Bastian Jäckl, Daniel A. Keim, Maximilian T. Fischer, Ladislav Peska, Jakub LokočAuthors Info & Claims
Lucas Joos, Bastian Jäckl, Daniel A. Keim, Maximilian T. Fischer, Ladislav Peska, Jakub LokočAuthors Info & ClaimsPages 311 - 319
Abstract
Deep learning has revolutionized multimedia retrieval, yet effectively searching within large video collections remains a complex challenge. This paper focuses on the design and evaluation of known-item search systems, leveraging the strengths of CLIP-based deep neural networks for ranking. At events like the Video Browser Showdown, these models have shown promise in effectively ranking the video frames. While ranking models can be pre-selected automatically based on a benchmark collection, the selection of an optimal browsing interface, crucial for refining top-ranked items, is complex and heavily influenced by user behavior. Our study addresses this by presenting an eye tracking-based analysis of user interaction with different image grid layouts. This approach offers novel insights into search patterns and user preferences, particularly examining the trade-off between displaying fewer but larger images versus more but smaller images. Our findings reveal a preference for grids with fewer images and detail how image similarity and grid position affect user search behavior. These results not only enhance our understanding of effective video retrieval interface design but also set the stage for future advancements in the field.
References
[1]
Richard Andersson, Linnea Larsson, Kenneth Holmqvist, Martin Stridh, and Marcus Nyström. 2017. One algorithm to rule them all? An evaluation and discussion of ten eye movement event-detection algorithms. Behavior research methods, Vol. 49 (2017), 616--637. https://doi.org/10.3758/s13428-016-0738-9
[2]
Mahmoud Artemi and Haiming Liu. 2021. A User Study on User Attention for an Interactive Content-based Image Search System. In CEUR Workshop Proceedings, Vol. 2863. CEUR-WS.org, 26--39.
[3]
Kai Uwe Barthel, Nico Hezel, Jung-Hyun Kim, and Konstantin Schall. 2023. Improved Evaluation and Generation Of Grid Layouts Using Distance Preservation Quality and Linear Assignment Sorting. Computer Graphics Forum, Vol. 42, 1 (2023), 261--276. https://doi.org/10.1111/CGF.14718
[4]
Jan-Eike Golenia, Markus A Wenzel, Mihail Bogojeski, and Benjamin Blankertz. 2018. relevance feedback from electroencephalography and eye tracking in image search. Journal of Neural Engineering, Vol. 15, 2 (2018), 1--10. https://doi.org/10.1088/1741-2552/aa9999
[5]
Ian Goodfellow, Yoshua Bengio, and Aaron Courville. 2016. Deep Learning. MIT Press. http://www.deeplearningbook.org.
[6]
James Hafner, Harpreet S. Sawhney, William Equitz, Myron Flickner, and Wayne Niblack. 1995. Efficient color histogram indexing for quadratic form distance functions. IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 17, 7 (1995), 729--736. https://doi.org/10.1109/34.391417
[7]
Gabriel Ilharco, Mitchell Wortsman, Ross Wightman, Cade Gordon, Nicholas Carlini, Rohan Taori, Achal Dave, Vaishaal Shankar, Hongseok Namkoong, John Miller, Hannaneh Hajishirzi, Ali Farhadi, and Ludwig Schmidt. 2021. OpenCLIP (v0.1). Zenodo. https://doi.org/10.5281/zenodo.5143773
[8]
Dirk Lewandowski and Yvonne Kammerer. 2021. Factors influencing viewing behaviour on search engine results pages: a review of eye-tracking research. Behaviour & Information Technology, Vol. 40, 14 (2021), 1485--1515. https://doi.org/10.1080/0144929X.2020.1761450
[9]
Zhen Liang, Hong Fu, Yun Zhang, Zheru Chi, and Dagan Feng. 2010. Content-Based Image Retrieval Using a Combination of Visual Features and Eye Tracking Data. In Proceedings of the 2010 Symposium on Eye-Tracking Research & Applications (ETRA '10). ACM, 41--44. https://doi.org/10.1145/1743666.1743675
[10]
Jakub Lokovc, Stelios Andreadis, Werner Bailer, Aaron Duane, Cathal Gurrin, Zhixin Ma, Nicola Messina, Thao-Nhu Nguyen, Ladislav Pevs ka, Luca Rossetto, Loris Sauter, Konstantin Schall, Klaus Schoeffmann, Omar Shahbaz Khan, Florian Spiess, Lucia Vadicamo, and Stefanos Vrochidis. 2023. Interactive video retrieval in the age of effective joint embedding deep models: lessons from the 11th VBS. Multimedia Systems, Vol. 29, 6 (2023), 3481--3504. https://doi.org/10.1007/S00530-023-01143-5
[11]
Jakub Lokoć, Tomávs Souçek, Patrik Veselý, Frantivsek Mejzlík, Jiaqi Ji, Chaoxi Xu, and Xirong Li. 2020. A W2VV Case Study with Automated and Interactive Text-to-Video Retrieval. In Proceedings of the 28th ACM International Conference on Multimedia (MM '20). ACM, 2553--2561. https://doi.org/10.1145/3394171.3414002
[12]
Jakub Lokovc, Werner Bailer, Klaus Schoeffmann, Bernd Mü nzer, and George Awad. 2018. On Influential Trends in Interactive Video Retrieval: Video Browser Showdown 2015--2017. IEEE Transactions on Multimedia, Vol. 20, 12 (2018), 3361--3376. https://doi.org/10.1109/TMM.2018.2830110
[13]
Wanxuan Lu and Yunde Jia. 2014. An Eye-Tracking Study of User Behavior in Web Image Search. In PRICAI 2014: Trends in Artificial Intelligence. Springer, 170--182. https://doi.org/10.1007/978-3-319-13560-1_14
[14]
Marc Mahy, Luc Van Eycken, and André Oosterlinck. 1994. Evaluation of uniform color spaces developed after the adoption of CIELAB and CIELUV. Color Research & Application, Vol. 19, 2 (1994), 105--121. https://doi.org/10.1111/j.1520--6378.1994.tb00070.x
[15]
Henning Müller, Nicolas Michoux, David Bandon, and Antoine Geissbuhler. 2004. A review of content-based image retrieval systems in medical applications-clinical benefits and future directions. International Journal of Medical Informatics, Vol. 73, 1 (2004), 1--23. https://doi.org/10.1016/j.ijmedinf.2003.11.024
[16]
Georgios Th. Papadopoulos, Konstantinos C. Apostolakis, and Petros Daras. 2014. Gaze-Based Relevance Feedback for Realizing Region-Based Image Retrieval. IEEE Trans. Multim., Vol. 16, 2 (2014), 440--454. https://doi.org/10.1109/TMM.2013.2291535
[17]
Stanislav Popelka, Zdeněk Stachoň, čeněk ?a?inka, and Jitka Dole?alová. 2016. EyeTribe Tracker Data Accuracy Evaluation and Its Interconnection with Hypothesis Software for Cartographic Purposes. Computational Intelligence and Neuroscience, Vol. 2016 (2016), 1--14. https://doi.org/10.1155/2016/9172506
[18]
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. 2021. Learning Transferable Visual Models From Natural Language Supervision. In Proceedings of the 38th International Conference on Machine Learning, ICML, Vol. 139. PMLR, 8748--8763.
[19]
Dario D. Salvucci and Joseph H. Goldberg. 2000. Identifying fixations and saccades in eye-tracking protocols. In Proceedings of the 2000 Symposium on Eye Tracking Research & Applications (ETRA '00). ACM, 71--78. https://doi.org/10.1145/355017.355028
[20]
Konstantin Schall, Kai Uwe Barthel, Nico Hezel, and Klaus Jung. 2023. Improving Image Encoders for General-Purpose Nearest Neighbor Search and Classification. In Proceedings of the 2023 ACM International Conference on Multimedia Retrieval (ICMR '23). ACM, 57--66. https://doi.org/10.1145/3591106.3592266
[21]
Arnold W. M. Smeulders, Marcel Worring, Simone Santini, Amarnath Gupta, and Ramesh Jain. 2000. Content-based image retrieval at the end of the early years. IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 22, 12 (2000), 1349--1380. https://doi.org/10.1109/34.895972
[22]
The Eye Tribe. 2016. The Eye Tribe. https://theeyetribe.com. Accessed: 2024-01-31.
[23]
Quang-Trung Truong, Tuan-Anh Vu, Tan-Sang Ha, Jakub Lokovc, Yue-Him Wong, Ajay Joneja, and Sai-Kit Yeung. 2023. Marine Video Kit: A New Marine Video Dataset for Content-Based Analysis and Retrieval. In MultiMedia Modeling - 29th International Conference, MMM 2023, Vol. 13833. Springer, 539--550. https://doi.org/10.1007/978-3-031-27077-2_42
[24]
Colin C. Venters and Matthew Cooper. 2000. A review of content-based image retrieval systems. Technical Report. Manchester Visualization Centre, University of Manchester, UK.
[25]
Xi Wang, Andreas Ley, Sebastian Koch, David Lindlbauer, James Hays, Kenneth Holmqvist, and Marc Alexa. 2019. The Mental Image Revealed by Gaze Tracking. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (CHI '19). ACM, 1--12. https://doi.org/10.1145/3290605.3300839
[26]
Zhijing Wu, Xiaohui Xie, Yiqun Liu, Min Zhang, and Shaoping Ma. 2017. A Study of User Image Search Behavior Based on Log Analysis. In Information Retrieval, Vol. 10390. Springer, 69--80. https://doi.org/10.1007/978-3-319-68699-8_6
[27]
Xiaohui Xie, Yiqun Liu, Xiaochuan Wang, Meng Wang, Zhijing Wu, Yingying Wu, Min Zhang, and Shaoping Ma. 2017. Investigating Examination Behavior of Image Search Users. In Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR '17). ACM, 275--284. https://doi.org/10.1145/3077136.3080799
[28]
Ying Zhou, Jiajun Wang, and Zheru Chi. 2018. Content-based image retrieval based on eye-tracking. In Proceedings of the Workshop on Communication by Gaze Interaction (COGAIN '18). ACM, Article 9, bibinfonumpages7 pages. https://doi.org/10.1145/3206343.3206353
[29]
Yi Zhuang and Nan Jiang. 2023. Effective and efficient content-based similarity retrieval of large lung CT images based on WSSLN model. PLOS ONE, Vol. 18, 9 (2023), 1--18. https://doi.org/10.1371/journal.pone.0285573
Index Terms
- Known-Item Search in Video: An Eye Tracking-Based Study
Recommendations
A Framework for Effective Known-item Search in Video
MM '19: Proceedings of the 27th ACM International Conference on MultimediaSearching for one particular scene in a large video collection (known-item search) represents a challenging task for video retrieval systems. According to the recent results reached at evaluation campaigns, even respected approaches based on machine ...
What Is the Role of Similarity for Known-Item Search at Video Browser Showdown?
Similarity Search and ApplicationsAbstractAcross many domains, machine learning approaches start to compete with human experts in tasks originally considered as very difficult for automation. However, effective retrieval of general video shots still represents an issue due to their ...
Balancing thread based navigation for targeted video search
CIVR '08: Proceedings of the 2008 international conference on Content-based image and video retrievalVarious query methods for video search exist. Because of the semantic gap each method has its limitations. We argue that for effective retrieval query methods need to be combined at retrieval time. However, switching query methods often involves a ...
Comments
Information & Contributors
Information
Published In
May 2024
1379 pages
ISBN:9798400706196
DOI:10.1145/3652583
- General Chairs:
- Cathal Gurrin,
- Rachada Kongkachandra,
- Klaus Schoeffmann,
- Program Chairs:
- Duc-Tien Dang-Nguyen,
- Luca Rossetto,
- Shin'ichi Satoh,
- Liting Zhou
Copyright © 2024 Owner/Author.
This work is licensed under a Creative Commons Attribution International 4.0 License.
Sponsors
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 07 June 2024
Check for updates
Author Tags
Qualifiers
- Research-article
Conference
Acceptance Rates
Overall Acceptance Rate 254 of 830 submissions, 31%
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 73Total Downloads
- Downloads (Last 12 months)73
- Downloads (Last 6 weeks)25
Reflects downloads up to 18 Aug 2024
Other Metrics
Citations
View Options
Get Access
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in