research-article

Interactive Lifelog Retrieval with vitrivr

Authors:

Mahnaz Amiri Parian,

Heiko SchuldtAuthors Info & Claims

LSC '20: Proceedings of the Third Annual Workshop on Lifelog Search Challenge

Pages 1 - 6

https://doi.org/10.1145/3379172.3391715

Published: 09 June 2020 Publication History

Abstract

The variety and amount of data being collected in our everyday life poses unique challenges for multimedia retrieval. In the Lifelog Search Challenge (LSC), multimedia retrieval systems compete in finding events based on descriptions containing hints about structured, semi-structured an unstructured data. In this paper, we present the multimedia retrieval system vitrivr with a focus on the changes and additions made based on the new dataset, and our successful participation at LSC 2019. Specifically, we show how the new dataset can be used for retrieval in different modalities without sacrificing efficiency, describe two recent additions, temporal scoring and staged querying, and discuss the deep learning methods used to enrich the dataset.

References

[1]

Michael Calonder, Vincent Lepetit, Mustafa Ö zuysal, Tomasz Trzcinski, Christoph Strecha, and Pascal Fua. 2012. BRIEF: Computing a Local Binary Descriptor Very Fast. IEEE Trans. Pattern Anal. Mach. Intell., Vol. 34, 7 (2012), 1281--1298.

Digital Library

[2]

Claudiu Cobâ rzan, Klaus Schoeffmann, Werner Bailer, Wolfgang Hü rst, Adam Blazek, Jakub Lokoc, Stefanos Vrochidis, Kai Uwe Barthel, and Luca Rossetto. 2017. Interactive Video Search Tools: a Detailed Analysis of the Video Browser Showdown 2015. Multimedia Tools Appl., Vol. 76, 4 (2017), 5539--5571.

[3]

Aaron Duane and Cathal Gurrin. 2020. Baseline Analysis of a Conventional and Virtual Reality Lifelog Retrieval System. In Proceedings of the 26th International Conference on MultiMedia Modeling (MMM 2020), Part II (Lecture Notes in Computer Science), Vol. 11962. Springer, Daejeon, South Korea, 412--423.

Digital Library

[4]

Aaron Duane, Cathal Gurrin, and Wolfgang Hü rst. 2018. Virtual Reality Lifelog Explorer: Lifelog Search Challenge at ACM ICMR 2018. In Proceedings of the 2018 ACM Workshop on The Lifelog Search Challenge (LSC@ICMR 2018). ACM, Yokohama, Japan, 20--23.

Digital Library

[5]

Ralph Gasser, Luca Rossetto, and Heiko Schuldt. 2019. Multimodal Multimedia Retrieval with vitrivr. In Proceedings of the 2019 on International Conference on Multimedia Retrieval (ICMR 2019). ACM, Ottawa, ON, Canada, 391--394.

Digital Library

[6]

Ivan Giangreco. 2018. Database support for large-scale multimedia retrieval. Ph.D. Dissertation. University of Basel.

[7]

Cathal Gurrin, Hideo Joho, Frank Hopfgartner, Liting Zhou, and Rami Albatal. 2016. NTCIR Lifelog: The First Test Collection for Lifelog Research. In Proceedings of the 39th International ACM SIGIR conference on Research and Development in Information Retrieval (SIGIR 2016). ACM, Pisa, Italy, 705--708.

Digital Library

[8]

Cathal Gurrin, Tu-Khiem Le, Van-Tu Ninh, Duc-Tien Dang-Nguyen, Björn Þór Jónsson, Jakub Loko, Wolfgang Hurst, Minh-Triet Tran, and Klaus Schoeffmann. 2020. An Introduction to the Third Annual Lifelog Search Challenge, LSC'20. In ICMR '20, The 2020 International Conference on Multimedia Retrieval. ACM, Dublin, Ireland.

Digital Library

[9]

Cathal Gurrin, Klaus Schoeffmann, Hideo Joho, Andreas Leibetseder, Liting Zhou, Aaron Duane, Dang Nguyen, Duc Tien, Michael Riegler, Luca Piras, Minh-Triet Tran, Jakub Lokoc, and Wolfgang Hürst. 2019 a. Comparing Approaches to Interactive Lifelog Search at the Lifelog Search Challenge (LSC2018). ITE Transactions on Media Technology and Applications, Vol. 7, 2 (2019), 46--59.

[10]

Cathal Gurrin, Klaus Schoeffmann, Hideo Joho, Bernd Münzer, Rami Albatal, Frank Hopfgartner, Liting Zhou, and Duc-Tien Dang-Nguyen. 2019 b. A Test Collection for Interactive Lifelog Retrieval. In Proceedings of the 25th International Conference on MultiMedia Modeling (MMM 2019), Part I (Lecture Notes in Computer Science), Vol. 11295. Springer, Thessaloniki, Greece, 312--324.

[11]

Cathal Gurrin, Alan F. Smeaton, and Aiden R. Doherty. 2014. LifeLogging: Personal Big Data. Foundations and Trends in Information Retrieval, Vol. 8, 1 (2014), 1--125.

Digital Library

[12]

Silvan Heller, Loris Sauter, Heiko Schuldt, and Luca Rossetto. 2020. Multi-Stage Queries and Temporal Scoring in vitrivr. In Proceesings of the IEEE International Conference on Multimedia & Expo Workshops (ICME Workshops 2020). IEEE .

[13]

Tim Jacquemard, Peter Novitzky, Fiachra O'Brolchá in, Alan F. Smeaton, and Bert Gordijn. 2014. Challenges and Opportunities of Lifelog Technologies: A Literature Review and Critical Analysis. Science and Engineering Ethics, Vol. 20, 2 (2014), 379--409.

[14]

Isadora Nguyen Van Khan, Pranita Shrestha, Min Zhang, Yiqun Liu, and Shaoping Ma. 2019 b. A Two-Level Lifelog Search Engine at the LSC 2019. In Proceedings of the ACM Workshop on Lifelog Search Challenge (LSC@ICMR 2019). ACM, Ottawa, ON, Canada, 19--23.

Digital Library

[15]

Omar Shahbaz Khan, Bjö rn Þó r Jó nsson, Jan Zahá lka, Stevan Rudinac, and Marcel Worring. 2019 a. Exquisitor at the Lifelog Search Challenge 2019. In Proceedings of the ACM Workshop on Lifelog Search Challenge (LSC@ICMR 2019). ACM, Ottawa, ON, Canada, 7--11.

[16]

Tsung-Yi Lin, Piotr Dollár, Ross B. Girshick, Kaiming He, Bharath Hariharan, and Serge J. Belongie. 2017. Feature Pyramid Networks for Object Detection. In Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR 2017). IEEE Computer Society, Honolulu, HI, USA, 936--944.

[17]

Tsung-Yi Lin, Michael Maire, Serge J. Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollá r, and C. Lawrence Zitnick. 2014. Microsoft COCO: Common Objects in Context. In Proceeding of the 13th European Conference on Computer Vision (ECCV 2014), Part V (Lecture Notes in Computer Science), Vol. 8693. Springer, Zürich, Switzerland, 740--755.

[18]

Jakub Lokoc, Werner Bailer, Klaus Schoeffmann, Bernd Münzer, and George Awad. 2018. On Influential Trends in Interactive Video Retrieval: Video Browser Showdown 2015--2017. IEEE Trans. Multimedia, Vol. 20, 12 (2018), 3361--3376.

Digital Library

[19]

Jakub Lokoc, Gregor Kovalcik, Bernd Mü nzer, Klaus Schöffmann, Werner Bailer, Ralph Gasser, Stefanos Vrochidis, Phuong Anh Nguyen, Sitapa Rujikietgumjorn, and Kai Uwe Barthel. 2019 a. Interactive Search or Sequential Browsing? A Detailed Analysis of the Video Browser Showdown 2018. TOMM, Vol. 15, 1 (2019), 29:1--29:18.

[20]

Jakub Lokoc, Gregor Kovalcik, Tomás Soucek, Jaroslav Moravec, and Premysl Cech. 2019 b. A Framework for Effective Known-item Search in Video. In Proceedings of the 27th ACM International Conference on Multimedia (MM 2019). ACM, Nice, France, 1777--1785.

Digital Library

[21]

Jakub Lokoc, Gregor Kovalcik, Tomás Soucek, Jaroslav Moravec, and Premysl Cech. 2019 c. VIRET: A Video Retrieval Tool for Interactive Known-item Search. In Proceedings of the 2019 on International Conference on Multimedia Retrieval (ICMR 2019). ACM, Ottawa, ON, Canada, 177--181.

Digital Library

[22]

Phuong Anh Nguyen, Jiaxin Wu, Chong-Wah Ngo, Danny Francis, and Benoit Huet. 2020. VIREO @ Video Browser Showdown 2020. In Proceedings of the 26th International Conference on MultiMedia Modeling (MMM 2020), Part II (Lecture Notes in Computer Science), Vol. 11962. Springer, Daejeon, South Korea, 772--777.

Digital Library

[23]

Joseph Redmon, Santosh Kumar Divvala, Ross B. Girshick, and Ali Farhadi. 2016. You Only Look Once: Unified, Real-Time Object Detection. In Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPR Workshops 2016). IEEE Computer Society, Las Vegas, NV, USA, 779--788.

[24]

Shaoqing Ren, Kaiming He, Ross B. Girshick, and Jian Sun. 2015. Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks. In Proceedings of the Annual Conference on Neural Information Processing Systems: Advances in Neural Information Processing Systems. Montreal, Quebec, Canada, 91--99.

[25]

Luca Rossetto, Ralph Gasser, Silvan Heller, Mahnaz Amiri Parian, and Heiko Schuldt. 2019 b. Retrieval of Structured and Unstructured Data with vitrivr. In Proceedings of the ACM Workshop on Lifelog Search Challenge (LSC@ICMR 2019). ACM, Ottawa, ON, Canada, 27--31.

Digital Library

[26]

L. Rossetto, R. Gasser, J. Lokoc, W. Bailer, K. Schoeffmann, B. Muenzer, T. Soucek, P. A. Nguyen, P. Bolettieri, A. Leibetseder, and S. Vrochidis. 2020. Interactive Video Retrieval in the Age of Deep Learning - Detailed Evaluation of VBS 2019. IEEE Transactions on Multimedia (2020), 1--1.

[27]

Luca Rossetto, Ralph Gasser, and Heiko Schuldt. 2019 a. Query by Semantic Sketch. CoRR, Vol. abs/1909.12526 (2019). arxiv: 1909.12526

[28]

Luca Rossetto, Ivan Giangreco, Silvan Heller, Claudiu Tanase, and Heiko Schuldt. 2016a. Searching in Video Collections Using Sketches and Sample Images -- The Cineast System. In Proceedings of the 22nd International Conference on MultiMedia Modeling (MMM 2016), Part II (Lecture Notes in Computer Science), Vol. 9517. Springer, Miami, FL, USA, 336--341.

[29]

L. Rossetto, I. Giangreco, and H. Schuldt. 2014. Cineast: A Multi-feature Sketch-Based Video Retrieval Engine. In 2014 IEEE International Symposium on Multimedia. 18--23.

[30]

Luca Rossetto, Ivan Giangreco, Claudiu Tanase, and Heiko Schuldt. 2016b. vitrivr: A Flexible Retrieval Stack Supporting Multiple Query Modes for Searching in Multimedia Collections. In Proceedings of the 2016 ACM Conference on Multimedia Conference (MM 2016). ACM, Amsterdam, The Netherlands, 1183--1186.

Digital Library

[31]

Luca Rossetto, Mahnaz Amiri Parian, Ralph Gasser, Ivan Giangreco, Silvan Heller, and Heiko Schuldt. 2019 c. Deep Learning-Based Concept Detection in vitrivr. In Proceedings of the 25th International Conference on MultiMedia Modeling (MMM 2019) (Lecture Notes in Computer Science), Vol. 11296. Springer, Thessaloniki, Greece, 616--621.

[32]

Loris Sauter, Mahnaz Amiri Parian, Ralph Gasser, Silvan Heller, Luca Rossetto, and Heiko Schuldt. 2020. Combining Boolean and Multimedia Retrieval in vitrivr for Large-Scale Video Search. In Proceedings of the 26th International Conference on MultiMedia Modeling (MMM 2020), Part II (Lecture Notes in Computer Science), Vol. 11962. Springer, Daejeon, South Korea, 760--765.

Digital Library

[33]

Loris Sauter, Luca Rossetto, and Heiko Schuldt. 2018. Exploring Cultural Heritage in Augmented Reality with GoFind!. In Proceedings of the 2018 IEEE International Conference on Artificial Intelligence and Virtual Reality (AIVR 2018). IEEE Computer Society, Taichung, Taiwan, 187--188.

[34]

Kenneth Tran, Xiaodong He, Lei Zhang, and Jian Sun. 2016. Rich Image Captioning in the Wild. In Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPR Workshops 2016). IEEE Computer Society, Las Vegas, NV, USA, 434--441.

[35]

Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron C. Courville, Ruslan Salakhutdinov, Richard S. Zemel, and Yoshua Bengio. 2015. Show, Attend and Tell: Neural Image Caption Generation with Visual Attention. In Proceedings of the 32nd International Conference on Machine Learning (ICML 2015), JMLR Workshop and Conference Proceedings, Vol. 37. JMLR.org, Lille, France, 2048--2057.

[36]

Quanzeng You, Hailin Jin, Zhaowen Wang, Chen Fang, and Jiebo Luo. 2016. Image Captioning with Semantic Attention. In Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR 2016). IEEE Computer Society, Las Vegas, NV, USA, 4651--4659.

Cited By

陈亮(2024)A Lifelog Management Model Based on EventsComputer Science and Application10.12677/CSA.2024.14100514:01(29-40)Online publication date: 2024
https://doi.org/10.12677/CSA.2024.141005
Sauter LGasser RRettig LSchuldt HRossetto L(2024)General Purpose Multimedia Retrieval with vitrivr at LSC'24Proceedings of the 7th Annual ACM Workshop on the Lifelog Search Challenge10.1145/3643489.3661120(47-52)Online publication date: 10-Jun-2024
https://dl.acm.org/doi/10.1145/3643489.3661120
Tran MTo TThai VCao TNguyen T(2023)AGAIN: A Multimodal Human-Centric Event Retrieval System using dual image-to-text representationsProceedings of the 12th International Symposium on Information and Communication Technology10.1145/3628797.3628975(931-937)Online publication date: 7-Dec-2023
https://dl.acm.org/doi/10.1145/3628797.3628975
Show More Cited By

Index Terms

Interactive Lifelog Retrieval with vitrivr
1. Information systems
  1. Information retrieval
    1. Specialized information retrieval
      1. Multimedia and multimodal retrieval
        Image search
        Speech / audio search
        Video search
  2. Information systems applications
    1. Multimedia information systems
      1. Multimedia databases

Recommendations

Exploring Intuitive Lifelog Retrieval and Interaction Modes in Virtual Reality with vitrivr-VR
LSC '21: Proceedings of the 4th Annual on Lifelog Search Challenge

The multimodal nature of lifelog data collections poses unique challenges for multimedia management and retrieval systems. The Lifelog Search Challenge (LSC) offers an annual evaluation platform for such interactive retrieval systems. They compete ...
Interactive Multimodal Lifelog Retrieval with vitrivr at LSC 2021
LSC '21: Proceedings of the 4th Annual on Lifelog Search Challenge

The Lifelog Search Challenge (LSC) is an annual benchmarking competition for interactive multimedia retrieval systems, where participating systems compete in finding events based on textual descriptions containing hints about structured, semi-structured,...
General Purpose Multimedia Retrieval with vitrivr at LSC'24
LSC '24: Proceedings of the 7th Annual ACM Workshop on the Lifelog Search Challenge

The collection of lifelog data --- visual and multi-sensory data, including biometric and spatiotemporal metadata --- becomes easier and more supported by commercial products every year. Naturally, lifelog data is multi-modal, with arguably a major audio-...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

LSC '20: Proceedings of the Third Annual Workshop on Lifelog Search Challenge

June 2020

89 pages

ISBN:9781450371360

DOI:10.1145/3379172

General Chairs:
Cathal Gurrin
Dublin City University, Ireland
,
Klaus Schoeffmann
Klagenfurt University, Austria
,
Björn Þór Jónsson
IT University of Copenhagen, Denmark
,
Duc-Tien Dang-Nguyen
University of Bergen, Norway
,
Jakub Lokoč
Charles University, Czech Republic
,
Minh-Triet Tran
HCM University of Science, Vietnam
,
Wolfgang Hürst
Utrecht University, Netherlands

Copyright © 2020 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

SIGMM: ACM Special Interest Group on Multimedia

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 09 June 2020

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

ICMR '20

Sponsor:

SIGMM

ICMR '20: International Conference on Multimedia Retrieval

June 9, 2020

Dublin, Ireland

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

25
Total Citations
View Citations
178
Total Downloads

Downloads (Last 12 months)9
Downloads (Last 6 weeks)0

Reflects downloads up to 10 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

陈亮(2024)A Lifelog Management Model Based on EventsComputer Science and Application10.12677/CSA.2024.14100514:01(29-40)Online publication date: 2024
https://doi.org/10.12677/CSA.2024.141005
Sauter LGasser RRettig LSchuldt HRossetto L(2024)General Purpose Multimedia Retrieval with vitrivr at LSC'24Proceedings of the 7th Annual ACM Workshop on the Lifelog Search Challenge10.1145/3643489.3661120(47-52)Online publication date: 10-Jun-2024
https://dl.acm.org/doi/10.1145/3643489.3661120
Tran MTo TThai VCao TNguyen T(2023)AGAIN: A Multimodal Human-Centric Event Retrieval System using dual image-to-text representationsProceedings of the 12th International Symposium on Information and Communication Technology10.1145/3628797.3628975(931-937)Online publication date: 7-Dec-2023
https://dl.acm.org/doi/10.1145/3628797.3628975
Tran LNguyen MDang-Nguyen DHeller SSpiess FLokoč JPeška LNguyen TKhan ODuane AJónsson BRossetto LYen AAlateeq AAlam NTran MHealy GSchoeffmann KGurrin C(2023)Comparing Interactive Retrieval Approaches at the Lifelog Search Challenge 2021IEEE Access10.1109/ACCESS.2023.324828411(30982-30995)Online publication date: 2023
https://doi.org/10.1109/ACCESS.2023.3248284
Heller SSpiess FSchuldt H(2023)A tale of two interfaces: vitrivr at the lifelog search challengeMultimedia Tools and Applications10.1007/s11042-023-15082-w82:24(37829-37853)Online publication date: 6-Apr-2023
https://doi.org/10.1007/s11042-023-15082-w
Alam NGraham Y(2023)Memento: a prototype search engine for LSC 2021Multimedia Tools and Applications10.1007/s11042-023-15067-982:24(37807-37828)Online publication date: 1-Apr-2023
https://doi.org/10.1007/s11042-023-15067-9
Ribeiro RTrifan ANeves A(2022)Lifelog Retrieval From Daily Digital Data: Narrative ReviewJMIR mHealth and uHealth10.2196/3051710:5(e30517)Online publication date: 2-May-2022
https://doi.org/10.2196/30517
Gasser RRossetto LHeller SSchuldt H(2022)Multimedia Retrieval and Analysis with Cottontail DBACM SIGMultimedia Records10.1145/3577934.357794013:1(1-1)Online publication date: 19-Dec-2022
https://dl.acm.org/doi/10.1145/3577934.3577940
Khan OZahálka JJónsson BRossetto LBailer WSchoeffmann KLokoč J(2022)Influence of Late Fusion of High-Level Features on User Relevance Feedback for VideosProceedings of the 2nd International Workshop on Interactive Multimedia Retrieval10.1145/3552467.3554795(17-24)Online publication date: 14-Oct-2022
https://dl.acm.org/doi/10.1145/3552467.3554795
Jónsson BDuane AMertz N(2022)Relational Database Performance for Multimedia: A Case StudyProceedings of the 19th International Conference on Content-based Multimedia Indexing10.1145/3549555.3549558(186-190)Online publication date: 14-Sep-2022
https://dl.acm.org/doi/10.1145/3549555.3549558
Show More Cited By

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents