research-article

LifeInsight: An Interactive Lifelog Retrieval System with Comprehensive Spatial Insights and Query Assistance

Authors:

Tien-Thanh Nguyen-Dang,

Xuan-Dang Thai,

Minh-Triet Tran,

Minh-Khoi Pham,

Graham HealyAuthors Info & Claims

LSC '23: Proceedings of the 6th Annual ACM Lifelog Search Challenge

Pages 59 - 64

https://doi.org/10.1145/3592573.3593106

Published: 12 June 2023 Publication History

Abstract

In this paper, we introduce LifeInsight – an interactive lifelog retrieval system developed for the sixth annual Lifelog Search Challenge (LSC’23). LifeInsight incorporates semantic search mechanisms from state-of-the-art lifelog retrieval systems while focusing on providing insights into the lifelogger’s routine using spatial information to support question-answering tasks. The system employs the Bootstrapping Language-Image Pre-training (BLIP) model for zero-shot image-text retrieval, which has been shown to achieve higher recall scores than the CLIP model on the Flickr30K dataset. In addition, the Elastic Search filtering mechanism is utilized to remove irrelevant images. Apart from semantic search mechanisms, the system also supports visual similarity search by comparing the inner product distance between the vectors in the lifelog image corpus and the query image. Furthermore, the system includes an explicit relevance feedback function, AI-based query description rewriting, and visual-example-generating features to re-phrase the query to describe it better and support end-users envisioning the targeted image for retrieval.

References

[1]

Naushad Alam, Yvette Graham, and Cathal Gurrin. 2021. Memento: A Prototype Lifelog Search Engine for LSC’21. In Proceedings of the 4th Annual on Lifelog Search Challenge (Taipei, Taiwan) (LSC ’21). Association for Computing Machinery, New York, NY, USA, 53–58. https://doi.org/10.1145/3463948.3469069

Digital Library

[2]

Ahmed Alateeq, Mark Roantree, and Cathal Gurrin. 2022. Voxento 3.0: A Prototype Voice-Controlled Interactive Search Engine for Lifelog. In Proceedings of the 5th Annual on Lifelog Search Challenge (Newark, NJ, USA) (LSC ’22). Association for Computing Machinery, New York, NY, USA, 43–47. https://doi.org/10.1145/3512729.3533009

Digital Library

[3]

Wei-Hong Ang, An-Zi Yen, Tai-Te Chu, Hen-Hsen Huang, and Hsin-Hsi Chen. 2021. LifeConcept: An Interactive Approach for Multimodal Lifelog Retrieval through Concept Recommendation. In Proceedings of the 4th Annual on Lifelog Search Challenge. 47–51.

Digital Library

[4]

Aaron Duane, Cathal Gurrin, and Wolfgang Huerst. 2018. Virtual reality lifelog explorer: lifelog search challenge at ACM ICMR 2018. In Proceedings of the 2018 ACM Workshop on The Lifelog Search Challenge. 20–23.

Digital Library

[5]

Cathal Gurrin, Alan F. Smeaton, and Aiden R. Doherty. 2014. LifeLogging: Personal Big Data. Found. Trends Inf. Retr. 8, 1 (jun 2014), 1–125. https://doi.org/10.1561/1500000033

Digital Library

[6]

Cathal Gurrin, Liting Zhou, Graham Healy, Björn Þór Jónsson, Duc-Tien Dang-Nguyen, Jakub Lokoč, Minh-Triet Tran, Wolfgang Hürst, Luca Rossetto, and Klaus Schöffmann. 2022. Introduction to the Fifth Annual Lifelog Search Challenge, LSC’22. In Proc. International Conference on Multimedia Retrieval (ICMR’22). ACM, Newark, NJ. https://doi.org/10.1145/3512527.3531439

Digital Library

[7]

Cathal Gurrin, Björn Þór Jónsson, Duc Tien Dang Nguyen, Graham Healy, Jakub Lokoc, Liting Zhou, Luca Rossetto, Minh-Triet Tran, Wolfgang Hürst, Werner Bailer, and Klaus Schoeffmann. 2023. Introduction to the Sixth Annual Lifelog Search Challenge, LSC’23. In Proc. International Conference on Multimedia Retrieval (ICMR’23) (Thessaloniki, Greece) (ICMR ’23). New York, NY, USA. https://doi.org/10.1145/3591106.3592304

Digital Library

[8]

Silvan Heller, Ralph Gasser, Mahnaz Parian-Scherb, Sanja Popovic, Luca Rossetto, Loris Sauter, Florian Spiess, and Heiko Schuldt. 2021. Interactive multimodal lifelog retrieval with Vitrivr at LSC 2021. In Proceedings of the 4th Annual on Lifelog Search Challenge. 35–39.

Digital Library

[9]

Nhat Hoang-Xuan, Hoang-Phuc Trang-Trung, E.-Ro Nguyen, Thanh-Cong Le, Mai-Khiem Tran, Tu-Khiem Le, Van-Tu Ninh, Cathal Gurrin, and Minh-Triet Tran. 2022. Flexible Interactive Retrieval SysTem 3.0 for Visual Lifelog Exploration at LSC 2022. In LSC@ICMR 2022: Proceedings of the 5th Annual on Lifelog Search Challenge, Newark, NJ, USA, June 27 - 30, 2022, Cathal Gurrin, Graham Healy, Liting Zhou, Björn Þór Jónsson, Duc-Tien Dang-Nguyen, Jakub Lokoc, Minh-Triet Tran, Wolfgang Hürst, Luca Rossetto, and Klaus Schoeffmann (Eds.). ACM, 20–26. https://doi.org/10.1145/3512729.3533013

Digital Library

[10]

Tu-Khiem Le, Tu Ninh, Duc Tien Dang Nguyen, Minh-Triet Tran, Liting Zhou, Pablo Redondo, Sinead Smyth, and Cathal Gurrin. 2019. LifeSeeker: Interactive Lifelog Search Engine at LSC 2019. 37–40. https://doi.org/10.1145/3326460.3329162

Digital Library

[11]

Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. 2022. Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation. In International Conference on Machine Learning. PMLR, 12888–12900.

[12]

Liunian Harold Li, Mark Yatskar, Da Yin, Cho-Jui Hsieh, and Kai-Wei Chang. 2019. Visualbert: A simple and performant baseline for vision and language. arXiv preprint arXiv:1908.03557 (2019).

[13]

Bernd Münzer, Andreas Leibetseder, Sabrina Kletz, Manfred Jürgen Primus, and Klaus Schoeffmann. 2018. lifexplore at the lifelog search challenge 2018. In Proceedings of the 2018 ACM Workshop on The Lifelog Search Challenge. 3–8.

Digital Library

[14]

Thao-Nhu Nguyen, Tu-Khiem Le, Van-Tu Ninh, Minh-Triet Tran, Thanh Binh Nguyen, Graham Healy, Sinéad Smyth, Annalina Caputo, and Cathal Gurrin. 2022. LifeSeeker 4.0: An Interactive Lifelog Search Engine for LSC’22. In Proceedings of the 5th Annual on Lifelog Search Challenge (Newark, NJ, USA) (LSC ’22). Association for Computing Machinery, New York, NY, USA, 14–19. https://doi.org/10.1145/3512729.3533014

Digital Library

[15]

Thao-Nhu Nguyen, Tu-Khiem Le, Van-Tu Ninh, Minh-Triet Tran, Nguyen Thanh Binh, Graham Healy, Annalina Caputo, and Cathal Gurrin. 2021. Lifeseeker 3.0: An interactive lifelog search engine for LSC’21. (2021).

[16]

Bryan A Plummer, Liwei Wang, Chris M Cervantes, Juan C Caicedo, Julia Hockenmaier, and Svetlana Lazebnik. 2015. Flickr30k entities: Collecting region-to-phrase correspondences for richer image-to-sentence models. In Proceedings of the IEEE international conference on computer vision. 2641–2649.

Digital Library

[17]

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. 2021. Learning Transferable Visual Models From Natural Language Supervision. CoRR abs/2103.00020 (2021). arXiv:2103.00020https://arxiv.org/abs/2103.00020

[18]

Joseph John Rocchio Jr. 1971. Relevance feedback in information retrieval. The SMART retrieval system: experiments in automatic document processing (1971).

[19]

Luca Rossetto, Matthias Baumgartner, Ralph Gasser, Lucien Heitz, Ruijie Wang, and Abraham Bernstein. 2021. Exploring Graph-querying approaches in LifeGraph. In Proceedings of the 4th Annual on Lifelog Search Challenge. 7–10.

Digital Library

[20]

Florian Spiess, Ralph Gasser, Silvan Heller, Luca Rossetto, Loris Sauter, Milan Van Zanten, and Heiko Schuldt. 2021. Exploring intuitive lifelog retrieval and interaction modes in virtual reality with VITRIVR-VR. In Proceedings of the 4th Annual on Lifelog Search Challenge. 17–22.

Digital Library

[21]

Ly-Duyen Tran, Manh-Duy Nguyen, Nguyen Thanh Binh, Hyowon Lee, and Cathal Gurrin. 2020. Myscéal: an experimental interactive lifelog retrieval system for LSC’20. In Proceedings of the Third Annual Workshop on Lifelog Search Challenge. 23–28.

Digital Library

[22]

Ly-Duyen Tran, Manh-Duy Nguyen, Binh Nguyen, Hyowon Lee, Liting Zhou, and Cathal Gurrin. 2022. E-Myscéal: Embedding-Based Interactive Lifelog Retrieval System for LSC’22. In Proceedings of the 5th Annual on Lifelog Search Challenge (Newark, NJ, USA) (LSC ’22). Association for Computing Machinery, New York, NY, USA, 32–37. https://doi.org/10.1145/3512729.3533012

Digital Library

[23]

Minh-Triet Tran, Nhat Hoang-Xuan, Hoang-Phuc Trang-Trung, Thanh-Cong Le, Mai-Khiem Tran, Minh-Quan Le, Tu-Khiem Le, Van-Tu Ninh, and Cathal Gurrin. 2022. V-FIRST: A Flexible Interactive Retrieval System for Video at VBS 2022. In MultiMedia Modeling - 28th International Conference, MMM 2022, Phu Quoc, Vietnam, June 6-10, 2022, Proceedings, Part II(Lecture Notes in Computer Science), Björn Þór Jónsson, Cathal Gurrin, Minh-Triet Tran, Duc-Tien Dang-Nguyen, Anita Min-Chun Hu, Huynh Thi Thanh Binh, and Benoit Huet (Eds.). Vol. 13142. Springer, 562–568. https://doi.org/10.1007/978-3-030-98355-0_55

Digital Library

[24]

Minh-Triet Tran, Thanh-An Nguyen, Quoc-Cuong Tran, Mai-Khiem Tran, Khanh Nguyen, Van-Tu Ninh, Tu-Khiem Le, Hoang-Phuc Trang-Trung, Hoang-Anh Le, Hai-Dang Nguyen, Trong-Le Do, Viet-Khoa Vo-Ho, and Cathal Gurrin. 2020. FIRST - Flexible Interactive Retrieval SysTem for Visual Lifelog Exploration at LSC 2020. In Proceedings of the Third ACM Workshop on Lifelog Search Challenge, LSC@ICMR 2020, Dublin, Ireland, June 8-11, 2020, Cathal Gurrin, Klaus Schöffmann, Björn Þór Jónsson, Duc-Tien Dang-Nguyen, Jakub Lokoc, Minh-Triet Tran, and Wolfgang Hürst (Eds.). ACM, 67–72. https://doi.org/10.1145/3379172.3391726

Digital Library

[25]

Hoang-Phuc Trang-Trung, Thanh-Cong Le, Mai-Khiem Tran, Van-Tu Ninh, Tu-Khiem Le, Cathal Gurrin, and Minh-Triet Tran. 2021. Flexible Interactive Retrieval SysTem 2.0 for Visual Lifelog Exploration at LSC 2021. In Proceedings of the 4th Annual on Lifelog Search Challenge, LSC@ICMR 2021, Taipei, Taiwan, 21 August 2021, Cathal Gurrin, Klaus Schoeffmann, Björn Þór Jónsson, Duc-Tien Dang-Nguyen, Jakub Lokoc, Minh-Triet Tran, Wolfgang Hürst, Luca Rossetto, and Graham Healy (Eds.). ACM, 81–87. https://doi.org/10.1145/3463948.3469072

Digital Library

Cited By

Vopálková ZYaghob JStroh MSchlegel ULokoc J(2024)Searching Temporally Distant Activities in Lifelog Data With PraK Tool V2Proceedings of the 7th Annual ACM Workshop on the Lifelog Search Challenge10.1145/3643489.3661131(111-116)Online publication date: 10-Jun-2024
https://dl.acm.org/doi/10.1145/3643489.3661131
Alateeq ARoantree MGurrin C(2024)Voxento-Pro: An Advanced Voice Lifelog Retrieval Interaction for Multimodal LifelogsProceedings of the 7th Annual ACM Workshop on the Lifelog Search Challenge10.1145/3643489.3661130(105-110)Online publication date: 10-Jun-2024
https://dl.acm.org/doi/10.1145/3643489.3661130
Tran LNguyen TGurrin CZhou L(2024)MyEachtraX: Lifelog Question Answering on MobileProceedings of the 7th Annual ACM Workshop on the Lifelog Search Challenge10.1145/3643489.3661128(93-98)Online publication date: 10-Jun-2024
https://dl.acm.org/doi/10.1145/3643489.3661128
Show More Cited By

Index Terms

LifeInsight: An Interactive Lifelog Retrieval System with Comprehensive Spatial Insights and Query Assistance
1. Human-centered computing
  1. Human computer interaction (HCI)
    1. Interactive systems and tools
2. Information systems
  1. Information retrieval
    1. Users and interactive retrieval
      1. Search interfaces
  2. Information systems applications
    1. Multimedia information systems
      1. Multimedia databases

Recommendations

MyEachtra: Event-Based Interactive Lifelog Retrieval System for LSC’23
LSC '23: Proceedings of the 6th Annual ACM Lifelog Search Challenge

Retrieval is a fundamental challenge within the research community of lifelog and the Lifelog Search Challenge (LSC) has been an important annual benchmarking activity for interactive lifelog retrieval systems since 2018. This paper proposes MyEachtra (...
MemoriEase: An Interactive Lifelog Retrieval System for LSC’23
LSC '23: Proceedings of the 6th Annual ACM Lifelog Search Challenge

Lifelogging is an activity of recording all events that happen in the daily life of an individual. The events can contain images, audio, health index, etc which are collected through various devices such as wearable cameras, smartwatches, and other ...
NewsInsight: A Comprehensive Video Event Retrieval System with Spatial Insights and Query Assistance
SOICT '23: Proceedings of the 12th International Symposium on Information and Communication Technology

Video event retrieval is the task of finding videos that are relevant to a given query. It is a challenging problem because videos are typically much larger than images, and they can contain a variety of different objects and scenes. However, there are ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

LSC '23: Proceedings of the 6th Annual ACM Lifelog Search Challenge

June 2023

74 pages

ISBN:9798400701887

DOI:10.1145/3592573

Copyright © 2023 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

SIGMM: ACM Special Interest Group on Multimedia

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 12 June 2023

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed limited

Funding Sources

Science Foundation Ireland
Vingroup Innovation Foundation (VINIF)

Conference

ICMR '23

Sponsor:

SIGMM

ICMR '23: International Conference on Multimedia Retrieval

June 12 - 15, 2023

Thessaloniki, Greece

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

15
Total Citations
View Citations
167
Total Downloads

Downloads (Last 12 months)134
Downloads (Last 6 weeks)3

Reflects downloads up to 01 Sep 2024

Other Metrics

View Author Metrics

Citations

Cited By

Vopálková ZYaghob JStroh MSchlegel ULokoc J(2024)Searching Temporally Distant Activities in Lifelog Data With PraK Tool V2Proceedings of the 7th Annual ACM Workshop on the Lifelog Search Challenge10.1145/3643489.3661131(111-116)Online publication date: 10-Jun-2024
https://dl.acm.org/doi/10.1145/3643489.3661131
Alateeq ARoantree MGurrin C(2024)Voxento-Pro: An Advanced Voice Lifelog Retrieval Interaction for Multimodal LifelogsProceedings of the 7th Annual ACM Workshop on the Lifelog Search Challenge10.1145/3643489.3661130(105-110)Online publication date: 10-Jun-2024
https://dl.acm.org/doi/10.1145/3643489.3661130
Tran LNguyen TGurrin CZhou L(2024)MyEachtraX: Lifelog Question Answering on MobileProceedings of the 7th Annual ACM Workshop on the Lifelog Search Challenge10.1145/3643489.3661128(93-98)Online publication date: 10-Jun-2024
https://dl.acm.org/doi/10.1145/3643489.3661128
Alam NGraham YGurrin C(2024)Memento 4.0: A Prototype Conversational Search System for LSC'24Proceedings of the 7th Annual ACM Workshop on the Lifelog Search Challenge10.1145/3643489.3661126(82-87)Online publication date: 10-Jun-2024
https://dl.acm.org/doi/10.1145/3643489.3661126
Ly DDuong-Le DVuong GHo VNinh VTran MLe K(2024)CollaXRSearch: A Collaborative Virtual Reality System for Lifelog RetrievalProceedings of the 7th Annual ACM Workshop on the Lifelog Search Challenge10.1145/3643489.3661125(76-81)Online publication date: 10-Jun-2024
https://dl.acm.org/doi/10.1145/3643489.3661125
Rader MSchoeffmann K(2024)lifeXplore at the Lifelog Search Challenge 2024Proceedings of the 7th Annual ACM Workshop on the Lifelog Search Challenge10.1145/3643489.3661123(64-69)Online publication date: 10-Jun-2024
https://dl.acm.org/doi/10.1145/3643489.3661123
Le HNguyen TLe TTran MNguyen TNinh VZhou LGurrin C(2024)LifeSeeker 6.0: Leveraging the linguistic aspect of the lifelog system in LSC'24Proceedings of the 7th Annual ACM Workshop on the Lifelog Search Challenge10.1145/3643489.3661121(53-57)Online publication date: 10-Jun-2024
https://dl.acm.org/doi/10.1145/3643489.3661121
Steffensen INystad SLiahjell MDang-Nguyen DKesavulu MNinh VGurrin CVuong GHo VTran M(2024)T@Retrospect: A Journey Through TimeProceedings of the 7th Annual ACM Workshop on the Lifelog Search Challenge10.1145/3643489.3661118(36-40)Online publication date: 10-Jun-2024
https://dl.acm.org/doi/10.1145/3643489.3661118
Pagani Vavik EWilhelmsen MDal Østensen JDang-Nguyen DNinh VKesavulu MGurrin CVuong GHo VTran M(2024)VitaChronicle: Applying UX/UI Principles and Guidelines to Enhance Lifelog Retrieval System DesignProceedings of the 7th Annual ACM Workshop on the Lifelog Search Challenge10.1145/3643489.3661117(30-35)Online publication date: 10-Jun-2024
https://dl.acm.org/doi/10.1145/3643489.3661117
Ho-Le MDo-Huu HHo DLe-Hinh NVo-Hoang HNinh VTran M(2024)SnapSeek: An Interactive Lifelog Acquisition System for LSC'24Proceedings of the 7th Annual ACM Workshop on the Lifelog Search Challenge10.1145/3643489.3661116(24-29)Online publication date: 10-Jun-2024
https://dl.acm.org/doi/10.1145/3643489.3661116
Show More Cited By

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

HTML Format

View this article in HTML Format.

Media

Figures

Other

Tables

View Table of Contents