research-article

Open access

LifeSeeker 6.0: Leveraging the linguistic aspect of the lifelog system in LSC'24

Authors:

Thao-Nhu Nguyen,

Minh-Triet Tran,

Thanh-Binh Nguyen,

Cathal GurrinAuthors Info & Claims

LSC '24: Proceedings of the 7th Annual ACM Workshop on the Lifelog Search Challenge

Pages 53 - 57

https://doi.org/10.1145/3643489.3661121

Published: 18 June 2024 Publication History

Abstract

Supporting effective access to digital lifelogs is a challenging research task because of both the volume and variety of multimodal lifelog data, as well as the many and diverse types of information need that should be supported. In this paper, we introduce a new version of LifeSeeker called LifeSeeker 6.0, for the 2024 edition of the ACM Lifelog Search Challenge. Our enhancements include the improvements to the user interface and the backend reconstruction by combining the E-LifeSeeker structure with using contrastive learning between texts. These adjustments are aimed at accelerating the correlation between the huge image collection and the text input, thereby enhancing the retrieval accuracy and efficiency.

References

[1]

Naushad Alam, Yvette Graham, and Cathal Gurrin. 2023. Memento 3.0: An enhanced lifelog search engine for LSC'23. In Proceedings of the 6th Annual ACM Lifelog Search Challenge. 41--46.

Digital Library

[2]

Ahmed Alateeq, Mark Roantree, and Cathal Gurrin. 2023. Voxento 4.0: A More Flexible Visualisation and Control for Lifelogs. In Proceedings of the 6th Annual ACM Lifelog Search Challenge. 7--12.

Digital Library

[3]

Youngmin Baek, Bado Lee, Dongyoon Han, Sangdoo Yun, and Hwalsuk Lee. 2019. Character region awareness for text detection. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 9365--9374.

[4]

Duc Tien Dang Nguyen Graham Healy Jakub Lokoc Liting Zhou Luca Rossetto Minh-Triet Tran Wolfgang Hürst Werner Bailer Klaus Schoeffmann Cathal Gurrin, Björn Þór Jónsson. 2023. Introduction to the Sixth Annual Lifelog Search Challenge, LSC'23. In Proc. International Conference on Multimedia Retrieval (ICMR'23) (Thessaloniki, Greece) (ICMR '23). New York, NY, USA.

[5]

Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. 2020. A Simple Framework for Contrastive Learning of Visual Representations. arXiv:2002.05709 [cs.LG]

[6]

Mehdi Cherti, Romain Beaumont, Ross Wightman, Mitchell Wortsman, Gabriel Ilharco, Cade Gordon, Christoph Schuhmann, Ludwig Schmidt, and Jenia Jitsev. 2023. Reproducible scaling laws for contrastive language-image learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2818--2829.

[7]

Hongchao Fang, Sicheng Wang, Meng Zhou, Jiayuan Ding, and Pengtao Xie. 2020. CERT: Contrastive Self-supervised Learning for Language Understanding. arXiv:2005.12766 [cs.CL]

[8]

Tianyu Gao, Xingcheng Yao, and Danqi Chen. 2021. Simcse: Simple contrastive learning of sentence embeddings. arXiv preprint arXiv:2104.08821 (2021).

[9]

Cathal Gurrin, Liting Zhou, Graham Healy, Bailer. Werner, Duc-Tien Dang-Nguyen, Steve Hodges, Björn Þór Jónsson, Jakub Lokoč, Luca Rossetto, Minh-Triet Tran, and Klaus Schöffmann. 2024. Introduction to the Seventh Annual Lifelog Search Challenge, LSC'24. International Conference on Multimedia Retrieval (ICMR'24).

Digital Library

[10]

Nhat Hoang-Xuan, Thang-Long Nguyen-Ho, Cathal Gurrin, and Minh-Triet Tran. 2023. Lifelog Discovery Assistant: Suggesting Prompts and Indexing Event Sequences for FIRST at LSC 2023. In Proceedings of the 6th Annual ACM Lifelog Search Challenge. 47--52.

Digital Library

[11]

Maria Tysse Hordvik, Julie Sophie Teilstad Østby, Manoj Kesavulu, Thao-Nhu Nguyen, Tu-Khiem Le, and Duc-Tien Dang-Nguyen. 2023. LifeLens: Transforming Lifelog Search with Innovative UX/UI Design. In Proceedings of the 6th Annual ACM Lifelog Search Challenge. 1--6.

Digital Library

[12]

Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. 2021. Scaling up visual and vision-language representation learning with noisy text supervision. In International conference on machine learning. PMLR, 4904--4916.

[13]

Tu-Khiem Le, Van-Tu Ninh, Duc-Tien Dang-Nguyen, Minh-Triet Tran, Liting Zhou, Pablo Redondo, Sinead Smyth, and Cathal Gurrin. 2019. Lifeseeker: Interactive lifelog search engine at lsc 2019. In Proceedings of the ACM Workshop on Lifelog Search Challenge. 37--40.

Digital Library

[14]

Tu-Khiem Le, Van-Tu Ninh, Minh-Triet Tran, Thanh-An Nguyen, Hai-Dang Nguyen, Liting Zhou, Graham Healy, and Cathal Gurrin. 2020. Lifeseeker 2.0: Interactive lifelog search engine at lsc 2020. In Proceedings of the Third Annual Workshop on Lifelog Search Challenge. 57--62.

Digital Library

[15]

Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. 2022. BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation. https://arxiv.org/abs/2201.12086

[16]

Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013. Efficient estimation of word representations in vector space. arXiv preprint arXiv:1301.3781 (2013).

[17]

Thao-Nhu Nguyen, Tu-Khiem Le, Van-Tu Ninh, Cathal Gurrin, Minh-Triet Tran, Thanh Binh Nguyen, Graham Healy, Annalina Caputo, and Sinead Smyth. 2023. E-LifeSeeker: An interactive lifelog search engine for lsc'23. In Proceedings of the 6th Annual ACM Lifelog Search Challenge. 13--17.

Digital Library

[18]

Thao-Nhu Nguyen, Tu-Khiem Le, Van-Tu Ninh, Minh-Triet Tran, Thanh Binh Nguyen, Graham Healy, Sinéad Smyth, Annalina Caputo, and Cathal Gurrin. 2022. LifeSeeker 4.0: An Interactive Lifelog Search Engine for LSC'22. In Proceedings of the 5th Annual on Lifelog Search Challenge. 14--19.

Digital Library

[19]

Thao-Nhu Nguyen, Tu-Khiem Le, Van-Tu Ninh, Minh-Triet Tran, Nguyen Thanh Binh, Graham Healy, Annalina Caputo, and Cathal Gurrin. 2021. Life-Seeker 3.0: An Interactive Lifelog Search Engine for LSC'21. In Proceedings of the 4th annual on lifelog search challenge. 41--46.

Digital Library

[20]

Tien-Thanh Nguyen-Dang, Xuan-Dang Thai, Gia-Huy Vuong, Van-Son Ho, Minh-Triet Tran, Van-Tu Ninh, Minh-Khoi Pham, Tu-Khiem Le, and Graham Healy. 2023. LifeInsight: an interactive lifelog retrieval system with comprehensive spatial insights and query assistance. In Proceedings of the 6th Annual ACM Lifelog Search Challenge. 59--64.

Digital Library

[21]

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. 2021. Learning transferable visual models from natural language supervision. In International conference on machine learning. PMLR, 8748--8763.

[22]

Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. 2023. Robust speech recognition via large-scale weak supervision. In International Conference on Machine Learning. PMLR, 28492--28518.

[23]

Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. 2022. Hierarchical Text-Conditional Image Generation with CLIP Latents. arXiv:2204.06125 [cs.CV]

[24]

Ricardo Ribeiro, Luísa Amaral, Wei Ye, Alina Trifan, António JR Neves, and Pedro Iglésias. 2023. MEMORIA: A Memory Enhancement and MOment RetrIeval Application for LSC 2023. In Proceedings of the 6th Annual ACM Lifelog Search Challenge. 18--23.

[25]

Joseph John Rocchio Jr. 1971. Relevance feedback in information retrieval. The SMART retrieval system: experiments in automatic document processing (1971).

[26]

Luca Rossetto, Oana Inel, Svenja Lange, Florian Ruosch, Ruijie Wang, and Abraham Bernstein. 2023. Multi-Mode Clustering for Graph-Based Lifelog Retrieval. In Proceedings of the 6th Annual ACM Lifelog Search Challenge. 36--40.

Digital Library

[27]

Noam Rotstein, David Bensaid, Shaked Brody, Roy Ganz, and Ron Kimmel. 2023. Fusecap: Leveraging large language models to fuse visual data into enriched image captions. arXiv preprint arXiv:2305.17718 (2023).

[28]

Klaus Schoeffmann. 2023. lifexplore at the lifelog search challenge 2023. In Proceedings of the 6th Annual ACM Lifelog Search Challenge. 53--58.

Digital Library

[29]

Florian Spiess, Ralph Gasser, Heiko Schuldt, and Luca Rossetto. 2023. The best of both worlds: Lifelog retrieval with a desktop-virtual reality hybrid system. In Proceedings of the 6th Annual ACM Lifelog Search Challenge. 65--68.

Digital Library

[30]

Mingxing Tan and Quoc Le. 2019. Efficientnet: Rethinking model scaling for convolutional neural networks. In International conference on machine learning. PMLR, 6105--6114.

[31]

Ly Duyen Tran, Binh Nguyen, Liting Zhou, and Cathal Gurrin. 2023. MyEachtra: Event-based interactive lifelog retrieval system for lsc'23. In Proceedings of the 6th Annual ACM Lifelog Search Challenge. 24--29.

Digital Library

[32]

Quang-Linh Tran, Ly-Duyen Tran, Binh Nguyen, and Cathal Gurrin. 2023. MemoriEase: An Interactive Lifelog Retrieval System for LSC'23. In Proceedings of the 6th Annual ACM Lifelog Search Challenge. 30--35.

[33]

Chien-Yao Wang, Alexey Bochkovskiy, and Hong-Yuan Mark Liao. 2023. YOLOv7: Trainable bag-of-freebies sets new state-of-the-art for real-time object detectors. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 7464--7475.

[34]

Jiahui Yu, Zirui Wang, Vijay Vasudevan, Legg Yeung, Mojtaba Seyedhosseini, and Yonghui Wu. 2022. Coca: Contrastive captioners are image-text foundation models. arXiv preprint arXiv:2205.01917 (2022).

Cited By

Gurrin CZhou LHealy GBailer WDang Nguyen DHodges SJónsson BLokoč JRossetto LTran MSchöffmann KGurrin CKongkachandra RSchoeffmann KDang-Nguyen DRossetto LSatoh SZhou L(2024)Introduction to the Seventh Annual Lifelog Search Challenge, LSC'24Proceedings of the 2024 International Conference on Multimedia Retrieval10.1145/3652583.3658891(1334-1335)Online publication date: 30-May-2024
https://dl.acm.org/doi/10.1145/3652583.3658891

Index Terms

LifeSeeker 6.0: Leveraging the linguistic aspect of the lifelog system in LSC'24
1. Human-centered computing
  1. Human computer interaction (HCI)
    1. Interactive systems and tools
2. Information systems
  1. Information retrieval
    1. Users and interactive retrieval
      1. Search interfaces
  2. Information systems applications
    1. Multimedia information systems
      1. Multimedia databases

Recommendations

MemoriEase 2.0: A Conversational Lifelog Retrieve System for LSC'24
LSC '24: Proceedings of the 7th Annual ACM Workshop on the Lifelog Search Challenge

Lifelog retrieval plays an important role in memory support for lifeloggers. It helps the lifeloggers to browse, search and navigate their life moments from the lifelog data. However, the volume and variety of lifelog data are enormous and range in ...
MemoriEase: An Interactive Lifelog Retrieval System for LSC’23
LSC '23: Proceedings of the 6th Annual ACM Lifelog Search Challenge

Lifelogging is an activity of recording all events that happen in the daily life of an individual. The events can contain images, audio, health index, etc which are collected through various devices such as wearable cameras, smartwatches, and other ...
Visual lifelog retrieval: humans and machines interpretation on first-person images
Abstract
People usually forget the details of life experiences and encounter situations where they require to recall their past experiences. Therefore, lifelog retrieval turns out to be an emerging task in the AI community. Nowadays, people can record ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

LSC '24: Proceedings of the 7th Annual ACM Workshop on the Lifelog Search Challenge

June 2024

128 pages

ISBN:9798400705502

DOI:10.1145/3643489

This work is licensed under a Creative Commons Attribution International 4.0 License.

Sponsors

SIGMM: ACM Special Interest Group on Multimedia

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 18 June 2024

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

Science Foundation Ireland

Conference

LSC '24

Sponsor:

SIGMM

LSC '24: 7th Annual ACM Workshop on the Lifelog Search Challenge

June 10, 2024

Phuket, Thailand

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

1
Total Citations
View Citations
72
Total Downloads

Downloads (Last 12 months)72
Downloads (Last 6 weeks)28

Reflects downloads up to 04 Oct 2024

Other Metrics

View Author Metrics

Citations

Cited By

Gurrin CZhou LHealy GBailer WDang Nguyen DHodges SJónsson BLokoč JRossetto LTran MSchöffmann KGurrin CKongkachandra RSchoeffmann KDang-Nguyen DRossetto LSatoh SZhou L(2024)Introduction to the Seventh Annual Lifelog Search Challenge, LSC'24Proceedings of the 2024 International Conference on Multimedia Retrieval10.1145/3652583.3658891(1334-1335)Online publication date: 30-May-2024
https://dl.acm.org/doi/10.1145/3652583.3658891

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Media

Figures

Other

Tables

View Table of Contents