research-article

LifeConcept: An Interactive Approach for Multimodal Lifelog Retrieval through Concept Recommendation

Authors:

Hsin-Hsi ChenAuthors Info & Claims

LSC '21: Proceedings of the 4th Annual on Lifelog Search Challenge

Pages 47 - 51

https://doi.org/10.1145/3463948.3469070

Published: 21 August 2021 Publication History

Get Access

Abstract

The major challenge in visual lifelog retrieval is the semantic gap between textual queries and visual concepts. This paper presents our work on the Lifelog Search Challenge 2021 (LSC'21), an annual comparative benchmarking activity for comparing approaches to interactive retrieval from multimodal lifelogs. We propose LifeConcept, an interactive lifelog search system that is aimed at accelerating the retrieval process and retrieving more precise results. In this work, we introduce several new features such as the number of people, location cluster, and object with color. Moreover, we obtain visual concepts from the images with computer vision models and propose a concept recommendation method to reduce the semantic gap. In this way, users can efficiently set up the related conditions for their requirements and search the desired images with appropriate query terms based on the suggestion.

References

[1]

Marc Bolanos, Ricard Mestre, Estefan'ia Talavera, Xavier Giró-i Nieto, and Petia Radeva. 2015. Visual summary of egocentric photostreams by representative keyframes. In 2015 IEEE International Conference on Multimedia & Expo Workshops (ICMEW). IEEE, 1--6.

Crossref

Google Scholar

[2]

M. Bolaños, M. Dimiccoli, and P. Radeva. 2017. Toward Storytelling From Visual Lifelogging: An Overview. IEEE Transactions on Human-Machine Systems, Vol. 47, 1 (2017), 77--90. https://doi.org/10.1109/THMS.2016.2616296

Crossref

Google Scholar

[3]

G. Bradski. 2000. The OpenCV Library. Dr. Dobb's Journal of Software Tools (2000).

Google Scholar

[4]

Tai-Te Chu, Chia-Chun Chang, An-Zi Yen, Hen-Hsen Huang, and Hsin-Hsi Chen. 2020. Multimodal Retrieval through Relations between Subjects and Objects in Lifelog Images. In Proceedings of the Third Annual Workshop on Lifelog Search Challenge. 51--55.

Digital Library

Google Scholar

[5]

Cathal Gurrin, Bjorn Þór Jonsson, Klaus Schoffmann, Duc-Tien Dang-Nguyen, Jakub Lokoč, Minh-Triet Tran, Wolfgang Hürst, Luca Rossetto, and Graham Healy. 2021. Introduction to the Fourth Annual Lifelog Search Challenge, LSC'21. In Proc. International Conference on Multimedia Retrieval (ICMR'21). ACM, Taipei, Taiwan.

Digital Library

Google Scholar

[6]

T Karthikeyan, P Manikandaprabhu, and S Nithya. 2014. A survey on text and content based image retrieval system for image mining. International Journal of Engineering, Vol. 3 (2014).

Google Scholar

[7]

Afshan Latif, Aqsa Rasheed, Umer Sajid, Jameel Ahmed, Nouman Ali, Naeem Iqbal Ratyal, Bushra Zafar, Saadat Hanif Dar, Muhammad Sajid, and Tehmina Khalil. 2019. Content-based image retrieval and feature extraction: a comprehensive review. Mathematical Problems in Engineering, Vol. 2019 (2019).

Google Scholar

[8]

Jiayu Li, Min Zhang, Weizhi Ma, Yiqun Liu, and Shaoping Ma. 2020. A Multi-level Interactive Lifelog Search Engine with User Feedback. In Proceedings of the Third Annual Workshop on Lifelog Search Challenge. 29--35.

Digital Library

Google Scholar

[9]

Yair Poleg, Chetan Arora, and Shmuel Peleg. 2014. Temporal segmentation of egocentric videos. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2537--2544.

Digital Library

Google Scholar

[10]

Peng Wang and Alan F Smeaton. 2012. Semantics-based selection of everyday concepts in visual lifelogging. International Journal of Multimedia Information Retrieval, Vol. 1, 2 (2012), 87--101.

Crossref

Google Scholar

[11]

Bo Xiong and Kristen Grauman. 2014. Detecting snap points in egocentric video with a web photo prior. In European conference on computer vision. Springer, 282--298.

Crossref

Google Scholar

[12]

An-Zi Yen, Hen-Hsen Huang, and Hsin-Hsi Chen. 2021. Ten Questions in Lifelog Mining and Information Recall. In Proceedings of ACM International Conference on Multimedia Retrieval 2021 (ICMR 2021) (2021).

Digital Library

Google Scholar

Cited By

View all

Vuong GHo VNguyen Dang TThai XNguyen-Ho TPham MLe TNinh VHealy GGurrin CTran M(2024)LifeInsight2.0: An Enhanced Approach for Automated Lifelog Retrieval in LSC'24Proceedings of the 7th Annual ACM Workshop on the Lifelog Search Challenge10.1145/3643489.3661112(1-6)Online publication date: 10-Jun-2024
https://dl.acm.org/doi/10.1145/3643489.3661112
Tran SMinh Nguyen DHuynh Minh Nguyen TPhuc Ngo DMinh Nguyen TVo HLe KDo TDuc Ngo T(2023)Diverse Search Methods and Multi-Modal Fusion for High-Performance Video RetrievalProceedings of the 12th International Symposium on Information and Communication Technology10.1145/3628797.3629021(997-1002)Online publication date: 7-Dec-2023
https://dl.acm.org/doi/10.1145/3628797.3629021
Nguyen-Dang TThai XVuong GHo VTran MNinh VPham MLe THealy G(2023)LifeInsight: An Interactive Lifelog Retrieval System with Comprehensive Spatial Insights and Query AssistanceProceedings of the 6th Annual ACM Lifelog Search Challenge10.1145/3592573.3593106(59-64)Online publication date: 12-Jun-2023
https://dl.acm.org/doi/10.1145/3592573.3593106
Show More Cited By

Index Terms

LifeConcept: An Interactive Approach for Multimodal Lifelog Retrieval through Concept Recommendation
1. Human-centered computing
  1. Human computer interaction (HCI)
    1. Interactive systems and tools

Recommendations

Interactive Multimodal Lifelog Retrieval with vitrivr at LSC 2021
LSC '21: Proceedings of the 4th Annual on Lifelog Search Challenge

The Lifelog Search Challenge (LSC) is an annual benchmarking competition for interactive multimedia retrieval systems, where participating systems compete in finding events based on textual descriptions containing hints about structured, semi-structured,...
Multimodal retrieval with relevance feedback based on genetic programming

This paper presents a framework for multimodal retrieval with relevance feedback based on genetic programming. In this supervised learning-to-rank framework, genetic programming is used for the discovery of effective combination functions of (multimodal)...
Visual lifelog retrieval: humans and machines interpretation on first-person images
Abstract
People usually forget the details of life experiences and encounter situations where they require to recall their past experiences. Therefore, lifelog retrieval turns out to be an emerging task in the AI community. Nowadays, people can record ...

Comments

Information & Contributors

Information

Published In

LSC '21: Proceedings of the 4th Annual on Lifelog Search Challenge

August 2021

102 pages

ISBN:9781450385336

DOI:10.1145/3463948

General Chairs:
Cathal Gurrin
Dublin City University, Ireland
,
Klaus Schoeffmann
Klagenfurt University, Austria
,
Björn Jónsson
IT University of Copenhagen, Denmark
,
Duc-Tien Dang-Nguyen
University of Bergen, Norway
,
Jakub Lokoč
Charles University, Czech Republic
,
Minh-Triet Tran
HCM University of Science, Vietnam
,
Wolfgang Hürst
Utrecht University, Netherlands
,
Luca Rossetto
University of Zurich, Switzerland
,
Graham Healy
Dublin City University, Ireland

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 21 August 2021

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

Ministry of Science and Technology, Taiwan

Conference

ICMR '21

Sponsor:

SIGMM

ICMR '21: International Conference on Multimedia Retrieval

August 21, 2021

Taipei, Taiwan

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

7
Total Citations
View Citations
131
Total Downloads

Downloads (Last 12 months)11
Downloads (Last 6 weeks)0

Reflects downloads up to 11 Feb 2025

Other Metrics

View Author Metrics

Citations

Cited By

View all

Vuong GHo VNguyen Dang TThai XNguyen-Ho TPham MLe TNinh VHealy GGurrin CTran M(2024)LifeInsight2.0: An Enhanced Approach for Automated Lifelog Retrieval in LSC'24Proceedings of the 7th Annual ACM Workshop on the Lifelog Search Challenge10.1145/3643489.3661112(1-6)Online publication date: 10-Jun-2024
https://dl.acm.org/doi/10.1145/3643489.3661112
Tran SMinh Nguyen DHuynh Minh Nguyen TPhuc Ngo DMinh Nguyen TVo HLe KDo TDuc Ngo T(2023)Diverse Search Methods and Multi-Modal Fusion for High-Performance Video RetrievalProceedings of the 12th International Symposium on Information and Communication Technology10.1145/3628797.3629021(997-1002)Online publication date: 7-Dec-2023
https://dl.acm.org/doi/10.1145/3628797.3629021
Nguyen-Dang TThai XVuong GHo VTran MNinh VPham MLe THealy G(2023)LifeInsight: An Interactive Lifelog Retrieval System with Comprehensive Spatial Insights and Query AssistanceProceedings of the 6th Annual ACM Lifelog Search Challenge10.1145/3592573.3593106(59-64)Online publication date: 12-Jun-2023
https://dl.acm.org/doi/10.1145/3592573.3593106
Nguyen TLe TNinh VGurrin CTran MNguyen THealy GCaputo ASmyth S(2023)E-LifeSeeker: An Interactive Lifelog Search Engine for LSC’23Proceedings of the 6th Annual ACM Lifelog Search Challenge10.1145/3592573.3593098(13-17)Online publication date: 12-Jun-2023
https://dl.acm.org/doi/10.1145/3592573.3593098
Tran LNguyen MDang-Nguyen DHeller SSpiess FLokoč JPeška LNguyen TKhan ODuane AJónsson BRossetto LYen AAlateeq AAlam NTran MHealy GSchoeffmann KGurrin C(2023)Comparing Interactive Retrieval Approaches at the Lifelog Search Challenge 2021IEEE Access10.1109/ACCESS.2023.324828411(30982-30995)Online publication date: 2023
https://doi.org/10.1109/ACCESS.2023.3248284
Heller SSpiess FSchuldt H(2023)A tale of two interfaces: vitrivr at the lifelog search challengeMultimedia Tools and Applications10.1007/s11042-023-15082-w82:24(37829-37853)Online publication date: 6-Apr-2023
https://dl.acm.org/doi/10.1007/s11042-023-15082-w
Tran LNguyen MNguyen BLee HZhou LGurrin CGurrin CHealy GZhou LÞór Jónsson BDang-Nguyen DLokoč JTran MHürst WRossetto LSchoeffmann K(2022)E-Myscéal: Embedding-based Interactive Lifelog Retrieval System for LSC'22Proceedings of the 5th Annual on Lifelog Search Challenge10.1145/3512729.3533012(32-37)Online publication date: 27-Jun-2022
https://dl.acm.org/doi/10.1145/3512729.3533012

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Abstract

References

Cited By

Index Terms

Recommendations

Interactive Multimodal Lifelog Retrieval with vitrivr at LSC 2021

Multimodal retrieval with relevance feedback based on genetic programming

Visual lifelog retrieval: humans and machines interpretation on first-person images

Comments

Information

Published In

Sponsors

Publisher

Publication History

Permissions

Check for updates

Author Tags

Qualifiers

Funding Sources

Conference

Contributors

Other Metrics

Bibliometrics

Article Metrics

Other Metrics

Citations

Cited By

Login options

Full Access

View options

PDF

eReader

Share

Share this Publication link

Share on social media

Affiliations