research-article

A Multi-level Interactive Lifelog Search Engine with User Feedback

Authors:

Shaoping MaAuthors Info & Claims

LSC '20: Proceedings of the Third Annual Workshop on Lifelog Search Challenge

Pages 29 - 35

https://doi.org/10.1145/3379172.3391720

Published: 09 June 2020 Publication History

Abstract

With the rise of portable wearable devices, it is easier for users to save their lifelog data. As lifelog is usually disorganized with multi-modal information (even noisy sometimes), an interactive search engine is crucial for users to review and explore their lifelog. Unlike traditional search engines, lifelog search includes multi-modality information of images, text and other data from sensors, which brings challenges to data arrangement and search. Accordingly, users' information need is also multi-level. Hence, a single interaction mechanism may not be able to satisfy users' requirements. As the data set is highly personalized, interaction and feedback from users should also be considered in the search engine. Therefore, in this paper we present an interactive multi-modality lifelog search engine to help users manage and find lifelog data. To this end, lifelog data is clustered and processed in multi-level processing. Then, we build an interactive search engine, includingtext as query, image as query, andtimeline view modules. Besides, the system is able to adopt user feedback mechanisms in multi-round queries. Our system shows promising experimental results on LSC'20 dataset and development topics. The text-based search module gives correct results on more than 60% of the development topics at LSC'20.

References

[1]

Münzer Bernd, Leibetseder Andreas, Kletz Sabrina, Primus Manfred Jürgen, and Schoeffmann Klaus. 2018. lifeXplore at the lifelog search challenge 2018. In Proceedings of the 2018 ACM Workshop on The Lifelog Search Challenge. 3--8.

[2]

Gurrin Cathal, Smeaton Alan F, Doherty Aiden R, et al. 2014. Lifelogging: Personal big data. Foundations and Trends® in information retrieval, Vol. 8, 1 (2014), 1--125.

[3]

Gurrin Cathal, Le Tu-Khiem, Ninh Van-Tu, Dang-Nguyen Duc-Tien, Jónsson Björn Þór, Lokoc Jakub, Hurst Wolfgang, Tran Minh-Triet, and Schoeffmann Klaus. 2020. An Introduction to the Third Annual Lifelog Search Challenge, LSC'20. In ICMR '20, The 2020 International Conference on Multimedia Retrieval. ACM, Dublin, Ireland.

[4]

Dang-Nguyen Duc-Tien, Schoeffmann Klaus, and Hurst Wolfgang. 2018. LSE2018 Panel-Challenges of Lifelog Search and Access. In Proceedings of the 2018 ACM Workshop on The Lifelog Search Challenge. 1--2.

[5]

Long Fuhui, Zhang Hongjiang, and Feng David Dagan. 2003. Fundamentals of content-based image retrieval. In Multimedia information retrieval and management. Springer, 1--26.

[6]

Nguyen Van Khan Isadora, Shrestha Pranita, Zhang Min, Liu Yiqun, and Ma Shaoping. 2019. A Two-Level Lifelog Search Engine at the LSC 2019. In Proceedings of the ACM Workshop on Lifelog Search Challenge. 19--23.

Digital Library

[7]

Lokovc Jakub, Soucek Tomávs, Cech Premysl, and Kovalvcik Gregor. 2019. Enhanced VIRET tool for lifelog data. In Proceedings of the ACM Workshop on Lifelog Search Challenge. 25--26.

[8]

Rossetto Luca, Gasser Ralph, Heller Silvan, Amiri Parian Mahnaz, and Schuldt Heiko. 2019. Retrieval of structured and unstructured data with vitrivr. In Proceedings of the ACM Workshop on Lifelog Search Challenge. 27--31.

[9]

Dodge Martin and Kitchin Rob. 2007. 'Outlines of a world coming into existence': pervasive computing and the ethics of forgetting. Environment and planning B: planning and design, Vol. 34, 3 (2007), 431--445.

[10]

Le Nguyen-Khang, Nguyen Dieu-Hien, Hoang Trung-Hieu, Nguyen Thanh-An, Truong Thanh-Dat, Dinh Duy-Tung, Luong Quoc-An, Vo-Ho Viet-Khoa, Nguyen Vinh-Tiep, and Tran Minh-Triet. 2019. Smart lifelog retrieval system with habit-based concepts and moment visualization. In Proceedings of the ACM Workshop on Lifelog Search Challenge. 1--6.

[11]

Vasconcelos Nuno and Lippman Andrew. 2000. Learning from user feedback in image retrieval systems. In Advances in neural information processing systems. 977--986.

[12]

Khan Omar Shahbaz, Jónsson Björn Þór, Zahálka Jan, Rudinac Stevan, and Worring Marcel. 2019. Exquisitor at the lifelog search challenge 2019. In Proceedings of the ACM Workshop on Lifelog Search Challenge. 7--11.

[13]

Ren Shaoqing, He Kaiming, Girshick Ross, and Sun Jian. 2015. Faster r-cnn: Towards real-time object detection with region proposal networks. In Advances in neural information processing systems. 91--99.

[14]

Guo Xiaoxiao, Wu Hui, Cheng Yu, Rennie Steven, Tesauro Gerald, and Feris Rogerio. 2018. Dialog-based interactive image retrieval. In Advances in Neural Information Processing Systems. 678--688.

[15]

Rui Yong, Huang Thomas S, and Chang Shih-Fu. 1999. Image retrieval: Current techniques, promising directions, and open issues. Journal of visual communication and image representation, Vol. 10, 1 (1999), 39--62.

Digital Library

Cited By

Alam NGraham Y(2023)Memento: a prototype search engine for LSC 2021Multimedia Tools and Applications10.1007/s11042-023-15067-982:24(37807-37828)Online publication date: 1-Apr-2023
https://doi.org/10.1007/s11042-023-15067-9
Yen AFu MAng WChu TTsai SHuang HChen H(2023)Visual lifelog retrieval: humans and machines interpretation on first-person imagesMultimedia Tools and Applications10.1007/s11042-023-14344-x82:24(37757-37787)Online publication date: 16-Feb-2023
https://doi.org/10.1007/s11042-023-14344-x
Tran LAlam NGraham YVo LDiep NNguyen BZhou LGurrin C(2022)An Exploration into the Benefits of the CLIP model for Lifelog RetrievalProceedings of the 19th International Conference on Content-based Multimedia Indexing10.1145/3549555.3549593(15-22)Online publication date: 14-Sep-2022
https://dl.acm.org/doi/10.1145/3549555.3549593
Show More Cited By

Index Terms

A Multi-level Interactive Lifelog Search Engine with User Feedback
1. Human-centered computing
  1. Interaction design
    1. Interaction design process and methods
      1. User interface design
2. Information systems
  1. Information retrieval
    1. Specialized information retrieval
      1. Multimedia and multimodal retrieval
    2. Users and interactive retrieval

Recommendations

An Interactive Lifelog Search Engine for LSC2018
LSC '18: Proceedings of the 2018 ACM Workshop on The Lifelog Search Challenge

In this work, we describe an interactive lifelog search engine developed for the LSC 2018 search challenge at ACM ICMR 2018. The paper introduces the four-step process required to support lifelog search engines and describes the source data for the ...
Search Engine Optimization by Re-Ranking the Product Search Result Based on User Click Data
AISS '21: Proceedings of the 3rd International Conference on Advanced Information Science and System

Blibli.com provides a search engine for its customers. It used Solr search engine with only plain BM25 similarity function which is based on probability. In order to improve search engine performance, this research tried to implement an algorithm that ...
A New Algorithm for Inferring User Search Goals with Feedback Sessions

For a broad-topic and ambiguous query, different users may have different search goals when they submit it to a search engine. The inference and analysis of user search goals can be very useful in improving search engine relevance and user experience. In ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

LSC '20: Proceedings of the Third Annual Workshop on Lifelog Search Challenge

June 2020

89 pages

ISBN:9781450371360

DOI:10.1145/3379172

General Chairs:
Cathal Gurrin
Dublin City University, Ireland
,
Klaus Schoeffmann
Klagenfurt University, Austria
,
Björn Þór Jónsson
IT University of Copenhagen, Denmark
,
Duc-Tien Dang-Nguyen
University of Bergen, Norway
,
Jakub Lokoč
Charles University, Czech Republic
,
Minh-Triet Tran
HCM University of Science, Vietnam
,
Wolfgang Hürst
Utrecht University, Netherlands

Copyright © 2020 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGMM: ACM Special Interest Group on Multimedia

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 09 June 2020

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

National Key Research and Development Program of China
Natural Science Foundation of China

Conference

ICMR '20

Sponsor:

SIGMM

ICMR '20: International Conference on Multimedia Retrieval

June 9, 2020

Dublin, Ireland

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

10
Total Citations
View Citations
180
Total Downloads

Downloads (Last 12 months)4
Downloads (Last 6 weeks)1

Reflects downloads up to 08 Feb 2025

Other Metrics

View Author Metrics

Citations

Cited By

Alam NGraham Y(2023)Memento: a prototype search engine for LSC 2021Multimedia Tools and Applications10.1007/s11042-023-15067-982:24(37807-37828)Online publication date: 1-Apr-2023
https://doi.org/10.1007/s11042-023-15067-9
Yen AFu MAng WChu TTsai SHuang HChen H(2023)Visual lifelog retrieval: humans and machines interpretation on first-person imagesMultimedia Tools and Applications10.1007/s11042-023-14344-x82:24(37757-37787)Online publication date: 16-Feb-2023
https://doi.org/10.1007/s11042-023-14344-x
Tran LAlam NGraham YVo LDiep NNguyen BZhou LGurrin C(2022)An Exploration into the Benefits of the CLIP model for Lifelog RetrievalProceedings of the 19th International Conference on Content-based Multimedia Indexing10.1145/3549555.3549593(15-22)Online publication date: 14-Sep-2022
https://dl.acm.org/doi/10.1145/3549555.3549593
Li JMa WZhang MWang PLiu YMa S(2021)Know Yourself: Physical and Psychological Self-Awareness With LifelogFrontiers in Digital Health10.3389/fdgth.2021.6768243Online publication date: 11-Aug-2021
https://doi.org/10.3389/fdgth.2021.676824
Jaiswal ALiu HFrommholz IHasibi FFang YAizawa A(2021)Semantic Hilbert Space for Interactive Image RetrievalProceedings of the 2021 ACM SIGIR International Conference on Theory of Information Retrieval10.1145/3471158.3472253(307-315)Online publication date: 11-Jul-2021
https://dl.acm.org/doi/10.1145/3471158.3472253
Ang WYen AChu THuang HChen HGurrin CSchoeffmann KJónsson BDang-Nguyen DLokoč JTran MHürst WRossetto LHealy G(2021)LifeConcept: An Interactive Approach for Multimodal Lifelog Retrieval through Concept RecommendationProceedings of the 4th Annual on Lifelog Search Challenge10.1145/3463948.3469070(47-51)Online publication date: 21-Aug-2021
https://dl.acm.org/doi/10.1145/3463948.3469070
Alam NGraham YGurrin CGurrin CSchoeffmann KJónsson BDang-Nguyen DLokoč JTran MHürst WRossetto LHealy G(2021)Memento: A Prototype Lifelog Search Engine for LSC'21Proceedings of the 4th Annual on Lifelog Search Challenge10.1145/3463948.3469069(53-58)Online publication date: 21-Aug-2021
https://dl.acm.org/doi/10.1145/3463948.3469069
Tran LNguyen MThanh Binh NLee HGurrin CGurrin CSchoeffmann KJónsson BDang-Nguyen DLokoč JTran MHürst WRossetto LHealy G(2021)Myscéal 2.0: A Revised Experimental Interactive Lifelog Retrieval System for LSC'21Proceedings of the 4th Annual on Lifelog Search Challenge10.1145/3463948.3469064(11-16)Online publication date: 21-Aug-2021
https://dl.acm.org/doi/10.1145/3463948.3469064
Heller SGasser RParian-Scherb MPopovic SRossetto LSauter LSpiess FSchuldt HGurrin CSchoeffmann KJónsson BDang-Nguyen DLokoč JTran MHürst WRossetto LHealy G(2021)Interactive Multimodal Lifelog Retrieval with vitrivr at LSC 2021Proceedings of the 4th Annual on Lifelog Search Challenge10.1145/3463948.3469062(35-39)Online publication date: 21-Aug-2021
https://dl.acm.org/doi/10.1145/3463948.3469062
Leibetseder ASchoeffmann KGurrin CSchoeffmann KJónsson BDang-Nguyen DLokoč JTran MHürst WRossetto LHealy G(2021)lifeXplore at the Lifelog Search Challenge 2021Proceedings of the 4th Annual on Lifelog Search Challenge10.1145/3463948.3469060(23-28)Online publication date: 21-Aug-2021
https://dl.acm.org/doi/10.1145/3463948.3469060

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten