research-article

Public Access

Investigating per Topic Upper Bound for Session Search Evaluation

Authors:

Zhiwen Tang,

Grace Hui YangAuthors Info & Claims

ICTIR '17: Proceedings of the ACM SIGIR International Conference on Theory of Information Retrieval

Pages 185 - 192

https://doi.org/10.1145/3121050.3121069

Published: 01 October 2017 Publication History

PDF eReader

Abstract

Session search is a complex Information Retrieval (IR) task. As a result, its evaluation is also complex. A great number of factors need to be considered in the evaluation of session search. They include document relevance, document novelty, aspect-related novelty discounting, and user's efforts in examining the documents. Due to increased complexity, most existing session search evaluation metrics are NP-hard. Consequently, the optimal value, i.e. the upper bound, of a metric highly varies with the actual search topics. In Cranfield-like settings such as the Text REtrieval Conference (TREC), scores for systems are usually averaged across all search topics. With undetermined upper bound values, however, it could be unfair to compare IR systems across different topics. This paper addresses the problem by investigating the actual per topic upper bounds of existing session search metrics. Through decomposing the metrics, we derive the upper bounds via mathematical optimization. We show that after being normalized by the bounds, the NP-hard session search metrics are then able to provide robust comparison across various search topics. The new normalized metrics are experimented on official runs submitted to the TREC 2016 Dynamic Domain (DD) Track.

References

[1]

Lars Backstrom and Jure Leskovec 2011. Supervised random walks: predicting and recommending links in social networks Proceedings of the fourth ACM international conference on Web search and data mining. ACM, 635--644.

Digital Library

Google Scholar

[2]

Shumeet Baluja, Rohan Seth, D Sivakumar, Yushi Jing, Jay Yagnik, Shankar Kumar, Deepak Ravichandran, and Mohamed Aly. 2008. Video suggestion and discovery for youtube: taking random walks through the view graph Proceedings of the 17th international conference on World Wide Web. ACM, 895--904.

Digital Library

Google Scholar

[3]

Roi Blanco and Christina Lioma 2012. Graph-based term weighting for information retrieval. Information retrieval Vol. 15, 1 (2012), 54--92.

Digital Library

Google Scholar

[4]

Jiajun Bu, Shulong Tan, Chun Chen, Can Wang, Hao Wu, Lijun Zhang, and Xiaofei He. 2010. Music recommendation by unified hypergraph: combining social media information and music content. In Proceedings of the international conference on Multimedia. ACM, 391--400.

Digital Library

Google Scholar

[5]

Pablo Castells, Miriam Fernández, David Vallet, Phivos Mylonas, and Yannis Avrithis. 2005. Self-tuning personalized information retrieval in an ontology-based framework On the Move to Meaningful Internet Systems 2005: OTM 2005 Workshops. Springer, 977--986.

Digital Library

Google Scholar

[6]

Kevyn Collins-Thompson and Jamie Callan 2005. Query expansion using random walk models. In Proceedings of the 14th ACM international conference on Information and knowledge management. ACM, 704--711.

Digital Library

Google Scholar

[7]

Nick Craswell and Martin Szummer 2007. Random walks on the click graph. In Proceedings of the 30th annual international ACM SIGIR conference on Research and development in information retrieval. ACM, 239--246.

Digital Library

Google Scholar

[8]

Gideon Dror, Yahoo Labs, Noam Koenigstein, Yehuda Koren, and Markus Weimer 2012. The Yahoo! music dataset and KDDCup'11. In JMLR Workshop and Conference Proceedings: Proceedings of KDD Cup 2011 Competition. 3--18.

Digital Library

Google Scholar

[9]

Carsten Eickhoff, Jaime Teevan, Ryen White, and Susan Dumais. 2014. Lessons from the journey: A query log analysis of within-session learning Proceedings of the 7th ACM international conference on Web search and data mining. ACM, 223--232.

Digital Library

Google Scholar

[10]

Yasuhiro Fujiwara, Makoto Nakatsuji, Hiroaki Shiokawa, Takeshi Mishima, and Makoto Onizuka. 2013. Efficient ad-hoc search for personalized pagerank. Proceedings of the 2013 ACM SIGMOD International Conference on Management of Data. ACM, 445--456.

Digital Library

Google Scholar

[11]

Susan Gauch, Jason Chaffee, and Alexander Pretschner. 2003. Ontology-based personalized search and browsing. Web Intelligence and Agent Systems Vol. 1, 3--4 (2003), 219--234.

Digital Library

Google Scholar

[12]

Chun Guo and Xiaozhong Liu 2015. Automatic Feature Generation on Heterogeneous Graph for Music Recommendation Proceedings of the 38th International ACM SIGIR Conference on Research and Development in Information Retrieval. ACM, 807--810.

Digital Library

Google Scholar

[13]

Aniko Hannak, Piotr Sapiezynski, Arash Molavi Kakhki, Balachander Krishnamurthy, David Lazer, Alan Mislove, and Christo Wilson. 2013. Measuring personalization of web search. In Proceedings of the 22nd international conference on World Wide Web. International World Wide Web Conferences Steering Committee, 527--538.

Digital Library

Google Scholar

[14]

Yifan Hu, Y. Koren, and C. Volinsky 2008. Collaborative Filtering for Implicit Feedback Datasets Eighth IEEE International Conference on Data Mining. 263--272.

Digital Library

Google Scholar

Cited By

View all

Lipani ACarterette BYilmaz E(2021)How Am I Doing?: Evaluating Conversational Search Systems OfflineACM Transactions on Information Systems10.1145/345116039:4(1-22)Online publication date: 17-Aug-2021
https://dl.acm.org/doi/10.1145/3451160
Zhang FMao JLiu YMa WZhang MMa SHuang JChang YCheng XKamps JMurdock VWen JLiu Y(2020)Cascade or RecencyProceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval10.1145/3397271.3401163(389-398)Online publication date: 25-Jul-2020
https://dl.acm.org/doi/10.1145/3397271.3401163
Sayed MOard DPiwowarski BChevalier MGaussier EMaarek YNie JScholer F(2019)Jointly Modeling Relevance and Sensitivity for Search Among Sensitive ContentProceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval10.1145/3331184.3331256(615-624)Online publication date: 18-Jul-2019
https://dl.acm.org/doi/10.1145/3331184.3331256
Show More Cited By

Index Terms

Investigating per Topic Upper Bound for Session Search Evaluation
1. General and reference
  1. Cross-computing tools and techniques
    1. Evaluation
    2. Metrics
2. Information systems
  1. Information retrieval
    1. Evaluation of retrieval results
      1. Retrieval effectiveness
      2. Retrieval efficiency

Recommendations

Towards Designing Better Session Search Evaluation Metrics
SIGIR '18: The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval

User satisfaction has been paid much attention to in recent Web search evaluation studies and regarded as the ground truth for designing better evaluation metrics. However, most existing studies are focused on the relationship between satisfaction and ...
Query change as relevance feedback in session search
SIGIR '13: Proceedings of the 36th international ACM SIGIR conference on Research and development in information retrieval

Session search is the Information Retrieval (IR) task that performs document retrieval for an entire session. During a session, users often change queries to explore and investigate the information needs. In this paper, we propose to use query change as ...
Investigating Cognitive Effects in Session-level Search User Satisfaction
KDD '19: Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining

User satisfaction is an important variable in Web search evaluation studies and has received more and more attention in recent years. Many studies regard user satisfaction as the ground truth for designing better evaluation metrics. However, most of the ...

Comments

Information & Contributors

Information

Published In

ICTIR '17: Proceedings of the ACM SIGIR International Conference on Theory of Information Retrieval

October 2017

348 pages

ISBN:9781450344906

DOI:10.1145/3121050

General Chairs:
Jaap Kamps
University of Amsterdam, The Netherlands
,
Evangelos Kanoulas
University of Amsterdam, The Netherlands
,
Maarten de Rijke
University of Amsterdam, The Netherlands
,
Program Chairs:
Hui Fang
University of Delaware, USA
,
Emine Yilmaz
University College London, UK

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 01 October 2017

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Badges

Best Student Paper

Author Tags

Qualifiers

Research-article

Funding Sources

Conference

ICTIR '17

Sponsor:

SIGIR

ICTIR '17: ACM SIGIR International Conference on the Theory of Information Retrieval

October 1 - 4, 2017

Amsterdam, The Netherlands

Acceptance Rates

ICTIR '17 Paper Acceptance Rate 27 of 54 submissions, 50%;

Overall Acceptance Rate 235 of 527 submissions, 45%

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

7
Total Citations
View Citations
340
Total Downloads

Downloads (Last 12 months)47
Downloads (Last 6 weeks)10

Reflects downloads up to 26 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

View all

Lipani ACarterette BYilmaz E(2021)How Am I Doing?: Evaluating Conversational Search Systems OfflineACM Transactions on Information Systems10.1145/345116039:4(1-22)Online publication date: 17-Aug-2021
https://dl.acm.org/doi/10.1145/3451160
Zhang FMao JLiu YMa WZhang MMa SHuang JChang YCheng XKamps JMurdock VWen JLiu Y(2020)Cascade or RecencyProceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval10.1145/3397271.3401163(389-398)Online publication date: 25-Jul-2020
https://dl.acm.org/doi/10.1145/3397271.3401163
Sayed MOard DPiwowarski BChevalier MGaussier EMaarek YNie JScholer F(2019)Jointly Modeling Relevance and Sensitivity for Search Among Sensitive ContentProceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval10.1145/3331184.3331256(615-624)Online publication date: 18-Jul-2019
https://dl.acm.org/doi/10.1145/3331184.3331256
Zhang ZHuang MZhao ZJi FChen HZhu X(2019)Memory-Augmented Dialogue Management for Task-Oriented Dialogue SystemsACM Transactions on Information Systems10.1145/331761237:3(1-30)Online publication date: 8-Jul-2019
https://dl.acm.org/doi/10.1145/3317612
van Dijk DFerrante MFerro NKanoulas E(2019)A Markovian Approach to Evaluate Session-Based IR SystemsAdvances in Information Retrieval10.1007/978-3-030-15712-8_40(621-635)Online publication date: 14-Apr-2019
https://dl.acm.org/doi/10.1007/978-3-030-15712-8_40
Albahem ASpina DScholer FCavedon L(2019)Meta-evaluation of Dynamic Search: How Do Metrics Capture Topical Relevance, Diversity and User Effort?Advances in Information Retrieval10.1007/978-3-030-15712-8_39(607-620)Online publication date: 7-Apr-2019
https://doi.org/10.1007/978-3-030-15712-8_39
Fang HKamps JKanoulas Ede Rijke MYilmaz E(2018)Report on the 2017 ACM SIGIR International Conference Theory of Information Retrieval (ICTIR?17)ACM SIGIR Forum10.1145/3190580.319059151:3(78-87)Online publication date: 22-Feb-2018
https://dl.acm.org/doi/10.1145/3190580.3190591

View Options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Cited By

Index Terms

Recommendations

Towards Designing Better Session Search Evaluation Metrics

Query change as relevance feedback in session search

Investigating Cognitive Effects in Session-level Search User Satisfaction