short-paper

Public Access

Correlation Between System and User Metrics in a Session

Authors:

James AllanAuthors Info & Claims

CHIIR '16: Proceedings of the 2016 ACM on Conference on Human Information Interaction and Retrieval

Pages 285 - 288

https://doi.org/10.1145/2854946.2855005

Published: 13 March 2016 Publication History

Abstract

We investigate the correlations between system-oriented evaluation metrics and a few user experience metrics for a search session. The system-oriented metrics include session-based DCG (sDCG), normalized sDCG (nsDCG), estimated session nDCG (esNDCG), and a few variants of these metrics. We also look into statistics (e.g., the mean, maximum, and minimum values) of individual queries' nDCG scores, as well as the first and the last query's nDCG in a session. These system-oriented metrics are compared with users' self-rated search performance and task difficulty for a session. Experimental results show that nsDCG and esNDCG have reasonable but weak correlations with the user metrics, while the worst and the last query's nDCG in a session have comparably strong correlations. This suggests future work may better measure users' search experience in a session by modeling each query in the session differently.

References

[1]

M. Ageev, Q. Guo, D. Lagun, and E. Agichtein. Find it if you can: A game for modeling different types of web search success using interaction data. In SIGIR '11, pages 345--354, 2011.

Digital Library

[2]

J. Arguello. Predicting search task difficulty. In ECIR '14, pages 88--99, 2014.

Digital Library

[3]

L. Azzopardi. Modelling interaction with economic models of search. In SIGIR '14, pages 3--12, 2014.

Digital Library

[4]

H. A. Feild, J. Allan, and R. Jones. Predicting searcher frustration. In SIGIR '10, pages 34--41, 2010.

Digital Library

[5]

A. Hassan, R. Jones, and K. L. Klinkner. Beyond DCG: User behavior as a predictor of a successful search. In WSDM '10, pages 221--230, 2010.

Digital Library

[6]

S. B. Huffman and M. Hochster. How well does result relevance predict session satisfaction? In SIGIR '07, pages 567--574, 2007.

Digital Library

[7]

K. Järvelin and J. Kekäläinen. Cumulated gain-based evaluation of IR techniques. ACM Transactions on Information Systems, 20(4):422--446, 2002.

Digital Library

[8]

K. Järvelin, S. L. Price, L. M. L. Delcambre, and M. L. Nielsen. Discounted cumulated gain based evaluation of multiple-query IR sessions. In ECIR '08, pages 4--15, 2008.

Digital Library

[9]

J. Jiang, A. Hassan Awadallah, X. Shi, and R. W. White. Understanding and predicting graded search satisfaction. In WSDM '15, pages 57--66, 2015.

Digital Library

[10]

J. Jiang, D. He, and J. Allan. Searching, browsing, and clicking in a search session: Changes in user behavior by task and over time. In SIGIR '14, pages 607--616, 2014.

Digital Library

[11]

E. Kanoulas, B. Carterette, P. Clough, and M. Sanderson. Overview of the TREC 2010 session track. In TREC 2010.

[12]

E. Kanoulas, B. Carterette, P. D. Clough, and M. Sanderson. Evaluating multi-query sessions. In SIGIR '11, pages 1053--1062, 2011.

Digital Library

[13]

E. Kanoulas, B. Carterette, M. Hall, P. Clough, and M. Sanderson. Overview of the TREC 2012 session track. In TREC 2012.

[14]

Y. Li and N. J. Belkin. A faceted approach to conceptualizing tasks in information seeking. Information Processing & Management, 44(6):1822--1837, 2008.

Digital Library

[15]

C. Liu, J. Liu, and N. J. Belkin. Predicting search task difficulty at different search stages. In CIKM '14, pages 569--578, 2014.

Digital Library

[16]

T. Sakai and Z. Dou. Summaries, ranked retrieval and sessions: A unified framework for information access evaluation. In SIGIR '13, pages 473--482, 2013.

Digital Library

[17]

M. D. Smucker and C. L. Clarke. Time-based calibration of effectiveness measures. In SIGIR '12, pages 95--104, 2012.

Digital Library

Cited By

Thomas PKazai GCraswell NSpielman SHui Yang GWang HHan SHauff CZuccon GZhang Y(2024)What Matters in a Measure? A Perspective from Large-Scale Search EvaluationProceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval10.1145/3626772.3657845(282-292)Online publication date: 10-Jul-2024
https://dl.acm.org/doi/10.1145/3626772.3657845
Moffat AMackenzie J(2024)How much freedom does an effectiveness metric really have?Journal of the Association for Information Science and Technology10.1002/asi.24874Online publication date: 15-Feb-2024
https://doi.org/10.1002/asi.24874
Breuer TFuhr NSchaer P(2023)Validating Synthetic Usage Data in Living Lab EnvironmentsJournal of Data and Information Quality10.1145/3623640Online publication date: 24-Sep-2023
https://dl.acm.org/doi/10.1145/3623640
Show More Cited By

Index Terms

Correlation Between System and User Metrics in a Session
1. Information systems
  1. Information retrieval
    1. Evaluation of retrieval results
      1. Retrieval effectiveness

Recommendations

Metrics, User Models, and Satisfaction
WSDM '20: Proceedings of the 13th International Conference on Web Search and Data Mining

User satisfaction is an important factor when evaluating search systems, and hence a good metric should give rise to scores that have a strong positive correlation with user satisfaction ratings. A metric should also correspond to a plausible user model,...
Towards a graph-based user profile modeling for a session-based personalized search

Most Web search engines use the content of the Web documents and their link structures to assess the relevance of the document to the user's query. With the growth of the information available on the web, it becomes difficult for such Web search engines ...
User model-based metrics for offline query suggestion evaluation
SIGIR '13: Proceedings of the 36th international ACM SIGIR conference on Research and development in information retrieval

Query suggestion or auto-completion mechanisms are widely used by search engines and are increasingly attracting interest from the research community. However, the lack of commonly accepted evaluation methodology and metrics means that it is not ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

CHIIR '16: Proceedings of the 2016 ACM on Conference on Human Information Interaction and Retrieval

March 2016

400 pages

ISBN:9781450337519

DOI:10.1145/2854946

General Chairs:
Diane Kelly
University of North Carolina at Chapel Hill, USA
,
Rob Capra
University of North Carolina at Chapel Hill, USA
,
Program Chairs:
Nick Belkin
Rutgers University, USA
,
Jaime Teevan
Microsoft Research, USA
,
Pertti Vakkari
University of Tampere, Finland

Copyright © 2016 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGIR: ACM Special Interest Group on Information Retrieval

In-Cooperation

SIGCHI: ACM Special Interest Group on Computer-Human Interaction

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 13 March 2016

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Short-paper

Funding Sources

National Science Foundation

Conference

CHIIR '16

Sponsor:

SIGIR

CHIIR '16: Conference on Human Information Interaction and Retrieval

March 13 - 17, 2016

North Carolina, Carrboro, USA

Acceptance Rates

CHIIR '16 Paper Acceptance Rate 23 of 58 submissions, 40%;

Overall Acceptance Rate 55 of 163 submissions, 34%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

23
Total Citations
View Citations
590
Total Downloads

Downloads (Last 12 months)88
Downloads (Last 6 weeks)20

Reflects downloads up to 15 Oct 2024

Other Metrics

View Author Metrics

Citations

Cited By

Thomas PKazai GCraswell NSpielman SHui Yang GWang HHan SHauff CZuccon GZhang Y(2024)What Matters in a Measure? A Perspective from Large-Scale Search EvaluationProceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval10.1145/3626772.3657845(282-292)Online publication date: 10-Jul-2024
https://dl.acm.org/doi/10.1145/3626772.3657845
Moffat AMackenzie J(2024)How much freedom does an effectiveness metric really have?Journal of the Association for Information Science and Technology10.1002/asi.24874Online publication date: 15-Feb-2024
https://doi.org/10.1002/asi.24874
Breuer TFuhr NSchaer P(2023)Validating Synthetic Usage Data in Living Lab EnvironmentsJournal of Data and Information Quality10.1145/3623640Online publication date: 24-Sep-2023
https://dl.acm.org/doi/10.1145/3623640
Shah CWhite RThomas PMitra BSarkar SBelkin N(2023)Taking Search to TaskProceedings of the 2023 Conference on Human Information Interaction and Retrieval10.1145/3576840.3578288(1-13)Online publication date: 19-Mar-2023
https://dl.acm.org/doi/10.1145/3576840.3578288
Ai QWang XBendersky MChen HDuh WHuang HKato MMothe JPoblete B(2023)Metric-agnostic Ranking OptimizationProceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval10.1145/3539618.3591935(2669-2680)Online publication date: 19-Jul-2023
https://dl.acm.org/doi/10.1145/3539618.3591935
Zou JAliannejadi MKanoulas EPera MLiu Y(2023)Users Meet Clarifying Questions: Toward a Better Understanding of User Interactions for Search ClarificationACM Transactions on Information Systems10.1145/352411041:1(1-25)Online publication date: 9-Jan-2023
https://dl.acm.org/doi/10.1145/3524110
Gomez CWang PFu Y(2023)Metric-agnostic Learning-to-Rank via Boosting and Rank Approximation2023 IEEE International Conference on Data Mining (ICDM)10.1109/ICDM58522.2023.00121(1043-1048)Online publication date: 1-Dec-2023
https://doi.org/10.1109/ICDM58522.2023.00121
Wu DDong JYuan FCheng L(2022)Users’ search performance prediction in cross-device searchJournal of Librarianship and Information Science10.1177/0961000622109095655:2(464-477)Online publication date: 27-Apr-2022
https://doi.org/10.1177/09610006221090956
Liu YThomas PGedeon TRusnachenko NElsweiler DKruschwitz ULudwig B(2022)Search Interfaces for Biomedical SearchingProceedings of the 2022 Conference on Human Information Interaction and Retrieval10.1145/3498366.3505769(78-89)Online publication date: 14-Mar-2022
https://dl.acm.org/doi/10.1145/3498366.3505769
Zerhoudi SGranitzer MSeifert CSchloetterer J(2022)Evaluating Simulated User Interaction and Search BehaviourAdvances in Information Retrieval10.1007/978-3-030-99739-7_28(240-247)Online publication date: 5-Apr-2022
https://doi.org/10.1007/978-3-030-99739-7_28
Show More Cited By

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Media

Figures

Other

Tables

View Table of Contents