research-article

Periodicity in User Engagement with a Search Engine and Its Application to Online Controlled Experiments

Authors:

Pavel SerdyukovAuthors Info & Claims

ACM Transactions on the Web (TWEB), Volume 11, Issue 2

Article No.: 9, Pages 1 - 35

https://doi.org/10.1145/2856822

Published: 14 April 2017 Publication History

Abstract

Nowadays, billions of people use the Web in connection with their daily needs. A significant part of these needs are constituted by search tasks that are usually addressed by search engines. Thus, daily search needs result in regular user engagement with a search engine. User engagement with web services was studied in various aspects, but there appears to be little work devoted to its regularity and periodicity. In this article, we study periodicity of user engagement with a popular search engine through applying spectrum analysis to temporal sequences of different engagement metrics. First, we found periodicity patterns of user engagement and revealed classes of users whose periodicity patterns do not change over a long period of time. In addition, we give an exhaustive analysis of the stability and quality of identified clusters. Second, we used the spectrum series as key metrics to evaluate search quality. We found that the novel periodicity metrics outperform the state-of-the-art quality metrics both in terms of significance level (p-value) and sensitivity to a large set of larges-scale A/B experiments conducted on real search engine users.

References

[1]

Eytan Adar, Jaime Teevan, and Susan T. Dumais. 2008. Large scale analysis of web revisitation patterns. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM, 1197--1206.

Digital Library

[2]

Eytan Adar, Jaime Teevan, and Susan T. Dumais. 2009a. Resonance on the web: Web dynamics and revisitation patterns. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM, 1381--1390.

Digital Library

[3]

Eytan Adar, Jaime Teevan, Susan T. Dumais, and Jonathan L. Elsas. 2009b. The web changes everything: Understanding the dynamics of web content. In Proceedings of the 2nd ACM International Conference on Web Search and Data Mining. ACM, 282--291.

Digital Library

[4]

Olga Arkhipova, Lidia Grauer, Igor Kuralenok, and Pavel Serdyukov. 2015. Search engine evaluation based on search engine switching prediction. In Proceedings of the 38th International ACM SIGIR Conference on Research and Development in Information Retrieval. ACM, 723--726.

Digital Library

[5]

Sunandan Chakraborty, Filip Radlinski, Milad Shokouhi, and Paul Baecke. 2014. On correlation of absence time and search effectiveness. In Proceedings of the 37th International ACM SIGIR Conference on Research 8 Development in Information Retrieval. ACM, 1163--1166.

Digital Library

[6]

William T. Cochran, James W. Cooley, David L. Favin, Howard D. Helms, Reg A. Kaenel, William W. Lang, George C. Maling Jr., David E. Nelson, Charles M. Rader, and Peter D. Welch. 1967. What is the fast fourier transform? Proc. IEEE 55, 10 (1967), 1664--1674.

[7]

James W. Cooley and John W. Tukey. 1965. An algorithm for the machine calculation of complex Fourier series. Math. Comput. 19, 90 (1965), 297--301.

[8]

Thomas Crook, Brian Frasca, Ron Kohavi, and Roger Longbotham. 2009. Seven pitfalls to avoid when running controlled experiments on the web. In Proceedings of the 15th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. ACM, 1105--1114.

Digital Library

[9]

David L. Davies and Donald W. Bouldin. 1979. A cluster separation measure. IEEE Trans. Pattern Anal. Mach. Intell. 2 (1979), 224--227.

Digital Library

[10]

Alex Deng. 2015. Objective bayesian two sample hypothesis testing for online controlled experiments. In Proceedings of the 24th International Conference on World Wide Web Companion. International World Wide Web Conferences Steering Committee, 923--928.

Digital Library

[11]

Alex Deng and Victor Hu. 2015. Diluted treatment effect estimation for trigger analysis in online controlled experiments. In Proceedings of the 8th ACM International Conference on Web Search and Data Mining. ACM, 349--358.

Digital Library

[12]

Alex Deng, Tianxi Li, and Yu Guo. 2014. Statistical inference in two-stage online controlled experiments with treatment selection and validation. In Proceedings of the 23rd International Conference on World Wide Web. ACM, 609--618.

Digital Library

[13]

Alex Deng, Ya Xu, Ron Kohavi, and Toby Walker. 2013. Improving the sensitivity of online controlled experiments by utilizing pre-experiment data. In Proceedings of the 6th ACM International Conference on Web Search and Data Mining. ACM, 123--132.

Digital Library

[14]

Abdigani Diriye, Ryen White, Georg Buscher, and Susan Dumais. 2012. Leaving so soon?: Understanding and predicting web search abandonment rationales. In Proceedings of the 21st ACM International Conference on Information and Knowledge Management. ACM, 1025--1034.

Digital Library

[15]

Alexey Drutsa. 2015. Sign-aware periodicity metrics of user engagement for online search quality evaluation. In Proceedings of the 38th International ACM SIGIR Conference on Research and Development in Information Retrieval. ACM, 779--782.

Digital Library

[16]

Alexey Drutsa, Gleb Gusev, and Pavel Serdyukov. 2015a. Engagement periodicity in search engine usage: Analysis and its application to search quality evaluation. In Proceedings of the 8th ACM International Conference on Web Search and Data Mining. ACM, 27--36.

Digital Library

[17]

Alexey Drutsa, Gleb Gusev, and Pavel Serdyukov. 2015b. Future user engagement prediction and its application to improve the sensitivity of online experiments. In Proceedings of the 24th International Conference on World Wide Web. International World Wide Web Conferences Steering Committee, 256--266.

Digital Library

[18]

Alexey Drutsa, Anna Ufliand, and Gleb Gusev. 2015c. Practical aspects of sensitivity in online experimentation with user engagement metrics. In Proceedings of the 24th ACM International on Conference on Information and Knowledge Management. ACM, 763--772.

Digital Library

[19]

Georges Dupret and Mounia Lalmas. 2013. Absence time and user engagement: Evaluating ranking functions. In Proceedings of the Sixth ACM International Conference on Web Search and Data Mining. ACM, 173--182.

Digital Library

[20]

Henry A. Feild, James Allan, and Rosie Jones. 2010. Predicting searcher frustration. In Proceedings of the 33rd International ACM SIGIR Conference on Research and Development in Information Retrieval. ACM, 34--41.

Digital Library

[21]

David A. Freedman, David Collier, Jasjeet S. Sekhon, and Philip B. Stark. 2010. Statistical Models and Causal Inference: A Dialogue with the Social Sciences. Cambridge University Press.

[22]

Qi Guo, Ryen W. White, Yunqiao Zhang, Blake Anderson, and Susan T. Dumais. 2011. Why searchers switch: Understanding and predicting engine switching rationales. In Proceedings of the 34th International ACM SIGIR Conference on Research and Development in Information Retrieval. ACM, 335--344.

Digital Library

[23]

Ahmed Hassan, Xiaolin Shi, Nick Craswell, and Bill Ramsey. 2013. Beyond clicks: Query reformulation as a predictor of search satisfaction. In Proceedings of the 22nd ACM International Conference on Conference on Information 8 Knowledge Management. ACM, 2019--2028.

Digital Library

[24]

Ahmed Hassan, Yang Song, and Li-wei He. 2011. A task level metric for measuring web search satisfaction and its application on improving relevance estimation. In Proceedings of the 20th ACM International Conference on Information and Knowledge Management. ACM, 125--134.

Digital Library

[25]

Ahmed Hassan and Ryen W. White. 2013. Personalized models of search satisfaction. In Proceedings of the 22nd ACM International Conference on Conference on Information 8 Knowledge Management. ACM, 2009--2018.

Digital Library

[26]

Henning Hohnhold, Deirdre O’Brien, and Diane Tang. 2015. Focusing on the long-term: It’s good for users and business. In Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. ACM, 1849--1858.

Digital Library

[27]

Victor Hu, Maria Stone, Jan Pedersen, and Ryen W White. 2011. Effects of search success on search engine re-use. In Proceedings of the 20th ACM International Conference on Information and Knowledge Management. ACM, 1841--1846.

Digital Library

[28]

Bernard J. Jansen, Amanda Spink, and Vinish Kathuria. 2007. How to define searching sessions on web search engines. In Advances in Web Mining and Web Usage Analysis. Springer, 92--109.

[29]

Rosie Jones and Kristina Lisa Klinkner. 2008. Beyond the session timeout: Automatic hierarchical segmentation of search topics in query logs. In Proceedings of the 17th ACM Conference on Information and Knowledge Management. ACM, 699--708.

Digital Library

[30]

Eugene Kharitonov, Craig Macdonald, Pavel Serdyukov, and Iadh Ounis. 2015a. Optimised scheduling of online experiments. In Proceedings of the 38th International ACM SIGIR Conference on Research and Development in Information Retrieval. ACM, 453--462.

Digital Library

[31]

Eugene Kharitonov, Aleksandr Vorobev, Craig Macdonald, Pavel Serdyukov, and Iadh Ounis. 2015b. Sequential testing for early stopping of online experiments. In Proceedings of the 38th International ACM SIGIR Conference on Research and Development in Information Retrieval. ACM, 473--482.

Digital Library

[32]

Ronny Kohavi, Thomas Crook, Roger Longbotham, Brian Frasca, Randy Henne, Juan Lavista Ferres, and Tamir Melamed. 2009. Online experimentation at microsoft. (2009), 11--23.

[33]

Ron Kohavi, Alex Deng, Brian Frasca, Roger Longbotham, Toby Walker, and Ya Xu. 2012. Trustworthy online controlled experiments: Five puzzling outcomes explained. In Proceedings of the 18th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. ACM, 786--794.

Digital Library

[34]

Ron Kohavi, Alex Deng, Brian Frasca, Toby Walker, Ya Xu, and Nils Pohlmann. 2013. Online controlled experiments at large scale. In Proceedings of the 19th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. ACM, 1168--1176.

Digital Library

[35]

Ron Kohavi, Alex Deng, Roger Longbotham, and Ya Xu. 2014. Seven rules of thumb for web site experimenters. In Proceedings of the 20th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. ACM, 1857--1866.

Digital Library

[36]

Ron Kohavi, Randal M. Henne, and Dan Sommerfield. 2007. Practical guide to controlled experiments on the web: Listen to your customers not to the hippo. In Proceedings of the 13th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. ACM, 959--967.

Digital Library

[37]

Ron Kohavi, Roger Longbotham, Dan Sommerfield, and Randal M. Henne. 2009. Controlled experiments on the web: Survey and practical guide. Data Min. Knowl. Discov. 18, 1 (2009), 140--181.

Digital Library

[38]

Ron Kohavi, David Messner, Seth Eliot, Juan Lavista Ferres, Randy Henne, Vignesh Kannappan, and Justin Wang. 2010. Tracking Users Clicks and Submits: Tradeoffs between User Experience and Data Loss. (2010).

[39]

Tomáš Kramár and Mária Bieliková. 2014. Context of seasonality in web search. In Advances in Information Retrieval. Springer, 644--649.

Digital Library

[40]

Janette Lehmann, Mounia Lalmas, Georges Dupret, and Ricardo Baeza-Yates. 2013. Online multitasking and user engagement. In Proceedings of the 22nd ACM International Conference on Conference on Information 8 Knowledge Management. ACM, 519--528.

Digital Library

[41]

Janette Lehmann, Mounia Lalmas, Elad Yom-Tov, and Georges Dupret. 2012. Models of user engagement. In User Modeling, Adaptation, and Personalization. Springer, 164--175.

Digital Library

[42]

Claudio Lucchese, Franco Maria Nardini, Salvatore Orlando, and Gabriele Tolomei. 2016. Learning to rank user queries to detect search tasks. In Proceedings of the 2016 ACM on International Conference on the Theory of Information Retrieval. ACM, 157--166.

Digital Library

[43]

Claudio Lucchese, Salvatore Orlando, Raffaele Perego, Fabrizio Silvestri, and Gabriele Tolomei. 2011. Identifying task-based sessions in search engine query logs. In Proceedings of the 4th ACM International Conference on Web Search and Data Mining. ACM, 277--286.

Digital Library

[44]

Claudio Lucchese, Salvatore Orlando, Raffaele Perego, Fabrizio Silvestri, and Gabriele Tolomei. 2013. Discovering tasks from search engine query logs. ACM Trans. Inf. Syst. 31, 3 (2013), 14.

Digital Library

[45]

Stephen L. Morgan and Christopher Winship. 2014. Counterfactuals and Causal Inference. Cambridge University Press.

[46]

Kirill Nikolaev, Alexey Drutsa, Ekaterina Gladkikh, Alexander Ulianov, Gleb Gusev, and Pavel Serdyukov. 2015. Extreme states distribution decomposition method for search engine online evaluation. In Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. ACM, 845--854.

Digital Library

[47]

Eric T. Peterson. 2004. Web Analytics Demystified: A Marketer’s Guide to Understanding How Your Web Site Affects Your Business. Ingram.

[48]

Alexey Poyarkov, Alexey Drutsa, Andrey Khalyavin, Gleb Gusev, and Pavel Serdyukov. 2016. Boosted decision tree regression adjustment for variance reduction in online controlled experiments. In Proceedings of the 22th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. ACM, 235--244.

Digital Library

[49]

Kira Radinsky, Krysta Svore, Susan Dumais, Jaime Teevan, Alex Bocharov, and Eric Horvitz. 2012. Modeling and predicting behavioral dynamics on the web. In Proceedings of the 21st International Conference on World Wide Web. ACM, 599--608.

Digital Library

[50]

Kerry Rodden, Hilary Hutchinson, and Xin Fu. 2010. Measuring the user experience on a large scale: User-centered metrics for web applications. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM, 2395--2398.

Digital Library

[51]

Peter J. Rousseeuw. 1987. Silhouettes: A graphical aid to the interpretation and validation of cluster analysis. J. Comput. Appl. Math. 20 (1987), 53--65.

Digital Library

[52]

Tetsuya Sakai. 2006. Evaluating evaluation metrics based on the bootstrap. In Proceedings of the 29th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval. ACM, 525--532.

Digital Library

[53]

Denis Savenkov, Dmitry Lagun, and Qiaoling Liu. 2013. Search engine switching detection based on user personal preferences and behavior patterns. In Proceedings of the 36th International ACM SIGIR Conference on Research and Development in Information Retrieval. ACM, 33--42.

Digital Library

[54]

Milad Shokouhi. 2011. Detecting seasonal queries by time-series analysis. In Proceedings of the 34th International ACM SIGIR Conference on Research and Development in Information Retrieval. ACM, 1171--1172.

Digital Library

[55]

Yang Song, Xiaolin Shi, and Xin Fu. 2013. Evaluating and predicting user engagement change with degraded search relevance. In Proceedings of the 22nd International Conference on World Wide Web. International World Wide Web Conferences Steering Committee, 1213--1224.

Digital Library

[56]

Diane Tang, Ashish Agarwal, Deirdre O’Brien, and Mike Meyer. 2010. Overlapping experiment infrastructure: More, better, faster experimentation. In Proceedings of the 16th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. ACM, 17--26.

Digital Library

[57]

Jaime Teevan, Susan T. Dumais, Daniel J. Liebling, and Richard L. Hughes. 2009. Changing how people view changes on the web. In Proceedings of the 22nd Annual ACM Symposium on User Interface Software and Technology. ACM, 237--246.

Digital Library

[58]

Michail Vlachos, Christopher Meek, Zografoula Vagena, and Dimitrios Gunopulos. 2004. Identifying similarities, periodicities and bursts for online search queries. In Proceedings of the 2004 ACM SIGMOD International Conference on Management of Data. ACM, 131--142.

Digital Library

[59]

Hongning Wang, Yang Song, Ming-Wei Chang, Xiaodong He, Ryen W. White, and Wei Chu. 2013. Learning to extract cross-session search tasks. In Proceedings of the 22nd International Conference on World Wide Web. ACM, 1353--1364.

Digital Library

[60]

William Wu-Shyong Wei. 1994. Time Series Analysis. Addison-Wesley Redwood City, CA.

[61]

Robert West, Ryen W. White, and Eric Horvitz. 2013. From cookies to cooks: Insights on dietary patterns via analysis of web usage logs. In Proceedings of the 22nd International Conference on World Wide Web. International World Wide Web Conferences Steering Committee, 1399--1410.

Digital Library

[62]

Ryen W. White and Susan T. Dumais. 2009. Characterizing and predicting search engine switching behavior. In Proceedings of the 18th ACM Conference on Information and Knowledge Management. ACM, 87--96.

Digital Library

[63]

Ryen W. White, Ashish Kapoor, and Susan T. Dumais. 2010. Modeling long-term search engine usage. In UMAP. 28--39.

Digital Library

[64]

Ying Zhang, Bernard J. Jansen, and Amanda Spink. 2009. Time series analysis of a Web search engine transaction log. Inf. Process. Manag. 45, 2 (2009), 230--245.

Digital Library

Cited By

Chen JChen H(2024)Unlocking the secrets of daily app switching: a comprehensive guide to mastering both intra-app and inter-app search strategiesLibrary Hi Tech10.1108/LHT-03-2024-0136Online publication date: 9-Jul-2024
https://doi.org/10.1108/LHT-03-2024-0136
Quin FWeyns DGalster MSilva C(2024)A/B testingJournal of Systems and Software10.1016/j.jss.2024.112011211:COnline publication date: 2-Jul-2024
https://dl.acm.org/doi/10.1016/j.jss.2024.112011
Chen JChen HLi Y(2023)Transitions in daily search tactics: during the cross-app interaction search processInformation Technology & People10.1108/ITP-10-2022-081437:3(1379-1402)Online publication date: 9-May-2023
https://doi.org/10.1108/ITP-10-2022-0814
Show More Cited By

Index Terms

Periodicity in User Engagement with a Search Engine and Its Application to Online Controlled Experiments

Recommendations

Practical Aspects of Sensitivity in Online Experimentation with User Engagement Metrics
CIKM '15: Proceedings of the 24th ACM International on Conference on Information and Knowledge Management

Online controlled experiments, e.g., A/B testing, is the state-of-the-art approach used by modern Internet companies to improve their services based on data-driven decisions. The most challenging problem is to define an appropriate online metric of user ...
Using the Delay in a Treatment Effect to Improve Sensitivity and Preserve Directionality of Engagement Metrics in A/B Experiments
WWW '17: Proceedings of the 26th International Conference on World Wide Web

State-of-the-art user engagement metrics (such as session-per-user) are widely used by modern Internet companies to evaluate ongoing updates of their web services via A/B testing. These metrics are predictive of companies' long-term goals, but suffer ...
Engagement Periodicity in Search Engine Usage: Analysis and its Application to Search Quality Evaluation
WSDM '15: Proceedings of the Eighth ACM International Conference on Web Search and Data Mining

Nowadays, billions of people use the Web in connection with their daily needs. A significant part of the needs are constituted by search tasks that are usually addressed by search engines. Thus, daily search needs result in regular user engagement with ...

Comments

Information & Contributors

Information

Published In

cover image ACM Transactions on the Web

ACM Transactions on the Web Volume 11, Issue 2

May 2017

199 pages

ISSN:1559-1131

EISSN:1559-114X

DOI:10.1145/3079924

Editors:
Brian D. Davison
Lehigh University, USA
,
Marianne Winslett
University of Illinois at Urbana-Champaign

Issue’s Table of Contents

Copyright © 2017 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 14 April 2017

Accepted: 01 December 2016

Revised: 01 September 2016

Received: 01 December 2015

Published in TWEB Volume 11, Issue 2

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

18
Total Citations
View Citations
340
Total Downloads

Downloads (Last 12 months)21
Downloads (Last 6 weeks)3

Reflects downloads up to 10 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Chen JChen H(2024)Unlocking the secrets of daily app switching: a comprehensive guide to mastering both intra-app and inter-app search strategiesLibrary Hi Tech10.1108/LHT-03-2024-0136Online publication date: 9-Jul-2024
https://doi.org/10.1108/LHT-03-2024-0136
Quin FWeyns DGalster MSilva C(2024)A/B testingJournal of Systems and Software10.1016/j.jss.2024.112011211:COnline publication date: 2-Jul-2024
https://dl.acm.org/doi/10.1016/j.jss.2024.112011
Chen JChen HLi Y(2023)Transitions in daily search tactics: during the cross-app interaction search processInformation Technology & People10.1108/ITP-10-2022-081437:3(1379-1402)Online publication date: 9-May-2023
https://doi.org/10.1108/ITP-10-2022-0814
Hadi Mogavi RHaq EGujar SHui PMa X(2022)More Gamification Is Not Always BetterProceedings of the ACM on Human-Computer Interaction10.1145/35555536:CSCW2(1-32)Online publication date: 11-Nov-2022
https://dl.acm.org/doi/10.1145/3555553
Hadi Mogavi RZhang YHaq EWu YHui PMa X(2022)What Do Users Think of Promotional Gamification Schemes? A Qualitative Case Study in a Question Answering WebsiteProceedings of the ACM on Human-Computer Interaction10.1145/35551246:CSCW2(1-34)Online publication date: 11-Nov-2022
https://dl.acm.org/doi/10.1145/3555124
Kerpicci MPrvulovic MZajic A(2022)A Hierarchical Approach for Multiple Periodicity Detection in Software Code AnalysisIEEE Access10.1109/ACCESS.2022.321240110(106936-106945)Online publication date: 2022
https://doi.org/10.1109/ACCESS.2022.3212401
Jansen BSalminen JJung SGuan KJansen BSalminen JJung SGuan K(2022)Getting Your Organization Data-Driven Persona ReadyData-Driven Personas10.1007/978-3-031-02231-9_2(29-60)Online publication date: 8-Mar-2022
https://doi.org/10.1007/978-3-031-02231-9_2
Jansen BSalminen JJung SGuan K(2021)Data-Driven PersonasSynthesis Lectures on Human-Centered Informatics10.2200/S01072ED1V01Y202101HCI04814:1(i-317)Online publication date: 5-Feb-2021
https://doi.org/10.2200/S01072ED1V01Y202101HCI048
Wen QHe KSun LZhang YKe MXu HLi GLi ZIdreos SSrivastava D(2021)RobustPeriod: Robust Time-Frequency Mining for Multiple Periodicity DetectionProceedings of the 2021 International Conference on Management of Data10.1145/3448016.3452779(2328-2337)Online publication date: 9-Jun-2021
https://dl.acm.org/doi/10.1145/3448016.3452779
Jansen BJung SChowdhury SSalminen J(2021)Persona analyticsExpert Systems with Applications: An International Journal10.1016/j.eswa.2021.115611185:COnline publication date: 15-Dec-2021
https://dl.acm.org/doi/10.1016/j.eswa.2021.115611
Show More Cited By

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Issue’s Table of Contents