Improving expert search effectiveness: Comparing ways to rank and present search results
Authors: 
Thomas Schoegje, 
Lynda Hardman, 
Arjen De Vries, and Toine PietersAuthors Info & Claims
Thomas Schoegje, Lynda Hardman, Arjen De Vries, and Toine PietersAuthors Info & ClaimsMarch 2024
Pages 56 - 65
Abstract
Expert search systems help professionals find colleagues with specific expertise. Expert search results can be presented as a list of documents with their associated experts, or as a list of candidate experts with evidence for their expertise based on documents they authored. The type of result may affect search behaviour, and therefore search task performance. Previous work has not considered such effects from the result presentation, focusing instead on how to rank experts or on ways to interact with the search results.
We compare the task performance of novice users using either a document-centric interface (where each search result is a document and its associated expert) or a candidate-centric interface (where each search result is a candidate expert and their associated documents). We also compare candidate-centric and document-centric ranking functions per interface.
A post-experiment survey indicated that two variables affect which interface participants preferred: the retrieval unit (candidates or documents) and the complexity (number of documents per search result). These variables affected participants’ search strategy, and consequently their task performance. A quantitative analysis revealed that 1) using the candidate-centric interface results in a higher rate of correctly completed tasks, as users evaluate candidates more thoroughly, and 2) the document-centric ranking yields faster task completion. Weak evidence of a statistical interaction effect was found that prevents a straightforward combination of the most effective interface type and the most efficient ranking type. Present work resulted in a more effective, albeit less efficient, search engine for expert search at the municipality of Utrecht.
Supplemental Material
PDF File
Appendix with supplementary details on the implementation and descriptive statistics that were not part of the main analysis
- Download
- 316.77 KB
References
[1]
Krisztian Balog, Leif Azzopardi, and Maarten de Rijke. 2006. Formal models for expert finding in enterprise corpora. In SIGIR 2006: Proceedings of the 29th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, August 6-11, 2006, Efthimis N. Efthimiadis, Susan T. Dumais, David Hawking, and Kalervo Järvelin (Eds.). ACM, Seattle, Washington, USA, 43–50. https://doi.org/10.1145/1148170.1148181
[2]
Krisztian Balog, Yi Fang, Maarten de Rijke, Pavel Serdyukov, and Luo Si. 2012. Expertise Retrieval. Found. Trends Inf. Retr. 6, 2-3 (2012), 127–256. https://doi.org/10.1561/1500000024
[3]
David Bawden and Lyn Robinson. 2011. Individual differences in information-related behaviour: what do we know about information styles?New directions in information behaviour 1 (2011), 127–158.
[4]
Nicholas J. Belkin. 2015. People, Interacting with Information. SIGIR Forum 49, 2 (2015), 13–27. https://doi.org/10.1145/2888422.2888424
[5]
Mark Berger, Jakub Zavrel, and Paul Groth. 2020. Effective distributed representations for academic expert search. In Proceedings of the First Workshop on Scholarly Document Processing, SDP@EMNLP 2020, November 19, 2020, Muthu Kumar Chandrasekaran, Anita de Waard, Guy Feigenblat, Dayne Freitag, Tirthankar Ghosal, Eduard H. Hovy, Petr Knoth, David Konopnicki, Philipp Mayr, Robert M. Patton, and Michal Shmueli-Scheuer (Eds.). Association for Computational Linguistics, Online, 56–71. https://doi.org/10.18653/v1/2020.sdp-1.7
[6]
John Brooke 1996. SUS-A quick and dirty usability scale. Usability evaluation in industry 189, 194 (1996), 4–7.
[7]
Bogeum Choi, Jaime Arguello, and Robert Capra. 2023. Understanding Procedural Search Tasks "in the Wild". In Proceedings of the 2023 Conference on Human Information Interaction and Retrieval, CHIIR 2023, March 19-23, 2023, Jacek Gwizdka and Soo Young Rieh (Eds.). ACM, Austin, TX, USA, 24–33. https://doi.org/10.1145/3576840.3578302
[8]
Bogeum Choi, Sarah Casteel, Robert Capra, and Jaime Arguello. 2022. Procedural Knowledge Search by Intelligence Analysts. In CHIIR ’22: ACM SIGIR Conference on Human Information Interaction and Retrieval, March 14 - 18, 2022, David Elsweiler (Ed.). ACM, Regensburg, Germany, 169–179. https://doi.org/10.1145/3498366.3505810
[9]
Sujatha Das Gollapalli, Prasenjit Mitra, and C. Lee Giles. 2012. Similar researcher search in academic environments. In Proceedings of the 12th ACM/IEEE-CS Joint Conference on Digital Libraries, JCDL ’12, June 10-14, 2012, Karim B. Boughida, Barrie Howard, Michael L. Nelson, Herbert Van de Sompel, and Ingeborg Sølvberg (Eds.). ACM, Washington, DC, USA, 167–170. https://doi.org/10.1145/2232817.2232849
[10]
Rodrigo Gonçalves and Carina Friedrich Dorneles. 2019. Automated Expertise Retrieval: A Taxonomy-Based Survey and Open Issues. ACM Comput. Surv. 52, 5 (2019), 96:1–96:30. https://doi.org/10.1145/3331000
[11]
Shuguang Han, Daqing He, Jiepu Jiang, and Zhen Yue. 2013. Supporting exploratory people search: a study of factor transparency and user control. In 22nd ACM International Conference on Information and Knowledge Management, CIKM’13, October 27 - November 1, 2013, Qi He, Arun Iyengar, Wolfgang Nejdl, Jian Pei, and Rajeev Rastogi (Eds.). ACM, San Francisco, CA, USA, 449–458. https://doi.org/10.1145/2505515.2505684
[12]
Shuguang Han, Danchen Zhang, Daqing He, and Qikai Cheng. 2016. User exploration of slider facets in interactive people search system. IConference 2016 Proceedings 1, 1 (2016), 5 pages.
[13]
Faegheh Hasibi, Krisztian Balog, and Svein Erik Bratsberg. 2017. Dynamic Factual Summaries for Entity Cards. In Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval, August 7-11, 2017, Noriko Kando, Tetsuya Sakai, Hideo Joho, Hang Li, Arjen P. de Vries, and Ryen W. White (Eds.). ACM, Shinjuku, Tokyo, Japan, 773–782. https://doi.org/10.1145/3077136.3080810
[14]
Morten Hertzum. 2014. Expertise seeking: A review. Inf. Process. Manag. 50, 5 (2014), 775–795. https://doi.org/10.1016/j.ipm.2014.04.003
[15]
Katja Hofmann, Krisztian Balog, Toine Bogers, and Maarten de Rijke. 2010. Contextual factors for finding similar experts. J. Assoc. Inf. Sci. Technol. 61, 5 (2010), 994–1014. https://doi.org/10.1002/asi.21292
[16]
Omayma Husain, Naomie Salim, Rose Alinda Alias, Samah Abdelsalam, and Alzubair Hassan. 2019. Expert finding systems: A systematic review. Applied Sciences 9, 20 (2019), 4250.
[17]
Peter Ingwersen and Kalervo Järvelin. 2005. The Turn - Integration of Information Seeking and Retrieval in Context. The Kluwer International Series on Information Retrieval, Vol. 18. Springer, Dordrecht, Netherlands. https://doi.org/10.1007/1-4020-3851-8
[18]
Peter Ingwersen and Kalervo Järvelin. 2006. The turn: Integration of information seeking and retrieval in context. Vol. 18. Springer Science & Business Media, Dordrecht, Netherlands.
[19]
W Iso. 1998. 9241-11. Ergonomic requirements for office work with visual display terminals (VDTs). The international organization for standardization 45, 9 (1998), 22 pages.
[20]
Udo Kruschwitz and Charlie Hull. 2017. Searching the Enterprise. Found. Trends Inf. Retr. 11, 1 (2017), 1–142. https://doi.org/10.1561/1500000053
[21]
James R. Lewis. 2018. The System Usability Scale: Past, Present, and Future. Int. J. Hum. Comput. Interact. 34, 7 (2018), 577–590. https://doi.org/10.1080/10447318.2018.1455307
[22]
Ruud Liebregts and Toine Bogers. 2009. Design and Evaluation of a University-Wide Expert Search Engine. In Advances in Information Retrieval, 31th European Conference on IR Research, ECIR 2009, April 6-9, 2009. Proceedings(Lecture Notes in Computer Science, Vol. 5478), Mohand Boughanem, Catherine Berrut, Josiane Mothe, and Chantal Soulé-Dupuy (Eds.). Springer, Toulouse, France, 587–594. https://doi.org/10.1007/978-3-642-00958-7_54
[23]
Rennan C. Lima and Rodrygo L. T. Santos. 2022. On Extractive Summarization for Profile-centric Neural Expert Search in Academia. In SIGIR ’22: The 45th International ACM SIGIR Conference on Research and Development in Information Retrieval, July 11 - 15, 2022, Enrique Amigó, Pablo Castells, Julio Gonzalo, Ben Carterette, J. Shane Culpepper, and Gabriella Kazai (Eds.). ACM, Madrid, Spain, 2331–2335. https://doi.org/10.1145/3477495.3531713
[24]
Han-Chin Liu. 2018. Investigating the impact of cognitive style on multimedia learners’ understanding and visual search patterns: an eye-tracking approach. Journal of Educational Computing Research 55, 8 (2018), 1053–1068.
[25]
Marianne Lykke, Ann Bygholm, Louise Bak Søndergaard, and Katriina Byström. 2022. The role of historical and contextual knowledge in enterprise search. J. Documentation 78, 5 (2022), 1053–1074. https://doi.org/10.1108/JD-08-2021-0170
[26]
Vítor Mangaravite and Rodrygo L. T. Santos. 2016. On Information-Theoretic Document-Person Associations for Expert Search in Academia. In Proceedings of the 39th International ACM SIGIR conference on Research and Development in Information Retrieval, SIGIR 2016, July 17-21, 2016, Raffaele Perego, Fabrizio Sebastiani, Javed A. Aslam, Ian Ruthven, and Justin Zobel (Eds.). ACM, Pisa,Italy, 925–928. https://doi.org/10.1145/2911451.2914751
[27]
Junichiro Mori, Nathalie Basselin, Alexander Kröner, and Anthony Jameson. 2008. Find me if you can: designing interfaces for people search. In Proceedings of the 13th International Conference on Intelligent User Interfaces, IUI 2008, January 13-16, 2008, Jeffrey M. Bradshaw, Henry Lieberman, and Steffen Staab (Eds.). ACM, Gran Canaria, Canary Islands, Spain, 377–380. https://doi.org/10.1145/1378773.1378834
[28]
Harumi Murakami, Hiroshi Ueda, Shin’ichi Kataoka, Yuya Takamori, and Shoji Tatsumi. 2010. Summarizing and Visualizing Web People Search Results. In ICAART 2010 - Proceedings of the International Conference on Agents and Artificial Intelligence, Volume 1 - Artificial Intelligence, January 22-24, 2010, Joaquim Filipe, Ana L. N. Fred, and Bernadette Sharp (Eds.). INSTICC Press, Valencia, Spain, 640–643.
[29]
Jakob Nielsen. 2012. Usability 101: Introduction to usability. https://www.nngroup.com/articles/usability-101-introduction-to-usability/
[30]
Anne Oeldorf-Hirsch, Brent J. Hecht, Meredith Ringel Morris, Jaime Teevan, and Darren Gergle. 2014. To search or to ask: the routing of information needs between traditional search engines and social networks. In Computer Supported Cooperative Work, CSCW ’14, February 15-19, 2014, Susan R. Fussell, Wayne G. Lutters, Meredith Ringel Morris, and Madhu C. Reddy (Eds.). ACM, Baltimore, MD, USA, 16–27. https://doi.org/10.1145/2531602.2531706
[31]
Georg Pardi, Steffen Gottschling, Peter Gerjets, and Yvonne Kammerer. 2023. The moderating effect of knowledge type on search result modality preferences in web search scenarios. Computers and Education Open 4 (2023), 100126.
[32]
Sharoda A. Paul. 2016. Find an Expert: Designing Expert Selection Interfaces for Formal Help-Giving. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems, May 7-12, 2016, Jofish Kaye, Allison Druin, Cliff Lampe, Dan Morris, and Juan Pablo Hourcade (Eds.). ACM, San Jose, CA, USA, 3038–3048. https://doi.org/10.1145/2858036.2858131
[33]
Marisa Peacock. 2009. The search for expert knowledge continues. https://www.cmswire.com/cms/enterprise-cms/the-search-for-expert-knowledge-continues-004594.php
[34]
Richard Riding and Indra Cheema. 1991. Cognitive Styles—an overview and integration. Educational Psychology 11, 3-4 (1991), 193–215. https://doi.org/10.1080/0144341910110301 arXiv:https://doi.org/10.1080/0144341910110301
[35]
Nirmal Roy, David Maxwell, and Claudia Hauff. 2022. Users and Contemporary SERPs: A (Re-)Investigation. In SIGIR ’22: The 45th International ACM SIGIR Conference on Research and Development in Information Retrieval, July 11 - 15, 2022, Enrique Amigó, Pablo Castells, Julio Gonzalo, Ben Carterette, J. Shane Culpepper, and Gabriella Kazai (Eds.). ACM, Madrid, Spain, 2765–2775. https://doi.org/10.1145/3477495.3531719
[36]
Miamaria Saastamoinen and Sanna Kumpulainen. 2014. Expected and materialised information source use by municipal officials: intertwining with task complexity. Inf. Res. 19, 4 (2014). http://www.informationr.net/ir/19-4/paper646.html
[37]
Thomas Schoegje, Arjen de Vries, Lynda Hardman, and Toine Pieters. 2023. Improving the Effectiveness and Efficiency of Web-Based Search Tasks for Policy Workers. Information 14, 7 (2023), 371.
[38]
Thomas Schoegje, Arjen P. de Vries, and Toine Pieters. 2022. Adapting a Faceted Search Task Model for the Development of a Domain-Specific Council Information Search Engine. In Electronic Government - 21st IFIP WG 8.5 International Conference, EGOV 2022, September 6-8, 2022, Proceedings(Lecture Notes in Computer Science, Vol. 13391), Marijn Janssen, Csaba Csáki, Ida Lindgren, Euripides N. Loukis, Ulf Melin, Gabriela Viale Pereira, Manuel Pedro Rodríguez Bolívar, and Efthimios Tambouris (Eds.). Springer, Linköping, Sweden, 402–418. https://doi.org/10.1007/978-3-031-15086-9_26
[39]
Yunqiu Shao, Jiaxin Mao, Yiqun Liu, Min Zhang, and Shaoping Ma. 2022. From linear to non-linear: investigating the effects of right-rail results on complex SERPs. Advances in Computational Intelligence 2, 1 (2022), 14.
[40]
Pertti Vakkari. 2020. The Usefulness of Search Results: A Systematization of Types and Predictors. In CHIIR ’20: Conference on Human Information Interaction and Retrieval, March 14-18, 2020, Heather L. O’Brien, Luanne Freund, Ioannis Arapakis, Orland Hoeber, and Irene Lopatovska (Eds.). ACM, Vancouver, BC, Canada, 243–252. https://doi.org/10.1145/3343413.3377955
[41]
Pertti Vakkari, Michael Völske, Martin Potthast, Matthias Hagen, and Benno Stein. 2019. Modeling the usefulness of search results as measured by information use. Inf. Process. Manag. 56, 3 (2019), 879–894. https://doi.org/10.1016/j.ipm.2019.02.001
[42]
Dan Wu, Shu Fan, and Fang Yuan. 2021. Research on pathways of expert finding on academic social networking sites. Inf. Process. Manag. 58, 2 (2021), 102475. https://doi.org/10.1016/j.ipm.2020.102475
[43]
Zimeng Yang, Song Yan, Abhimanyu Lad, Xiaowei Liu, and Weiwei Guo. 2021. Cascaded Deep Neural Ranking Models in LinkedIn People Search. In CIKM ’21: The 30th ACM International Conference on Information and Knowledge Management, November 1 - 5, 2021, Gianluca Demartini, Guido Zuccon, J. Shane Culpepper, Zi Huang, and Hanghang Tong (Eds.). ACM, Virtual Event, Queensland, Australia, 4312–4320. https://doi.org/10.1145/3459637.3481899
[44]
Jin-ge Yao, Xiaojun Wan, and Jianguo Xiao. 2017. Recent advances in document summarization. Knowl. Inf. Syst. 53, 2 (2017), 297–336. https://doi.org/10.1007/s10115-017-1042-4
Index Terms
- Improving expert search effectiveness: Comparing ways to rank and present search results
Recommendations
Re-ranking search results using query logs
CIKM '06: Proceedings of the 15th ACM international conference on Information and knowledge managementThis work addresses two common problems in search, frequently occurring with underspecified user queries: the top-ranked results for such queries may not contain documents relevant to the user's search intent, and fresh and relevant pages may not get ...
The influence of the document ranking in expert search
The retrieval effectiveness of the underlying document search component of an expert search engine can have an important impact on the effectiveness of the generated expert search results. In this large-scale study, we perform novel experiments in the ...
On perfect document rankings for expert search
SIGIR '09: Proceedings of the 32nd international ACM SIGIR conference on Research and development in information retrievalExpert search systems often employ a document search component to identify on-topic documents, which are then used to identify people likely to have relevant expertise. This work investigates the impact of the retrieval effectiveness of the underlying ...
Comments
Information & Contributors
Information
Published In
March 2024
481 pages
Copyright © 2024 Owner/Author.
This work is licensed under a Creative Commons Attribution International 4.0 License.
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 10 March 2024
Check for updates
Author Tags
Qualifiers
- Research-article
- Research
- Refereed limited
Data Availability
Appendix with supplementary details on the implementation and descriptive statistics that were not part of the main analysis https://dl.acm.org/doi/10.1145/3627508.3638296#CHIIR_Expert_search___appendix.pdf
Conference
CHIIR '24
CHIIR '24: 2024 ACM SIGIR Conference on Human Information Interaction and Retrieval
March 10 - 14, 2024
Sheffield, United Kingdom
Acceptance Rates
Overall Acceptance Rate 55 of 163 submissions, 34%
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 100Total Downloads
- Downloads (Last 12 months)100
- Downloads (Last 6 weeks)27
Other Metrics
Citations
View Options
View options
View or Download as a PDF file.
PDFeReader
View online with eReader.
eReaderHTML Format
View this article in HTML Format.
HTML FormatGet Access
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in