Abstract
Knowledge acquisition passage retrieval is a task that captures search in a learning or educational setting, where users seek to find key educational information within their field of interest. Traditional relevance assessments used in ad-hoc retrieval tasks tend to focus on topical relevance, often overlooking other factors such as the “informativeness” of the retrieved educational content in relation to the user’s knowledge acquisition needs. This paper presents a new test collection for the knowledge acquisition passage retrieval (KAPR) task, constructed using the data and production systems of a large academic publisher containing: First, a set of search requests covering key educational topics/concepts across different science domains. Second, a large corpus of passages extracted from review (survey) articles published in over 2, 700 journals as well as the content of 43, 000 books published in a wide range of science domains. Third, relevance assessments on both topical relevance as well as informativeness, reflecting the task-specific relevance. This resource enables direct evaluation of the user’s utility of the retrieved content and provides a comparative analysis with traditional topical relevance. Our findings indicate a strong correlation between relevance and informativeness, although the distribution of these labels varies per domain.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
Notes
- 1.
The dataset can be found here: https://github.com/acapari/KAPR.git.
- 2.
Agricultural and Biological Sciences, Biochemistry, Genetics and Molecular Biology, Chemical Engineering, Chemistry, Computer Science, Earth and Planetary Sciences, Economics, Econometrics and Finance, Engineering, Food Science, Immunology and Microbiology, Materials Science, Mathematics, Medicine and Dentistry, Neuroscience, Nursing and Health Professions, Pharmacology, Toxicology and Pharmaceutical Science, Physics and Astronomy, Psychology, Social Science, Veterinary Science and Veterinary Medicine.
References
Arabzadeh, N., Vtyurina, A., Yan, X., Clarke, C.L.: Shallow pooling for sparse labels. Inf. Retrieval J. 25(4), 365–385 (2022)
Borlund, P.: The concept of relevance in IR. J. Am. Soc. Inform. Sci. Technol. 54(10), 913–925 (2003)
Devlin, J., Chang, M.W., Lee, K., Toutanova, K.: Bert: pre-training of deep bidirectional transformers for language understanding. arXiv:1810.04805 (2018)
Ghafourian, Y.; Knoth, P., Hanbury, A.: Information retrieval evaluation in knowledge acquisition tasks. WEPIR 2021: The 3rd Workshop on Evaluation of Personalisation in Information Retrieval at CHIIR, pp. 88–95 (2021)
Ghafourian, Y.: Relevance models based on the knowledge gap. In: ECIR, pp. 488–495 (2022)
Hjørland, B.: The foundation of the concept of relevance. JASIST 61(2), 217–237 (2010)
Hofstätter, S., Lin, S.C., Yang, J.H., Lin, J., Hanbury, A.: Efficiently teaching an effective dense retriever with balanced topic aware sampling. In: SIGIR, pp. 113–122 (2021)
Huang, X., Soergel, D.: Relevance judges’ understanding of topical relevance types: an explication of an enriched concept of topical relevance. JASIST 41(1), 156–167 (2004)
Ingwersen, P., Järvelin, K.: The turn: integration of information seeking and retrieval in context, vol. 18. Springer, Cham (2006)
Karpukhin, V., et al.: Dense passage retrieval for open-domain question answering. arXiv:2004.04906 (2020)
Li, C., Yates, A., MacAvaney, S., He, B., Sun, Y.: Parade: passage representation aggregation for document reranking. preprint arXiv:2008.09093 (2020)
Liu, Y., et al.: Roberta: a robustly optimized BERT pretraining approach. CoRR abs/1907.11692 (2019), http://arxiv.org/abs/1907.11692
MacAvaney, S., Yates, A., Cohan, A., Goharian, N.: CEDR: contextualized embeddings for document ranking. In: SIGIR, pp. 1101–1104 (2019)
Malaisé, V., Otten, A., Coupet, P.: Omniscience and extensions–lessons learned from designing a multi-domain, multi-use case knowledge representation system. In: European Knowledge Acquisition Workshop, pp. 228–242 (2018)
Ni, J., et al.: Sentence-t5: scalable sentence encoders from pre-trained text-to-text models. arXiv:2108.08877 (2021a)
Ni, J., et al.: Large dual encoders are generalizable retrievers. arXiv:2112.07899 (2021b)
Nogueira, R., Cho, K.: Passage re-ranking with bert. arXiv:1901.04085 (2019)
Reimers, N., Gurevych, I.: Sentence-bert: sentence embeddings using siamese bert-networks. In: EMNLP, Association for Computational Linguistics (2019)
Robertson, S.E., Walker, S., Beaulieu, M., Willett, P.: Okapi at TREC-7: automatic ad hoc, filtering, VLC and interactive track. Nist Special Publication SP, pp. 253–264 (1999)
Sanh, V., Debut, L., Chaumond, J., Wolf, T.: Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter. ArXiv abs/1910.01108 (2019)
Saracevic, T.: Relevance reconsidered. In: Proceedings of the Second Conference on Conceptions of Library and Information Science (CoLIS 2), pp. 201–218 (1996)
Saracevic, T.: Relevance: a review of the literature and a framework for thinking on the notion in information science. part ii. Adv. Librarianship 30, 3–71 (2006)
Schwartz, A.S., Hearst, M.A.: A simple algorithm for identifying abbreviation definitions in biomedical text. In: Biocomputing 2003, pp. 451–462, World Scientific (2002)
Sormunen, E.: Liberal relevance criteria of trec- counting on negligible documents? In: SIGIR, pp. 324–330 (2002)
Thakur, N., Reimers, N., Rücklé, A., Srivastava, A., Gurevych, I.: Beir: a heterogenous benchmark for zero-shot evaluation of information retrieval models. arXiv:2104.08663 (2021)
Voorhees, E.M., Craswell, N., Lin, J.: Too many relevants: Whither cranfield test collections? In: SIGIR, pp. 2970–2980 (2022)
Wang, X., Macdonald, C., Ounis, I.: Improving zero-shot retrieval using dense external expansion. Inf. Process. Manage. 59(5), 103026 (2022)
Wang, Y., Wang, L., Li, Y., He, D., Liu, T.Y.: A theoretical analysis of NDCG type ranking measures. In: Conference on Learning Theory, pp. 25–54 (2013)
Acknowledgments
The KAPR test collection is constructed using the data and production systems of Elsevier’s ScienceDirect. The constructed dataset is available here: https://github.com/acapari/KAPR.git. Jaap Kamps is partly funded by the Netherlands Organization for Scientific Research (NWO CI # CISC.CC.016, NWO NWA # 1518.22.105), the University of Amsterdam (AI4FinTech program), and ICAI (AI for Open Government Lab). Views expressed in this paper are not necessarily shared or endorsed by those funding the research.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2024 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Capari, A., Azarbonyad, H., Tsatsaronis, G., Afzal, Z., Dunham, J., Kamps, J. (2024). Knowledge Acquisition Passage Retrieval: Corpus, Ranking Models, and Evaluation Resources. In: Goeuriot, L., et al. Experimental IR Meets Multilinguality, Multimodality, and Interaction. CLEF 2024. Lecture Notes in Computer Science, vol 14958. Springer, Cham. https://doi.org/10.1007/978-3-031-71736-9_3
Download citation
DOI: https://doi.org/10.1007/978-3-031-71736-9_3
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-71735-2
Online ISBN: 978-3-031-71736-9
eBook Packages: Computer ScienceComputer Science (R0)