Article

Complement Lexical Retrieval Model with Semantic Residual Embeddings

Authors:

Benjamin Van Durme,

Jamie CallanAuthors Info & Claims

Advances in Information Retrieval: 43rd European Conference on IR Research, ECIR 2021, Virtual Event, March 28 – April 1, 2021, Proceedings, Part I

Pages 146 - 160

https://doi.org/10.1007/978-3-030-72113-8_10

Published: 28 March 2021 Publication History

Abstract

This paper presents clear, a retrieval model that seeks to complement classical lexical exact-match models such as BM25 with semantic matching signals from a neural embedding matching model.clear explicitly trains the neural embedding to encode language structures and semantics that lexical retrieval fails to capture with a novel residual-based embedding learning method. Empirical evaluations demonstrate the advantages of clear over state-of-the-art retrieval models, and that it can substantially improve the end-to-end accuracy and efficiency of reranking pipelines.

References

[1]

Bahdanau, D., Cho, K., Bengio, Y.: Neural machine translation by jointly learning to align and translate. CoRR abs/1409.0473 (2015)

[2]

Bromley J, Guyon I, LeCun Y, Säckinger E, and Shah R Signature verification using a Siamese time delay neural network Adv. Neural Inf. Process. Syst. 1993 6 737-744

[3]

Caid WR, Dumais ST, and Gallant SI Learned vector-space models for document retrieval Inf. Process. Manag. 1995 31 3 419-429

Digital Library

[4]

Chang, W., Yu, F.X., Chang, Y., Yang, Y., Kumar, S.: Pre-training tasks for embedding-based large-scale retrieval. In: 8th International Conference on Learning Representations (2020)

[5]

Chen, T., Van Durme, B.: Discriminative information retrieval for question answering sentence selection. In: Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics, pp. 719–725 (2017)

[6]

Craswell, N., Mitra, B., Yilmaz, E., Campos, D.: Overview of the TREC 2019 deep learning track. In: TREC (to appear) (2019)

[7]

Dai, Z., Callan, J.: Context-aware document term weighting for ad-hoc search. In: WWW 2020: The Web Conference 2020, pp. 1897–1907 (2020)

[8]

Dai, Z., Callan, J.: Context-aware term weighting for first-stage passage retrieval. In: The 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval (to appear) (2020)

[9]

Deerwester SC, Dumais ST, Landauer TK, Furnas GW, and Harshman RA Indexing by latent semantic analysis J. Am. Soc. Inf. Sci. 1990 41 6 391-407

[10]

Devlin, J., Chang, M., Lee, K., Toutanova, K.: BERT: pre-training of deep bidirectional transformers for language understanding. In: Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 4171–4186 (2019)

[11]

Guo, J., Fan, Y., Ai, Q., Croft, W.B.: A deep relevance matching model for ad-hoc retrieval. In: Proceedings of the 25th ACM International Conference on Information and Knowledge Management, pp. 55–64 (2016)

[12]

Guo, R., et al.: Accelerating large-scale inference with anisotropic vector quantization. In: Proceedings of the 37th International Conference on Machine Learning (2020)

[13]

Guu, K., Lee, K., Tung, Z., Pasupat, P., Chang, M.: REALM: retrieval-augmented language model pre-training. CoRR abs/2002.08909 (2020)

[14]

Hochreiter S and Schmidhuber J Long short-term memory Neural Comput. 1997 9 1735-1780

Digital Library

[15]

Johnson, J., Douze, M., Jégou, H.: Billion-scale similarity search with GPUs. CoRR abs/1702.08734 (2017)

[16]

Kim, Y.: Convolutional neural networks for sentence classification. In: EMNLP (2014)

[17]

Kingma, D.P., Ba, J.: Adam: A method for stochastic optimization. In: 3rd International Conference on Learning Representations (2015)

[18]

Kuzi, S., Zhang, M., Li, C., Bendersky, M., Najork, M.: Leveraging semantic and lexical matching to improve the recall of document retrieval systems: A hybrid approach. ArXiv abs/2010.01195 (2020)

[19]

Lafferty, J.D., Zhai, C.: Document language models, query models, and risk minimization for information retrieval. In: SIGIR 2001: Proceedings of the 24th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 111–119 (2001)

[20]

Lavrenko, V., Croft, W.B.: Relevance-based language models. In: SIGIR 2001: Proceedings of the 24th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 120–127 (2001)

[21]

Lee, K., Chang, M., Toutanova, K.: Latent retrieval for weakly supervised open domain question answering. In: Proceedings of the 57th Conference of the Association for Computational Linguistics, pp. 6086–6096 (2019)

[22]

Lin, J.: The neural hype and comparisons against weak baselines. In: SIGIR Forum, pp. 40–51 (2018)

[23]

Luan, Y., Eisenstein, J., Toutanova, K., Collins, M.: Sparse, dense, and attentional representations for text retrieval. Transactions of the Association of Computational Linguistics (2020)

[24]

Malkov, Y.A., Yashunin, D.A.: Efficient and robust approximate nearest neighbor search using hierarchical navigable small world graphs. IEEE Trans. Pattern Anal. Mach. Intell. 42(4), 824-836 (2018)

[25]

Metzler, D., Croft, W.B.: A markov random field model for term dependencies. In: SIGIR 2005: Proceedings of the 28th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 472–479 (2005)

[26]

Mitra, B., Diaz, F., Craswell, N.: Learning to match using local and distributed representations of text for web search. In: Proceedings of the 26th International Conference on World Wide Web, pp. 1291–1299 (2017)

[27]

Nguyen, T., et al.: MS MARCO: A human generated machine reading comprehension dataset. In: Proceedings of the Workshop on Cognitive Computation: Integrating Neural and Symbolic Approaches 2016 Co-Located with the 30th Annual Conference on Neural Information Processing Systems (2016)

[28]

Nogueira, R., Cho, K.: Passage re-ranking with bert. arXiv:1901.04085 (2019)

[29]

Nogueira, R., Yang, W., Lin, J., Cho, K.: Document expansion by query prediction. CoRR abs/1904.08375 (2019)

[30]

Rajashekar TB and Croft WB Combining automatic and manual index representations in probabilistic retrieval J. Am. Soc. Inf. Sci. 1995 46 4 272-283

[31]

Reimers, N., Gurevych, I.: Sentence-Bert: Sentence embeddings using Siamese Bert-networks. In: Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pp. 3980–3990 (2019)

[32]

Robertson, S.E., Walker, S.: Some simple effective approximations to the 2-Poisson model for probabilistic weighted retrieval. In: Proceedings of the 17th Annual International ACM-SIGIR Conference on Research and Development in Information Retrieval, pp. 232–241 (1994)

[33]

Salton, G., McGill, M.: Introduction to Modern Information Retrieval. McGraw-Hill Book Company (1984)

[34]

Shrivastava A and Li P Asymmetric LSH (ALSH) for sublinear time maximum inner product search (MIPS) Adv. Neural Inf. Process. Syst. 2014 27 2321-2329

Digital Library

[35]

Vaswani, A., et al.: Attention is all you need. In: Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, pp. 5998–6008 (2017)

[36]

Weston, J., Watkins, C.: Support vector machines for multi-class pattern recognition. In: ESANN 1999, 7th European Symposium on Artificial Neural Networks, pp. 219–224 (1999)

[37]

Wolf, T., et al.: Huggingface’s transformers: State-of-the-art natural language processing. CoRR abs/1910.03771 (2019)

[38]

Yang, P., Fang, H., Lin, J.: Anserini: enabling the use of Lucene for information retrieval research. In: Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 1253–1256 (2017)

[39]

Yao, X., Van Durme, B., Clark, P.: Automatic coupling of answer extraction and information retrieval. In: Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics, pp. 159–165 (2013)

[40]

Zamani, H., Dehghani, M., Croft, W.B., Learned-Miller, E.G., Kamps, J.: From neural re-ranking to neural ranking: Learning a sparse representation for inverted indexing. In: Proceedings of the 27th ACM International Conference on Information and Knowledge Management, pp. 497–506 (2018)

Cited By

Leonhardt JMüller HRudra KKhosla MAnand AAnand A(2024)Efficient Neural Ranking Using Forward Indexes and Lightweight EncodersACM Transactions on Information Systems10.1145/363193942:5(1-34)Online publication date: 29-Apr-2024
https://dl.acm.org/doi/10.1145/3631939
Luo JQian CGlass LMa FHui Yang GWang HHan SHauff CZuccon GZhang Y(2024)Clinical Trial Retrieval via Multi-grained Similarity LearningProceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval10.1145/3626772.3661366(2950-2954)Online publication date: 10-Jul-2024
https://dl.acm.org/doi/10.1145/3626772.3661366
McKechnie JMcDonald GMacdonald CHui Yang GWang HHan SHauff CZuccon GZhang Y(2024)Bi-Objective Negative Sampling for Sensitivity-Aware SearchProceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval10.1145/3626772.3657895(2296-2300)Online publication date: 10-Jul-2024
https://dl.acm.org/doi/10.1145/3626772.3657895
Show More Cited By

Recommendations

Lexical ambiguity and information retrieval

Lexical ambiguity is a pervasive problem in natural language processing. However, little quantitative information is available about the extent of the problem or about the impact that it has on information retrieval systems. We report on an analysis of ...
Learning Word Embeddings from Portuguese Lexical-Semantic Knowledge Bases
Computational Processing of the Portuguese Language
Abstract
This paper describes the creation of PT-LKB, new Portuguese word embeddings learned from a large lexical-semantic knowledge base (LKB), using the node2vec method. Resulting embeddings combine the strengths of word vector representations and, even ...
Exploring Portuguese Word Embeddings for Discovering Lexical-Semantic Relations
Computational Processing of the Portuguese Language
Abstract
Word2vec-like word embeddings are known for keeping linguistic regularities and thus good for solving analogies. Following this, we explore such embeddings for Portuguese in the discovery of lexical-semantic relations, which can be used for ...

Comments

Information & Contributors

Information

Published In

cover image Guide Proceedings

Advances in Information Retrieval: 43rd European Conference on IR Research, ECIR 2021, Virtual Event, March 28 – April 1, 2021, Proceedings, Part I

Mar 2021

807 pages

ISBN:978-3-030-72112-1

DOI:10.1007/978-3-030-72113-8

Editors:
Djoerd Hiemstra
Radboud University Nijmegen, Nijmegen, The Netherlands
,
Marie-Francine Moens
Department of Computer Science, Katholieke Universiteit Leuven, Heverlee, Belgium
,
Josiane Mothe
Toulouse Institute of Computer Science Research, Toulouse, France
,
Raffaele Perego
Istituto di Scienza e Tecnologie dell’Informazione, Consiglio Nazionale delle Ricerche, Pisa, Italy
,
Martin Potthast
Leipzig University, Leipzig, Germany
,
Fabrizio Sebastiani
Istituto di Scienza e Tecnologie dell’Informazione, Consiglio Nazionale delle Ricerche, Pisa, Italy

© Springer Nature Switzerland AG 2021.

Publisher

Springer-Verlag

Berlin, Heidelberg

Publication History

Published: 28 March 2021

Qualifiers

Article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

21
Total Citations
View Citations
0
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 17 Oct 2024

Other Metrics

View Author Metrics

Citations

Cited By

Leonhardt JMüller HRudra KKhosla MAnand AAnand A(2024)Efficient Neural Ranking Using Forward Indexes and Lightweight EncodersACM Transactions on Information Systems10.1145/363193942:5(1-34)Online publication date: 29-Apr-2024
https://dl.acm.org/doi/10.1145/3631939
Luo JQian CGlass LMa FHui Yang GWang HHan SHauff CZuccon GZhang Y(2024)Clinical Trial Retrieval via Multi-grained Similarity LearningProceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval10.1145/3626772.3661366(2950-2954)Online publication date: 10-Jul-2024
https://dl.acm.org/doi/10.1145/3626772.3661366
McKechnie JMcDonald GMacdonald CHui Yang GWang HHan SHauff CZuccon GZhang Y(2024)Bi-Objective Negative Sampling for Sensitivity-Aware SearchProceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval10.1145/3626772.3657895(2296-2300)Online publication date: 10-Jul-2024
https://dl.acm.org/doi/10.1145/3626772.3657895
Upadhyay RAskari APasi GViviani M(2024)Beyond Topicality: Including Multidimensional Relevance in Cross-encoder Re-rankingAdvances in Information Retrieval10.1007/978-3-031-56027-9_16(262-277)Online publication date: 24-Mar-2024
https://dl.acm.org/doi/10.1007/978-3-031-56027-9_16
Kishore VWan CLovelace JArtzi YWeinberger KKrause ABrunskill ECho KEngelhardt BSabato SScarlett J(2023)IncDSIProceedings of the 40th International Conference on Machine Learning10.5555/3618408.3619112(17122-17134)Online publication date: 23-Jul-2023
https://dl.acm.org/doi/10.5555/3618408.3619112
Mei LMao JHu JTan NChai HWen J(2023)Improving First-stage Retrieval of Point-of-interest Search by Pre-training ModelsACM Transactions on Information Systems10.1145/363193742:3(1-27)Online publication date: 29-Dec-2023
https://dl.acm.org/doi/10.1145/3631937
Bassani ETonellotto NPasi G(2023)Personalized Query Expansion with Contextual Word EmbeddingsACM Transactions on Information Systems10.1145/362498842:2(1-35)Online publication date: 11-Dec-2023
https://dl.acm.org/doi/10.1145/3624988
Shen TGeng XTao CXu CLong GZhang KJiang DSingh ASun YAkoglu LGunopulos DYan XKumar ROzcan FYe J(2023)UnifieR: A Unified Retriever for Large-Scale RetrievalProceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining10.1145/3580305.3599927(4787-4799)Online publication date: 6-Aug-2023
https://dl.acm.org/doi/10.1145/3580305.3599927
Huang RZhang DLu WLi HWang MShi DFan JCheng ZGu SYin DSingh ASun YAkoglu LGunopulos DYan XKumar ROzcan FYe J(2023)Learning Discrete Document Representations in Web SearchProceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining10.1145/3580305.3599854(4185-4194)Online publication date: 6-Aug-2023
https://dl.acm.org/doi/10.1145/3580305.3599854
Kulkarni HMacAvaney SGoharian NFrieder OChen HDuh WHuang HKato MMothe JPoblete B(2023)Lexically-Accelerated Dense RetrievalProceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval10.1145/3539618.3591715(152-162)Online publication date: 19-Jul-2023
https://dl.acm.org/doi/10.1145/3539618.3591715
Show More Cited By

View Options

View options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Media

Figures

Other

Tables

View Table of Contents