research-article

Semi-Siamese Bi-encoder Neural Ranking Model Using Lightweight Fine-Tuning

Authors:

Wonjong RheeAuthors Info & Claims

WWW '22: Proceedings of the ACM Web Conference 2022

Pages 502 - 511

https://doi.org/10.1145/3485447.3511978

Published: 25 April 2022 Publication History

Abstract

A BERT-based Neural Ranking Model (NRM) can be either a cross-encoder or a bi-encoder. Between the two, bi-encoder is highly efficient because all the documents can be pre-processed before the actual query time. In this work, we show two approaches for improving the performance of BERT-based bi-encoders. The first approach is to replace the full fine-tuning step with a lightweight fine-tuning. We examine lightweight fine-tuning methods that are adapter-based, prompt-based, and hybrid of the two. The second approach is to develop semi-Siamese models where queries and documents are handled with a limited amount of difference. The limited difference is realized by learning two lightweight fine-tuning modules, where the main language model of BERT is kept common for both query and document. We provide extensive experiment results for monoBERT, TwinBERT, and ColBERT where three performance metrics are evaluated over Robust04, ClueWeb09b, and MS-MARCO datasets. The results confirm that both lightweight fine-tuning and semi-Siamese are considerably helpful for improving BERT-based bi-encoders. In fact, lightweight fine-tuning is helpful for cross-encoder, too.1

References

[1]

Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, 2020. Language models are few-shot learners. arXiv preprint arXiv:2005.14165(2020).

[2]

Jamie Callan, Mark Hoy, Changkuk Yoo, and Le Zhao. 2009. Clueweb09 data set.

[3]

Hang Du, Hailin Shi, Yuchi Liu, Jun Wang, Zhen Lei, Dan Zeng, and Tao Mei. 2020. Semi-siamese training for shallow face learning. In European Conference on Computer Vision. Springer, 36–53.

Digital Library

[4]

Ruidan He, Linlin Liu, Hai Ye, Qingyu Tan, Bosheng Ding, Liying Cheng, Jia-Wei Low, Lidong Bing, and Luo Si. 2021. On the Effectiveness of Adapter-based Tuning for Pretrained Language Model Adaptation. arXiv preprint arXiv:2106.03164(2021).

[5]

Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. 2019. Parameter-efficient transfer learning for NLP. In International Conference on Machine Learning. PMLR, 2790–2799.

[6]

Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, and Weizhu Chen. 2021. LoRA: Low-Rank Adaptation of Large Language Models. arXiv preprint arXiv:2106.09685(2021).

[7]

Samuel Huston and W Bruce Croft. 2014. Parameters learned in the comparison of retrieval models using term dependencies. Ir, University of Massachusetts(2014).

[8]

Omar Khattab and Matei Zaharia. 2020. Colbert: Efficient and effective passage search via contextualized late interaction over bert. In Proceedings of the 43rd International ACM SIGIR conference on research and development in Information Retrieval. 39–48.

Digital Library

[9]

Diederik P Kingma and Jimmy Ba. 2015. Adam: A Method for Stochastic Optimization. In ICLR (Poster).

[10]

Jaejun Lee, Raphael Tang, and Jimmy Lin. 2019. What would elsa do? freezing layers during transformer fine-tuning. arXiv preprint arXiv:1911.03090(2019).

[11]

Brian Lester, Rami Al-Rfou, and Noah Constant. 2021. The power of scale for parameter-efficient prompt tuning. arXiv preprint arXiv:2104.08691(2021).

[12]

Xiang Lisa Li and Percy Liang. 2021. Prefix-tuning: Optimizing continuous prompts for generation. arXiv preprint arXiv:2101.00190(2021).

[13]

Zongxian Li, Sheng Li, Lantian Xue, and Yonghong Tian. 2019. Semi-Siamese Network for Content-Based Video Relevance Prediction. In 2019 IEEE International Symposium on Circuits and Systems (ISCAS). IEEE, 1–5.

[14]

Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. 2021. Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing. arXiv preprint arXiv:2107.13586(2021).

[15]

Yuhan Liu, Saurabh Agarwal, and Shivaram Venkataraman. 2021. AutoFreeze: Automatically Freezing Model Blocks to Accelerate Fine-tuning. arXiv preprint arXiv:2102.01386(2021).

[16]

Wenhao Lu, Jian Jiao, and Ruofei Zhang. 2020. Twinbert: Distilling knowledge to twin-structured bert models for efficient retrieval. arXiv preprint arXiv:2002.06275(2020).

[17]

Sean MacAvaney, Andrew Yates, Arman Cohan, and Nazli Goharian. 2019. CEDR: Contextualized embeddings for document ranking. In Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval. 1101–1104.

Digital Library

[18]

Tri Nguyen, Mir Rosenberg, Xia Song, Jianfeng Gao, Saurabh Tiwary, Rangan Majumder, and Li Deng. 2016. MS MARCO: A human generated machine reading comprehension dataset. In CoCo@ NIPS.

[19]

Rodrigo Nogueira, Kyunghyun Cho, and CIFAR Azrieli Global Scholar. 2019. PASSAGE RE-RANKING WITH BERT. arXiv preprint arXiv:1901.04085(2019).

[20]

Evani Radiya-Dixit and Xin Wang. 2020. How fine can fine-tuning be? learning efficient language models. In International Conference on Artificial Intelligence and Statistics. PMLR, 2435–2443.

[21]

Ellen M Voorhees 2004. Overview of TREC 2004. In Trec.

[22]

Elad Ben Zaken, Shauli Ravfogel, and Yoav Goldberg. 2021. BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-models. arXiv preprint arXiv:2106.10199(2021).

[23]

Tianyi Zhang, Felix Wu, Arzoo Katiyar, Kilian Q Weinberger, and Yoav Artzi. 2020. Revisiting few-sample BERT fine-tuning. arXiv preprint arXiv:2006.05987(2020).

[24]

Yichi Zhang and Zhiyao Duan. 2017. IMINET: Convolutional semi-Siamese networks for sound search by vocal imitation. In 2017 IEEE Workshop on Applications of Signal Processing to Audio and Acoustics (WASPAA). IEEE, 304–308.

Cited By

Leonhardt JMüller HRudra KKhosla MAnand AAnand A(2024)Efficient Neural Ranking Using Forward Indexes and Lightweight EncodersACM Transactions on Information Systems10.1145/363193942:5(1-34)Online publication date: 29-Apr-2024
https://dl.acm.org/doi/10.1145/3631939
Chang WJiang JZhang JAl-Darabsah MTeo CHsieh CYu HVishwanathan SAngélica LLattanzi SMuñoz Medina AAkoglu LGionis AVassilvitskii S(2024)PEFA: Parameter-Free Adapters for Large-scale Embedding-based Retrieval ModelsProceedings of the 17th ACM International Conference on Web Search and Data Mining10.1145/3616855.3635791(77-86)Online publication date: 4-Mar-2024
https://dl.acm.org/doi/10.1145/3616855.3635791
Pourbahman ZMomtazi SBagheri A(2023)Deep neural ranking model using distributed smoothingExpert Systems with Applications: An International Journal10.1016/j.eswa.2023.119913224:COnline publication date: 15-Aug-2023
https://dl.acm.org/doi/10.1016/j.eswa.2023.119913
Show More Cited By

Index Terms

Semi-Siamese Bi-encoder Neural Ranking Model Using Lightweight Fine-Tuning

Index terms have been assigned to the content through auto-classification.

Recommendations

Deep neural ranking model using distributed smoothing
Abstract
Information Retrieval (IR) provides access to unstructured data to satisfy user information needs within an extensive collection. In probabilistic IR models, the probability of an N-gram query is estimated by increasing the background ...
Highlights
- Providing a comprehensive study on ranking models in text information retrieval.
Fine-tuning deep convolutional neural networks for distinguishing illustrations from photographs

Automatically detecting illustrations is needed for the target system.Deep Convolutional Neural Networks have been successful in computer vision tasks.DCNN with fine-tuning outperformed the other models including handcrafted features. Systems for ...
On fine-tuning convolutional neural networks for smartphone based ocular recognition
2017 IEEE International Joint Conference on Biometrics (IJCB)
Recent reported advances in smartphone based ocular biometric recognition in visible spectrum demonstrated the efficacy of deep-learning schemes. In this paper, we evaluate convolutional neural networks (CNNs) pretrained for large scale object recognition,...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

WWW '22: Proceedings of the ACM Web Conference 2022

April 2022

3764 pages

ISBN:9781450390965

DOI:10.1145/3485447

Editors:
Frédérique Laforest
INSA Lyon, France
,
Raphaël Troncy
EURECOM, France
,
Elena Simperl
King’s College London, UK
,
Deepak Agarwal
Pinterest, USA
,
Aristides Gionis
KTH Royal Institute of Technology, Sweden
,
Ivan Herman
W3C / retired
,
Lionel Médini
Université Lyon 1, France

Copyright © 2022 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGWEB: ACM Special Interest Group on Hypertext, Hypermedia, and Web

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 25 April 2022

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed limited

Funding Sources

National Research Foundation of Korea (NRF)
Institute of Information & Communications Technology Planning & Evaluation (IITP)
Naver corporation

Conference

WWW '22

Sponsor:

SIGWEB

WWW '22: The ACM Web Conference 2022

April 25 - 29, 2022

Virtual Event, Lyon, France

Acceptance Rates

Overall Acceptance Rate 1,899 of 8,196 submissions, 23%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

5
Total Citations
View Citations
363
Total Downloads

Downloads (Last 12 months)87
Downloads (Last 6 weeks)10

Reflects downloads up to 27 Jul 2024

Other Metrics

View Author Metrics

Citations

Cited By

Leonhardt JMüller HRudra KKhosla MAnand AAnand A(2024)Efficient Neural Ranking Using Forward Indexes and Lightweight EncodersACM Transactions on Information Systems10.1145/363193942:5(1-34)Online publication date: 29-Apr-2024
https://dl.acm.org/doi/10.1145/3631939
Chang WJiang JZhang JAl-Darabsah MTeo CHsieh CYu HVishwanathan SAngélica LLattanzi SMuñoz Medina AAkoglu LGionis AVassilvitskii S(2024)PEFA: Parameter-Free Adapters for Large-scale Embedding-based Retrieval ModelsProceedings of the 17th ACM International Conference on Web Search and Data Mining10.1145/3616855.3635791(77-86)Online publication date: 4-Mar-2024
https://dl.acm.org/doi/10.1145/3616855.3635791
Pourbahman ZMomtazi SBagheri A(2023)Deep neural ranking model using distributed smoothingExpert Systems with Applications: An International Journal10.1016/j.eswa.2023.119913224:COnline publication date: 15-Aug-2023
https://dl.acm.org/doi/10.1016/j.eswa.2023.119913
Wu YLu BTian LLiang S(2022)Learning to Co-Embed Queries and DocumentsElectronics10.3390/electronics1122369411:22(3694)Online publication date: 11-Nov-2022
https://doi.org/10.3390/electronics11223694
Ma XGuo JZhang RFan YCheng XAl Hasan MXiong L(2022)Scattered or Connected? An Optimized Parameter-efficient Tuning Approach for Information RetrievalProceedings of the 31st ACM International Conference on Information & Knowledge Management10.1145/3511808.3557445(1471-1480)Online publication date: 17-Oct-2022
https://dl.acm.org/doi/10.1145/3511808.3557445

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

HTML Format

View this article in HTML Format.

Media

Figures

Other

Tables

View Table of Contents