short-paper

Joint Learning of Deep Retrieval Model and Product Quantization based Embedding Index

Authors:

Yunjiang Jiang,

Wen-Yun YangAuthors Info & Claims

SIGIR '21: Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval

Pages 1718 - 1722

https://doi.org/10.1145/3404835.3462988

Published: 11 July 2021 Publication History

Abstract

Embedding index that enables fast approximate nearest neighbor(ANN) search, serves as an indispensable component for state-of-the-art deep retrieval systems. Traditional approaches, often separating the two steps of embedding learning and index building, incur additional indexing time and decayed retrieval accuracy. In this paper, we propose a novel method called Poeem, which stands for product quantization based embedding index jointly trained with deep retrieval model, to unify the two separate steps within an end-to-end training, by utilizing a few techniques including the gradient straight-through estimator, warm start strategy, optimal space decomposition and Givens rotation. Extensive experimental results show that the proposed method not only improves retrieval accuracy significantly but also reduces the indexing time to almost none. We have open sourced our approach for the sake of comparison and reproducibility.

Supplementary Material

MP4 File (SIGIR21-sp1192.mp4)

The video is about the work "Joint Learning of Deep Retrieval Model and Product Quantization based Embedding Index" from JD.com. This work proposed a novel method, called Poeem, to learn embedding indexes jointly with deep retrieval models which can avoid precision decay incurred by quantization distortion and reduce embedding indexing time. The standalone embedding indexing layer can be easily plugged into any retrieval models. In the video, we introduce our model architecture in detail, show the results about experiments, and present the model performance in a very explicit and intuitive way of embedding visualization. The open source of the model is offered in the end of the video.

Download
72.23 MB

References

[1]

Amir Beck and Luba Tetruashvili. 2013. On the convergence of block coordinate descent type methods. SIAM journal on Optimization 23, 4 (2013), 2037--2060.

Digital Library

[2]

Yoshua Bengio, Nicholas Léonard, and Aaron Courville. 2013. Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation. In arXiv. 1308.3432.

[3]

Erik Bernhardsson. 2018. Annoy: Approximate Nearest Neighbors in C++/Python. https://pypi.org/project/annoy/ Python package version 1.13.0.

[4]

Yue Cao, Mingsheng Long, Jianmin Wang, Han Zhu, and Qingfu Wen. 2016. Deep quantization network for efficient image retrieval. In AAAI, Vol. 30.

[5]

Ting Chen, Lala Li, and Yizhou Sun. 2020. Differentiable product quantization for end-to-end embedding compression. In ICML. 1617--1626.

[6]

Paul Covington, Jay Adams, and Emre Sargin. 2016. Deep Neural Networks for YouTube Recommendations. In RecSys. 191--198.

[7]

Mayur Datar, Nicole Immorlica, Piotr Indyk, and Vahab S Mirrokni. 2004. Locality-sensitive hashing scheme based on p-stable distributions. In Proceedings of the twentieth annual symposium on Computational geometry. 253--262.

Digital Library

[8]

Jeffrey Dean. 2009. Challenges in building large-scale information retrieval systems. In WSDM, Vol. 10.

Digital Library

[9]

Tiezheng Ge, Kaiming He, Qifa Ke, and Jian Sun. 2013. Optimized product quantization for approximate nearest neighbor search. InCVPR. 2946--2953.

[10]

Gene H. Golub and Charles F. Van Loan. 1996.Matrix Computations (3rd Ed.).Johns Hopkins University Press, USA.

Digital Library

[11]

Ruiqi Guo, Philip Sun, Erik Lindgren, Quan Geng, David Simcha, Felix Chern, and Sanjiv Kumar. 2020. Accelerating large-scale inference with anisotropic vector quantization. In ICML. 3887--3896.

[12]

F Maxwell Harper and Joseph A Konstan. 2015. The movie lens datasets: History and context. Acm transactions on interactive intelligent systems (tiis)5, 4 (2015), 1--19.

[13]

Ruining He and Julian McAuley. 2016. Ups and downs: Modeling the visual evolution of fashion trends with one-class collaborative filtering. In WWW. 507--517.

[14]

Jui-Ting Huang, Ashish Sharma, Shuying Sun, Li Xia, David Zhang, Philip Pronin,Janani Padmanabhan, Giuseppe Ottaviano, and Linjun Yang. 2020. Embedding-based retrieval in facebook search. In SIGKDD. 2553--2561.

[15]

Po-Sen Huang, Xiaodong He, Jianfeng Gao, Li Deng, Alex Acero, and LarryHeck. 2013. Learning deep structured semantic models for web search using clickthrough data. In CIKM. 2333--2338.

[16]

Adolf Hurwitz. 1963. Ueber die erzeugung der invarianten durch integration. In Mathematische Werke. Springer, 546--564.

[17]

Herve Jegou, Matthijs Douze, and Cordelia Schmid. 2010. Product quantization for nearest neighbor search. IEEE transactions on pattern analysis and machine intelligence 33, 1 (2010), 117--128.

[18]

Jeff Johnson, Matthijs Douze, and Hervé Jégou. 2019. Billion-scale similarity search with GPUs. IEEE Transactions on Big Data(2019).

[19]

Benjamin Klein and Lior Wolf. 2017. In defense of product quantization. arXiv preprint arXiv:1711.085892, 3 (2017), 4.

[20]

Chao Li, Zhiyuan Liu, Mengmeng Wu, Yuchi Xu, Huan Zhao, Pipei Huang, Guoliang Kang, Qiwei Chen, Wei Li, and Dik Lun Lee. 2019. Multi-Interest Network with Dynamic Routing for Recommendation at Tmall. In CIKM. 2615--2623.

[21]

Peter Schönemann. 1966. A generalized solution of the orthogonal procrustes problem. Psychometrika 31, 1 (1966), 1--10.

[22]

Aaron Van Den Oord, Oriol Vinyals, et al. 2017. Neural discrete representation learning. In NIPS. 6306--6315.

[23]

Laurens van der Maaten and Geoffrey Hinton. 2008. Visualizing Data using t-SNE. Journal of Machine Learning Research9 (2008), 2579--2605.

[24]

Stephen J Wright. 2015. Coordinate descent algorithms. Mathematical Programming 151, 1 (2015), 3--34.

Digital Library

[25]

Tan Yu, Junsong Yuan, Chen Fang, and Hailin Jin. 2018. Product quantization network for fast image retrieval. In ECCV. 186--201.

[26]

Han Zhang, Songlin Wang, Kang Zhang, Zhiling Tang, Yunjiang Jiang, Yun Xiao, Weipeng Yan, and Wen-Yun Yang. 2020. Towards Personalized and Semantic Retrieval: An End-to-End Solution for E-commerce Search via Embedding Learning. arXiv preprint arXiv:2006.02282(2020).

[27]

Han Zhu, Xiang Li, Pengye Zhang, Guozheng Li, Jie He, Han Li, and Kun Gai.2018. Learning tree-based deep model for recommender systems. In SIGKDD. 1079--1088.

[28]

Jingwei Zhuo, Ziru Xu, Wei Dai, Han Zhu, Han Li, Jian Xu, and Kun Gai. 2020. Learning Optimal Tree Models under Beam Search. In ICML, Vol. 119. 11650--11659.

Cited By

Zhao WLiu JRen RWen J(2024)Dense Text Retrieval Based on Pretrained Language Models: A SurveyACM Transactions on Information Systems10.1145/363787042:4(1-60)Online publication date: 9-Feb-2024
https://dl.acm.org/doi/10.1145/3637870
Sun ZFeng KYang JQu XFang HOng YLiu WHui Yang GWang HHan SHauff CZuccon GZhang Y(2024)Adaptive In-Context Learning with Large Language Models for Bundle GenerationProceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval10.1145/3626772.3657808(966-976)Online publication date: 10-Jul-2024
https://dl.acm.org/doi/10.1145/3626772.3657808
Zhang FChen CHua XLuo X(2024)FATE: Learning Effective Binary Descriptors With Group FairnessIEEE Transactions on Image Processing10.1109/TIP.2024.340613433(3648-3661)Online publication date: 2024
https://doi.org/10.1109/TIP.2024.3406134
Show More Cited By

Index Terms

Joint Learning of Deep Retrieval Model and Product Quantization based Embedding Index
1. Information systems
  1. Information retrieval
    1. Retrieval models and ranking
      1. Novelty in information retrieval

Recommendations

Quantization Index Modulation Methods for Digital Watermarking and Information Embedding of Multimedia
Special issue on multimedia signal processing

Copyright notification and enforcement, authentication, covert communication, and hybrid transmission applications such as digital audio broadcasting are examples of emerging multimedia applications for digital watermarking and information embedding ...
Fast Forward Index Methods for Pseudo-Relevance Feedback Retrieval

The inverted index is the dominant indexing method in information retrieval systems. It enables fast return of the list of all documents containing a given query term. However, for retrieval schemes involving query expansion, as in pseudo-relevance ...
Vector Quantization Based Index Cube Model for Image Retrieval
PSIVT '10: Proceedings of the 2010 Fourth Pacific-Rim Symposium on Image and Video Technology

We propose a Vector quantization (VQ) based index cube model for content based image retrieval. VQ captures the pixel intensity and the spatial information of the image blocks. An indexing and retrieval algorithm is implemented and different similarity ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

SIGIR '21: Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval

July 2021

2998 pages

ISBN:9781450380379

DOI:10.1145/3404835

General Chairs:
Fernando Diaz
(Google)
,
Chirag Shah
University of Washington
,
Torsten Suel
New York University
,
Program Chairs:
Pablo Castells
Universidad Autónoma de Madrid, Amazon
,
Rosie Jones
Spotify
,
Tetsuya Sakai
Waseda University

Copyright © 2021 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGIR: ACM Special Interest Group on Information Retrieval

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 11 July 2021

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Short-paper

Conference

SIGIR '21

Sponsor:

SIGIR

SIGIR '21: The 44th International ACM SIGIR Conference on Research and Development in Information Retrieval

July 11 - 15, 2021

Virtual Event, Canada

Acceptance Rates

Overall Acceptance Rate 792 of 3,983 submissions, 20%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

19
Total Citations
View Citations
354
Total Downloads

Downloads (Last 12 months)55
Downloads (Last 6 weeks)3

Reflects downloads up to 17 Oct 2024

Other Metrics

View Author Metrics

Citations

Cited By

Zhao WLiu JRen RWen J(2024)Dense Text Retrieval Based on Pretrained Language Models: A SurveyACM Transactions on Information Systems10.1145/363787042:4(1-60)Online publication date: 9-Feb-2024
https://dl.acm.org/doi/10.1145/3637870
Sun ZFeng KYang JQu XFang HOng YLiu WHui Yang GWang HHan SHauff CZuccon GZhang Y(2024)Adaptive In-Context Learning with Large Language Models for Bundle GenerationProceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval10.1145/3626772.3657808(966-976)Online publication date: 10-Jul-2024
https://dl.acm.org/doi/10.1145/3626772.3657808
Zhang FChen CHua XLuo X(2024)FATE: Learning Effective Binary Descriptors With Group FairnessIEEE Transactions on Image Processing10.1109/TIP.2024.340613433(3648-3661)Online publication date: 2024
https://doi.org/10.1109/TIP.2024.3406134
Zhao CJiang YQiu YZhang HYang WFrommholz IHopfgartner FLee MOakes MLalmas MZhang MSantos R(2023)Differentiable Retrieval Augmentation via Generative Language Modeling for E-commerce Query Intent ClassificationProceedings of the 32nd ACM International Conference on Information and Knowledge Management10.1145/3583780.3615210(4445-4449)Online publication date: 21-Oct-2023
https://dl.acm.org/doi/10.1145/3583780.3615210
Huang RZhang DLu WLi HWang MShi DFan JCheng ZGu SYin DSingh ASun YAkoglu LGunopulos DYan XKumar ROzcan FYe J(2023)Learning Discrete Document Representations in Web SearchProceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining10.1145/3580305.3599854(4185-4194)Online publication date: 6-Aug-2023
https://dl.acm.org/doi/10.1145/3580305.3599854
Aoyama KAmagata DFujita SHara TChen HDuh WHuang HKato MMothe JPoblete B(2023)Simpler is Much Faster: Fair and Independent Inner Product SearchProceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval10.1145/3539618.3592061(2379-2383)Online publication date: 19-Jul-2023
https://dl.acm.org/doi/10.1145/3539618.3592061
Zhou ZDing NFan XShang YQiu YZhuo JGe ZWang SLiu LXu SZhang HChen HDuh WHuang HKato MMothe JPoblete B(2023)Semantic-enhanced Modality-asymmetric Retrieval for Online E-commerce SearchProceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval10.1145/3539618.3591863(3405-3409)Online publication date: 19-Jul-2023
https://dl.acm.org/doi/10.1145/3539618.3591863
Li MYuan CWang BZhuo JWang SLiu LXu SChen HDuh WHuang HKato MMothe JPoblete B(2023)Learning Query-aware Embedding Index for Improving E-commerce Dense RetrievalProceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval10.1145/3539618.3591834(3265-3269)Online publication date: 19-Jul-2023
https://dl.acm.org/doi/10.1145/3539618.3591834
Kulkarni HMacAvaney SGoharian NFrieder OChen HDuh WHuang HKato MMothe JPoblete B(2023)Lexically-Accelerated Dense RetrievalProceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval10.1145/3539618.3591715(152-162)Online publication date: 19-Jul-2023
https://dl.acm.org/doi/10.1145/3539618.3591715
Su LYan FZhu JXiao XDuan HZhao ZDong ZTang RChen HDuh WHuang HKato MMothe JPoblete B(2023)Beyond Two-Tower Matching: Learning Sparse Retrievable Cross-Interactions for RecommendationProceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval10.1145/3539618.3591643(548-557)Online publication date: 19-Jul-2023
https://dl.acm.org/doi/10.1145/3539618.3591643
Show More Cited By

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents