-
MS MARCO Web Search: a Large-scale Information-rich Web Dataset with Millions of Real Click Labels
Authors:
Qi Chen,
Xiubo Geng,
Corby Rosset,
Carolyn Buractaon,
Jingwen Lu,
Tao Shen,
Kun Zhou,
Chenyan Xiong,
Yeyun Gong,
Paul Bennett,
Nick Craswell,
Xing Xie,
Fan Yang,
Bryan Tower,
Nikhil Rao,
Anlei Dong,
Wenqi Jiang,
Zheng Liu,
Mingqin Li,
Chuanjie Liu,
Zengzhong Li,
Rangan Majumder,
Jennifer Neville,
Andy Oakley,
Knut Magne Risvik
, et al. (6 additional authors not shown)
Abstract:
Recent breakthroughs in large models have highlighted the critical significance of data scale, labels and modals. In this paper, we introduce MS MARCO Web Search, the first large-scale information-rich web dataset, featuring millions of real clicked query-document labels. This dataset closely mimics real-world web document and query distribution, provides rich information for various kinds of down…
▽ More
Recent breakthroughs in large models have highlighted the critical significance of data scale, labels and modals. In this paper, we introduce MS MARCO Web Search, the first large-scale information-rich web dataset, featuring millions of real clicked query-document labels. This dataset closely mimics real-world web document and query distribution, provides rich information for various kinds of downstream tasks and encourages research in various areas, such as generic end-to-end neural indexer models, generic embedding models, and next generation information access system with large language models. MS MARCO Web Search offers a retrieval benchmark with three web retrieval challenge tasks that demand innovations in both machine learning and information retrieval system research domains. As the first dataset that meets large, real and rich data requirements, MS MARCO Web Search paves the way for future advancements in AI and system research. MS MARCO Web Search dataset is available at: https://github.com/microsoft/MS-MARCO-Web-Search.
△ Less
Submitted 13 May, 2024;
originally announced May 2024.
-
Multilingual E5 Text Embeddings: A Technical Report
Authors:
Liang Wang,
Nan Yang,
Xiaolong Huang,
Linjun Yang,
Rangan Majumder,
Furu Wei
Abstract:
This technical report presents the training methodology and evaluation results of the open-source multilingual E5 text embedding models, released in mid-2023. Three embedding models of different sizes (small / base / large) are provided, offering a balance between the inference efficiency and embedding quality. The training procedure adheres to the English E5 model recipe, involving contrastive pr…
▽ More
This technical report presents the training methodology and evaluation results of the open-source multilingual E5 text embedding models, released in mid-2023. Three embedding models of different sizes (small / base / large) are provided, offering a balance between the inference efficiency and embedding quality. The training procedure adheres to the English E5 model recipe, involving contrastive pre-training on 1 billion multilingual text pairs, followed by fine-tuning on a combination of labeled datasets. Additionally, we introduce a new instruction-tuned embedding model, whose performance is on par with state-of-the-art, English-only models of similar sizes. Information regarding the model release can be found at https://github.com/microsoft/unilm/tree/master/e5 .
△ Less
Submitted 8 February, 2024;
originally announced February 2024.
-
Improving Text Embeddings with Large Language Models
Authors:
Liang Wang,
Nan Yang,
Xiaolong Huang,
Linjun Yang,
Rangan Majumder,
Furu Wei
Abstract:
In this paper, we introduce a novel and simple method for obtaining high-quality text embeddings using only synthetic data and less than 1k training steps. Unlike existing methods that often depend on multi-stage intermediate pre-training with billions of weakly-supervised text pairs, followed by fine-tuning with a few labeled datasets, our method does not require building complex training pipelin…
▽ More
In this paper, we introduce a novel and simple method for obtaining high-quality text embeddings using only synthetic data and less than 1k training steps. Unlike existing methods that often depend on multi-stage intermediate pre-training with billions of weakly-supervised text pairs, followed by fine-tuning with a few labeled datasets, our method does not require building complex training pipelines or relying on manually collected datasets that are often constrained by task diversity and language coverage. We leverage proprietary LLMs to generate diverse synthetic data for hundreds of thousands of text embedding tasks across 93 languages. We then fine-tune open-source decoder-only LLMs on the synthetic data using standard contrastive loss. Experiments demonstrate that our method achieves strong performance on highly competitive text embedding benchmarks without using any labeled data. Furthermore, when fine-tuned with a mixture of synthetic and labeled data, our model sets new state-of-the-art results on the BEIR and MTEB benchmarks.
△ Less
Submitted 31 May, 2024; v1 submitted 30 December, 2023;
originally announced January 2024.
-
Development and Evaluation of Ensemble Learning-based Environmental Methane Detection and Intensity Prediction Models
Authors:
Reek Majumder,
Jacquan Pollard,
M Sabbir Salek,
David Werth,
Gurcan Comert,
Adrian Gale,
Sakib Mahmud Khan,
Samuel Darko,
Mashrur Chowdhury
Abstract:
The environmental impacts of global warming driven by methane (CH4) emissions have catalyzed significant research initiatives in developing novel technologies that enable proactive and rapid detection of CH4. Several data-driven machine learning (ML) models were tested to determine how well they identified fugitive CH4 and its related intensity in the affected areas. Various meteorological charact…
▽ More
The environmental impacts of global warming driven by methane (CH4) emissions have catalyzed significant research initiatives in developing novel technologies that enable proactive and rapid detection of CH4. Several data-driven machine learning (ML) models were tested to determine how well they identified fugitive CH4 and its related intensity in the affected areas. Various meteorological characteristics, including wind speed, temperature, pressure, relative humidity, water vapor, and heat flux, were included in the simulation. We used the ensemble learning method to determine the best-performing weighted ensemble ML models built upon several weaker lower-layer ML models to (i) detect the presence of CH4 as a classification problem and (ii) predict the intensity of CH4 as a regression problem.
△ Less
Submitted 17 December, 2023;
originally announced December 2023.
-
Large Search Model: Redefining Search Stack in the Era of LLMs
Authors:
Liang Wang,
Nan Yang,
Xiaolong Huang,
Linjun Yang,
Rangan Majumder,
Furu Wei
Abstract:
Modern search engines are built on a stack of different components, including query understanding, retrieval, multi-stage ranking, and question answering, among others. These components are often optimized and deployed independently. In this paper, we introduce a novel conceptual framework called large search model, which redefines the conventional search stack by unifying search tasks with one la…
▽ More
Modern search engines are built on a stack of different components, including query understanding, retrieval, multi-stage ranking, and question answering, among others. These components are often optimized and deployed independently. In this paper, we introduce a novel conceptual framework called large search model, which redefines the conventional search stack by unifying search tasks with one large language model (LLM). All tasks are formulated as autoregressive text generation problems, allowing for the customization of tasks through the use of natural language prompts. This proposed framework capitalizes on the strong language understanding and reasoning capabilities of LLMs, offering the potential to enhance search result quality while simultaneously simplifying the existing cumbersome search stack. To substantiate the feasibility of this framework, we present a series of proof-of-concept experiments and discuss the potential challenges associated with implementing this approach within real-world search systems.
△ Less
Submitted 2 January, 2024; v1 submitted 23 October, 2023;
originally announced October 2023.
-
Exploring Diverse Coping Mechanisms in 2023: A Comprehensive Survey Across Backgrounds and Cultures
Authors:
Abhijit Paul,
Rony Majumder
Abstract:
This study presents a pioneering investigation into the wide array of coping mechanisms employed by individuals in the year 2023, with a focus on data collected through the popular social media platform TikTok. Coping mechanisms are essential strategies that people adopt to navigate the challenges and stressors of everyday life, yet little research has been conducted on their comprehensive compila…
▽ More
This study presents a pioneering investigation into the wide array of coping mechanisms employed by individuals in the year 2023, with a focus on data collected through the popular social media platform TikTok. Coping mechanisms are essential strategies that people adopt to navigate the challenges and stressors of everyday life, yet little research has been conducted on their comprehensive compilation across different backgrounds, countries, and experiences.
Using TikTok as a data collection tool allowed us to access a diverse and extensive pool of participants, representing various cultural, social, and demographic backgrounds. Our study collates coping mechanisms reported by users from different parts of the world, facilitating the identification of both universal and culture-specific strategies.
This research contributes to the existing literature by providing a holistic view of coping mechanisms without being limited to specific fields or populations. By analyzing the coping methods shared on TikTok, we reveal a comprehensive list of strategies employed by people from diverse walks of life. The findings of this study not only shed light on how individuals cope with challenges in the modern era but also offer insights into the evolving coping trends and the role of social media in disseminating coping strategies. Understanding these coping mechanisms can have implications for mental health professionals, practitioners, and policymakers seeking to provide support and resources to individuals facing different stressors and hardships.
△ Less
Submitted 4 September, 2023;
originally announced September 2023.
-
Inference with Reference: Lossless Acceleration of Large Language Models
Authors:
Nan Yang,
Tao Ge,
Liang Wang,
Binxing Jiao,
Daxin Jiang,
Linjun Yang,
Rangan Majumder,
Furu Wei
Abstract:
We propose LLMA, an LLM accelerator to losslessly speed up Large Language Model (LLM) inference with references. LLMA is motivated by the observation that there are abundant identical text spans between the decoding result by an LLM and the reference that is available in many real world scenarios (e.g., retrieved documents). LLMA first selects a text span from the reference and copies its tokens t…
▽ More
We propose LLMA, an LLM accelerator to losslessly speed up Large Language Model (LLM) inference with references. LLMA is motivated by the observation that there are abundant identical text spans between the decoding result by an LLM and the reference that is available in many real world scenarios (e.g., retrieved documents). LLMA first selects a text span from the reference and copies its tokens to the decoder and then efficiently checks the tokens' appropriateness as the decoding result in parallel within one decoding step. The improved computational parallelism allows LLMA to achieve over 2x speed-up for LLMs with identical generation results as greedy decoding in many practical generation scenarios where significant overlap between in-context reference and outputs exists (e.g., search engines and multi-turn conversations).
△ Less
Submitted 10 April, 2023;
originally announced April 2023.
-
LEAD: Liberal Feature-based Distillation for Dense Retrieval
Authors:
Hao Sun,
Xiao Liu,
Yeyun Gong,
Anlei Dong,
Jingwen Lu,
Yan Zhang,
Linjun Yang,
Rangan Majumder,
Nan Duan
Abstract:
Knowledge distillation is often used to transfer knowledge from a strong teacher model to a relatively weak student model. Traditional methods include response-based methods and feature-based methods. Response-based methods are widely used but suffer from lower upper limits of performance due to their ignorance of intermediate signals, while feature-based methods have constraints on vocabularies,…
▽ More
Knowledge distillation is often used to transfer knowledge from a strong teacher model to a relatively weak student model. Traditional methods include response-based methods and feature-based methods. Response-based methods are widely used but suffer from lower upper limits of performance due to their ignorance of intermediate signals, while feature-based methods have constraints on vocabularies, tokenizers and model architectures. In this paper, we propose a liberal feature-based distillation method (LEAD). LEAD aligns the distribution between the intermediate layers of teacher model and student model, which is effective, extendable, portable and has no requirements on vocabularies, tokenizers, or model architectures. Extensive experiments show the effectiveness of LEAD on widely-used benchmarks, including MS MARCO Passage Ranking, TREC 2019 DL Track, MS MARCO Document Ranking and TREC 2020 DL Track. Our code is available in https://github.com/microsoft/SimXNS/tree/main/LEAD.
△ Less
Submitted 11 December, 2023; v1 submitted 10 December, 2022;
originally announced December 2022.
-
Text Embeddings by Weakly-Supervised Contrastive Pre-training
Authors:
Liang Wang,
Nan Yang,
Xiaolong Huang,
Binxing Jiao,
Linjun Yang,
Daxin Jiang,
Rangan Majumder,
Furu Wei
Abstract:
This paper presents E5, a family of state-of-the-art text embeddings that transfer well to a wide range of tasks. The model is trained in a contrastive manner with weak supervision signals from our curated large-scale text pair dataset (called CCPairs). E5 can be readily used as a general-purpose embedding model for any tasks requiring a single-vector representation of texts such as retrieval, clu…
▽ More
This paper presents E5, a family of state-of-the-art text embeddings that transfer well to a wide range of tasks. The model is trained in a contrastive manner with weak supervision signals from our curated large-scale text pair dataset (called CCPairs). E5 can be readily used as a general-purpose embedding model for any tasks requiring a single-vector representation of texts such as retrieval, clustering, and classification, achieving strong performance in both zero-shot and fine-tuned settings. We conduct extensive evaluations on 56 datasets from the BEIR and MTEB benchmarks. For zero-shot settings, E5 is the first model that outperforms the strong BM25 baseline on the BEIR retrieval benchmark without using any labeled data. When fine-tuned, E5 obtains the best results on the MTEB benchmark, beating existing embedding models with 40x more parameters.
△ Less
Submitted 22 February, 2024; v1 submitted 7 December, 2022;
originally announced December 2022.
-
SimANS: Simple Ambiguous Negatives Sampling for Dense Text Retrieval
Authors:
Kun Zhou,
Yeyun Gong,
Xiao Liu,
Wayne Xin Zhao,
Yelong Shen,
Anlei Dong,
Jingwen Lu,
Rangan Majumder,
Ji-Rong Wen,
Nan Duan,
Weizhu Chen
Abstract:
Sampling proper negatives from a large document pool is vital to effectively train a dense retrieval model. However, existing negative sampling strategies suffer from the uninformative or false negative problem. In this work, we empirically show that according to the measured relevance scores, the negatives ranked around the positives are generally more informative and less likely to be false nega…
▽ More
Sampling proper negatives from a large document pool is vital to effectively train a dense retrieval model. However, existing negative sampling strategies suffer from the uninformative or false negative problem. In this work, we empirically show that according to the measured relevance scores, the negatives ranked around the positives are generally more informative and less likely to be false negatives. Intuitively, these negatives are not too hard (\emph{may be false negatives}) or too easy (\emph{uninformative}). They are the ambiguous negatives and need more attention during training. Thus, we propose a simple ambiguous negatives sampling method, SimANS, which incorporates a new sampling probability distribution to sample more ambiguous negatives. Extensive experiments on four public and one industry datasets show the effectiveness of our approach. We made the code and models publicly available in \url{https://github.com/microsoft/SimXNS}.
△ Less
Submitted 24 October, 2022; v1 submitted 21 October, 2022;
originally announced October 2022.
-
PROD: Progressive Distillation for Dense Retrieval
Authors:
Zhenghao Lin,
Yeyun Gong,
Xiao Liu,
Hang Zhang,
Chen Lin,
Anlei Dong,
Jian Jiao,
Jingwen Lu,
Daxin Jiang,
Rangan Majumder,
Nan Duan
Abstract:
Knowledge distillation is an effective way to transfer knowledge from a strong teacher to an efficient student model. Ideally, we expect the better the teacher is, the better the student. However, this expectation does not always come true. It is common that a better teacher model results in a bad student via distillation due to the nonnegligible gap between teacher and student. To bridge the gap,…
▽ More
Knowledge distillation is an effective way to transfer knowledge from a strong teacher to an efficient student model. Ideally, we expect the better the teacher is, the better the student. However, this expectation does not always come true. It is common that a better teacher model results in a bad student via distillation due to the nonnegligible gap between teacher and student. To bridge the gap, we propose PROD, a PROgressive Distillation method, for dense retrieval. PROD consists of a teacher progressive distillation and a data progressive distillation to gradually improve the student. We conduct extensive experiments on five widely-used benchmarks, MS MARCO Passage, TREC Passage 19, TREC Document 19, MS MARCO Document and Natural Questions, where PROD achieves the state-of-the-art within the distillation methods for dense retrieval. The code and models will be released.
△ Less
Submitted 24 June, 2023; v1 submitted 27 September, 2022;
originally announced September 2022.
-
SimLM: Pre-training with Representation Bottleneck for Dense Passage Retrieval
Authors:
Liang Wang,
Nan Yang,
Xiaolong Huang,
Binxing Jiao,
Linjun Yang,
Daxin Jiang,
Rangan Majumder,
Furu Wei
Abstract:
In this paper, we propose SimLM (Similarity matching with Language Model pre-training), a simple yet effective pre-training method for dense passage retrieval. It employs a simple bottleneck architecture that learns to compress the passage information into a dense vector through self-supervised pre-training. We use a replaced language modeling objective, which is inspired by ELECTRA, to improve th…
▽ More
In this paper, we propose SimLM (Similarity matching with Language Model pre-training), a simple yet effective pre-training method for dense passage retrieval. It employs a simple bottleneck architecture that learns to compress the passage information into a dense vector through self-supervised pre-training. We use a replaced language modeling objective, which is inspired by ELECTRA, to improve the sample efficiency and reduce the mismatch of the input distribution between pre-training and fine-tuning. SimLM only requires access to unlabeled corpus, and is more broadly applicable when there are no labeled data or queries. We conduct experiments on several large-scale passage retrieval datasets, and show substantial improvements over strong baselines under various settings. Remarkably, SimLM even outperforms multi-vector approaches such as ColBERTv2 which incurs significantly more storage cost. Our code and model check points are available at https://github.com/microsoft/unilm/tree/master/simlm .
△ Less
Submitted 12 May, 2023; v1 submitted 6 July, 2022;
originally announced July 2022.
-
Development of Decision Support System for Effective COVID-19 Management
Authors:
shuvrangshu Jana,
Rudrashis Majumder,
Aashay Bhise,
Nobin Paul,
Stuti Garg,
Debasish Ghose
Abstract:
This paper discusses a Decision Support System (DSS) for cases prediction, allocation of resources, and lockdown management for managing COVID-19 at different levels of a government authority. Algorithms incorporated in the DSS are based on a data-driven modeling approach and independent of physical parameters of the region, and hence the proposed DSS is applicable to any area. Based on predicted…
▽ More
This paper discusses a Decision Support System (DSS) for cases prediction, allocation of resources, and lockdown management for managing COVID-19 at different levels of a government authority. Algorithms incorporated in the DSS are based on a data-driven modeling approach and independent of physical parameters of the region, and hence the proposed DSS is applicable to any area. Based on predicted active cases, the demand of lower-level units and total availability, allocation, and lockdown decision is made. A MATLAB-based GUI is developed based on the proposed DSS and could be implemented by the local authority.
△ Less
Submitted 12 March, 2022;
originally announced March 2022.
-
In situ process quality monitoring and defect detection for direct metal laser melting
Authors:
Sarah Felix,
Saikat Ray Majumder,
H. Kirk Mathews,
Michael Lexa,
Gabriel Lipsa,
Xiaohu Ping,
Subhrajit Roychowdhury,
Thomas Spears
Abstract:
Quality control and quality assurance are challenges in Direct Metal Laser Melting (DMLM). Intermittent machine diagnostics and downstream part inspections catch problems after undue cost has been incurred processing defective parts. In this paper we demonstrate two methodologies for in-process fault detection and part quality prediction that can be readily deployed on existing commercial DMLM sys…
▽ More
Quality control and quality assurance are challenges in Direct Metal Laser Melting (DMLM). Intermittent machine diagnostics and downstream part inspections catch problems after undue cost has been incurred processing defective parts. In this paper we demonstrate two methodologies for in-process fault detection and part quality prediction that can be readily deployed on existing commercial DMLM systems with minimal hardware modification. Novel features were derived from the time series of common photodiode sensors along with standard machine control signals. A Bayesian approach attributes measurements to one of multiple process states and a least squares regression model predicts severity of certain material defects.
△ Less
Submitted 3 December, 2021;
originally announced December 2021.
-
Game-Theoretic Model Based Resource Allocation During Floods
Authors:
Rudrashis Majumder,
Rakesh R Warier,
Debasish Ghose
Abstract:
For multiple emergencies caused by natural disasters, it is crucial to allocate resources equitably to each emergency location, especially when the availability of resources is limited in quantity. This paper has developed a multi-event crisis management system using a non-cooperative, complete information, strategic form game model. In the proposed system, each emergency event is assumed to occur…
▽ More
For multiple emergencies caused by natural disasters, it is crucial to allocate resources equitably to each emergency location, especially when the availability of resources is limited in quantity. This paper has developed a multi-event crisis management system using a non-cooperative, complete information, strategic form game model. In the proposed system, each emergency event is assumed to occur in different locations simultaneously. These locations are represented as the players in the game, competing with the other players for an optimal allocation of scarce resources available at different resource stations. The players incur a non-monetary cost for obtaining resource units. The objective of the proposed game is to derive optimal strategies for an effective and fair allocation of resources to the respective players.
△ Less
Submitted 2 December, 2021;
originally announced December 2021.
-
Hybrid Classical-Quantum Deep Learning Models for Autonomous Vehicle Traffic Image Classification Under Adversarial Attack
Authors:
Reek Majumder,
Sakib Mahmud Khan,
Fahim Ahmed,
Zadid Khan,
Frank Ngeni,
Gurcan Comert,
Judith Mwakalonge,
Dimitra Michalaka,
Mashrur Chowdhury
Abstract:
Image classification must work for autonomous vehicles (AV) operating on public roads, and actions performed based on image misclassification can have serious consequences. Traffic sign images can be misclassified by an adversarial attack on machine learning models used by AVs for traffic sign recognition. To make classification models resilient against adversarial attacks, we used a hybrid deep-l…
▽ More
Image classification must work for autonomous vehicles (AV) operating on public roads, and actions performed based on image misclassification can have serious consequences. Traffic sign images can be misclassified by an adversarial attack on machine learning models used by AVs for traffic sign recognition. To make classification models resilient against adversarial attacks, we used a hybrid deep-learning model with both the quantum and classical layers. Our goal is to study the hybrid deep-learning architecture for classical-quantum transfer learning models to support the current era of intermediate-scale quantum technology. We have evaluated the impacts of various white box adversarial attacks on these hybrid models. The classical part of hybrid models includes a convolution network from the pre-trained Resnet18 model, which extracts informative features from a high dimensional LISA traffic sign image dataset. The output from the classical processor is processed further through the quantum layer, which is composed of various quantum gates and provides support to various quantum mechanical features like entanglement and superposition. We have tested multiple combinations of quantum circuits to provide better classification accuracy with decreasing training data and found better resiliency for our hybrid classical-quantum deep learning model during attacks compared to the classical-only machine learning models.
△ Less
Submitted 2 August, 2021;
originally announced August 2021.
-
XGLUE: A New Benchmark Dataset for Cross-lingual Pre-training, Understanding and Generation
Authors:
Yaobo Liang,
Nan Duan,
Yeyun Gong,
Ning Wu,
Fenfei Guo,
Weizhen Qi,
Ming Gong,
Linjun Shou,
Daxin Jiang,
Guihong Cao,
Xiaodong Fan,
Ruofei Zhang,
Rahul Agrawal,
Edward Cui,
Sining Wei,
Taroon Bharti,
Ying Qiao,
Jiun-Hung Chen,
Winnie Wu,
Shuguang Liu,
Fan Yang,
Daniel Campos,
Rangan Majumder,
Ming Zhou
Abstract:
In this paper, we introduce XGLUE, a new benchmark dataset that can be used to train large-scale cross-lingual pre-trained models using multilingual and bilingual corpora and evaluate their performance across a diverse set of cross-lingual tasks. Comparing to GLUE(Wang et al., 2019), which is labeled in English for natural language understanding tasks only, XGLUE has two main advantages: (1) it pr…
▽ More
In this paper, we introduce XGLUE, a new benchmark dataset that can be used to train large-scale cross-lingual pre-trained models using multilingual and bilingual corpora and evaluate their performance across a diverse set of cross-lingual tasks. Comparing to GLUE(Wang et al., 2019), which is labeled in English for natural language understanding tasks only, XGLUE has two main advantages: (1) it provides 11 diversified tasks that cover both natural language understanding and generation scenarios; (2) for each task, it provides labeled data in multiple languages. We extend a recent cross-lingual pre-trained model Unicoder(Huang et al., 2019) to cover both understanding and generation tasks, which is evaluated on XGLUE as a strong baseline. We also evaluate the base versions (12-layer) of Multilingual BERT, XLM and XLM-R for comparison.
△ Less
Submitted 22 May, 2020; v1 submitted 3 April, 2020;
originally announced April 2020.
-
MS MARCO: A Human Generated MAchine Reading COmprehension Dataset
Authors:
Payal Bajaj,
Daniel Campos,
Nick Craswell,
Li Deng,
Jianfeng Gao,
Xiaodong Liu,
Rangan Majumder,
Andrew McNamara,
Bhaskar Mitra,
Tri Nguyen,
Mir Rosenberg,
Xia Song,
Alina Stoica,
Saurabh Tiwary,
Tong Wang
Abstract:
We introduce a large scale MAchine Reading COmprehension dataset, which we name MS MARCO. The dataset comprises of 1,010,916 anonymized questions---sampled from Bing's search query logs---each with a human generated answer and 182,669 completely human rewritten generated answers. In addition, the dataset contains 8,841,823 passages---extracted from 3,563,535 web documents retrieved by Bing---that…
▽ More
We introduce a large scale MAchine Reading COmprehension dataset, which we name MS MARCO. The dataset comprises of 1,010,916 anonymized questions---sampled from Bing's search query logs---each with a human generated answer and 182,669 completely human rewritten generated answers. In addition, the dataset contains 8,841,823 passages---extracted from 3,563,535 web documents retrieved by Bing---that provide the information necessary for curating the natural language answers. A question in the MS MARCO dataset may have multiple answers or no answers at all. Using this dataset, we propose three different tasks with varying levels of difficulty: (i) predict if a question is answerable given a set of context passages, and extract and synthesize the answer as a human would (ii) generate a well-formed answer (if possible) based on the context passages that can be understood with the question and passage context, and finally (iii) rank a set of retrieved passages given a question. The size of the dataset and the fact that the questions are derived from real user search queries distinguishes MS MARCO from other well-known publicly available datasets for machine reading comprehension and question-answering. We believe that the scale and the real-world nature of this dataset makes it attractive for benchmarking machine reading comprehension and question-answering models.
△ Less
Submitted 31 October, 2018; v1 submitted 28 November, 2016;
originally announced November 2016.
