QuAC

Question Answering in Context is a dataset for modeling, understanding, and participating in information seeking dialog. Data instances consist of an interactive dialog between two crowd workers: (1) a student who poses a sequence of freeform questions to learn as much as possible about a hidden Wikipedia text, and (2) a teacher who answers the questions by providing short excerpts (spans) from the text. QuAC introduces challenges not found in existing machine comprehension datasets: its questions are often more open-ended, unanswerable, or only meaningful within the dialog context.

---

QuAC paper

QuAC poster

---

QuAC is meant to be an academic resource and has significant limitations. Please read our detailed datasheet before considering it for any practical application.

---

Datasheet

No, QuAC shares many principles with SQuAD 2.0 such as span based evaluation and unanswerable questions (including website design principles! Big thanks for sharing the code!) but incorporates a new dialog component. We expect models can be easily evaluated on both resources and have tried to make our evaluation protocol as similar as possible to their own.

Download a copy of the dataset (distributed under the CC BY-SA 4.0 license):

• Training Set
• Val Set

To evaluate your models, we have also made available the evaluation script we will use for official evaluation, along with a sample prediction file that the script will take as input. To run the evaluation, use
python scorer.py --val_file <path_to_val> --model_output <path_to_predictions> --o eval.json; .

• Evaluation Script
• Sample Prediction File (on Dev)

Once you have a built a model that works to your expectations on the dev set, you submit it to get official scores on the dev and a hidden test set. To preserve the integrity of test results, we do not release the test set to the public. Instead, we require you to submit your model so that we can run it on the test set for you. The submission process is very similar to SQuaD 2.0 (Live!):

Submission Tutorial

All baseline models are available through AllenNLP. Specifically, model is here and the configuration is here. AllenNLP Model

First, download the duck The Duck Then, put this macro in your latex: \newcommand{\daffy}[0]{\includegraphics[width=.04\textwidth]{path_to_daffy/daffyhand.pdf}}
Finally, enjoy the command \daffy in your paper!

Ask us questions at our google group or at eunsol@cs.washington.edu hehe@stanford.edu
miyyer@cs.umass.edu marky@allenai.org

There can be only one duck.

Rank	Model	F1	HEQQ	HEQD
	Human Performance (Choi et al. EMNLP '18)	81.1	100	100
Aug 24, 2022	CKT-QA (ensemble) Ant Group, ZhiXiaoBao & Ada	76.3	73.6	17.9
2 Feb 2, 2022	CDQ-DeBERTa (single model) Anonymous	75.8	73.1	15.9
3 Nov 9, 2021	AERmodel (essemble) Protago Labs AI	75.2	72.5	16.5
4 Jan 27, 2021	RoR (single model) JD AI Research https://arxiv.org/abs/2109.04780	74.9	72.2	16.4
5 Sep 3, 2020	EL-QA (single model) JD AI Research	74.6	71.6	16.3
6 Jul 29, 2020	HistoryQA (single model) PAII Inc.	74.2	71.5	13.9
7 Dec 16, 2019	TR-MT (ensemble) WeChat AI	74.4	71.3	13.6
8 Nov 11, 2019	RoBERTa + DA (ensemble) Microsoft Dynamics 365 AI	74.0	70.7	13.1
9 Aug 9, 2022	GHR_ELECTRA (single model) SUDA NLP & A*STAR https://github.com/jaytsien/GHR	73.7	69.9	13.7
10 June 22, 2022	MarCQAp (single model) Technion - Israel Institute of Technology & Google Research https://arxiv.org/abs/2206.14796	74.0	70.7	12.5
11 Sep 15, 2019	History-Attentive-TransBERT (single model) Alibaba AI Labs	72.9	69.7	13.6
12 Nov 11, 2019	RoBERTa + DA (single model) Microsoft Dynamics 365 AI	73.5	69.8	12.1
13 Nov 25, 2019	BertMT (ensemble) WeChat AI	72.3	69.4	13.1
14 Nov 1, 2019	XLNet + Augmentation (single model) Xiaoming	71.2	67.5	11.8
15 Aug 31, 2019	TransBERT (single model) Anonymous	71.4	68.1	10.0
16 Nov 22, 2019	BertMT (single model) WeChat AI	69.4	66.0	9.8
17 June 13, 2019	Context-Aware-BERT (single model) Anonymous	69.6	65.7	8.1
18 Sep 9, 2019	BertInfoFlow (single model) PINGAN Omni-Sinitic	69.3	65.2	8.5
19 Mar 14, 2019	ConvBERT (single model) Joint Laboratory of HIT and iFLYTEK Research	68.0	63.5	9.1
20 Aug 22, 2019	zhiboBERT (single model) Anonymous	67.0	63.5	8.6
21 May 21, 2019	HAM (single model) UMass Amherst, Alibaba PAI, Rutgers University https://arxiv.org/abs/1908.09456	65.4	61.8	6.7
22 Jan 10, 2020	Bert-FlowDelta (single model) National Taiwan University, MiuLab https://arxiv.org/abs/1908.05117	65.5	61.0	6.9
23 Dec 4, 2019	BERT w/ 4-conversation history (single) Zhengzhou University	64.5	60.2	6.7
24 Mar 7, 2019	BERT w/ 2-context (single model) NTT Media Intelligence Labs	64.9	60.2	6.1
25 Nov 27, 2019	AHBert (single) ZZU	64.0	59.9	6.6
26 Feb 21, 2019	GraphFlow (single model) RPI-IBM https://arxiv.org/abs/1908.00059	64.9	60.3	5.1
27 Sep 26, 2018	FlowQA (single model) Allen Institute of AI https://arxiv.org/abs/1810.06683	64.1	59.6	5.8
28 Aug 20, 2018	BERT + History Answer Embedding (single model) UMass Amherst, Alibaba PAI, Rutgers University https://arxiv.org/abs/1905.05412	62.4	57.8	5.1
29 Aug 20, 2018	BiDAF++ w/ 2-Context (single model) baseline	60.1	54.8	4.0
30 Aug 20, 2018	BiDAF++ (single model) baseline	50.2	43.3	2.2