Retrospective Reader for Machine Reading Comprehension

Zhuosheng Zhang1,2,3 , Junjie Yang2,3,4 , Hai Zhao1,2,3,*

1
Department of Computer Science and Engineering, Shanghai Jiao Tong University
2
Key Laboratory of Shanghai Education Commission for Intelligent Interaction
and Cognitive Engineering, Shanghai Jiao Tong University, Shanghai, China
3
MoE Key Lab of Artificial Intelligence, AI Institute, Shanghai Jiao Tong University, Shanghai, China
4
SJTU-ParisTech Elite Institute of Technology, Shanghai Jiao Tong University, Shanghai, China
zhangzs@sjtu.edu.cn, jj-yang@sjtu.edu.cn, zhaohai@cs.sjtu.edu.cn
arXiv:2001.09694v4 [cs.CL] 11 Dec 2020

Abstract Passage:
Computational complexity theory is a branch of the the-
Machine reading comprehension (MRC) is an AI challenge ory of computation in theoretical computer science that
that requires machines to determine the correct answers to
questions based on a given passage. MRC systems must
focuses on classifying computational problems according
not only answer questions when necessary but also tact- to their inherent difficulty, and relating those classes to
fully abstain from answering when no answer is available each other. A computational problem is understood to be
according to the given passage. When unanswerable ques- a task that is in principle amenable to being solved by a
tions are involved in the MRC task, an essential verifica- computer, which is equivalent to stating that the problem
tion module called verifier is especially required in addition may be solved by mechanical application of mathemati-
to the encoder, though the latest practice on MRC modeling cal steps, such as an algorithm.
still mostly benefits from adopting well pre-trained language Question:
models as the encoder block by only focusing on the “read- What cannot be solved by mechanical application of
ing”. This paper devotes itself to exploring better verifier de-
mathematical steps?
sign for the MRC task with unanswerable questions. Inspired
by how humans solve reading comprehension questions, we Gold Answer: hno answeri
proposed a retrospective reader (Retro-Reader) that integrates Plausible answer: algorithm
two stages of reading and verification strategies: 1) sketchy
reading that briefly investigates the overall interactions of Table 1: An unanswerable MRC example.
passage and question, and yields an initial judgment; 2) inten-
sive reading that verifies the answer and gives the final pre-
diction. The proposed reader is evaluated on two benchmark
MRC challenge datasets SQuAD2.0 and NewsQA, achieving tems (Zhang et al. 2018; Choi et al. 2018; Reddy, Chen, and
new state-of-the-art results. Significance tests show that our Manning 2019; Zhang, Huang, and Zhao 2018; Xu, Zhao,
model is significantly better than strong baselines. and Zhang 2021; Zhu et al. 2018). The early MRC systems
(Kadlec et al. 2016; Chen, Bolton, and Manning 2016; Dhin-
1 Introduction gra et al. 2017; Wang et al. 2017; Seo et al. 2016) were de-
Be certain of what you know and be aware what you signed on a latent hypothesis that all questions can be an-
don’t. That is wisdom. swered according to the given passage (Figure 1-[a]), which
is not always true for real-world cases. The recent progress
Confucius (551 BC - 479 BC) on the MRC task has required that the model must be capa-
Machine reading comprehension (MRC) aims to teach ble of distinguishing those unanswerable questions to avoid
machines to answer questions after comprehending given giving plausible answers (Rajpurkar, Jia, and Liang 2018).
passages (Hermann et al. 2015; Joshi et al. 2017; Rajpurkar, MRC task with unanswerable questions may be usually de-
Jia, and Liang 2018), which is a fundamental and long- composed into two subtasks: 1) answerability verification
standing goal of natural language understanding (NLU) and 2) reading comprehension. To determine unanswerable
(Zhang, Zhao, and Wang 2020). It has significant applica- questions requires a deep understanding of the given text
tion scenarios such as question answering and dialog sys- and requires more robust MRC models, making MRC much
closer to real-world applications. Table 1 shows an unan-
* Corresponding author. This paper was partially supported
swerable example from SQuAD2.0 MRC task (Rajpurkar,
by National Key Research and Development Program of China Jia, and Liang 2018).
(No. 2017YFB0304100), Key Projects of National Natural Science
Foundation of China (U1836222 and 61733011), Huawei-SJTU
So far, a common reading system (reader) which solves
long term AI project, Cutting-edge Machine reading comprehen- MRC problem generally consists of two modules or building
sion and language model. steps as shown in Figure 1-[a]: 1) building a robust language
Copyright © 2021, Association for the Advancement of Artificial model (LM) as encoder; 2) designing ingenious mechanisms
Intelligence (www.aaai.org). All rights reserved. as decoder according to MRC task characteristics.
 Part I: Model Desgins Part II: Retrospective Reader Architecture
Input Sketchy Reading Module Decoder
Encoder Decoder
(External Front Verification, E-FV) (Rear Verification)
[a] Encoder+Decoder
Encoding Interaction
Encoder t1
Decoder
Verifier

scoreext
[b] (Encoder+E-FV)-Decoder

Decoder
t2
Encoder
Verifier
[c] Encoder-(Decoder+I-FV) t3

Answer Prediction
Sketchy Decoder
Intensive Reading Module
(Internal Front Vefification, I-FV)
Intensive

[d] Sketchy and Internsive Reading t4 Encoding Interaction

scorehas
Sketchy Reading

E-FV
t5
Decoder
(R-V)
Encoder
...

scorenull
I-FV Intensive Reading
tn
[e] (Encoder+FV)+FV-(Decoder+RV)

Figure 1: Reader overview. For the left part, models [a-c] summarize the instances in previous work, and model [d] is ours,
with the implemented version [e]. In the names of models [a-e], “(·)” represents a module, “+” means the parallel module and
“-” is the pipeline. The right part is the detailed architecture of our proposed Retro-Reader.

Pre-trained language models (PrLMs) such as BERT (De- a pipeline or concatenation way (Figure 1-[b-c]), which is
vlin et al. 2019) and XLNet (Yang et al. 2019) have achieved shown suboptimal for installing a verifier.
success on various natural language processing tasks, which As a natural practice of how humans solve complex read-
broadly play the role of a powerful encoder (Zhang et al. ing comprehension (Zheng et al. 2019; Guthrie and Mosen-
2019; Li et al. 2020; Zhou, Zhang, and Zhao 2019). How- thal 1987), the first step is to read through the full passage
ever, it is quite time-consuming and resource-demanding to along with the question and grasp the general idea; then,
impart massive amounts of general knowledge from external people re-read the full text and verify the answer if not so
corpora into a deep language model via pre-training. sure. Inspired by such a reading and comprehension pattern,
Recently, most MRC readers keep the primary focus on we proposed a retrospective reader (Retro-Reader, Figure 1-
the encoder side, i.e., the deep PrLMs (Devlin et al. 2019; [d]) that integrates two stages of reading and verification
Yang et al. 2019; Lan et al. 2020), as readers may simply strategies: 1) sketchy reading that briefly touches the rela-
and straightforwardly benefit from a strong enough encoder. tionship of passage and question, and yields an initial judg-
Meanwhile, little attention is paid to the decoder side1 of ment; 2) intensive reading that verifies the answer and gives
MRC models (Hu et al. 2019; Back et al. 2020; Reddy et al. the final prediction. Our major contributions are three folds:2
2020), though it has been shown that better decoder or bet- 1. We propose a new retrospective reader design which is
ter manner of using encoder still has a significant impact on capable of effectively performing answer verification in-
MRC performance, no matter how strong the encoder it is stead of simply stacking verifier in existing readers.
(Zhang et al. 2020a; Liu et al. 2021; Li et al. 2019, 2018;
Zhu, Zhao, and Li 2020). 2. Experiments show that our reader can yield substan-
For the concerned MRC challenge with unanswerable tial improvements over strong baselines and achieve new
questions, a reader has to handle two aspects carefully: state-of-the-art results on benchmark MRC tasks.
1) give the accurate answers for answerable questions; 3. For the first time, we apply the significance test for the
2) effectively distinguish the unanswerable questions, and concerned MRC task and show that our models are sig-
then refuse to answer. Such requirements lead to the re- nificantly better than the baselines.
cent reader’s design by introducing an extra verifier mod-
ule or answer-verification mechanism. Most readers simply 2 Related Work
stack the verifier along with encoder and decoder parts in The research of machine reading comprehension have at-
1
tracted great interest with the release of a variety of bench-
We define decoder here as the task-specific part in an MRC
2
system, such as passage and question interaction and answer veri- Our source code is available at https://github.com/cooelf/
fication. AwesomeMRC.
mark datasets (Hill et al. 2015; Hermann et al. 2015; Ra- sitions in the passage P and extract span as answer A but
jpurkar et al. 2016; Joshi et al. 2017; Rajpurkar, Jia, and also return a null string when the question is unanswerable.
Liang 2018; Lai et al. 2017). The early trend is a variety Our retrospective reader is composed of two parallel mod-
of attention-based interactions between passage and ques- ules: a sketchy reading module and an intensive reading
tion, including Attention Sum (Kadlec et al. 2016), Gated module to conduct a two-stage reading process. The intu-
attention (Dhingra et al. 2017), Self-matching (Wang et al. ition behind the design is that the sketchy reading makes
2017), Attention over Attention (Cui et al. 2017) and Bi- a coarse judgment (external front verification) about the an-
attention (Seo et al. 2016). Recently, PrLMs dominate the swerability, whether the question is answerable; and then the
encoder design for MRC and achieve great success. These intensive reading jointly predicts the candidate answers and
PrLMs include ELMo (Peters et al. 2018), GPT (Radford combines its answerability confidence (internal front veri-
et al. 2018), BERT (Devlin et al. 2019), XLNet (Yang et al. fication) with the sketchy judgment score to yield the final
2019), RoBERTa (Liu et al. 2019), ALBERT (Lan et al. answer (rear verification). 3
2020), and ELECTRA (Clark et al. 2020). They show strong
capacity for capturing the contextualized sentence-level lan- 3.1 Sketchy Reading Module
guage representations and greatly boost the benchmark per-
Embedding We concatenate question and passage texts as
formance of current MRC. Following this line, we take
the input, which is firstly represented as embedding vec-
PrLMs as our backbone encoder.
tors to feed an encoder (i.e., a PrLM). In detail, the input
In the meantime, the study of the decoder mechanisms texts are first tokenized to word pieces (subword tokens).
has come to a bottleneck due to the already powerful PrLM Let T = {t1 , . . . , tn } denote a sequence of subword tokens
encoder. Thus this work focuses on the non-encoder part, of length n. For each token, the input embedding is the sum
such as passage and question attention interactions, and es- of its token embedding, position embedding, and token-type
pecially the answer verification. embedding. Let X = {x1 , . . . , xn } be the outputs of the en-
To solve the MRC task with unanswerable questions is coder, which are embedding features of encoding sentence
though important, only a few studies paid attention to this tokens of length n. The input embeddings are then fed to the
topic with straightforward solutions. Mostly, a treatment is interaction layer to obtain the contextual representations.
to adopt an extra answer verification layer, the answer span
prediction and answer verification are trained jointly with
multi-task learning (Figure 1-[c]). Such an implemented ver- Interaction Following Devlin et al. (2019), the encoded
ification mechanism can also be as simple as an answerable sequence X is processed to a multi-layer Transformer
threshold setting broadly used by powerful enough PrLMs (Vaswani et al. 2017) for learning contextual representa-
for quickly building readers (Devlin et al. 2019; Zhang et al. tions. For the following part, we use H = {h1 , . . . , hn } to
2020b). Liu et al. (2018) appended an empty word token denote the last-layer hidden states of the input sequence.
to the context and added a simple classification layer to the
reader. Hu et al. (2019) used two types of auxiliary loss, External Front Verification After reading, the sketchy
independent span loss to predict plausible answers and in- reader will make a preliminary judgment, whether the ques-
dependent no-answer loss the to decide answerability of tion is answerable given the context. We implement this
the question. Further, an extra verifier is adopted to de- reader as an external front verifier (E-FV) to identify unan-
cide whether the predicted answer is entailed by the input swerable questions. The pooled first token (the special sym-
snippets (Figure 1-[b]). Back et al. (2020) developed an bol, [CLS]) representation h1 ∈ H, as the overall repre-
attention-based satisfaction score to compare question em- sentation of the sequence, is passed to a fully connection
beddings with the candidate answer embeddings (Figure 1- layer to get classification logits ŷi composed of answer-
[c]). Zhang et al. (2020c) proposed a verifier layer, which able (logitans ) and unanswerable (logitna ) elements. We
is a linear layer applied to context embedding weighted by use cross entropy as training objective:
start and end distribution over the context words representa-
tions concatenated to [CLS] token representation for BERT 1 X
N
(Figure 1-[c]). Lans = − [yi log ŷi + (1 − yi ) log(1 − ŷi )] (1)
Different from these existing studies which stack the ver- N i=1
ifier module in a simple way or just jointly learn answer
location and non-answer losses, our Retro-Reader adopts a where ŷi ∝ SoftMax(FFN(h1 )) denotes the prediction and
two-stage humanoid design (Zheng et al. 2019; Guthrie and yi is the target indicating whethter the question is answer-
Mosenthal 1987) based on a comprehensive survey over ex- bale or not. N is the number of examples. We calculate
isting answer verification solutions. the difference as the external verification score: scoreext =
logitna −logitans , which is used in the later rear verification
as effective indication factor.
3 Our Proposed Model
3
Intuitively, our model is supposed to be designed as shown
We focus on the span-based MRC task, which can be de- in Figure 1-[d]. In the implementation, we find that modeling the
scribed as a triplet hP, Q, Ai, where P is a passage, and Q entire reading process into two parallel modules is both simple and
is a query over P , in which a span is a right answer A. Our practicable with basically the same performance, which results in
system is supposed to not only predict the start and end po- a parallel reading module design at last as shown in Figure 1-[e].
3.2 Intensive Reading Module with SoftMax operation and feed H0 as the input to obtain
The objective of the intensive reader is to verify the answer- the start and end probabilities, s and e:
ability, produce candidate answer spans, and then give the
s, e ∝ SoftMax(FFN(H0 )). (3)
final answer prediction. It employs the same encoding and
interaction procedure as the sketchy reader, to obtain the rep-
The training objective of answer span prediction is de-
resentation H. In previous studies (Devlin et al. 2019; Yang
fined as cross entropy loss for the start and end predictions,
et al. 2019; Lan et al. 2020), H is directly fed to a linear layer
to yield the prediction.
N
1 X
Lspan = − [log(psyis ) + log(peyie )] (4)
Question-aware Matching Inspired by previous success N i
of explicit attention matching between passage and question
(Kadlec et al. 2016; Dhingra et al. 2017; Wang et al. 2017; where yis and yie are respectively ground-truth start and end
Seo et al. 2016), we are interested in whether the advance positions of example i. N is the number of examples.
still holds based on the strong PrLMs. Here we investigate
two alternative question-aware matching mechanisms as an
extra layer. Note that this part is only used for ablation in Internal Front Verification We adopted an internal front
Table 7 as a reference for interested readers. We do not use verifier (I-FV) such that the intensive reader can identify
any extra matching part in our submission on test evaluations unanswerable questions as well. In general, a verifier’s func-
(e.g., in Tables 2-3) for the sake of simplicity as our major tion can be implemented as a cross-entropy loss (I-FV-
focus is the verification. CE), binary cross-entropy loss (I-FV-BE), or regression-
To obtain the representation of each passage and ques- style mean square error loss (I-FV-MSE). The pooled rep-
0
tion, we split the last-layer hidden state H into HQ and HP resentation h1 ∈ H0 , is passed to a fully connected layer
as the representations of the question and passage, accord- to get the classification logits or regression score. Let ŷi ∝
ing to its position information. Both of the sequences are Linear(h1 ) denote the prediction and yi is the answerability
padded to the maximum length in a minibatch. Then, we target, the three alternative loss functions are as defined as
investigate two potential question-aware matching mecha- follows:
nisms, 1) Transformer-style multi-head cross attention (CA) (1) We use cross entropy as loss function for the classifi-
and 2) traditional matching attention (MA). cation verification:
• Cross Attention We feed the HQ and H to a revised 0

one-layer multi-head attention layer inspired by Lu et al. ȳi,k = SoftMax(FFN(h1 )),

(2019). Since the setting is Q = K = V in multi-head self N K
1 XX (5)
attention,4 which are all derived from the input sequence, we Lans = − [yi,k log ȳi,k ] ,
N i=1
replace the input to Q with H, and both of K and V with HQ k=1
to obtain the question-aware context representation H0 .
where K means the number of classes (K = 2 in this work).
• Matching Attention Another alternative is to feed HQ
N is the number of examples.
and H to a traditional matching attention layer (Wang et al.
(2) For binary cross entropy as loss function for the clas-
2017), by taking the question presentation HQ as the atten-
sification verification:
tion to the representation H:
ȳi = Sigmoid(FFN(h1 )),
M = SoftMax(H(WHQ + b ⊗ e)T ),
(2) 1 X
N
(6)
H0 = MHQ , Lans = − [yi log ȳi + (1 − yi ) log(1 − ȳi )] .
N i=1
where W and b are learnable parameters. e is a all-ones vec-
tor and used to repeat the bias vector into the matrix. M de- (3) For the regression verification, mean square error is
notes the weights assigned to the different hidden states in adopted as its loss function:
the concerned two sequences. H0 is the weighted sum of all
the hidden states and it represents how the vectors in H can 0
ȳi = FFN(h1 ), (7)
be aligned to each hidden state in HQ . Finally, the represen-
N
tation H0 is used for the later predictions. If we do not use 1 X
the above matching like the original use in BERT models, Lans = (yi − ȳi )2 . (8)
N
then H0 = H for the following part. i=1

During training, the joint loss function for FV is the

Span Prediction The aim of span-based MRC is to find a weighted sum of the span loss and verification loss:
span in the passage as answer, thus we employ a linear layer
L = α1 Lspan + α2 Lans , (9)
4
In this work, Q, K, V correspond to the items
Ql+1 l l+1 l l+1 l
m xi , Km xj , Vm xj , respectively. where α1 and α2 are weights.
Threshold-based Answerable Verification Following Dev Test
previous studies (Devlin et al. 2019; Yang et al. 2019; Liu Model
EM F1 EM F1
et al. 2019; Lan et al. 2020), we adopt threshold based an-
swerable verification (TAV), which is a heuristic strategy to Human - - 86.8 89.5
decide whether a question is answerable according to the BERT (Devlin et al. 2019) - - 82.1 84.8
predicted answer start and end logits finally. Given the out- NeurQuRI (Back et al. 2020) 80.0 83.1 81.3 84.3
put start and end probabilities s and e, and the verification XLNet (Yang et al. 2019) 86.1 88.8 86.4 89.1
probability v, we calculate the has-answer score scorehas RoBERTa (Liu et al. 2019) 86.5 89.4 86.8 89.8
and the no-answer score scorenull : SG-Net (Zhang et al. 2020c) - - 87.2 90.1
ALBERT (Lan et al. 2020) 87.4 90.2 88.1 90.9
scorehas = max(sk + el ), 1 < k ≤ l ≤ n,
(10) ELECTRA (Clark et al. 2020) 88.0 90.6 88.7 91.4
scorenull = s1 + e1 ,
Our implementation
We obtain a difference score between scorenull and ALBERT (+TAV) 87.0 90.2 - -
the scorehas as the final no-answer score: scoredif f = Retro-Reader on ALBERT 87.8 90.9 88.1 91.4
scorenull − scorehas . An answerable threshold δ is set and ELECTRA (+TAV) 88.0 90.6 - -
determined according to the development set. The model Retro-Reader on ELECTRA 88.8 91.3 89.6 92.1
predicts the answer span that gives the has-answer score if
the final score is above the threshold δ, and null string oth- Table 2: The results (%) for SQuAD2.0 dataset. The results
erwise. are from the official leaderboard. TAV: threshold-based an-
TAV is used in all our models as the last step for the swerable verification (§3.2).
answerability decision. We denote it in our baselines with
(+TAV) as default in Table 2-3, and omit the notation for
simplicity in analysis part to avoid misunderstanding to keep answerable threshold method described in §3.2. In the fol-
on the specific ablations. lowing part, +TAV (for all the baseline modes) denotes the
baseline verification for easy reference, which is equivalent
3.3 Rear Verification to the baseline implementations in public literatures (Devlin
Rear verification (RV) is the combination of predicted prob- et al. 2019; Lan et al. 2020; Clark et al. 2020).
abilities of E-FV and I-FV, which is an aggregated verifica-
tion for final answer. 4.2 Benchmark Datasets
v = β1 scoredif f + β2 scoreext , (11) Our proposed reader is evaluated in two benchmark MRC
challenges.
where β1 and β2 are weights. Our model predicts the answer
span if v > δ, and null string otherwise.
SQuAD2.0 As a widely used MRC benchmark dataset,
SQuAD2.0 (Rajpurkar, Jia, and Liang 2018) combines the
4 Experiments 100,000 questions in SQuAD1.1 (Rajpurkar et al. 2016) with
4.1 Setup over 50,000 new, unanswerable questions that are written
We use the available PrLMs as encoder to build baseline adversarially by crowdworkers to look similar to answerable
MRC models: BERT (Devlin et al. 2019), ALBERT (Lan ones. The training dataset contains 87k answerable and 43k
et al. 2020), and ELECTRA (Clark et al. 2020). Our imple- unanswerable questions.
mentations of BERT and ALBERT are based on the public
Pytorch implementation from Transformers.5 ELECTRA is NewsQA NewsQA (Trischler et al. 2017) is a question-
based on the Tensorflow release.6 We use the pre-trained LM answering dataset with 100,000 human-generated question-
weights in the encoder module of our reader, using all the answer pairs. The questions and answers are based on a
official hyperparameters.7 For the fine-tuning in our tasks, set of over 10,000 news articles from CNN supplied by
we set the initial learning rate in {2e-5, 3e-5} with a warm- crowdworkers. The paragraphs are about 600 words on aver-
up rate of 0.1, and L2 weight decay of 0.01. The batch size age, which tend to be longer than SQuAD2.0. The training
is selected in {32, 48}. The maximum number of epochs dataset has 20k unanswerable questions among 97k ques-
is set in 2 for all the experiments. Texts are tokenized us- tions.
ing wordpieces (Wu et al. 2016), with a maximum length of
512. Hyper-parameters were selected using the dev set. The 4.3 Evaluation
manual weights are α1 = α2 = β1 = β2 = 0.5 in this work. Metrics Two official metrics are used to evaluate the
For answer verification, we follow the same setting ac- model performance: Exact Match (EM) and a softer metric
cording to the corresponding literatures (Devlin et al. 2019; F1 score, which measures the average overlap between the
Lan et al. 2020; Clark et al. 2020), which simply adopts the prediction and ground truth answer at the token level.
5
https://github.com/huggingface/transformers.
6
https://github.com/google-research/electra. Significance Test With the rapid development of deep
7
BERTlarge ; ALBERTxxlarge ; ELECTRAlarge . MRC models, the dominant models have achieved very high
 Dev Test All HasAns NoAns
Model Model
EM F1 EM F1 EM F1 EM F1 EM F1
Existing Systems BERT 78.8 81.7 74.6 80.3 83.0 83.0
BARB (Trischler et al. 2017) 36.1 49.6 34.1 48.2 + E-FV 79.1 82.1 73.4 79.4 84.8 84.8
mLSTM (Wang and Jiang 2017) 34.4 49.6 34.9 50.0 + I-FV-CE 78.6 82.0 73.3 79.5 84.5 84.5
BiDAF (Seo et al. 2016) - - 37.1 52.3 + I-FV-BE 78.8 81.8 72.6 78.7 85.0 85.0
R2-BiLSTM (Weissenborn 2017) - - 43.7 56.7 + I-FV-MSE 78.5 81.7 73.0 78.6 84.8 84.8
AMANDA (Kundu and Ng 2018) 48.4 63.3 48.4 63.7 + RV 79.6 82.5 73.7 79.6 85.2 85.2
DECAPROP (Tay et al. 2018) 52.5 65.7 53.1 66.3
ALBERT 87.0 90.2 82.6 89.0 91.4 91.4
BERT (Devlin et al. 2019) - - 46.5 56.7
+ E-FV 87.4 90.6 82.4 88.7 92.4 92.4
NeurQuRI (Back et al. 2020) - - 48.2 59.5
+ I-FV-CE 87.2 90.3 81.7 87.9 92.7 92.7
Our implementation + I-FV-BE 87.2 90.2 82.2 88.4 92.1 92.1
ALBERT (+TAV) 57.1 67.5 55.3 65.9 + I-FV-MSE 87.3 90.4 82.4 88.5 92.3 92.3
Retro-Reader on ALBERT 58.5 68.6 55.9 66.8 + RV 87.8 90.9 83.1 89.4 92.4 92.4
ELECTRA (+TAV) 56.3 66.5 54.0 64.5
Retro-Reader on ELECTRA 56.9 67.0 54.7 65.7
Table 4: Results (%) with different answer verification
methods on the SQuAD2.0 dev set. CE, BE, and MSE are
Table 3: Results (%) for NewsQA dataset. The results except short for the two classification and one regression loss func-
ours are from Tay et al. (2018) and Back et al. (2020). TAV: tions defined in §3.2.
threshold based answerable verification (§3.2).
Method Prec. Rec. F1 Acc.
results (e.g., over 90% F1 scores on SQuAD2.0), and further ALBERT 91.70 93.42 92.55 86.14
advance has been very marginal. Thus a significance test Retro-Reader on ALBERT 94.30 92.38 93.33 87.49
would be beneficial for measuring the difference in model ELECTRA 92.71 92.58 92.64 86.30
performance. Retro-Reader on ELECTRA 93.27 93.51 93.39 87.60
For selecting evaluation metrics for the significance test,
since answers vary in length, using the F1 score would have
a bias when comparing models, i.e., if one model fails on one Table 5: Performance on the unanswerable questions from
severe example though works well on the others. Therefore, SQuAD2.0 dev set.
we use the tougher metric EM as the goodness measure. If
the EM is equal to 1, the prediction is regarded as right and
vice versa. Then the test is modeled as a binary classifica- bers reported in the corresponding papers (Lan et al. 2020;
tion problem to estimate the answer of the model is exactly Clark et al. 2020), ensuring that the proposed method can be
right (EM=1) or wrong (EM=0) for each question. Accord- fairly evaluated over the public strong baseline systems.
ing to our task setting, we used McNemar’s test (McNemar 2) In terms of powerful enough PrLMs like ALBERT and
1947) to test the statistical significance of our results. This ELECTRA, our Retro-Reader not only significantly outper-
test is designed for paired nominal observations, and it is ap- forms the baselines with p-value < 0.01,8 but also achieves
propriate for binary classification tasks (Ziser and Reichart new state-of-the-art on the SQuAD2.0 challenge.9
2017). 3) The results on NewsQA further verifies the general
The p-value is defined as the probability, under the null effectiveness of our proposed Retro-Reader. Our method
hypothesis, of obtaining a result equal to or more extreme shows consistent improvements over the baselines and
than what was observed. The smaller the p-value, the higher achieves new state-of-the-art results.
the significance. A commonly used level of reliability of the
result is 95%, written as p = 0.05. 5 Ablations
5.1 Evaluation on Answer Verification
4.4 Results Table 4 presents the results with different answer verification
Tables 2-3 compare the leading single models on SQuAD2.0 methods. We observe that either of the front verifiers boosts
and NewsQA. Retro-Reader on ALBERT and Retro-Reader the baselines, and integrating both as rear verification works
on ELECTRA denote our final models (i.e., our submissions the best. Note that we show the HasAns and NoAns only for
to SQuAD2.0 online evaluation), which are respectively the 8
Besides the McNemar’s test, we also used paired t-test for sig-
ALBERT and ELECTRA based retrospective reader com- nificance test, with consistent findings.
posed of both sketchy and intensive reading modules with- 9
When our models were submitted (Jan 10th 2020 and Apr
out question-aware matching for simplicity. According to 05, 2020 for ALBERT- and ELECTRA-based models, respec-
the results, we make the following observations: tively), our Retro-Reader achieved the first place on the SQuAD2.0
1) Our implemented ALBERT and ELECTRA baselines Leaderboard (https://rajpurkar.github.io/SQuAD-explorer/ ) for
show the similar EM and F1 scores with the original num- both single and ensemble models.
 Method EM F1 Passage:
Southern California consists of a heavily developed ur-
ALBERT 87.0 90.2 ban environment, home to some of the largest urban areas
Two-model Ensemble 87.6 90.6 in the state, along with vast areas that have been left un-
Retro-Reader 87.8 90.9 developed. It is the third most populated megalopolis in
the United States, after the Great Lakes Megalopolis and
Table 6: Comparisons with Equivalent Parameters on the the Northeastern megalopolis. Much of southern Califor-
dev set of SQuAD2.0. nia is famous for its large, spread-out, suburban commu-
nities and use of automobiles and highways. The domi-
nant areas are Los Angeles, Orange County, San Diego,
SQuAD2.0 NewsQA and Riverside-San Bernardino, each of which are the cen-
Method
EM F1 EM F1 ters of their respective metropolitan areas...
BERT 78.8 81.7 51.8 62.5 Question:
+ CA 78.8 81.7 52.1 62.7 What are the second and third most populated megalopo-
+ MA 78.3 81.4 52.4 62.6 lis after Southern California?
ALBERT 87.0 90.2 57.1 67.5 Answer:
+ CA 87.3 90.3 56.0 66.3 Gold: hno answeri
+ MA 86.8 90.0 55.8 66.1 ALBERT (+TAV): Great Lakes Megalopolis and the
Northeastern megalopolis.
Retro-Reader over ALBERT: hno answeri
Table 7: Results (%) with matching interaction methods on scorehas = 0.03, scorena = 1.73, δ = −0.98
the dev sets of SQuAD2.0 and NewsQA.
Table 8: Answer prediction examples from the ALBERT
baseline and Retro-Reader.
completeness. Since the final predictions are based on the
threshold search of answerability scores (§3.2), there exists
a tradeoff between the HasAns and NoAns accuracies. We
ers could not bring noticeable improvement, which indicates
see that the final RV that combines E-FV and I-FV shows
that simply adding more layers and parameters would not
the best performance, which we select as our final imple-
substantially benefit the model performance. The results ver-
mentation for testing.
ified the PrLMs’ strong ability to capture the relationships
We further conduct the experiments on our model per-
between passage and question after processing the paired in-
formance of the 5,945 unanswerable questions from the
put by deep self-attention layers. In contrast, answer verifi-
SQuAD 2.0 dev set. Results in Table 5 show that our method
cation could still give consistent and substantial advance.
improves the performance on unanswerable questions by a
large margin, especially in the primary F1 and accuracy met-
5.4 Comparison of Predictions
rics.
To have an intuitive observation of the predictions of Retro-
5.2 Comparisons with Equivalent Parameters Reader, we give a prediction example on SQuAD2.0 from
When using sketchy reading module for external verifica- baseline and Retro-Reader in Table 8, which shows that our
tion, we have two parallel modules that have independent method works better at judging whether the question is an-
parameters. For comparisons with equivalent parameters, we swerable on a given passage and gets rid of the plausible
add an ensemble of two baseline models, to see if the ad- answer.
vance is purely from the increase of parameters. Table 6
shows the results. We see that our model can still outperform 6 Conclusion
two ensembled models. Although the two modules share the As machine reading comprehension tasks with unanswer-
same design of the Transformer encoder, the training objec- able questions stress the importance of answer verification
tives (e.g., loss functions) are quite different, one for answer in MRC modeling, this paper devotes itself to better verifier-
span prediction, the other for answerable decision. The re- oriented MRC task-specific design and implementation for
sults indicate that our two-stage reading modules would be the first time. Inspired by human reading comprehension ex-
more effective for learning diverse aspects (verification and perience, we proposed a retrospective reader that integrates
span prediction) for solving MRC tasks with different train- both sketchy and intensive reading. With the latest PrLM
ing objectives. From the two modules, we can easily find the as encoder backbone and baseline, the proposed reader
effectiveness of either the span prediction or answer verifi- is evaluated on two benchmark MRC challenge datasets
cation, to improve the modules correspondingly. We believe SQuAD2.0 and NewsQA, achieving new state-of-the-art re-
this design would be quite useful for real-world applications. sults and outperforming strong baseline models in terms of
newly introduced statistical significance, which shows the
5.3 Evaluation on Matching Interactions choice of verification mechanisms has a significant impact
Table 7 shows the results with different interaction meth- for MRC performance and verifier is an indispensable reader
ods described in §3.2. We see that merely adding extra lay- component even for powerful enough PrLMs used as the en-
coder. In the future, we will investigate more decoder-side Joshi, M.; Choi, E.; Weld, D.; and Zettlemoyer, L. 2017.
problem-solving techniques to cooperate with the strong en- TriviaQA: A Large Scale Distantly Supervised Challenge
coders for more advanced MRC. Dataset for Reading Comprehension. In Proceedings of the
55th Annual Meeting of the Association for Computational
References Linguistics (Volume 1: Long Papers), 1601–1611.
Back, S.; Chinthakindi, S. C.; Kedia, A.; Lee, H.; and Choo, Kadlec, R.; Schmid, M.; Bajgar, O.; and Kleindienst, J.
J. 2020. NeurQuRI: Neural Question Requirement Inspector 2016. Text Understanding with the Attention Sum Reader
for Answerability Prediction in Machine Reading Compre- Network. In Proceedings of the 54th Annual Meeting of the
hension. In International Conference on Learning Repre- Association for Computational Linguistics (Volume 1: Long
sentations. Papers), 908–918.
Chen, D.; Bolton, J.; and Manning, C. D. 2016. A Thorough Kundu, S.; and Ng, H. T. 2018. A Question-Focused Multi-
Examination of the CNN/Daily Mail Reading Comprehen- Factor Attention Network for Question Answering. In McIl-
sion Task. In Proceedings of the 54th Annual Meeting of the raith, S. A.; and Weinberger, K. Q., eds., AAAI, 5828–5835.
Association for Computational Linguistics (Volume 1: Long Lai, G.; Xie, Q.; Liu, H.; Yang, Y.; and Hovy, E. 2017.
Papers), 2358–2367. RACE: Large-scale ReAding Comprehension Dataset From
Choi, E.; He, H.; Iyyer, M.; Yatskar, M.; Yih, W.-t.; Choi, Examinations. In Proceedings of the 2017 Conference on
Y.; Liang, P.; and Zettlemoyer, L. 2018. QuAC: Question Empirical Methods in Natural Language Processing, 785–
Answering in Context. In Proceedings of the 2018 Confer- 794.
ence on Empirical Methods in Natural Language Process- Lan, Z.; Chen, M.; Goodman, S.; Gimpel, K.; Sharma, P.;
ing, 2174–2184. and Soricut, R. 2020. ALBERT: A Lite BERT for Self-
Clark, K.; Luong, M.-T.; Le, Q. V.; and Manning, C. D. supervised Learning of Language Representations. In In-
2020. ELECTRA: Pre-training Text Encoders as Discrimi- ternational Conference on Learning Representations.
nators Rather Than Generators. In International Conference Li, Z.; He, S.; Cai, J.; Zhang, Z.; Zhao, H.; Liu, G.; Li, L.;
on Learning Representations. and Si, L. 2018. A unified syntax-aware framework for se-
Cui, Y.; Chen, Z.; Wei, S.; Wang, S.; Liu, T.; and Hu, G. mantic role labeling. In Proceedings of the 2018 Confer-
2017. Attention-over-Attention Neural Networks for Read- ence on Empirical Methods in Natural Language Process-
ing Comprehension. In Proceedings of the 55th Annual ing, 2401–2411.
Meeting of the Association for Computational Linguistics Li, Z.; He, S.; Zhao, H.; Zhang, Y.; Zhang, Z.; Zhou, X.; and
(Volume 1: Long Papers), 593–602. Zhou, X. 2019. Dependency or span, end-to-end uniform se-
Devlin, J.; Chang, M.-W.; Lee, K.; and Toutanova, K. 2019. mantic role labeling. In Proceedings of the AAAI Conference
BERT: Pre-training of Deep Bidirectional Transformers for on Artificial Intelligence, volume 33, 6730–6737.
Language Understanding. In Proceedings of the 2019 Con- Li, Z.; Wang, R.; Chen, K.; Utiyama, M.; Sumita, E.; Zhang,
ference of the North American Chapter of the Association Z.; and Zhao, H. 2020. Explicit Sentence Compression for
for Computational Linguistics: Human Language Technolo- Neural Machine Translation. Proceedings of the AAAI Con-
gies, Volume 1 (Long and Short Papers), 4171–4186. ference on Artificial Intelligence 34(05): 8311–8318.
Dhingra, B.; Liu, H.; Yang, Z.; Cohen, W. W.; and Salakhut- Liu, L.; Zhang, Z.; ; Zhao, H.; Zhou, X.; and Zhou, X. 2021.
dinov, R. 2017. Gated-Attention Readers for Text Compre- Filling the Gap of Utterance-aware and Speaker-aware Rep-
hension. In Proceedings of the 55th Annual Meeting of the resentation for Multi-turn Dialogue. In The Thirty-Fifth
Association for Computational Linguistics (Volume 1: Long AAAI Conference on Artificial Intelligence (AAAI-21).
Papers), 1832–1846.
Liu, X.; Li, W.; Fang, Y.; Kim, A.; Duh, K.; and Gao, J.
Guthrie, J. T.; and Mosenthal, P. 1987. Literacy as Multidi- 2018. Stochastic Answer Networks for SQuAD 2.0. In Pro-
mensional: Locating Information and Reading Comprehen- ceedings of the 56th Annual Meeting of the Association for
sion. Educational Psychologist 22(3-4): 279–297. Computational Linguistics (Volume 1: Long Papers), 1694–
Hermann, K. M.; Kocisky, T.; Grefenstette, E.; Espeholt, L.; 1704.
Kay, W.; Suleyman, M.; and Blunsom, P. 2015. Teaching Liu, Y.; Ott, M.; Goyal, N.; Du, J.; Joshi, M.; Chen, D.;
machines to read and comprehend. Advances in neural in- Levy, O.; Lewis, M.; Zettlemoyer, L.; and Stoyanov, V.
formation processing systems 28: 1693–1701. 2019. RoBERTa: A robustly optimized BERT pretraining
Hill, F.; Bordes, A.; Chopra, S.; and Weston, J. 2015. approach. arXiv preprint arXiv:1907.11692 .
The Goldilocks Principle: Reading Children’s Books Lu, J.; Batra, D.; Parikh, D.; and Lee, S. 2019. ViLBERT:
with Explicit Memory Representations. arXiv preprint Pretraining Task-Agnostic Visiolinguistic Representations
arXiv:1511.02301 . for Vision-and-Language Tasks. In Advances in Neural In-
Hu, M.; Wei, F.; Peng, Y.; Huang, Z.; Yang, N.; and Li, D. formation Processing Systems, 13–23.
2019. Read+ verify: Machine reading comprehension with McNemar, Q. 1947. Note on the sampling error of the dif-
unanswerable questions. In Proceedings of the AAAI Con- ference between correlated proportions or percentages. Psy-
ference on Artificial Intelligence, volume 33, 6529–6537. chometrika 12(2): 153–157.
Peters, M. E.; Neumann, M.; Iyyer, M.; Gardner, M.; Clark, Xu, Y.; Zhao, H.; and Zhang, Z. 2021. Topic-aware multi-
C.; Lee, K.; and Zettlemoyer, L. 2018. Deep contextualized turn dialogue modeling. In The Thirty-Fifth AAAI Confer-
word representations. In NAACL-HLT, 2227–2237. ence on Artificial Intelligence (AAAI-21).
Radford, A.; Narasimhan, K.; Salimans, T.; and Sutskever, Yang, Z.; Dai, Z.; Yang, Y.; Carbonell, J.; Salakhutdinov,
I. 2018. Improving language understanding by generative R. R.; and Le, Q. V. 2019. XLNET: Generalized autoregres-
pre-training. Technical report . sive pretraining for language understanding. In Advances in
neural information processing systems, 5753–5763.
Rajpurkar, P.; Jia, R.; and Liang, P. 2018. Know What You
Don’t Know: Unanswerable Questions for SQuAD. In Pro- Zhang, S.; Zhao, H.; Wu, Y.; Zhang, Z.; Zhou, X.; and Zhou,
ceedings of the 56th Annual Meeting of the Association for X. 2020a. DCMN+: Dual Co-Matching Network for Multi-
Computational Linguistics (Volume 2: Short Papers), 784– choice Reading Comprehension. Proceedings of the AAAI
789. Conference on Artificial Intelligence 34(05): 9563–9570.
Zhang, Z.; Huang, Y.; and Zhao, H. 2018. Subword-
Rajpurkar, P.; Zhang, J.; Lopyrev, K.; and Liang, P. 2016.
augmented Embedding for Cloze Reading Comprehension.
SQuAD: 100,000+ Questions for Machine Comprehension
In Proceedings of the 27th International Conference on
of Text. In Proceedings of the 2016 Conference on Empiri-
Computational Linguistics (COLING 2018), 1802––1814.
cal Methods in Natural Language Processing, 2383–2392.
Zhang, Z.; Li, J.; Zhu, P.; Zhao, H.; and Liu, G. 2018. Mod-
Reddy, R. G.; Sultan, M. A.; Kayi, E. S.; Zhang, R.; Castelli, eling Multi-turn Conversation with Deep Utterance Aggre-
V.; and Sil, A. 2020. Answer Span Correction in Machine gation. In Proceedings of the 27th International Conference
Reading Comprehension. In Findings of the Association for on Computational Linguistics, 3740–3752.
Computational Linguistics: EMNLP 2020.
Zhang, Z.; Wu, Y.; Li, Z.; and Zhao, H. 2019. Explicit Con-
Reddy, S.; Chen, D.; and Manning, C. D. 2019. CoQA: textual Semantics for Text Comprehension. In Proceedings
A Conversational Question Answering Challenge. Trans- of the 33rd Pacific Asia Conference on Language, Informa-
actions of the Association for Computational Linguistics 7: tion and Computation (PACLIC 33).
249–266.
Zhang, Z.; Wu, Y.; Zhao, H.; Li, Z.; Zhang, S.; Zhou, X.;
Seo, M.; Kembhavi, A.; Farhadi, A.; and Hajishirzi, H. 2016. and Zhou, X. 2020b. Semantics-aware BERT for language
Bidirectional Attention Flow for Machine Comprehension. understanding. Proceedings of the AAAI Conference on Ar-
In International Conference on Learning Representations. tificial Intelligence 34(05): 9628–9635.
Tay, Y.; Luu, A. T.; Hui, S. C.; and Su, J. 2018. Densely con- Zhang, Z.; Wu, Y.; Zhou, J.; Duan, S.; Zhao, H.; and Wang,
nected attention propagation for reading comprehension. In R. 2020c. SG-Net: Syntax-Guided Machine Reading Com-
Advances in neural information processing systems, 4906– prehension. Proceedings of the AAAI Conference on Artifi-
4917. cial Intelligence 34(05): 9628–9635.
Trischler, A.; Wang, T.; Yuan, X.; Harris, J.; Sordoni, A.; Zhang, Z.; Zhao, H.; and Wang, R. 2020. Machine Read-
Bachman, P.; and Suleman, K. 2017. NewsQA: A Machine ing Comprehension: The Role of Contextualized Language
Comprehension Dataset. In Proceedings of the 2nd Work- Models and Beyond. arXiv preprint arXiv:2005.06249 .
shop on Representation Learning for NLP, 191–200. Zheng, Y.; Mao, J.; Liu, Y.; Ye, Z.; Zhang, M.; and Ma, S.
Vaswani, A.; Shazeer, N.; Parmar, N.; Uszkoreit, J.; Jones, 2019. Human Behavior Inspired Machine Reading Com-
L.; Gomez, A. N.; Kaiser, L.; and Polosukhin, I. 2017. At- prehension. In Proceedings of the 42nd International ACM
tention is All you Need. Advances in neural information SIGIR Conference on Research and Development in Infor-
processing systems 30: 5998–6008. mation Retrieval, 425–434.
Zhou, J.; Zhang, Z.; and Zhao, H. 2019. LIMIT-BERT: Lin-
Wang, S.; and Jiang, J. 2017. Machine Comprehension
guistic Informed Multi-Task BERT. In Findings of the Asso-
Using Match-LSTM and Answer Pointer. In International
ciation for Computational Linguistics: EMNLP 2020, 4450–
Conference on Learning Representations.
4461.
Wang, W.; Yang, N.; Wei, F.; Chang, B.; and Zhou, M. 2017. Zhu, P.; Zhang, Z.; Li, J.; Huang, Y.; and Zhao, H. 2018.
Gated Self-Matching Networks for Reading Comprehension Lingke: A Fine-grained Multi-turn Chatbot for Customer
and Question Answering. In Proceedings of the 55th An- Service. In Proceedings of the 27th International Confer-
nual Meeting of the Association for Computational Linguis- ence on Computational Linguistics (COLING 2018), System
tics (Volume 1: Long Papers), 189–198. Demonstrations, 108––112.
Weissenborn, D. 2017. Reading Twice for Natural Language Zhu, P.; Zhao, H.; and Li, X. 2020. Dual multi-head co-
Understanding. CoRR abs/1706.02596. attention for multi-choice reading comprehension. arXiv
Wu, Y.; Schuster, M.; Chen, Z.; Le, Q. V.; Norouzi, M.; preprint arXiv:2001.09415 .
Macherey, W.; Krikun, M.; Cao, Y.; Gao, Q.; Macherey, K.; Ziser, Y.; and Reichart, R. 2017. Neural Structural Corre-
et al. 2016. Google’s neural machine translation system: spondence Learning for Domain Adaptation. In Proceedings
Bridging the gap between human and machine translation. of the 21st Conference on Computational Natural Language
arXiv preprint arXiv:1609.08144 . Learning (CoNLL 2017), 400–410.