Open-domain question answering (Open-QA) To mea-
sure a model’s ability to incorporate world knowledge, we
need a downstream task where world knowledge is criti-
cal. Perhaps one of the most knowledge-intensive tasks
in natural language processing is open-domain question
answering (Open-QA): given a question x such as “What
is the currency of the UK?”, a model must output
the correct answer string y, “pound”. The “open” part of
Open-QA refers to the fact that the model does not receive
a pre-identiﬁed document that is known to contain the an-
swer, unlike traditional reading comprehension (RC) tasks

1We use the term “document” loosely to refer to a passage from

the knowledge corpus, not necessarily a whole article.

2Strictly speaking, MLM is not a standard language model,
since it does not deﬁne a distribution over the entire sequence
of tokens. In the paper we sometimes abuse the term “language
model” slightly to make the phrase shorter.
REALM