Examples of retrieved documents Table 3 shows an ex-
ample of the REALM masked language model prediction.
In this example, “Fermat” is the correct word, and REALM
(row (c)) gives the word a much high probability compared
to the BERT model (row (a)). Since REALM manages to
retrieve some documents with a related fact (row (b)), the
marginalized probability of the correct answer dramatically
increases. This shows that REALM is able to retrieve doc-
ument to ﬁll in the masked word even though it is trained
with unsupervised text only.

5. Discussion and Related Work
We previously discussed related methods for Open-QA.
Here we present several alternate ways of viewing REALM
that connect it to a broader set of ideas beyond Open-QA:
REALM