(a) BERT
(b) REALM p(y = “Fermat”| x, z) = 1.0
(c) REALM p(y = “Fermat”| x) = 0.129

(No retrieval.)
(Conditional probability with document z =“257 is . . . a Fermat prime.
Thus a regular polygon with 257 sides is constructible with compass . . . ”)
(Marginal probability, marginalizing over top 8 retrieved documents.)

ﬁne-tuning. The latter metric more signiﬁcantly isolates the
contribution of improving the retriever during pre-training.

Encoder or Retriever We ﬁrst aim to determine whether
REALM pre-training improves the retriever or the encoder,
or both. To do so, we can reset the parameters of either
the retriever or the encoder to their baseline state before
REALM pre-training, and feed that into ﬁne-tuning. Reset-
ting both the retriever and encoder reduces the system to
our main baseline, ORQA. We ﬁnd that both the encoder
and retriever beneﬁt from REALM training separately, but
the best result requires both components acting in unison.
REALM