In more detail, we ﬁrst perform a similar proce-
dure to Lewis et al. (2020b) and discard questions
in the ﬁne-tuning training data that contain answers
which overlap with answers to questions in the dev
and test data. We end up with 9144/2308/3996 data
(train/dev/test) in FreebaseQA and 1348/151/1639
data in WebQuestionsSP. Next, to ensure that the
model will be unable to memorize paraphrases of
question-answer pairs that it observed in the pre-
training text, we remove all overlap between the
pretraining data and ﬁne-tuning test data: speciﬁ-
cally, for every question-answer entity pair in our
ﬁne-tuning dataset (from any split), we ﬁlter every
example from our Wikipedia pretraining corpus in
which that pair of entities co-occur. Additionally,
we ﬁlter every fact from our fact memory contain-
ing any of these entity pairs.
Adaptable and Interpretable Neural Memory Over Symbolic Knowledge