Even though entity prediction approach cannot
answer 15% of the data with unlinked answers for
TriviaQA, and 9% for WebQuestions, it outper-
forms all of the standard T5 models including one
that has 30× the parameters. This indicates that the
entity speciﬁc model architecture is more efﬁcient
in capturing the sort of information required for this
knowledge probing task than the general encoder-
decoder architecture used by T5. However, when
T5 is enhanced with an extra pre-training steps
focusing on likely answer spans from Wikipedia
(T5-11B + SSM) its performance leapfrogs that of
EAE. We note that the ‘salient spans’ included in
the SSM objective are likely to be entities (Guu
et al., 2020), and believe that there is signiﬁcant
future work to be done in combining methods of
entity prediction and text generation.

Though closed-book approaches are still be-
hind open-book approaches on TriviaQA, we be-
Entities as Experts- Sparse Memory Access with Entity Supervision