6 Analysis of TriviaQA Results
6.1 Entity-Based Analysis
We compare the performances of retrieval-based
GR, generation-based T5-11B, and our EAE model
on the TriviaQA Unﬁltered-dev set. Figure 3 (a)
shows that all models perform better on frequent
entities. EAE’s performance is much lower for en-
tities seen fewer than one hundred times, likely be-
cause their entity embedding do not contain enough
information. Figure 3 (b) shows that as the number
of named entity mentions grows in the question,
T5 performance decreases whereas EAE and GR
performance increases. We presume more entity
mentions makes it easier to retrieve relevant docu-
ments, thus contributing to better performance for
GR. For EAE, having more entity mentions allows
the model to access more entity knowledge. We
conduct further qualitative analysis below. Figure 3
(c) shows that closed-book models under-perform
on long questions. The trend disappears for open-
book models.
Entities as Experts- Sparse Memory Access with Entity Supervision