We focus on the problem of capturing declar-
ative knowledge about entities in the learned
parameters of a language model. We introduce
a new model—Entities as Experts (EAE)—
that can access distinct memories of the en-
tities mentioned in a piece of text. Unlike
previous efforts to integrate entity knowledge
into sequence models, EAE’s entity representa-
tions are learned directly from text. We show
that EAE’s learned representations capture suf-
ﬁcient knowledge to answer TriviaQA ques-
tions such as “Which Dr. Who villain has
been played by Roger Delgado, Anthony Ain-
ley, Eric Roberts?”, outperforming an encoder-
generator Transformer model with 10× the pa-
rameters. According to the LAMA knowledge
probes, EAE contains more factual knowledge
than a similarly sized BERT, as well as previ-
ous approaches that integrate external sources
of entity knowledge. Because EAE associates
parameters with speciﬁc entities, it only needs
to access a fraction of its parameters at infer-
Entities as Experts- Sparse Memory Access with Entity Supervision