that can access distinct and independent representa-
tions of the entities mentioned in text. Unlike other
efforts to inject entity speciﬁc knowledge into se-
quence models (Peters et al., 2019; Zhang et al.,
2019; Poerner et al., 2019) our model learns entity
representations from text along with all the other
model parameters. We call our model Entities as
Experts (EAE), since it divides the parameter space
according to entity identity. This name also reﬂects
EAE’s similarities with the Massive Mixture of Ex-
perts (Shazeer et al., 2017), as well as other work
that integrates learned memory stores into sequence
models (Weston et al., 2014; Lample et al., 2019).
To understand the motivation for distinct and in-
dependent entity representations, consider Figure 1.
A traditional Transformer (Vaswani et al., 2017)
needs to build an internal representation of Charles
Darwin from the words “Charles” and “Darwin”,
both of which can also refer to different entities
Entities as Experts- Sparse Memory Access with Entity Supervision