Introducing the entity memory layer in the mid-
dle of the transformer stack (EAE) improves per-
formance on both language modeling and entity
linking, compared to the No-EAE model. This
indicates that the entity representations are being
used effectively, and that the model is learning
inter-entity relations, using the output from the en-
tity memory layer to improve predictions at the
downstream entity and token prediction layers.

If the memory layer is not supervised (EAE-
unsup), performance on token prediction accuracy
and perplexity is signiﬁcantly worse than for EAE.
This underlines the importance of entity linking
supervision in teaching EAE how to best allocate
the parameters in the entity memory layer.

Finally, EAE performs better at predicting men-
tion tokens than the 24-layer MM-large, but does
marginally worse in terms of perplexity. We believe
that EAE is overconﬁdent in its token predictions
when wrong, and we leave investigation of this
phenomenon to future work.
Entities as Experts- Sparse Memory Access with Entity Supervision