4 Models Evaluated
We evaluate EAE on cloze knowledge probes, open-
domain question answering and relation extraction.
Here, we describe baselines from previous work
and ablations.

4.1 The Entities as Experts models
Our primary model, Entities As Experts embeds
the input using the token embedding layer, then
passes it through 4 transformer layers. This output
is used for the entity memory layer, which uses em-
beddings of size 256, then passed through an addi-
tional 8 transformer layers. Finally, the model has a
TokenPred head and an EntityPred head. In
EAE, hyper-parameters for the transformer layers
are identical to those in BERT-base (Devlin et al.,
2018).
Sparse Activation in EaE EAE only accesses
the entity embedding matrix for mention spans,
and we only retrieve k = 100 entity memories for
each mention (see Appendix 6.3 for analysis of this
Entities as Experts- Sparse Memory Access with Entity Supervision