2.3 Entity Memory
Our entity memory E ∈ R|E|×de is a matrix contain-
ing a vector for each entity in E and trained as an
entity-masked LM. The model input is a text span
containing unlinked entity mentions with known
boundaries3. Mentions are masked with some prob-
ability. Our entity memory follows Entity as Ex-
perts (EaE) (Févry et al., 2020) which interleaves
standard Transformer (Vaswani et al., 2017) layers
with layers that accesss the entity memory4.
Given a piece of text q = {w1, . . . , w|q|} the
is the output at the i’th

contextual embedding h(l)
i

2Although we use the term paragraph here, in our exper-
iments we use spans of 128 tokens, which need not follow
paragraph boundaries.

3Févry et al. (2020) also showed the model is capable of
learning to predict these boundaries. For simplicity, in this
work we assume they are given.

4We follow the implementation of Févry et al. (2020) and
have a single entity memory access between the fourth and
ﬁfth transformer layers.
Adaptable and Interpretable Neural Memory Over Symbolic Knowledge