j + WT

= h(l)

(1)

(2)

(3)

After the ﬁnal transformer layer T , h(T )

mi is used
to predict the context entities ˆemi and produce a
loss with Iemi
, the one-hot label of entity emi. Fol-
lowing Févry et al. (2020), we supervise the entity
access for the intermediate query vector in Eq. 1.

ˆemi = argmaxei∈E (cT

miei)

)

)
Adaptable and Interpretable Neural Memory Over Symbolic Knowledge