It is clear that learning entity representations is
beneﬁcial. However, initializing the entity memory
with Deep-Ed embeddings leads to gains on most
of the knowledge probing tasks, suggesting that
there are potential beneﬁts from combining train-
ing regimens. Meanwhile, the best entity typing
results come from initializing with Wikidata em-
beddings, possibly because Wikidata has high qual-
ity coverage of the types (‘person’, ‘organization’,
‘location’, etc.) used. Finally, we point out that
both ERNIE and KNOWBERT differ from EAE in
other ways (see Appendix A). KNOWBERT in par-
ticular incorporates WordNet synset embeddings as
well as entity embeddings, leading to entity typing
accuracy of 76.1. Future work will explore differ-
ent ways of combining learned embeddings with
knowledge from diverse external sources.
Entities as Experts- Sparse Memory Access with Entity Supervision