Task-Speciﬁc Heads The ﬁnal transformer layer
can be connected to multiple task speciﬁc heads.
In our experiments, we introduce two heads:
TokenPred and EntityPred.

The TokenPred head predicts masked tokens
for a cloze task. Each masked token’s ﬁnal repre-
sentation x4
i is fed to an output softmax over the
token vocabulary, as in BERT.

The EntityPred head predicts entity ids for
each entity mention span (i.e., entity linking). We
build the pseudo entity embedding (hmi) from the
last sequence output (X 4). Then, the model pre-
dicts the entity whose embedding in E is the closest
to the pseudo entity embedding.
Entities as Experts- Sparse Memory Access with Entity Supervision