Proceedingsofthe2021ConferenceoftheNorthAmericanChapteroftheAssociationforComputationalLinguistics:HumanLanguageTechnologies,pages3678–3691June6–11,2021.©2021AssociationforComputationalLinguistics3678Figure 1: Fact Injected Language Model architecture. The model takes a piece of text (a question during ﬁne-
tuning or arbitrary text during pre-training) and ﬁrst contextually encodes it with an entity enriched transformer.
FILM uses the contextually encoded MASK token as a query to the fact memory. In this case, the contextual query
chooses the fact key (Charles Darwin, born_in) which returns the set of values {United Kingdom} (The value set
can be multiple entity objects such as the case from calling the key [United Kingdom, has_city]) . The returned
object representation is incorporated back into the context in order to make the ﬁnal prediction. Note that the entity
representations in the facts (both in keys and values) are shared with the entity memory. The portion within the
Adaptable and Interpretable Neural Memory Over Symbolic Knowledge