Wenhan Xiong, Jingfei Du, William Yang Wang, and
Veselin Stoyanov. 2019a. Pretrained encyclopedia:
Weakly supervised knowledge-pretrained language
model. arXiv preprint 1912.09637.

Fabio Petroni, Tim Rockt¨aschel, Patrick Lewis, Anton
Bakhtin, Yuxiang Wu, Alexander H Miller, and Se-
bastian Riedel. 2019. Language models as knowl-
edge bases? arXiv preprint 1909.01066.

Wenhan Xiong, Jingfei Du, William Yang Wang, and
Veselin Stoyanov. 2019b. Pretrained encyclopedia:
Weakly supervised knowledge-pretrained language
model. arXiv preprint 1912.09637.

Zichao Yang, Phil Blunsom, Chris Dyer, and Wang
Reference-aware language models.

Ling. 2016.
arXiv preprint 1611.01628.

Yuhao Zhang, Victor Zhong, Danqi Chen, Gabor An-
geli, and Christopher D Manning. 2017. Position-
aware attention and supervised data improve slot ﬁll-
ing. In EMNLP.
Entities as Experts- Sparse Memory Access with Entity Supervision