Peters, M. E., Neumann, M., IV, R. L. L., Schwartz, R.,
Joshi, V., Singh, S., and Smith, N. A. Knowledge en-
hanced contextual word representations, 2019.

Petroni, F., Rockt¨aschel, T., Lewis, P., Bakhtin, A., Wu,
Y., Miller, A. H., and Riedel, S. Language models as
arXiv preprint arXiv:1909.01066,
knowledge bases?
2019.

Radford, A., Narasimhan, K., Salimans, T., and Sutskever,
I. Improving language understanding with unsupervised
learning. Technical report, OpenAI, 2018.

Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., and
Sutskever, I. Language models are unsupervised multitask
learners. OpenAI Blog, 2019.

Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S.,
Matena, M., Zhou, Y., Li, W., and Liu, P. J. Exploring
the limits of transfer learning with a uniﬁed text-to-text
transformer. arXiv preprint arXiv:1910.10683, 2019.
REALM