13

Preprint

Kevin Clark, Minh-Thang Luong, Quoc V. Le, and Christopher D. Manning. ELECTRA: Pre-
training Text Encoders as Discriminators Rather Than Generators. In International Conference
on Learning Representations, September 2019. URL https://openreview.net/for
um?id=r1xMH1BtvB.

Kevin Clark, Minh-Thang Luong, Quoc V. Le, and Christopher D. Manning. Pre-Training Trans-
formers as Energy-Based Cloze Models. arXiv:2012.08561 [cs], December 2020. URL http:
//arxiv.org/abs/2012.08561.

Zihang Dai, Guokun Lai, Yiming Yang, and Quoc Le. Funnel-Transformer: Filtering out Sequential
Redundancy for Efﬁcient Language Processing. In Advances in Neural Information Processing
Systems, volume 33, pp. 4271–4282. Curran Associates, Inc., 2020. URL https://papers
.nips.cc/paper/2020/hash/2cd2915e69546904e4e5d4a2ac9e1652-Abstra
ct.html.
CRAMMING-TRAININGALANGUAGEMODELONA SINGLEGPUINONEDAY