Masked language modeling (MLM) pre-training methods such as BERT corrupt the input by replacing some tokens with [MASK]  and then train a model to reconstruct the original tokens. While effective when transferred to downstream NLP tasks, they generally require large amounts of compute to be effective. As an alternative, we propose a more sample-efficient pre-training task called replaced token detection. Instead of masking the input, our approach corrupts it by replacing some tokens with plausible alternatives sampled from a small generator network. Then, instead of training a model that predicts the original identities of the corrupted tokens, we train a discriminative model that predicts whether each token in the corrupted input was replaced by a generator sample or not. Thorough experiments demonstrate this new pre-training task is more efficient than MLM because the task is defined over all input tokens rather than just the small subset that was masked out. As a result, the contextual representations learned by our approach substantially outperform the ones learned by methods such as BERT and XLNet given the same model size, data, and compute. The gains are particularly strong for small models; for example, we train a model on one GPU for 4 days that outperforms GPT (trained using 30x more compute) on the GLUE natural language understanding benchmark. Our approach also works well at scale, where we match the performance of RoBERTa while using less than 1/4 of the compute.
The authors propose replaced token detection, a novel self-supervised task, for learning text representations.
The principle advantage of the approach is that, in contrast with the standard masked language model (MLM) objective used by BERT and derivatives, there is a training signal for all tokens of the input (rather than a small fraction, when 10-20% of the input tokens are masked and then reconstructed under the MLM objective).
A smaller MLE-trained BERT-style generator is used to replace masked words with plausible alternatives, which the ELECTRA discriminator (the part that is retained and finetuned on downstream tasks) must detect (unmodified word slots are also in the objective, and must be detected as such).
In general the paper reads well, and the authors present ablations to reveal the source of gains. ELECTRA matches the performance of RobBERTa on the popular GLUE NLP task, with just 1/4 of the training compute.
Strengths:
-Simple but novel self-supervised task for learning text representations, strong results, adequate ablation.
Limitations:
-The authors limit their investigation of downstream performance to the GLUE set of tasks, which are classification tasks. This is a significant limitation of the current version of the paper, as it may be that replaced token detection is more suitable for these tasks, but inferior to MLM (a higher precision self-supervised task) for more involved tasks like question answering. The latter is arguably of much higher importance to the NLP research community at this point, and some consider the GLUE task to be essentially solved for all practical purposes, given inherent noise levels.
-In contrast with BERT, there is no mention of any plan to release ELECTRA (big or small versions), which is a disappointment, lowers the significance of the work
Overall:
An okay paper. Results on SQUAD or another more elaborate NLP task and/or the release of the ELECTRA models would make the paper much stronger.