2 TYING OUR HANDS BEHIND OUR BACK: A SETUP WITH LIMITED COMPUTE

Before we start this investigation, we want to outline the extent of limitations we are interested in.
The rules for cramming are as follows:

• A transformer-based language model of arbitrary size is trained with masked-language

modeling, completely from scratch.

• Existing pretrained models cannot be included in any part of the pipeline.
• Any raw text (excluding downstream data) can be included for training. This means that
one can achieve speedups by making judicious choices about how and when to sample data,
provided the sampling mechanism does not require a pre-trained model.

• The downloading and pre-processing of raw data is exempted from the total compute bud-
get. Pre-processing may include CPU-based tokenizer construction, tokenization, and ﬁl-
tering, but cannot include representation learning (e.g. pre-training a word embedding is
not allowed, unless it is counted towards the ﬁnal runtime).
CRAMMING-TRAININGALANGUAGEMODELONA SINGLEGPUINONEDAY