Initial Data Setup We start our investigation with a close analogue to the original raw text sources
of Devlin et al. (2019), using a recent dump of the English Wikipedia (20220301.en) and En-

4

Preprint

Figure 1: Various Transformer architectures and shapes, showing MLM loss versus number of tokens ingested.
Left: Global view. Right: Zoom onto 10e8 or more tokens. All models trained with the same budget. We see
that improvements through architectural reshaping are minimal; while there are some ﬂuctuations in loss early
in training, the rates of loss decay during most of training differ by a multiplicative constant (horizontal shift
due to logarithmic horizontal axis) that depends strongly on the model size and not model type.
CRAMMING-TRAININGALANGUAGEMODELONA SINGLEGPUINONEDAY