Table 8: Additional raw results for experiments considered in the main body. This table contains architec-
ture variants for a prelimary architecture setup which contained 4 heads in the attention block, 12 layers and
included rotary embeddings. First two blocks: Architectural variants as discussed in Section 4.2 (but for this
preliminary variant). Third block: Ablation study of this model. All experiments run with the training setup
described in Section 4.3 for a day on a single GPU with mixed precision. Batch size is 4032 and dataset is
bookcorpus-wikipedia. Downstream evaluation as described in Section 5. All values for pretraining on
an A4000.

25

Preprint
CRAMMING-TRAININGALANGUAGEMODELONA SINGLEGPUINONEDAY