5.2 WHAT HAPPENS WHEN TRAINING LONGER?

We also verify what happens if the cramming recipe discussed so far is used with more budget. To
this end, we train models for 48 hours on 8 A6000 GPUs, which ends up to be 208 total exaFLOP,
c.f. Table 1. We directly apply the setting described so far, simply scaling the learning rate schedules
to cover the new budget of 48 hours. In Table 6, we ﬁnd that the discussed recipe does immediately
generalize to larger compute budgets. This is surprising, not the least, as now, the dataset (which
was sorted in Section 4.4 is now too small and repeated multiple times. The newly trained models
have strong performances, especially on MNLI and SST-2, where they signiﬁcantly outperform the
original BERT checkpoint and fall into a similar range as the roBERTA-base checkpoint of Liu et al.
(2019), which was trained with much more compute. Yet, in other tasks, such as (again) CoLA, the
new models barely improve even in the larger compute regime.

6 LIMITATIONS
CRAMMING-TRAININGALANGUAGEMODELONA SINGLEGPUINONEDAY