sufﬁcient for the downstream applications that we are targeting and simpliﬁes attention computa-
tions. Packing data into full sequences limits us to simpler sequence losses, but uses the available
compute optimally Liu et al. (2019); Izsak et al. (2021). For the targeted compute settings, this
sequence length results in micro-batch sizes of 64 to 96 for most variations of the base BERT ar-
chitecture on the gtx2080ti, which we will accumulate into larger batch sizes. With our limited
compute budget, this produces enough samples to run single-epoch training (Komatsuzaki, 2019;
Hernandez et al., 2022) where no data point is revisited.
CRAMMING-TRAININGALANGUAGEMODELONA SINGLEGPUINONEDAY