Scaling laws hold in the low-resource regime A large corpus of research in recent years has
developed architectural improvements to speed up the original transformer. Many of these methods
have not been found to improve training for the large-scale T5 architecture Narang et al. (2021);
Tay et al. (2022a). But, in the low compute setting where data throughput is of utmost importance,
maybe this is the way forward? Scaling laws have been observed by Kaplan et al. (2020) in the high-
resource regime, and seem to hold strongly in the limit as resources grow. Surprisingly, these laws
also hold in the limit of extreme compute down-scaling, and they create a barrier to low-cost training.
We exemplify the effect of scaling laws for many transformer variants from the literature in Figure 1,
where we train each architecture variant with optimized training hyperparameters as described below
in Section 4.3. We apply these architecture variants to a shared baseline model that incorporates

5
CRAMMING-TRAININGALANGUAGEMODELONA SINGLEGPUINONEDAY