(2021) work in the same setting and evaluate the optimal shape of T5 derived architectures and its
relative effects on downstream performance as models are scaled. Further exploration of the scaling
behavior of various architectural improvements in Tay et al. (2022a) ﬁnd that only few modiﬁcations
outperform the original architecture of Vaswani et al. (2017) at all scales, especially when evaluating
downstream accuracy. The meta-study investigating improvements in preparation for extreme-scale
training in Scao et al. (2022) focuses on minor modiﬁcations to layout, positional embeddings and
data sources for autoregressive models, and other extremely-large scale training runs have so far
been similarly conservative in their settings (Brown et al., 2020; Black et al., 2022; Rae et al., 2022).
In general though, these evaluations target larger compute settings than we intend to use, and are
concerned with whether improvements (often from academic sources and proposed with evaluations
CRAMMING-TRAININGALANGUAGEMODELONA SINGLEGPUINONEDAY