of large-compute pretraining, then this opens up a host of further academic investigations that
are currently hard to realize for large-scale models. For example, research questions about the
differences between existing and new pre-training tasks, tracing model predictions to data points
(Ilyas et al., 2022), security questions such as membership inference (Carlini et al., 2022) and data
poisoning (Geiping et al., 2021), and a wide range of empirical investigations into topics such as
stability or generalization that arise during training (Nagarajan & Kolter, 2019; Jiang et al., 2019).
At the same time, we can imagine situations in which legal requirements make it unclear whether
models trained on public data with uncertain origin are permissible, and where a practitioner is
interested in retraining their language models using a specialized or trustworthy data source (Wilka
et al., 2017; Gold & Latonero, 2017).
CRAMMING-TRAININGALANGUAGEMODELONA SINGLEGPUINONEDAY