GPT-Neo (Black et al., 2021; Wang & Komatsuzaki, 2021; Black et al., 2022) These models strictly speaking do not
form a suite and have some non-negligible differences between them with respect to model architecture implementation,
training codebase, tokenizer, and training data setup and order. Despite that, they are commmonly used as if they were a
consistent model suite.

OPT (Zhang et al., 2022) While more checkpoints of OPT models exist (as is seen by their use in Xia et al. (2022)) they
largely are not publicly available (less than 10 checkpoints available, only for the 2.7b, 6.7b, and 13b parameter models).
Additionally, the training dataset for OPT is not public.

T5 (Raffel et al., 2020) The original paper did not release its training data, but it did release code for producing it which
was subsequently run and released by Dodge et al. (2021).
Pythia- A Suite for Analyzing Large Language Models Across Training and Scaling