issues with using batch sizes 4× to 8× what is considered
standard for models with less than 1 billion parameters.
Consequently, we use a batch size of 1024 samples with a
sequence length of 2048 (2,097,152 tokens) for all models,
in order to maintain consistency across all Pythia model
training runs.

Model Size GPU Count GPT-3 GPUs

Speed-Up

70 M
160 M
410 M
1.0 B

32
32
32
64

4
8
8
16

8×
4×
4×
4×

Table 3. Models in the Pythia suite, number of GPUs used during
training, and the number of GPUs we would have been able to use
had we used the GPT-3 suite’s batch sizes. Due to the ability of
GPT-NeoX to scale linearly as the number of GPUs increases, this
produces substantial wall-clock speed-ups for small models. All
GPUs are A100s with 40 GiB VRAM.
Pythia- A Suite for Analyzing Large Language Models Across Training and Scaling