050000100000150000200000Steps0.00.20.40.6WeightPile-CCPubMed CentralBooks3OpenWebText2ArXivGithubFreeLawStackExchangeUSPTO BackgroundsPubMed AbstractsGutenberg (PG-19)OpenSubtitlesWikipedia (en)DM MathematicsUbuntu IRCBookCorpus2EuroParlHackerNewsYoutubeSubtitlesPhilPapersNIH ExPorterEnron Emails050000100000150000200000Steps0.00.20.40.6WeightPile-CCPubMed CentralBooks3OpenWebText2ArXivGithubFreeLawStackExchangeUSPTO BackgroundsPubMed AbstractsGutenberg (PG-19)OpenSubtitlesWikipedia (en)DM MathematicsUbuntu IRCBookCorpus2EuroParlHackerNewsYoutubeSubtitlesPhilPapersNIH ExPorterEnron EmailsTable 8: Architecture hyperparameters for various model scales used in the paper. All models are
vanilla Transformer decoder-only models and use vocabulary size 256k.

Layers Attention heads Attention head dim Model dim Hidden dim
1024
2048
3072
8192
8192
8192
24576

256
512
768
1024
1280
2048
4096

64
64
64
64
64
64
128

4
8
12
16
20
32
32

3
6
12
12
12
16
32

70M
150M
280M
510M
760M
1B
8B
DoReMi- Optimizing Data Mixtures Speeds Up Language Model Pretraining