Table 3: Summary of per-domain perplexities on The Pile (22 total domains). Average perplexity is
an unweighted average of the per-domain perplexities.

(a) Varying the size of the proxy/reference model and
training at 8B.

Worst-case

# domains
besting baseline

Baseline (8B)
DoReMi (70M->8B)
DoReMi (150M->8B)
DoReMi (280M->8B)
DoReMi (1B->8B)

pplx Avg pplx
1.64
1.71
1.53
1.63
1.52
1.56
1.40
1.46
1.58
1.54

(b) Perplexity of the DoReMi main model and proxy model
of the same size. Although the 1B proxy model is relatively
poor quality, the resulting domain weights still improve
the main model.

0/22
22/22
22/22
22/22
22/22

Baseline (280M)
DoReMi (280M->280M)
Proxy (280M)
Baseline (1B)
DoReMi (1B->1B)
Proxy (1B)

Worst-case

# domains
besting baseline

pplx Avg pplx
2.32
2.39
2.13
2.19
2.27
2.33
1.87
1.94
1.92
1.83
2.02
2.11

0/22
22/22
19/22
0/22
19/22
0/22
DoReMi- Optimizing Data Mixtures Speeds Up Language Model Pretraining