(cid:2)λz(x)2 + 2λz(x)nzp∗(x | z) + nzp∗(x | z) + (n2

i=1

z − nz)p∗(x | z)2(cid:3)

Putting it all together, the parameter error can be written as

(cid:88)

x

E[(ˆθz(x) − p∗(x | z))2] =

(cid:88)
(cid:88)

x

z − nz)p∗(x | z)2 + λz(x)2 + (nz − 2szλz(x))p∗(x | z)
(s2

(nz + sz)2

(cid:16)

(cid:17)2

nzp∗(x | z)(1 − p∗(x | z)) + s2
z
(nz + sz)2

p∗(x | z) − λz(x)

sz

=

=

x
nzHz + s2
(nz + sz)2

z∆z

.

(2)

(3)

(4)

(5)

(6)

(7)

(8)

(9)

(10)

(11)

(12)

(13)

(cid:35)

22

No-tradeoff example. Suppose there are 3 domains z ∈ {1, 2, 3} and m = 3 vocabulary tokens
x ∈ {1, 2, 3}. We use a symmetric Dirichlet prior (preferring a uniform token distribution) where
x λz(x) = 1. In this setting, we show that
there is a set of domain weights that has strictly lower parameter error than the baseline where we
sample the same number of tokens from each domain: nz are equal for all domains z.
Suppose the ground truth paramaters for the unigram distributions are
DoReMi- Optimizing Data Mixtures Speeds Up Language Model Pretraining