The base initialization standard deviation
—

Layer width multiplier:

dmodel/dmodel,base

Emdedding output multiplier

memb
Empirically Tuned Values

Must tune for each model size

—

—

0.02
—

256
6e-3
0.08
10.0

Yemb = embed(X)

softmax(cid:16) QT K√

(cid:17)

V
Wqkv ∼ Ntrunc(0, σ2
base)

dhead

ηqkv = ηbase
Wo ∼ Ntrunc(0, σ2
Wﬀn1 ∼ Ntrunc(0, σ2

ηo = ηbase

base

2·nlayers )

base)

ηﬀn1 = ηbase

Wﬀn2 ∼ Ntrunc(0, σ2

base

2·nlayers )

ηﬀn2 = ηbase

Ylogits = WunembX

Wemb ∼ Ntrunc(0, σ2

ηemb = ηbase
Wγ ∼ 1, bβ ∼ 0
ηLN = ηbase

b ∼ 0

ηb = ηbase

base)
Yemb = memb · embed(X)

softmax(cid:16) QT K

dhead

(cid:17)

V

Wqkv ∼ Ntrunc(0, σ2

base/mwidth)

2mwidth·nlayers )

ηqkv = ηbase/mwidth

σ2
base
ηo = ηbase/mwidth

Wo ∼ Ntrunc(0,
Wﬀn1 ∼ Ntrunc(0, σ2

Wﬀn2 ∼ Ntrunc(0,

ηﬀn1 = ηbase/mwidth
σ2
base
ηﬀn2 = ηbase/mwidth

base/mwidth)

2mwidth·nlayers )

Ylogits = WunembX/mwidth

Variables

W
b
X, Y
dmodel,base
dmodel
dhead
embed
ηbase
σbase
mwidth
Cerebras-GPT- Open Compute-Optimal Language Models Trained on the Cerebras Wafer-Scale Cluster