Table 26: Theory of mind questions are often challenging for LaMDA, but with one example LaMDA gains some
understanding and answers a similar question correctly.

46

D Model hyper-parameters

Table 27: Hyper-parameters for pre-training 2B, 8B and 137B models. All models were trained with 256K tokens per
batch.

Parameters
2B
8B
137B

Layers

10
16
64

Units
2560
4096
8192

Heads

pre-train steps

pre-train chips

pre-train time (days)

ﬁne-tune chips

ﬁne-tune time (hours)

40
64
128

501k
521k
3M

64
64
1024

1.5
23
57.7

16
16
64

3
6
36

E Pre-training data composition
LaMDA- Language Models for Dialog Applications