Verbatim memorization We ﬁrst evaluate the training data extraction rate of PaLM 2 compared to PaLM on a shared
part of the English pre-training data. To do this, we follow a similar protocol to that of Chowdhery et al. (2022) and
sample 10,000 unique documents from this shared data. Then, we prompt the model with the ﬁrst P = 50 tokens from
each document with the target sufﬁx being the next S = 50 tokens. For our analysis, we use three comparable model
sizes at different scales, “small”, “medium”, and “large”, for both PaLM and PaLM 2. In Figure 8 (a) we compare

21

SmallMediumLarge(a) Model Size012345% Verbatim MemorizedModel FamilyPaLMPaLM 2110100(b) Number of Repetitions0.1110% Verbatim MemorizedModel FamilyPaLM 2PaLMSmallMediumLargeModel SizeTable 13: Distribution of canaries across languages.

Dataset

Size

Languages
Included

Canary Types

Included

Total Canaries per
Type per Language

shufﬂe

interleave

3090

Canaries per # of Repetitions

1

5

40

20

10

20

25

20

50
PaLM 2 Technical Report