Memorization of the tail Though multilinguality presents many beneﬁts to PaLM 2, it also presents a new risk for
memorization: data is often scarcer and of poorer quality (Kreutzer et al., 2022). Due to their presence in the tail of the
overall data distribution, it is plausible that the model may memorize this data (Feldman, 2019). Here, we analyze the

22

(a) Canary extraction rate. We use all available canaries inserted for a
language.

(b) Training data extraction rate.

Figure 9: Outlier ‘canaries’ need fewer repetitions to be memorized in the tail. But, on average, we observe no strong
correlation that under-represented languages may be memorized more than well-represented languages. Often, we
observe that lower resource languages have less memorization for unique (single repetition) sequences.
PaLM 2 Technical Report