79

Figure 36: Disaggregated analysis of TPC across variations of the task, models, and ﬁnal PaLM 2 checkpoints.

Finally, we evaluated PaLM 2 (L) continuously during training with a “tiny” variant of the task using only 500 examples
sampled from the small variant. Performance throughout training was stable, with mean TPC of 0.0676±0.0026, and
worst-case performance of 0.0725, as measured over 87% of training epochs (excluding initial epochs and unrelated
infrastructure issues). Notably, this is a slightly lower level than the evaluation on the “small” variation.

Figure 37: Toxicity Probability of Continuation throughout training, “tiny” variant (n=500).
PaLM 2 Technical Report