4. Develop evaluations. Within the set of tasks deﬁned above, we focus on potential harms observable within the
context of the language model performing that task. This means focusing on extrinsic measures of harm within
each task, although where possible we draw connections with work on interpretability and probing internal
representations. These measures are a limited approximation, and are by no means exhaustive or holistically
complete (Raji et al., 2021; Selbst et al., 2019).

5. Analyze bias. We evaluate the model separately on measures that may serve as proxies for more general forms of
bias or potential harms. We measure whether these proxies are related to measures of potential harm with speciﬁc
downstream uses.
PaLM 2 Technical Report