Code Generation We benchmark PaLM 2 on 3 coding datasets: HumanEval (Chen et al., 2021), MBPP (Austin et al.,
2021), and ARCADE (Yin et al., 2022). HumanEval and MBPP are natural language to code datasets which test the
model’s ability to generate self-contained Python programs that pass a set of held-out test cases. ARCADE is a Jupyter
Notebook completion task that requires the model to complete the next cell in a notebook given a textual description and
the preceding notebook cells. As in (Chen et al., 2021; Austin et al., 2021; Yin et al., 2022), we benchmark models in a
pass@1 and pass@k setting. We use greedy sampling for all pass@1 evals and temperature 0.8 with nucleus sampling
p = 0.95 for all pass@k evals. All samples are executed in a code sandbox with access to a small number of relevant
modules and careful isolation from the system environment. For ARCADE, we use the New Tasks split containing
problems from newly curated notebooks to avoid evaluation data leakage.
PaLM 2 Technical Report