7

Table 3: Code Llama pass@ scores on APPS. We list the two-shot pass@5, pass@10, and pass@100
scores of Code Llama on APPS. For our models, we use nucleus sampling with p=0.95 and a temperature
of 0.6. Code Llama is not fine-tuned on the training set of APPS and all results are calculated with raw
predictions without filtering by the test cases from the prompt. Fine-tuned GPT-Neo numbers are reported
by Hendrycks et al. (2021), one-shot Codex results by Chen et al. (2021), and fine-tuned AlphaCode numbers
by Li et al. (2022).

3.1 Code generation

3.1.1 Python code generation
CodeLlama2