49.0%
62.0%
73.0%

Table 6: Multilingual HumanEval single line infilling with MultiPL-E. Exact match rates on the line
infilling benchmark from Allal et al. (2023) with greedy decoding. Evaluated in both prefix-suffix-middle
(PSM) and suffix-prefix-middle (SPM) format. Numbers for InCoder, SantaCoder and StarCoder are reported
from Li et al. (2023).

(a)

(b)

Figure 4: Code Llama behavior on long sequences. (a) Perplexity on large source files (≥50 kB) from
the validation data from the code dataset. The dashed line marks the fine-tuning context length. Perplexity
decreases for up to 100K tokens for all Code Llama sizes. (b) Accuracy on a synthetic key retrieval task,
with a context of 16K tokens and comparison to gpt-3.5-turbo.
CodeLlama2