0
3.1
100.0
81.3
0.0
4.7
46.9

-

0.4
85.9
6.3
81.3
54.7
100.0
85.9

-

Table 17: Function Key Retrieval Accuracy (%) for Code Llama models.

def my_function() -> int:

"""Note that this function is used at the end
"""
return <VALUE>

We finish the prompt with “assert my_function() == ”. Accuracy is measured over 64 distinct examples
for each combination of prompt length and key position depending on whether it generated the correct value
or not.

LCC-balanced. The distribution of source file lengths in the LCC test and validation sets is heavily skewed
towards shorter files (Table 16). To better test the behavior of our models on long context, we resample data
points from the validation and test sets. This results in a corpus of 548, 412 and 512 data points for Python,
Java and C#, respectively.
CodeLlama2