F.2 Long context benchmarks

Synthetic Key Retrieval Task. We prompt the model with a variable number of tokens by concatenating
Python solutions from the CodeContest dataset (Li et al., 2022), which results in syntactically valid source
code. At a specified relative position within the prompt, we insert the following key, where <VALUE> is a
two-digit number that is randomly sampled based on the overall number of tokens in the prompt:

30

050001000015000200002500030000Relative Distance050100150200250Attention Expectation=104=106<44-88-1212-1616-2424-3232File Context Length x10³420246BLEU DifferenceCode Completion (LCC-balanced)7B13B34BLanguage

Average

Code Tokens
25% 50% 75%

Code Llama Tokens

Average

25% 50%

75%

LCC test set
Python
Java
C#
LCC-balanced
Python
Java
C#

1992.7
1904.6
2005.5

6954.8
7243.1
7458.3

1055
1083
1037

3249
3491
3503

1438
1437
1418

6532
6827
7048

2211
2061
2184

4689.1
4029.8
4378.6

10371
10128
10914

17791.1
16567.1
16971.1

2552
2347
2346
CodeLlama2