2211
2061
2184

4689.1
4029.8
4378.6

10371
10128
10914

17791.1
16567.1
16971.1

2552
2347
2346

8915
8728
8560

3300
2953
3072

5068
4247
4647

16775
15465
16038

24957
22854
23830

Table 16: LCC dataset statistics for different subsets. We compare the original test set from (Guo et al.,
2023) to our resampled “LCC-balanced” test set. Code tokens are determined by parsing the completion
context with tree_sitter.

Model

Code Llama
Code Llama
Code Llama
Code Llama - Instruct
Code Llama - Instruct
Code Llama - Instruct
gpt-3.5-turbo-16k-0630

Size

0

7B
100.0
13B 100.0
76.6
34B
7B
100.0
13B 100.0
34B
92.2
100.0
-

8,000
0.2
95.3
100.0
100.0
97.7
100.0
100.0
100.0

Context Length / Key Position

0.4
100.0
100.0
100.0
100.0
100.0
100.0
95.3

0
54.7
100.0
95.3
7.0
100.0
68.8
95.3

16,000

0.2
100.0
100.0
96.9
96.9
100.0
95.3
90.6

0.4
98.4
100.0
100.0
96.1
93.8
100.0
98.4

24,000

0.2
85.9
89.1
0.0
62.5
84.4
0.0
-

0
3.1
100.0
81.3
0.0
4.7
46.9

-

0.4
85.9
6.3
81.3
54.7
100.0
85.9

-
CodeLlama2