F.3 Extended Results
In Table 17, we list performance on our synthetic key retrieval task (Appendix F.2) for all Code Llama
models. While our models generally show strong performance for up to 16K tokens even after instruction
fine-tuning, Code Llama - Instruct 7B fails to retrieve keys placed at the start of the prompt for a
prompt length of 16K. With prompts longer then 16K tokens, we observe a decline in retrieval accuracy across
all models. GPT-3.5-Turbo (16K) exhibits small performance decreases with 16K token prompts, which
corresponds to a prompt length of 12K tokens with the GPT-3.5 tokenizer. 24K token prompts surpass the
limits of the API to GPT-3.5-Turbo.

31

Configuration

4,000
0.2

0.4

0

Context Length / Key Position
16,000
0.2

8,000
0.2

0.4

0

0

After code-training
95.3
θ = 104
θ = 106
95.3
Long context fine-tuning
θ = 104
33.6
100.0
freq. scaling 1/4
Ours (θ = 106)
95.3

100.0
100.0

100.0
100.0

0.0
0.0

93.0
100.0
95.3

97.7
100.0
100.0

0.0
100.0
100.0
CodeLlama2