F Long context fine-tuning

F.1 Further Discussion

For illustrating the effect of increasing the base period of rotary position embeddings, we plot expectations
for attention scores when varying the distance between key and query vectors in Figure 8a. Compared to
the default base period of 10,000, θ = 1, 000, 000 reduces the decay in attention scores, which helps far-away
tokens contribute to the current prediction. Notably, this change in rotation frequencies can be applied to
pretrained models, with loss curves stabilizing within a few gradient steps at a low learning rate. While the
uniform frequency scaling proposed by Chen et al. (2023b) is motivated by maintaining the overall range
of rotations when increasing the context from the sequence length used for pretraining, our modification
explicitly addresses the problem of performing attention over long distances.

F.2 Long context benchmarks
CodeLlama2