Performance impact on short contexts. While our models are effective on long sequences, we observe
that LCFT slightly hurts performance on standard code synthesis benchmarks consisting of short sequences.
In Table 10, we observe an average decrease of 0.52 percentage points on HumanEval pass@1 and 1.9 points
on MBPP for the pass@1 metric. Similarly, a breakdown of the code completion results in Table 7 by the
number of tokens in each example shows that for prompts shorter than 4k tokens, long context fine-tuning
induces a reduction of up to 2 BLEU points from base models after code training (Figure 8b). We observe
similar decreases in performance for infilling tasks (Table 14).
LCFT comes at a cost for short sequences, and slightly decreases our scores on standard coding benchmarks
such as HumanEval and MBPP. However, many real-world use cases are not captured by these benchmarks,
and we believe that this cost is more than offset by the potential of handling long sequences for real
CodeLlama2