B Additional Ablation Results

In Table 10 we report pass@1, pass@10, and pass@100 scores, for models with and without both infilling
(FIM) and long-context fine-tuning (LCFT). Results are reported for 7B, 13B, and 34B parameter models.
For the pass@1 we use greedy decoding, while for pass@10 and pass@100 we use temperature of 0.8, N = 200,
using nucleus sampling with p = 0.95.

To measure math-reasoning capabilities of the proposed method, we report results on the GSM8K bench-
mark Cobbe et al. (2021), which is comprised of a set of middle-school math word problems. Results are
summarised on Table 12.

C Math reasoning results

D Infilling
CodeLlama2