2.6 Training details
Optimization. Our optimizer is AdamW (Loshchilov & Hutter, 2019) with β1 and β2 values of 0.9 and
0.95. We use a cosine schedule with 1000 warm-up steps, and set the final learning rate to be 1/30th of the
peak learning rate. We use a batch size of 4M tokens which are presented as sequences of 4,096 tokens each.
Despite the standard practice of using lower learning rates in fine-tuning stages than in pre-training stages,
we obtained best results when retaining the original learning rate of the Llama 2 base model. We carry
these findings to the 13B and 34B models, and set their learning rates to 3e−4 and 1.5e−4, respectively. For
python fine-tuning, we set the initial learning rate to 1e−4 instead. For Code Llama - Instruct, we train
with a batch size of 524,288 tokens and on approx. 5B tokens in total.
Long context fine-tuning. For long context fine-tuning (LCFT), we use a learning rate of 2e−5, a sequence
CodeLlama2