The value of model specialization. We observe that model specialization is yields a boost in code
generation capabilities when comparing Llama 2 to Code Llama and Code Llama to Code Llama -
Python. Llama 2 was trained on 2T tokens, and training on only 500B of extra tokens from a code-heavy
dataset results in massive performance gains on both HumanEval and MBPP, to the point that Llama 2
70B is roughly equivalent to Code Llama 7B on Python coding benchmarks. Although Code Llama was
trained on more than two epochs of our code dataset, which contains our entire Python dataset, training
on 100B extra tokens of a Python-heavy data mix leads to significant gains on Python code generation
benchmarks, between 4.3% points and 8.3% points in HumanEval pass@1 and between 1.2% points and
6.4% points in MBPP pass@1. These gains are smaller than for the first code training step, but still allow
Code Llama - Python 7B to outperform even Code Llama 13B on MBPP and HumanEval. For the APPS
CodeLlama2