6.5. Loss is a poor proxy for solve rate
When ﬁnetuning AlphaCode models we observed that the validation language modelling loss starts
increasing after around 50k steps for an early training run of the 1B model, while the training loss
still decreases. This normally indicates overﬁtting. However, contrary to the validation loss, our target
metric solve rate continues to improve well past 50k steps as shown in Figure 13.
As discussed in Section 4.3, solving a problem is a one-of-many task, i.e. as long as one of many
samples solves a problem, the problem is considered solved. Our ﬁnetuning dataset CodeContests
contains many solutions per problem. Our main solve rate metric, 10@k, also uses 𝑘 samples rather
than a single sample. We hypothesize that the model reallocates probability mass from some atypical
solutions towards more typical solutions, leading to a worse validation loss overall, but a higher
probability of producing more typical solutions and therefore a better solve rate.
alphacode