method. We therefore followed the settings we used for pre-training on GitHub, and ﬁne-tuned our
pre-trained models on the APPS training set without using clustering, tags, ratings, value conditioning,
or prediction, and with sampling temperature 0.25 and nucleus sampling. Other settings were the
same as our main models.
Table 10 compares our model with existing large language models ﬁne-tuned on this dataset as
reported by Hendrycks et al. (2021), as well as the 1-shot performance of the Codex model reported
by Chen et al. (2021). A small 1B parameter model already outperforms the GPT-NEO baseline on
all diﬃculty levels, and outperforms Codex 12B on the interview and competition diﬃculty levels.
We highlight that AlphaCode still improves when increasing the number of samples per problem,
showing support for our claim of the importance of large scale sampling. Diﬀerences in performance
between APPS results and CodeContests could be attributed to dataset quality (e.g. the high APPS
alphacode