2. Fine-tune the model on our dataset of competitive programming data, using GOLD (Pang and
He, 2020) with tempering (Dabre and Fujita, 2020) as the training objective. This further
reduces the search space, and compensates for the small amount of competitive programming
data by leveraging pre-training.

3. Generate a very large number of samples from our models for each problem.
4. Filter the samples to obtain a small set of candidate submissions (at most 10), to be evaluated
on the hidden test cases, by using the example tests and clustering to pick samples based on
program behaviour.

Among these, the large-scale sampling followed by ﬁltering is unique to our setup, and we found that
this process greatly improves problem solve rate. Therefore many of our design decisions were made
to facilitate eﬃcient and eﬀective sampling.

9

Competition-Level Code Generation with AlphaCode
alphacode