5.1. Codeforces competitions evaluation
Evaluating on programming competitions checks program correctness more thoroughly, compared to
evaluating on our dataset which has known weaknesses including false positives, accepting algorith-
mically ineﬃcient solutions, and handling problems with multiple acceptable outputs. Additionally,
evaluating in the real setting allows us to benchmark against the best performers on this task: human
competitors.
We evaluated our best system on all Codeforces competitions from 2021/12/01 to 2021/12/28 with
more than 5,000 participants per contest, a total of 10 competitions. The system was an ensemble
of 41B and 9B models with clustering, which performed best on our validation set but turned out
to be slightly worse than using the 41B model alone with clustering (see Appendix C.1 for more on
ensembling). For each contest, we simulated running AlphaCode live, generating samples for each
alphacode