prestigious competitions in computer science, drawing hundreds of thousands of participants from
around the world. Using problems that humans ﬁnd challenging from such battle-tested competitions
ensures robustness against shortcuts and provides a meaningful benchmark for many aspects of
intelligence.
Early work using program synthesis for competitive programming has shown that large transformer
models can achieve low single-digit solve rates (Chen et al., 2021; Hendrycks et al., 2021), but could
not yet reliably generate solutions for the vast majority of problems. Furthermore, as we show in
Section 3.2.1, the lack of suﬃcient test cases in existing competitive programming datasets makes
alphacode