E. Additional analysis of AlphaCode’s capabilities and limitations
E.1. Model sample statistics
Table A7 shows the percentage of syntactically correct samples from the models. That is, samples that
compile for C++, and samples that do not generate a SyntaxError for Python.

E.2. Solve rate for diﬀerent problem diﬃculty ratings
Table A8 shows the 10@100k solve rates for our 9B and 41B models on the validation and test sets
in diﬀerent problem diﬃculty ratings. The validation and test sets contain 117 and 165 problems

50

Competition-Level Code Generation with AlphaCode

Contest Problem Problem Solved Incorrect

Submission

Problem

Letter

Rating
800
1100
800
1000
800
1300
1600
800
800
1800
800
1000
900
800
800
1300
1700
800
900
800
900
1000
1200

1623
1623
1622
1622
1615
1615
1615
1619
1619
1619
1620
1620
1617
1618
1618
1618
1618
1591
1591
1608
1613
1613
1613
alphacode