E.3. Sensitivity to the problem descriptions
We measured model performance in response to assorted changes in the problem description, to
see if the model responds appropriately and makes use of the description. Because of compute
limitations, we were unable to retrain models on these modiﬁcations, but instead sampled already
trained models. The main metric used for most analyses was the 10@1024 solve rate, that is, solve
rate using 10 submissions from 1024 samples. We found overall that the problem description is
critical, and AlphaCode does not simply brute force possible solutions.

52

Competition-Level Code Generation with AlphaCode

Split Bucket 800-1100 1200-1500 1600-1900 2000-2300 2400-2700 2800-3100 3200-3500

Valid

Test

#Probs.

9B
41B

9B
41B

#Probs.

29

60.5%
62.4%

37

64.7%
69.5%

18

18.0%
17.3%

21

34.1%
35.4%

20

10.8%
15.9%

25

17.7%
20.6%

19

26.6%
19.7%

31

15.7%
13.8%

15
8.8%
7.8%
26
8.4%
7.8%

8

12.5%
14.9%

13
0.0%
7.5%

8

21.9%
31.6%

12
0.0%
0.0%
alphacode