1
5
1
5
1000
1
5
1000
5
5
5

N/A
N/A
N/A
N/A
N/A
1000
1000
N/A
1000
10000
50000

GPT-Neo 2.7B
GPT-Neo 2.7B
Codex 12B
Codex 12B
Codex 12B
Codex 12B
Codex 12B
AlphaCode 1B
AlphaCode 1B
AlphaCode 1B
AlphaCode 1B
Table 10 | n@k results on APPS. If there is no ﬁltering, then 𝑛 = 𝑘 and the metric is pass@k.
Finetuned GPT-Neo numbers reported from Hendrycks et al. (2021), Codex numbers from Chen et al.
(2021). We used a time limit of 3 seconds per test to match Codex 12B, and report average numbers
over 3 diﬀerent ﬁne-tuning runs for AlphaCode. Note that this does not include all components
described in Section 4, and does not use the CodeContests dataset.

n@k
0.00%
0.00%
0.02%
0.09%
3.23%
3.04%
3.08%
7.06%
4.58%
6.65%
7.75%

n@k
0.57%
0.80%
0.14%
0.51%
3.70%
2.64%
3.23%
5.24%
5.63%
8.21%
9.66%
alphacode