20.99
7.77
12.4
203.7

60%
30%
62%
4%

70%
N/A
88%
46%

Table 2 | Dataset false positive rates. The bottom row is the dataset we used, while “CodeContests
raw” does not use generated tests and does not ﬁlter out problems with insuﬃcient tests. Validation
splits were used for CodeContests and APPS. We randomly selected 50 problems our 1B parameter
model solved (from 10,000 samples per problem for APPS, 200 for HumanEval, and 1,000,000 for
CodeContests), and manually examined one solution for each problem to check whether they are
false positives or slow solutions. HumanEval does not have timing constraints for most problems, so
there is no slow rate.
alphacode