samples. On problems where our models do ﬁnd a correct solution, the fraction of samples that
pass example tests roughly doubles but still remains at a low level. The non-uniform distribution of
𝑝pass example tests across problems is highlighted more in Appendix C.4.
Another observation from Table 9 is that larger and better quality models produce samples more
likely to pass example tests, and pass example tests for signiﬁcantly more problems. With 106
samples, our largest 41B models can generate solutions that pass example tests for over 90% of
problems, a remarkable success as ﬁnding programs that satisfy I/O example constraints remains a
very challenging problem.
Clustering. A solution has to pass hidden tests in addition to example tests, so we must further
select correct samples from those that pass all public tests. Filtering 99% of a million samples still
leaves thousands of samples per problem to select from. We cluster the remaining samples based on
alphacode