7.3. Scaling sampling
Similar to our sampling and ﬁltering, though on a much smaller scale, Chen et al. (2021), Austin et al.
(2021), and Cobbe et al. (2021) found that repeated sampling on the same problem signiﬁcantly
increases the probability of ﬁnding a correct solution. Cobbe et al. (2021) further introduced a way
of selecting a small number of ﬁnal submissions from multiple samples by majority voting. They also
demonstrate veriﬁers (value functions) used to judge the correctness of model samples as a way of
reranking samples.

28

Competition-Level Code Generation with AlphaCode
alphacode