2.2. Evaluation
Though running a system against a live programming competition is an unbiased evaluation, it adds a
large degree of complexity and is not a stable benchmark. To alleviate this issue, we developed a proxy
measure suitable for research iteration similar to the development sets present in most supervised
learning datasets. Our measure mirrors the fundamental structure of competitions while simplifying
incidental details. The metric we use is “percentage of problems solved using 𝑛 submissions from 𝑘
samples per problem”, denoted as 𝑛@𝑘.
This metric indicates the percentage of problems a model can solve if for each problem it is allowed
ﬁrst to create 𝑘 samples, and then to evaluate 𝑛 ≤ 𝑘 of these samples against the hidden tests. The
problem is considered solved if any of these 𝑛 evaluations passes all tests. The ﬁltering method is
up to the system itself, but should only be based on information available to competitors (e.g. the
alphacode