these created test inputs are not guaranteed to be valid, especially when problems have complex
constraints, imperfect and even invalid test inputs can still be useful for grouping sampled programs.
This learned test input generation model is diﬀerent from the mutation-based test generation process
used in Section 3.2.1 to augment our dataset. The latter requires correct solutions (which are not
available at test time) to ﬁlter out bad test cases.
After clustering on program behaviour we found that selecting one solution from each cluster from
largest to smallest performed best, perhaps because there are many ways solutions can be incorrect
while correct solutions tend to behave the same and therefore are grouped into larger clusters. If the
candidate solutions for a problem form less than 10 clusters (or more in the case of more than 10
submissions), after reaching the smallest cluster, we repeat from the ﬁrst cluster skipping samples
that have already been submitted.
alphacode