D.1. Simulation
Contest scoring is based not only on whether or not a problem is solved, but also on when in the contest
it is solved and how many incorrect submissions there were before a correct one. Our procedure takes
all three into account.
For each contest, we simulated running live with 3750 TPUv4 and 3750 TPUv4i (Jouppi et al., 2021)
chips for the duration of the contest. These devices continuously draw samples, using a pool of
workers to evaluate samples against example tests. Because clustering runs best when there are
enough samples that pass example tests, we cannot continuously submit submissions. Instead, for
each problem in the contest we clustered and submitted up to a total of 10 samples at three points
(Table A4), where the point was either based on the fraction of the time remaining in the competition
or based on the relative number of samples that pass example tests available compared to the default
alphacode