E.3.3. Capturing variables and their relations
Problem descriptions include multiple variable names to denote objects and quantities of interest, and
to specify relationships between them. For example, a problem might feature an array, 𝑎, of length
𝑛, or two integers called 𝑛 and 𝑚 with 𝑛 < 𝑚. Understanding these variables and their relations is
necessary for solving problems.
We investigated two changes: random variable name substitutions either consistently applied through-
out a problem, or applied at random for each instance such that consistency is not guaranteed (which
renders the problem ill-posed). Perturbations were done on up to 6 diﬀerent variables, maintaining
consistent character case, and variables were replaced by other existing variable names from the
same problem (see Figure A8 for a concrete example).
Figure A9 shows the results of this evaluation. The small (300M) model is largely unaﬀected by

54

Competition-Level Code Generation with AlphaCode
alphacode