E.4.2. Solution correctness
Value conditioning conditions the model with either a CORRECT SOLUTION or an INCORRECT
SOLUTION tag at training time depending on the solution, and always with a CORRECT SOLUTION
tag during sampling. Table 8 shows that it results in a performance improvement. Here, we investigate
whether our models are strongly conditioned on this tag by supplying the INCORRECT SOLUTION
tag or even no tag at all, instead of the CORRECT SOLUTION. The comparison is illustrated in Figure
A13. Conditioning on the INCORRECT SOLUTION tag hurts in the 10@𝑘 metric, although not the
𝑝𝑎𝑠𝑠@𝑘 metric, indicating that the model may produce more solutions that pass example tests but not
hidden tests. Removing conditioning hurts in both metrics, although not signiﬁcantly.

57
alphacode