the output logits of a model by a scalar temperature 𝑇 before the softmax layer. We observed that
when using 𝑇 = 0.2 < 1, tempering helps avoid overﬁtting to our ﬁne-tuning dataset by making
the training distribution sharper, and consequently the inference distribution smoother. Notably,
this is the opposite of the suggestion of Dabre and Fujita (2020) to use 𝑇 > 1 to make a sharper
inference distribution. At sampling time, we divided the logits by another temperature 𝑇(cid:48) tuned on
the validation set (𝑇(cid:48) = 0.12 for models trained with tempering only; 𝑇(cid:48) = 0.25 for models trained
with tempering and GOLD).
Value conditioning & prediction.
CodeContests contains both correct and incorrect problem
submissions. We used value conditioning and prediction to discriminate between these two types of
submissions, providing an additional training signal and allowing use of data which could otherwise
alphacode