∇LGOLD(𝜃) = − ∑︁

𝑠∈Solution tokens

𝑃𝜃(𝑠)∇ log 𝑃𝜃(𝑠),

where 𝜃 are the model parameters, and log 𝑃𝜃(𝑠) is the standard log-likelihood objective for predicting
the next token 𝑠. The additional 𝑃𝜃(𝑠) multiplicative importance weight allows the model to both
learn from tokens it already assigns high likelihood to, and to ignore tokens that are not in its
distribution. This way, the model can concentrate on precision rather than recall, and increase its
chance of getting at least one correct sample. To mitigate instabilities during training, we replace
𝑃𝜃(𝑠) in the importance weight with max(𝑃𝜃(𝑠)𝛼, 𝛽), 𝛼 = 1

2 , 𝛽 = 0.05.

(1)

44

Competition-Level Code Generation with AlphaCode
alphacode