C.3. GOLD
We want to train a model that is capable of ﬁnding any correct solution to a problem (like precision),
rather than one that over-focuses on capturing the entire training distribution (like recall). To avoid
the tendency of maximum-likelihood objectives to put some weight on every solution, we used a
variation of the 𝛿-reward version of GOLD (Pang and He, 2020), an oﬄine RL algorithm which adds
an oﬀ-policy importance weight to the standard maximum likelihood objective gradient:

∇LGOLD(𝜃) = − ∑︁

𝑠∈Solution tokens

𝑃𝜃(𝑠)∇ log 𝑃𝜃(𝑠),
alphacode