Figure 36 Elo scores for a 52B context-distilled model with rejection sampling (utilizing a 52B PM). For
each prompt, we generate k number of responses, and return the response with the highest PM score.

B.6 Elo Scores for Rejection Sampling Models

In Figure 36 we show helpfulness Elo scores for a 52B context distilled model with rejection sampling
(utilizing a 52B preference model trained on pure helpfulness) for k = 1, 4, 16, 64, showing that higher values
of k clearly perform better. Note that the context distilled model and the preference models discussed here
were trained during an earlier stage of our research with different datasets and settings from those discussed
elsewhere in the paper, so they are not directly comparable with other Elo results, though very roughly and
heuristically, our online models seem to perform about as well or better than k = 64 rejection sampling. Note
that k = 64 rejection sampling corresponds to DKL = log(64) ≈ 4.2.

B.7 Stack Overﬂow Results
Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback