This analysis suggests that these RL learning curves might be associated with changes in the RL policy that
behave very similarly to simply rejection sampling from the initial distribution.
We ﬁnd this simple relation quite striking, and believe it merits further study. At a conjectural level, it might
have a variety of implications and uses when RL-ﬁnetuning large generative models:

• These relations provide a rough prediction for ‘how much does the policy need to change to achieve a
speciﬁc reward’. Furthermore, if the lines corresponding to different model sizes really are parallel,
then one can use RL training of a small model along with the zero-shot performance of a larger
model to estimate the eventual performance of a larger RL policy. The slopes of these lines also
explain how RLHF training can produce such large effective gains in model size, and for example it
explains why the RLHF and context-distilled lines in Figure 1 are roughly parallel.
Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback