• One can ask a subtle, perhaps ill-deﬁned question about RLHF training – is it teaching the model
new skills or simply focusing the model on generating a sub-distribution of existing behaviors. We
might attempt to make this distinction sharp by associating the latter class of behaviors with the
region where RL reward remains linear in

KL.

√

• To make some bolder guesses – perhaps the linear relation actually provides an upper bound on RL
√
reward, as a function of the KL. One might also attempt to extend the relation further by replacing

KL with a geodesic length in the Fisher geometry.

By making RL learning more predictable and by identifying new quantitative categories of behavior, we
might hope to detect unexpected behaviors emerging during RL training.

4.4 Tension Between Helpfulness and Harmlessness in RLHF Training
Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback