were only trained on helpfulness. In ﬁgure 16, we compare Elo scores for various snapshots of both runs,
as determined by crowdworker preferences, showing that the policy trained on the iterated-online mixture is
Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback