(b) Difference in mean log-prob between good and bad
answers to Stack Overﬂow questions.

the RLHF models obtain worse loss. This is most likely due to optimizing a different objective rather than
pure language modeling.

B.8 Further Analysis of RLHF on Code-Model Snapshots

As discussed in Section 5.3, RLHF improves performance of base code models on code evals. In this ap-
pendix, we compare that with simply prompting the base code model with a sample of prompts designed to
elicit helpfulness, harmlessness, and honesty, which we refer to as ‘HHH’ prompts. In particular, they contain
a couple of coding examples. Below is a description of what this prompt looks like:

Below are a series of dialogues between various people and an AI assistant. The AI tries to be helpful,
Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback