Large unsupervised language models (LMs) trained on very large datasets acquire surprising capabili-
ties [11, 7, 40, 8]. However, these models are trained on data generated by humans with a wide variety
of goals, priorities, and skillsets. Some of these goals and skillsets may not be desirable to imitate; for
example, while we may want our AI coding assistant to understand common programming mistakes
in order to correct them, nevertheless, when generating code, we would like to bias our model toward
the (potentially rare) high-quality coding ability present in its training data. Similarly, we might want
our language model to be aware of a common misconception believed by 50% of people, but we
certainly do not want the model to claim this misconception to be true in 50% of queries about it!
In other words, selecting the model’s desired responses and behavior from its very wide knowledge
Direct Preference Optimization