Self-supervised language models of increasing scale learn to complete some tasks zero-shot [31] or
with few-shot prompts [6, 25, 11]. However, their performance on downstream tasks and alignment
with user intent can be significantly improved by fine-tuning on datasets of instructions and human-
written completions [23, 36, 13, 39]. This ‘instruction-tuning’ procedure enables LLMs to generalize
to instructions outside of the instruction-tuning set and generally increase their usability [13]. Despite
the success of instruction tuning, relative human judgments of response quality are often easier to
collect than expert demonstrations, and thus subsequent works have fine-tuned LLMs with datasets of
human preferences, improving proficiency in translation [18], summarization [38, 49], story-telling
[49], and instruction-following [26, 32]. These methods first optimize a neural network reward
function for compatibility with the dataset of preferences under a preference model such as the

2
Direct Preference Optimization