luated candidate
responses (see Section 2.2), following Xu et al. [2023], discarding the pair if their
scores are the same. These pairs can be used for training with a preference tuning
algorithm. We use DPO [Rafailov et al., 2023].
• Positive examples only: In this variant, we add additional examples of (instruction
prompt, response) curated by the model to the seed set for supervised fine-tuning,
following other approaches [Li et al., 2023a, Adolphs et al., 2023, Gulcehre et al.,
3
Review the user’s quest