ization and
Iterative DPO introduced in Xu et al. [2023]; however, an external fixed reward model was
used in that work.
4
3
Experiments
3.1
Experimental Setup
Base Model
In our experiments we use Llama 2 70B [Touvron et al., 2023] as our base
pretrained model.
3.1.1
Seed Training Data
IFT Seed Data
We use the human-authored examples provided in the Open Assistant
dataset [Köpf et al., 2023] for instruction fine-tuning. Following Li et al. [2023a] we use
3200 examples, by sampling only first conversational 