odel. We thus
define AIFT(Mt) to mean AI Feedback Training data created using model Mt.
Model Sequence
We thus define the models, and the training data they use as follows:
M0 : Base pretrained LLM with no fine-tuning.
M1 : Initialized with M0, then fine-tuned on the IFT+EFT seed data using SFT.
M2 : Initialized with M1, then trained with AIFT(M1) data using DPO.
M3 : Initialized with M2, then trained with AIFT(M2) data using DPO.
This iterative training resembles the procedure used in Pairwise Cringe Optim