as well (49.2% wins vs. 14.5% wins). Clearly,
there is a large jump in performance from M1 to M2 by using the preference data AIFT(M1)
provided by the reward model from Iteration 1.
Iteration 3 (M3) improves over Iteration 2 (M2)
We see a further gain in Iteration 3
over Iteration 2, with 47.7% wins for M3 compared to only 12.5% for M2 in a head to head
evaluation. Similarly, the win rate over the SFT Baseline for M3 increases to 62.5% wins vs.
9.8%, i.e. winning more often than the M2 model did. Overall, w