th Self-Training: We evaluate our
models using head-to-head win rates on diverse prompts using GPT-4. The SFT Baseline is
on par with Self-Rewarding Iteration 1 (M1). However, Iteration 2 (M2) outperforms both
Iteration 1 (M1) and the SFT Baseline. Iteration 3 (M3) gives further gains over Iteration 2
(M2), outperforming M1, M2 and the SFT Baseline by a large margin.
Data distribution analysis
We perform a t-SNE [Van der Maaten and Hinton, 2008]
visualization of the IFT, EFT and AIFT(M1) data, shown in Sect