Iteration 2 (Model M2), which is trained using the self-reward model derived
from its previous iteration M1 outperforms Iteration 1 (M1), while M1 itself outperforms a
standard SFT baseline model trained on only Instruction Fine-Tuning (IFT) data. Iteration
3 (Model M3) gives further improvements over Iteration 2.
Self-Rewarding Models
Model
SFT Baseline
Iter 1 (M1)
Iter 2 (M2)
Iter 3 (M3)
Training data
IFT
IFT+EFT
IFT+EFT
IFT+EFT
+AIFT(M1)
+AIFT(M2)
Pairwise accuracy (↑)
65.1%
78.7%
80.4%
81.7%
5-best % (↑