ditional data via AI (Self-)Feedback.
AI Feedback Training
After performing the self-instruction creation procedure, we can
then augment the seed data with additional examples for training, which we refer to as AI
Feedback Training (AIFT) data. We try two variants of such feedback:
• Preference pairs: We construct training data of the form (instruction prompt xi,
winning response yw
i , losing response yl
i). To form the winning and losing pair
we take the highest and lowest scoring responses from the N eva