Why do language models like InstructGPT and LLM utilize reinforcement learning instead of supervised learning to learn based on user-ranked examples? Why do language models like InstructGPT and LLM utilize reinforcement learning instead of supervised learning to learn based on user-ranked examples?
Language models like InstructGPT and ChatGPT are initially pretrained using self-supervised methods, followed by supervised fine-tuning. The researchers then train a reward model on responses that are ranked by humans on a scale of 1 to 5. After the reward model has been trained using supervised learning, reinforcement learning with proximal policy optimization (PPO) is used to update the language model.
Why do they then use RLL with PPO to update the LLM based on the reward signal. Why don't they train / fine-tune the LLM using the labels (ranks 1-5 provided by humans) in a supervised fashion using ranking or (ordinal) regression losses?

Supervised LLM training only gives the model positive examples, i.e. ones it should produce. It does not provide the negative ones, and a naive attempt to do so would probably fail due to the sheer volume of negatives in the space of possible outputs.
Indeed, you probably could somehow penalize the model for producing outputs like "afsjkafnkfkasfjk nasjfasfas" but that would be a poor negative sample as the model would probably not produce this gibberish in the first place. Coming up with a particular set of useful negative examples is hard and probably depends on a particular model. This is where RL comes in: it allows you to operate on the models' outputs themselves, which is exactly the thing you want to improve.
