Why do language models like InstructGPT and LLM utilize reinforcement learning instead of supervised learning to learn based on user-ranked examples? Why do language models like InstructGPT and LLM utilize reinforcement learning instead of supervised learning to learn based on user-ranked examples?
Language models like InstructGPT and ChatGPT are initially pretrained using self-supervised methods, followed by supervised fine-tuning. The researchers then train a reward model on responses that are ranked by humans on a scale of 1 to 5. After the reward model has been trained using supervised learning, reinforcement learning with proximal policy optimization (PPO) is used to update the language model.
Why do they then use RLL with PPO to update the LLM based on the reward signal. Why don't they train / fine-tune the LLM using the labels (ranks 1-5 provided by humans) in a supervised fashion using ranking or (ordinal) regression losses?

Your LLM will give you a categorical distribution as output, over which you can sample, and thus use RL to estimate the gradient...
What you are suggesting looks more like a GAN which instead of using a discriminator, you have a ordinal-NN, over which you take the gradient to maximize the ordinal output... however, this is unstable and usually the generator (LLM) will have a very easy time maximizing the output of the discriminator, which is known as mode alignment...
You can probably fine tune a LLM with a loss like:
$$
\nabla L(\theta) = \nabla(-D_{kl}(\pi_{\theta}(y|x)||\pi_{orig}(y|x))) + \nabla D(\pi_{\theta}(y|x)|x)
$$
where the second term is the gradient flowing from the conditional discriminator (which maximization should give you more human-like response), and the first one is just a penalization term to not go too far from the pre-trained model
However, in my opinion, this will just make the LLM overfit the discriminator...
