IF and RLHF procedures and find the methods they compare
perform roughly equally. They use an “off-the-shelf” LLM to perform LLM-as-a-Judge
prompting to build a training set to train a fixed reward model, which is then used for RL
training. They also experiment with using the fixed but separate LLM-as-a-Judge model
directly, which the authors report is computationally expensive due to using it within PPO
training (rather than the offline step in the iterative approach we use in our work, which
is relatively