Paper Review, Deep Learning, Rlhf, Llm, NLP.
for an imprecise reward proxy can result in “reward hacking,” where the model finds unintended ways to maximize its rewards. Misspecification and misgeneralization, along with the inherent complexity of real-world problems, contribute to the differences between reward models and human judgment.