Paper Review, Deep Learning, Rlhf, Llm, NLP.
under-represented groups. Reward Misgeneralization and Hacking Reward models can misgeneralize to be poor reward proxies, even from correctly labeled training data: There might be many ways to fit human feedback data, even with infinite training data. Reward models may compute rewards based on