Paper Review, Deep Learning, Rlhf, Llm, NLP.
unexpected features of the environment, leading to causal confusion and weak generalization outside the training distribution. Some reward models fail to train new agents, questioning their reliability for policy learning. Optimizing for an imperfect reward proxy leads to reward hacking: Optimizing