Paper Review, Deep Learning, Rlhf, Llm, NLP.
complex: If the reward model is trained on offline data, it might misgeneralize. If reward and policy are learned jointly, the system may suffer from “auto-induced distributional shift”. Errors in over- or underestimating rewards can become gradually prominent, and such mistakes accumulate, making