Paper Review, Deep Learning, Rlhf, Llm, NLP.
found that RLHF fine-tuning can negatively impact various aspects of models like GPT-4, such as calibration on question-answering or a narrow distribution of political views. Mode collapse might partially stem from switching from supervised pretraining to an RL objective, as RL pushes for