Paper Review, Deep Learning, Rlhf, Llm, NLP.
Models are trained to mirror human approval rather than genuine human benefit, which might result in undesirable outcomes. Strong optimization for an inaccurate proxy of a goal can cause failure in achieving the actual target goal. Specific problems like outputting nonsensical text have been