Paper Review, Deep Learning, Rlhf, Llm, NLP.
might differ from the training and evaluation environments. Policies can end up pursuing the wrong goals if these incorrect objectives correlate with other events. A specific example is when a system trained with RLHF pursues the reward mechanism itself rather than the actual intended goal. Optimal