Paper Review, Deep Learning, Rlhf, Llm, NLP.
observed without proper regularization, and reward hacking has been recognized in AI systems, including those trained with RLHF. Studies show that unhackable proxies are rare in intricate environments, and reward hacking is to be expected by default, especially as an agent’s capabilities increase.