Paper Review, Deep Learning, Rlhf, Llm, NLP.
Various factors contribute to this instability, including the random nature of exploration, violations in data collection assumptions, bias in value functions, and unpredictability in deep neural learning. Policies tend to be adversarially exploitable: Even well-trained policies that perform well