xuehy, I am facing the same problem on CartPole… the agent learns the most optimal behavior after 2K episodes, but then wanders off into less optimal regions. Can you please provide your learning rate, discount factor, exploration epsilon that worked for you? Thanks!