Thanks man.
I tried after I saw your reply. The nature DQN paper didn’t mention how frequently they updated their target network. But I tried every 2, 10, and 100 iterations. None of them solved my divergence problem. But they made my averages stay at 200 for a very long time. I guess it’s working, but not enough.
BTW I’m using exactly the same implementation given by PyTorch tutorials, which takes screens as inputs.