I have a similar problem where I try to get pytorch to work on Pong-v0, while tensorflow can get past 20, pytorch merely gets past -20 which means it learns nothing! The same pytorch code runs pretty well on another machine with an older version of pytorch and it does not work on a newer version of pytorch, so I think I should downgrade my pytorch version? After I downgrade, the problem solved. So I’d suggest to stay with the original pytorch version 0.1.x and wait for 0.2.x after you are sure it will work.