Double DQN might help. DQN is known to have a very high variance and there is no guarantee of convergence. Double DQN yields a much lower variance and a better policy from my experiments on CartPole. While I used state signals from Gym directly, this might also work for the images.