Interesting, for the loss function I just follow the dueling double dqn algorithm. And when I use only x,y,z,row,pitch,yaw as input everything was working properly, so I think my loss function is defined correctly. Maybe 2000 episodes is not enough for using images as input?