First, thanks for your reply. I am following a pytorch implementation (https://github.com/transedward/pytorch-dqn/blob/master/dqn_model.py 110) and these Stanford lectures notes (http://cs231n.github.io/convolutional-networks/ 99). I cant figure out what I am doing wrong.