I think  your loss function input argument is wrong.  As shown in pytorch documents, the shape of input is  BatchCHW, and the target is  BatchH*W.  http://pytorch.org/docs/master/nn.html?highlight=torch%20nn%20nll#torch.nn.NLLLoss2d 510