Well thats because you must have one-hot encoded it, just use a matrix having class labels for each sample (batch_size,1,H,W). Have a look at the documentation of Cross Entropy Loss, It would be much more clear to you. Your prediction output should be having 2 channels.