Naruto:

hey, I have a similar question regarding cross entropy, but for multi-class prediction. My output of network, the logits has shape of [5,6,256,256]. Number of classes is 6 here. The target has a shape of [5,256,256]. Since this criterion combines LogSoftMax and ClassNLLCriterion in one single class, cross entropy expects logits and target having different size, right? At least, criterion = nn.CrossEntropyLoss()  loss = criterion(logit, true_masks) didn’t give me error.


Yes, the shapes look good. The target should contain values in the range [0, 5], which seems to be the case.



 Naruto:

My initial values are larger than 1. According to some sanity checks post out there, it says that it is about bad initialization of weights.


I don’t think a loss value smaller than 1 is expected.
For a 6-class classification with a random accuracy of 1/6=0.167, the initial loss would be in the range -ln(1/6.) = 1.79 as described in the CS231n notes 3.
(Karpathy used CIFAR10 and thus used the random accuracy of 0.1)