This happens in the CrossEntropyLoss right?
This is because the input is not of the right shape. You should check the doc here 12.
If you want to do classification for each pixel given the score in the channel, you want to transpose the output to be of size nb_samples x nb_classes. And the label should be nb_samples. Where nb_samples = batch_size * height * width.