If you are using e.g. nn.CrossEntropyLoss as your criterion, your label image should contain the class indices and have the shape [batch_size, height, width]. ToTensor() should not be called, since this would normalize the values to the range [0, 1].