If you are using a multi-class segmentation use case and therefore nn.CrossEntropyLoss or nn.NLLLoss, your mask should not contain a channel dimension, but instead contain the class indices in the shape [batch_size, height, width].
PIL.NEAREST is a valid option, as it won’t distort your color codes or class indices.
Since you are loading the image via PIL, I assume your mask is an RGB image, where each color represents a class?
If so, you should map the color values to the corresponding class index.
Let me know, if that would work for you or if you need more information.