Yes, you should be able to transform the color masks to the mask targets containing indices and return them in the __getitem__.
No, nn.CrossEntropyLoss won’t raise an error, if you pass the model outputs in the shape [batch_size, nb_classes, height, width] and the masks as [batch_size, height, width] for a multi-class segmentation use case.