Thanks for the code.
It seems you are indeed dealing with “color” images as your masks. which won’t work out of the box.
I’m not sure, how the mask was created initially, but it seems the background is black, while the car class is a mixture of mostly gray and some artifacts on the borders, which is not good.
If you have mask images with a specific color for each class, e.g. black = background, white = car, red = …, then you could use the mentioned mapping to create class indices.
The result would be a new mask tensor only containing a class index for each pixel (also without the channel dimension). In this case the background could have values of 0, while the car would have the class index 1.
Once you get the mapping, note that you have to be careful about resizing operations, as they might use interpolation techniques by default, thus creating “between class” colors or indices.
E.g. the border of your car could slowly turn from the car class to the background class, which is not useful for a classification task, as you won’t be able to map these colors back to class indices.
The Resize transformation has an interpolation argument, which should be set to PIL.Image.NEAREST for the mask.



 alex_d:

One of the reasons I am confused is that since I am using a pre-trained model, wouldn’t there be an existing mapping of the colors already that I could refer to instead of creating my own indices?


The model doesn’t know anything about the class mapping. It has an output conv layer with a number of output channels, which correspond to the classes.
As far as I know, the pretrained models were trained on the COCO dataset, so the output channels would correspond to whatever the COCO dataset defines as class0, class1, etc.
Since you are dealing with two classes and would most likely replace the last conv layer with a new one returning two output channels, you are free to chose whatever mapping you want (for 2 classes you don’t have much choice  ).



 alex_d:

And if I do have to create new class indices, like you mention in this post in which part of the code should this happen? Like should I have a helper function creating the class indices (mapping each color to a class) and should I include that within the MyDataset class, and at what point should I call that function?


I would say it depends how you are loading the image and mask.
If you are lazily loading both in the __getitem__ method of your Dataset, I would convert the mask images there to the right format.