Did you make sure the target tensor do represent the valid segmentation mask?
Your current code is:


class MyDataset(Dataset):
    def __init__(self, image_paths, mask_paths, train=True):
        self.image_paths = image_paths
        self.mask_paths = mask_paths
        
    def transforms(self, image, mask):
        image = transforms.Resize(size=(256, 256))(image)
        mask = transforms.Resize(size=(256, 256))(mask)
        image = TF.to_tensor(image)
        mask = TF.to_tensor(mask)
        
        return image, mask
    
    def __getitem__(self, index):
        image = Image.open(self.image_paths[index])
        mask = Image.open(self.mask_paths[index])
        
        x, y = self.transforms(image, mask)
        return x, y
    
    def __len__(self):
        return len(self.image_paths)

I’m a little bit concerned about the transformation applied on the target.
If you are loading a valid segmentation mask containing only the class indices (in your case it should be zeros for background and ones for the car), the Resize and ToTensor transformations might mess up the target values.
For Resize you should use PIL.Image.NEAREST as the interpolation, since PIL.Image.BILINEAR is the default one and will interpolate to invalid class indices at the borders.
Depending which image type you are passing to ToTensor, this might also normalize the image, which might further cause trouble.
Could you print some target tensors and just make a sanity check, that they only contain your desired class indices?