Oh, I completely misunderstood you then. Then I guess you can just open the image with PIL with mode “P” and transform it into a tensor. your getitem function in Dataset class looks something like this.
        image_file_path = self.img_list[index]
        raw_img = Image.open(image_file_path)
        raw_img = raw_img.convert('RGB')  
        
        anno_file_path = self.mask_list[index]
        anns_img = Image.open(anno_file_path).convert('P')
        
        raw_img,anns_img  = self.transform(self.phase, raw_img, anns_img)

torchvision has totensor function.
https://pytorch.org/docs/stable/torchvision/transforms.html#torchvision.transforms.ToTensor 2