Hmm, lets see if we can figure this out together.
The input image has the mode ‘P’ as you say. This is a 1-channel image which maps to another colorspace and is represented by a uint8. This means that it can show a maximum of 256 colors. This is in fact very much the mode ‘L’ (gray image) which also is a 1-channel image represented by an uint8.
The difference here is that a gray image can show 256 shades of gray, ranging from black to white. The image you are working with can represent 256 colors (blue, green, yellow, etc) but never any mix of these colors. There aren’t any shades in between.
You will never get a normal color-image from the image you are working with.
Now, that we know what kind of data we are working with, what is the intended use of the data? I’m guessing you have a corresponding image, like a .jpg, that you use as input and this data as ground truth.
I wrote this little script that converts an image from PIL image mode P, to tensor and back. Just fill in one of the paths and it should show you a color image. Just note that I don’t know how this works if you have multiple images with different colors
from PIL import Image
from torchvision import transforms

im = Image.open('PATHFORIMAGE.png')
palette = im.getpalette()

transform = transforms.Compose([
  transforms.CenterCrop((128,128)),
  transforms.ToTensor(),
])

tr_im = transform(im)
to_pil = transforms.ToPILImage()
back_im = to_pil(tr_im)

p_im = back_im.convert('P')
p_im.putpalette(palette)
p_im.show()

Outputs ->