My question is why you are using ToPILImage transformation?
can you try this:
gt_transform = transforms.Compose([
transforms.Resize((256, 256), interpolation=0),
transforms.ToTensor()
])
label = torch.from_numpy(label.npy) # [143, 512, 512]
label_transform = gt_transform(label)