I got the answer and really sorry for posting the doubt without doing more work which I should have done.
Initially, I thought that converting images to tensor is distorting my mask, which is not because I am using torch.from_numpy. It was because of resizing. It seems default interpolation in transforms.Resize() is Image.BILINEAR, which in turn is changing the pixel values of masks.
Changing the line

resize = transforms.Resize(size = self.img_size)

to

resize = transforms.Resize(size = self.img_size, interpolation=Image.NEAREST)

solves my problem.
I am again sorry for this silly doubt but I think it might be helpful for someone else in the community.