Thanks I have implemented your tips and improved the code. The actual error was from the way the way I loaded in my masks. Initially they were just pixels from [0, 1, 2], but I used torchvision.transforms.functional.to_tensor which converts it between 0-1 so it became [0., 0.00392157, 0.00784314] instead. Using torch.from_numpy() worked for me which converted it without changing the numbers. And nn.CrossEntropyLoss takes unnormalised label scores. So now it’s working pretty decent