If you’ve resized the target image and the resizing operation added new values to it, it could be problematic, since the values in the target define the number of classes.
I don’t know, how and if you are mapping color codes of the target to class indices, but would need to make sure it contains the aforementioned valid class indices.