Yes. you should just normalize the inputs in my view.
The given ground truths are not images with ‘pixel values’, but they are labels that say which class the particular pixel belongs to. I am not seeing the reason why you need to normalize the labels.