Your input can and should probably be normalized to properly train the model.
The mask however, should most likely not be normalized, as it contains some kind of class information.
How is the other mask stored if not as RGB? Does it store the class indices directly for each pixel?
Could you post some values and the shape of an example mask you are currently using?