Jeffery_Ott:

I even checked pixel values and the my target is unique values 0-3 and my prediction is 0-20.


This sounds counter-intuitive to the higher overall accuracy or are the “additionally” predicted class indices quite sparse and do not lower the accuracy a lot?
I would have assumed your model would learn to predict only the used target classes, as it would never get any signal from classes 5-20 so unsure why the model is learning to predict them at all.