It seems that the model output contains the background class + all other 22 classes.
If I understand your work flow correctly, you have passed the background class index as ignore_index, so that your model never learned how to predict the background?
If that’s the case, I would expect the model never to output the background class during inference either.