Basically 1 category  is like a collection of class labels, who belongs to this category. For example flat as a category includes road, road line, sidewalk.  The pretrained model is trained on class level, but what I want to fine tune is to predict category label.
I just want to know modify the prediction before computing the loss, whether this is a legal operation, because I feel like manually modification is not differentiable. Maybe I can add a additional mappling layer right after the pretrained model, that learn how to map class label to category?