In a semantic segmentation use case the output of your model would represent logits and would thus not be one-hour encoded. To get the predicted class label you would use argmax. Since this operation is not differentiable it’s not included in the forward pass of the model.