I guess it would because of loss  function. Here you don’t have categorical cross-entropy loss.
You can use a custom loss function like this one and softmax.
def custom_categorical_cross_entropy(y_pred, y_true):
y_pred = torch.clamp(y_pred, 1e-9, 1 - 1e-9)
return -(y_true * torch.log(y_pred)).sum(dim=1).mean()