Yes I mean the integer output layer that you would add you would have to train it like all the other layers. You could freeze the rest of your model and just train that layer and it might work. But you would have to train it to see. One possibility is that you could apply a sigmoid function to that layer which would make it in the range of 0 to 1. Then you could multiply it by the number of labels. That would ensure the range is between 0 and 19 but then you would have to train it using something like an MSE loss to make sure it is outputting the correct classes. Overall it is not worth it softmax is a simpler solution in my opinion.