Just curious, what is the pre-trained model you are using? I would like to see the last layer (which can confirm your ‘high’ values are perhaps what is expected and your approach to take argmax is right).
It could also be you may need to fine-tune the ‘head’ of the model, to classify better?