Ok, I will try that out. Thanks for the help!
I know it sounds odd, but this classification is not the final answer of the network, only an intermediate selection of other downstream parts of the network. I am currently using a lower temperature on the softmax, and it does pretty well, but I don’t want the network to try and compensate for the colder softmax and still achive a mixture of answers. This is why I want to incentivize the network to stick with one choice and not a mixture. Thanks again for everything!