Removing redundant model weights
## 🐛 This is not a bug, more like implemetation detail

I am using BertForTokenClassification model for my binary token classification problem. If my understanding is right, BertForTokenClassification has one layer on top with num_classes output neurons (one for each class) with softmax activation function + CrossEntropyLoss().

Now, if your problem has >2 classes this is completely fine, but if num_classes=2 you are both modeling P(input = 0) and P(input = 1), and it's easy to see why this is redunant.

num_classes=2 is special case and it should be implemented with only one output neuron with sigmoid activation function + binary cross-entopy.


Please correct me if I am wrong :)

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
