Yes, that is also a good one. The problem that I am having is that this is not always intuitive. Take for instance the BertForSequenceClassification model. I would have expected that the classifier layer was pretrained (because of the NSP task mentioned in the paper), but it isn’t. A small note in the documentation would immediately make it clear to a user when they look it up.