This would have been a lot easier if you had simply used BertForSequenceClassification because then you could have easily used this with save_pretrained inside the trainer. Here is one example 39 but there are more on Github in the examples folder.