Solved by tom in post #4 
              
              
                You can look at what the BertForSequenceClassification model does in it’s forward. 
The pooled_output obtained from self.bert would seem to be the features you are looking for. 
Best regards 
Thomas