I got no suggestion but rather a question 
Would you mind showing me your whole code? I am currently trying to write a token classifier / NER with also very little data. But I’m not quite sure how to define the optimizer only with the linear layers and freezing all BERT weights.