I guess you directly want to use the embeddings and not really train or fine-tune the BERT model.


If it’s a pre-trained model, the embeddings are already trained too. So, you can directly proceed with model.eval() with torch.no_grad() like you would for any model.


(2.1) The last layer if you’re planning to further fine-tune your model.


(2.2) After layer norm, after dense function (depends on your task though).