Thank you for the reply.
Do you have an opinion on using batch normalization in the classification head? I don’t see any specific use of it in HuggingFace’s code for the BERT models. I’m using the HuggingFace code as a best practice for lack of other information.