Again, try and see if it gives you better results. The head is the one used in the BERT article for the fine-tuning, so it’s there for reproducibility. Note that transformers use LayerNorm inside so maybe this one could work well too for classification.