I don’t think there has been lots of research on this. Also, has the doc say, the pooler_output in BERT is not a good summary and should not be used in classification, yet we do that still.
When I have some time, I’d love to experiment a bit and see which kind of classification head works best, but if anyone wants to try different variants and report results here, that would be super helpful!