@BramVanroy In the BERT paper, the author use concatenation of the last 4 layers only for the feature-based approach (BERT layers are frozen). For the fine-tuning approach, he just use the last CLS representation. (Section 5.3 of BERT paper 15)
I’m wondering if you tried fine-tuning with only the last layer versus concatenation of the 4 last layers, and see any improvement ?

I’m also curious about the the effects of having a single Linear or 2 Linears with non-linearity on the classification scores.