+1 @dhpollack
From my experience, for classification tasks, usually the concatenation of last few encoder layers and then operating on this (using RNN or Linear) tend to give best results. You can also average or maxpool to reduce dimensions.
If you’re planning to use a pre-trained model like BERT, the output representation of [CLS]  token is your best bet. For some reason [CLS] captures the meaning of the entire sequence (no theoretical backing to this).