Hello,
I know that some transformer models are not pre trained with Next Sentence Prediction objective, like RoBERTa based models. In that case, CLS token does not mean anything, right?
Given CLS token is not pretrained, when I am developing my downstream classification task, would be better to fine tune this CLS token or perform average pooling on all the tokens?
Thanks in advance,
Bruno