My doubt is how the CLS token learns the context in the sequence if it is not used during the pre training (like RoBERTa based models).
If I understood correctly, this CLS token learns only during the finetunning, which is also good enough for text classification.