It’s probably because of the original paper. In the BERT paper, they discuss how NSP is beneficial for QA and NLI but that “[t]he vector C is not a meaningful sentence representation
without fine-tuning, since it was trained with NSP”, where C is “the final hidden
vector of the special [CLS] token” (p. 4).
In other words, CLS token without finetuning might not be a semantic representation of a given sentence because it was pretrained only on the NSP task. To use it in your own downstream task, you should finetune it. If you only use BERT as a feature extractor (e.g. for sentence embeddings), then you are probably better off by using mean/avg pooling. See this Twitter thread and the paper reference in there: https://twitter.com/JohnMGiorgi/status/1295472684353105920 5